fix off-by-one in CBC de-/encrypt; don't touch input buffer during CBC encryption

CBC_encrypt()/CBC_decrypt() wouldn't behave properly if the length was not a multiple of KEYLEN.
Also the input buffer is not modified inplace during CBC_encrypt().

Hi Andreas,
This is by design. I have omitted checking for valid input, I should have stated that in comments.
I am looking into your fork for the improvements you have accepted.
I will look closer into this.

Oh, glancing over it again, I see not all of it is by design... Sorry for the arrogant answer :/ I'll have a look at fixing this

No worries; I don't see anything arrogant there btw.

@andreas-wehrmann  Is this pull request still relevant? I think the bugs may have been fixed.

On 11/21/2017 01:25 AM, kokke wrote:

@andreas-wehrmann  Is this pull request still relevant? I think the bugs may have been fixed.

In short: nope.
I guess I should have closed it myself...
Regards

Hi @andreas-wehrmann I was just checking up after some of the latest updates.
I think the padding issues are gone and now the input buffer is leaved untouched as well.

Closing because issue is fixed after merging #76


Add documentation for Azure OpenAI

Thank you for helping make the Dapr documentation better!
Please follow this checklist before submitting:

[x] Commits are signed with Developer Certificate of Origin (DCO - learn more)
[ ] Read the contribution guide
[ ] Commands include options for Linux, MacOS, and Windows within codetabs
[x] New file and folder names are globally unique
[x] Page references use shortcodes instead of markdown or URL links
[ ] Images use HTML style and have alternative text
[x] Places where multiple code/command options are given have codetabs

In addition, please fill out the following to help reviewers understand this pull request:
Description
Add documentation for Azure OpenAI output binding
Refer to this PR for more details https://github.com/dapr/components-contrib/pull/2965
Issue reference
https://github.com/dapr/components-contrib/issues/2716

@ItalyPaleAle Please have a look.

@msfussell Please have a look.


feat(connector): [Novalnet] Integrate wallets Paypal and Googlepay

Type of Change

[ ] Bugfix
[X] New feature
[ ] Enhancement
[ ] Refactoring
[ ] Dependency updates
[ ] Documentation
[ ] CI/CD

Description

Integrated Googlepay and Paypal for Novalnet
Integrated mandates for Googlepay and Paypal

Additional Changes

[ ] This PR modifies the API contract
[ ] This PR modifies the database schema
[ ] This PR modifies application configuration/environment variables

Motivation and Context
https://github.com/juspay/hyperswitch-cloud/issues/6731

How did you test it?

Tested manually.
Test cases:

Googlepay payment

manual
automatic
mandates

Paypal

manual
automatic
mandates

Manual Testing Doc
Checklist

[X] I formatted the code cargo +nightly fmt --all
[X] I addressed lints thrown by cargo clippy
[X] I reviewed the submitted code
[ ] I added unit tests for my changes where possible

@cookieg13
Did you test Google Pay and Paypal Flows?

@cookieg13 Did you test Google Pay and Paypal Flows?

yes tested locally, webhooks and mandates are tested too

@cookieg13 You add the response aswell then only will get to know whether its worked or not

@cookieg13 Can you add the response aswell then only will get to know whether its worked or not. and add the mandate req and res also

added testing doc with req and responses in readme


Salesforce Integration

I followed the steps in https://docs.mautic.org/en/plugins/salesforce but the plugin could not connect. The error was in Client::authenticate().
To be able to connect I had to change the setting in Salesforce.  Then edit your app and change OAuth policies.

Thanks for flagging it up - can you make a PR to update the documentation please?


OE connections become disconnected when dragged between server groups

SQL Operations Studio Version: February Preview Insiders Build 0.26.1
Steps to Reproduce:

Create a server group
Save a connection outside of that server group
Connect to the saved connection in OE
Drag the connection into the server group

Expected behavior: The connection remains connected
Actual behavior: The connection appears to become disconnected, but if the password is saved you can still expand and interact with it without reconnecting. If the password is not saved, it will prompt you for the password when you interact with it

Closing older bugs that are planned for an upcoming milestone.  Thanks!


Expanded Playback and List controls

[Requires lbryio/lbry-redux/pull/426]

What is the new behavior?

Removed countdown on a list (Closes #6321)
Added "Replay" option on autoplay (#2921)

Added Loop option on Lists
Added Shuffle option on Lists
Added link to "View List" popup option, so can be opened on a new tab
Added Copy List option (Closes #6210)

Fixed an issue with hovering on audio files with gif thumbnails:

Clicking on the link of a floating player will take you back to the list
Added video shortcuts for numbers (0,1....8,9) to skip to certain parts of the video (0 goes back to the start)
Lists on sidebar: (Closes #6220)

Play Next/Previous (with shortcuts SHIFT+N SHIFT+P):

Other information

PR Type & Checklist
PR Type...
What kind of change does this PR introduce?

[X] Bugfix
[X] Feature
[X] Code style update (formatting)
[ ] Refactoring (no functional changes)
[ ] Documentation changes
[ ] Other - Please describe:

PR Checklist...

Please check all that apply to this PR using "x":

[X] I have checked that this PR is not a duplicate of an existing PR (open, closed or merged)
[ ] I added a line describing my change to CHANGELOG.md
[ ] I have checked that this PR does not introduce a breaking change
[ ] This PR introduces breaking changes and I have provided a detailed explanation below

this is a long one
🚀 the best is yet to come

This is huge.
I think copy list is its own thing.
modalCollectionAdd could conditionally return a simple collectionCopy component instead of the collectionAdd with the checkboxes etc.

Not sure about adding lists to sidebare in this way. Will discuss at planning.

It seems like coming up with a a shuffled next url can be done in the index.js of the a component, especially interacting and excluding based on content/history reducer. I'm not sure you need any work in lbry-redux for this.

I don't quite understand the case for selecting the previous url from collection with shuffle.

At the meeting we thought we needed to brainstorm the design for sidebar playlists more - take it out for now?

✔ Separated Copy List
✔ Took out Sidebar Lists

It seems like coming up with a a shuffled next url can be done in the index.js of the a component, especially interacting and excluding based on content/history reducer. I'm not sure you need any work in lbry-redux for this.

IMO on redux it avoids repetition of the same code every time you want to select a shuffled list

I don't quite understand the case for selecting the previous url from collection with shuffle.

going back on previously played shuffled videos, or else the ordering would be lost

Ah, I see.
How about just selecting the values right in the selector like is done here:
https://github.com/lbryio/lbry-redux/blob/d016d8057be1c676db4ad0d21bdbcacd39a678eb/src/redux/selectors/publish.js#L43
state => state.content.shuffledUris,
state => state.content.shouldLoop,

How about just selecting the values right in the selector like is done here:

✔  Done, and wow this does look way cleaner
➕ Also just added "separate control for autoplay next"

[ ]  if autoplay is off in settings and you're in playlist mode, we don't show the autoplay next button - should this assume autoplay next is on, or should we just always show the button (I know youtube hide it).
[ ]  the settings option will only control autoplay on load (adjust text), and autoplay next will be controlled from the player. We can consider adding it settings also.

These are up for discussion

[ ]  if you have autoplay off + playing a video + then go to settings to enable autoplay, it crashes at :         autoplayButton.removeClass('vjs-button--autoplay-next');  at videojs.jsx:670
  at oc (react-dom.production.min.js:211)```

This is the last one missing, I didn't manage to reproduce it @tzarebczan


avoid TypeError on when registering service workers without config

Fixes #5299
I have no idea how to unit test this. Just trying to get the ball rolling.

Thanks!


[Bug] StringReader.readQuotedString() does not reset cursor on exception

The method StringReader.readQuotedString does not reset the cursor to its start position when an exception is thrown. This is inconsistent with all other parsing methods.
This appears to be currently no problem because CommandDispatcher.parseNodes creates a new reader for every iteration and additionally resets the cursor when an exception is thrown (which is redundant).

I think this is intended.  The intention of changing the cursor position is so that the cursor is at the position where the issue is; for e.g. the case where there isn't a quote at the end of the string, the end of the string is the correct place for the cursor to be.  (Unless I'm misunderstanding what you're saying)

But how useful would that be? You cannot reliably use the reader afterwards without setting the cursor back to its original position and if you wanted to get the position of the incorrect value you could (and should) use the fields of the thrown exception.


Gatsby/feature/list of category detail

trello: https://trello.com/c/c1aHQmxn/178-life-situation-measure-list

Failuje build
01:43:37.508 | error There was an error in your GraphQL query:
-- | --
01:43:37.508 | Variable "$langCode" is never used in operation "vercelworkpath0GatsbysrctemplatesmeasureslistTsx382824416".
01:43:37.509 | GraphQL request:2:25
01:43:37.509 | 1 \|
01:43:37.509 | 2 \|   query($slug: String!, $langCode: String!) {
01:43:37.509 | \|                         ^
01:43:37.509 | 3 \|     taxonomyTermMeasureType(path: { alias: { eq: $slug } }) {
01:43:37.509 | See if $langCode has a typo or vercelworkpath0GatsbysrctemplatesmeasureslistTsx382824416 doesn't actually require this variable.


SerenityBDDReporter bug with some mocha hooks

the reporter cannot handle tasks in afterEach:
[15:46:15] E/launcher - Cannot read property 'append' of undefined
[15:46:15] E/launcher - TypeError: Cannot read property 'append' of undefined
    at /Users/ct/Projekte/xxx/packages/e2e-tests/node_modules/@serenity-js/core/src/reporting/rehearsal_report.ts:31:27

the same error occurs, when trying to define a before
I think the reporter doesn't know to which scene to append the activity to, with before it has not startet yet, with afterEach the scene already has concluded
edit: the afterEach Bug only appears if there are at least 2 its in the describe

Hi @ctaepper, yes, you're right. @Tom-Hudson's ticket #104 has a bit more detail so let's track it there if that's OK? I finally have some time to work on the reporters so am hoping to get quite a few of those related tickets done.


Build/Prop on a schedule

User Story
Regular builds propped to TEST

T-shirt size: SML
S

User Story
We should be able to have the latest build of NEXt available on the TEST env every day so that we can see changes as they roll in and provide early feedback

Description
I think a cron job was mentioned in the discussion...

Acceptance Criteria/Tests
TEST env has a stable build.  Tests passed.  Test data is available tool.  Push happens during low activity time of the day but should happen at same time every day.  We should make it clear to everyone how to tell if the daily drop succeeded (do I check travis?  see a version on the UI?  build log?).

Just finished. Set a cron job on the demo server to pull new source and redeploy every Tuesday at 6am.  The stg server pulls latest source and redeploys every day at 6am.

as per our discussion yesterday I removed the cron job on the demo server. Demo will have manual deployments at the end of each sprint until CI/CD is implemented and the stg server will have deployments every morning at 6am with a cron job. Added logging on each server to track deployment times.


config_override results in DispatchFailure

Describe the bug
I am trying to create a client with no credentials, then use config_override to pass in credentials on request. When trying to do config_override() to use new credentials for a call to KMS, the call results in a DispatchFailure. See code snippet for bug replication.
Expected Behavior
I expect the result of the call to KMS with config_override to give me back the correct decrypted result.
Current Behavior
The call using config_override returns
DispatchFailure(DispatchFailure { source: ConnectorError { kind: Other(None), source: NoMatchingAuthScheme, connection: Unknown } })

Reproduction Steps
use aws_sdk_kms::config::{Credentials, Region};
use aws_sdk_kms::Client;
use aws_sdk_kms::primitives::Blob;
use aws_credential_types::cache::CredentialsCache;

fn creds() -> Credentials {
    Credentials::new(
        "<Key ID>",
        "<Secret Key>",
        Some("<Session Token>".to_string()),
        None,
        "",
    )
}

#[tokio::main]
async fn main(){

    //Code to encrypt text (this works fine)

    let conf = aws_sdk_kms::Config::builder()
        .credentials_provider(creds())
        .region(Region::new("<region of KMS key>"))
        .build();

    let client = Client::from_conf(conf);

    let out = client
        .clone()
        .encrypt()
        .key_id("<Your KMS key>".to_owned())
        .plaintext(Blob::new("encrypted text".as_bytes().to_vec())) // text to be encrypted
        .send()
        .await;

    // Code to Decrypt the text that was encrypted above using an empty client, whose credentials get overridden
    // on decrypt call (where issue arises)

    // Create a client with no credentials
    let conf = aws_sdk_kms::Config::builder().credentials_cache(CredentialsCache::no_caching()).build();
    let client = Client::from_conf(conf);

    // Create a configuration override to be used on decrypt call
    let override_config = aws_sdk_kms::Config::builder()
     .credentials_provider(creds())
     .credentials_cache(CredentialsCache::no_caching())
     .region(Region::new("<region of KMS key>"));

    // Decrypt call returns DispatchFailure when it should return the decrypted result
    let _result = client
     .decrypt()
     .ciphertext_blob(out.unwrap().ciphertext_blob().unwrap().clone())
     .customize()
     .await
     .unwrap()
     .config_override(override_config)
     .send()
     .await
     .unwrap();
}

Possible Solution
No response
Additional Information/Context
In code snippet replicating the problem, you will have to use your own KMS key and AWS credentials.
 Here's what I think is going on:
Because of how you're constructing your config, it's not including an auth scheme. What is an auth scheme? It's a new orchestrator-related concept:

An authentication scheme consists of an ID (e.g. “aws.auth#sigv4”), logic to determine the IdentityResolver, and a Signer matching the authentication type.

In your case, since you're trying to use KMS, it requires the Sigv4 auth scheme. I'm guessing that when you perform the config override, that that's where the bug is. My guess is that config_override isn't respecting the auth schemes that are set by default when making requests with an AWS service client.
I'll put this issue on our backlog, although this is a busy week for us and it might take a week or two for us to take a look at this. If you absolutely can't wait that long, the workaround I recommend is to create a client + config the usual way instead of using the config override mechanism.

My goal is to create a client that gets passed in different credentials to be used for each request (ex. the client is not tied to any particular credentials, but instead uses different credentials on each request that is passed in).
If I understand correctly, creating a client + config the usual way would mean I would have to make a new client on each request because each client's creation is tied a specific set of credentials in a config.
Is there a work-around for where I can pass in credentials and the client uses those credentials (without using config_override())

I think the only workaround right now would be to construct a new client every time, and do it in such a way that all the new clients reuse the same HTTP connector so that you're not paying the price for TLS connect every time. This can be done by settings the http_connector on the Config. The rest of client construction should be cheap.

Still pending to release the fix

Bad bot.

@jdisanti this was actually a shockingly good bot—it saw the sync'd commit and closed the issue! Honestly, this may be the best way to close issues on the Rust SDK if we can get it to work reliably...

This issue still seems to be a problem.

https://github.com/rcoh/aws-sdk-rust-57-release/blob/main/src/bin/config_override_901.rs

@ysaito1001 @rcoh Is the fix released ? I was not able to run https://github.com/rcoh/aws-sdk-rust-57-release/blob/main/src/bin/config_override_901.rs in any known versions available.

is it possible to send a small reproducer?

I took the same example of yours and modified a little bit. Here is how it looks.
// https://github.com/awslabs/aws-sdk-rust/issues/901
use aws_sdk_kms::config::{Credentials, Region};
use aws_sdk_kms::Client;
use aws_sdk_kms::primitives::Blob;

fn valid_creds() -> Credentials {
    Credentials::new(
        "valid",
        "valid",
        Some("valid".to_string()),
        None,
        "valid",
    )
}

fn invalid_creds() -> Credentials {
    Credentials::new(
        "invalid",
        "invalid",
        Some("invalid".to_string()),
        None,
        "invalid",
    )
}

#[tokio::main]
async fn main(){

    //Code to encrypt text (this works fine)
    tracing_subscriber::fmt::init();

    // Create a client with no credentials
    let conf = aws_sdk_kms::Config::builder().region(Region::new("us-east-1")).credentials_provider(invalid_creds());.build();
    let client = Client::from_conf(conf);

    // Create a configuration override to be used on decrypt call
    let override_config = aws_sdk_kms::Config::builder()
        .credentials_provider(valid_creds());

    // Decrypt call returns DispatchFailure when it should return the decrypted result
    let _result = client
        .decrypt()
        .ciphertext_blob(Blob::new("blob"))
        .customize()
        .config_override(override_config)
        .send()
        .await
        .unwrap();
    // This should fail with an error about invalid credentials, but instead the request is not sent!
}

I supply invalid credentials in the first config and then override it with valid config.

Got it. I've reproduced this issue.

Great, fixed. We'll re-close this once the updated fix lands. Thanks for the reproducer!

This fix is now live in the latest version of this sdk. Please let us know if you have any other issues with using this sdk.


Migrate the project to use modules

Compilers which CMake natively supports module dependency scanning include:
MSVC toolset 14.34 and newer (provided with Visual Studio 17.4 and newer)
LLVM/Clang 16.0 and newer
GCC 14 (for the in-development branch, after 2023-09-20) and newer
We need to update to gcc14

We need to update to gcc14

gcc 14 rumored to be released in the early months of 2024, https://gcc.gnu.org/gcc-14/changes.html

We need to update to gcc14

gcc 14 rumored to be released in the early months of 2024, https://gcc.gnu.org/gcc-14/changes.html

Let's build it and modify the toolchain files accordingly, no need to wait.

Closing for now. This is diverging from main quickly and there are still quite a few barriers to doing this properly.


0 value bug

Hey, not sure if you're excepting pull requests on this but i'm using your solution (pretty much exactly what I was looking for so firstly thankyou for saving me the time) but found a bug and a tweak i'd like to make.
Bug:  when you have a key with 0 value, if there is a default set for that key, this will incorrectly get overwritten with the default.  In my branch (not submitted yet) i've tweaked that and the test cases to check for it.
Tweak:  I want to be able to pass no 'defaults' argument and have it delete keys where there's no value as 'undefined' is not valid JSON.
Let me know if you are interested and i'll push my code up.
Cheers,
Kevin

Should probably be closed, since #3 was merged

Truth, thanks!

Probably need to release a patch version as well.
P.S. I found this after I wrote https://github.com/knownasilya/parky

Also done--didn't realize I hadn't pushed it.  Again, thanks for the heads up.


Question about comparison with Jin et al.'s work (CVPR2018)

Hi, thank you for your interesting work!
I have a question about the comparison of methods in your work.
When making comparisons, did you retrain Jin et al.'s model ("Learning to Extract a Video Sequence from a Single Motion-Blurred Image" from CVPR 2018), or did you just use their pre-trained checkpoints?
I couldn't find the training code on their github page.

Got it.
Thank you :)


Fixing docker image building issues.

Don't mind me -- just stealing @milo-hyben 's commit from another branch to fix the docker build issues

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (6d01ce6) 70.56% compared to head (bbb5fc3) 70.56%.

Additional details and impacted files
@@           Coverage Diff           @@
##              dev     #612   +/-   ##
=======================================
  Coverage   70.56%   70.56%
=======================================
  Files         116      116
  Lines        9316     9316
=======================================
  Hits         6574     6574
  Misses       2742     2742

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


#2534 Linking stages

Closes #2534

Note, that I did not include the test execution in travis.yml as they are not executed right now anyway and it is unclear where/how we will run integration tests in the near future.


Fix python3.8 not found issue at build env

Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Thanks for the PR! This shouldn't be necessary if you follow the steps in the README for creating the Amplify application though. Amplify is configured via an env var to use a custom build image which already includes python3.8 (its a bit hidden, see here). This avoids the extra time it would take to build python3.8 from source for each build.
Let me know the steps you're taking if you're running into issues here, maybe I need to improve the docs.

Hey @cakepietoast , thanks for getting back on this. I was trying to deploy it directly from github using Amplify console and faced this python3.8 not found issue. Needed a quick fix and updated the build spec like that. I think others will try to deploy this app using console by few clicks like me and will face the same issue. Thus sent the quick PR.

Makes sense, I'm reluctant to merge this as-is due to the impact on time to build for those who are using a custom build image that already includes python3.8. With that said, there are definitely improvements we could make here! Would you be willing to alter this PR to perform the steps only on the condition that python3.8 isn't already installed? Perhaps by supplying the installation steps as a shell script and conditionally calling with something like: which python3.8 || ./install_python_3.8.sh
I'd be happy to merge that, then I can update the README to provide a simplified deploy-with-amplify instruction (hopefully as simple as adding a one click deploy button, with an optional extra step to supply a custom image (e.g. one that already has python3.8 installed).

@k-hasan-19 Amplify PM here. We are currently updating the build container to use Python 3.8 by default.

@k-hasan-19 Amplify PM here. We are currently updating the build container to use Python 3.8 by default.

Thanks @swaminator, thats great news. @k-hasan-19 I'll close this and make some improvements to simplify deploying with Amplify. Thanks for the inspiration!

Awesome!


docs(button): add missing props

📝 Description

Add missing props (type & ref) to the button docs.
⛳ Current behavior (updates)

No type & ref props in the docs.
🚀 New behavior

Add type & ref props to the docs
💣 Is this a breaking change (Yes/No):

No
📝 Additional Information

Summary by CodeRabbit

New Features

Enhanced documentation for the Button and ButtonGroup components, including new props for better customization.
Added detailed descriptions for various event handlers, improving clarity on usage.

Bug Fixes

Deprecated the onClick event handler, recommending the use of onPress instead for better functionality.

Documentation

Updated default value for the radius attribute in ButtonGroup from - to xl, reflecting a change in styling.

General props will not be included. We only include our component props there.


Fix tf2 code snippet

Previous code caused an error

@timkondratiev, actually, it's much more natural to import PoseStamped from the original module:
from geometry_msgs.msg import PoseStamped

Note, that import tf2_geometry_msgs is still needed, because this import has a side effect enabling the messages transforming functionality.


Internal - ported the .Net SDK benchmark tool

Goal of this effort is two-folded:

this benchmark tool has the ability to publish run results into a CosmosDB table - making it very easy to generate graphs (like when running continuous tests for example against different commits) or to generate comparison reports when doing runs for a candidate build against a baseline with different concurrency levels
I would like to have the ability to run an "apple-to-apple" comparison for core Read/Write scenarios between .Net and Java SDK to use the results as an signal to prioritize perf work

Next steps (post this PR)

Implementing core Read scenario
"Porting" readme.md with the instructions on how to run the tool
Create a Cosmic notebook that can be used to visualize candidate vs. baseline comparison

/azp run java - cosmos - ci

/azp run java - cosmos - ci


Change from NextAuth v4 syntax to v5 syntaxes

i still have a bug in middleware auth is always null


fix: allow user to define testMatch in package.json (#1067)

Waiting for it to be merged, nice work!

@laander with this additional commit it will respect options in either location

What is the suggested change for when the tests only fail on CI because of timeouts? just increase the timeouts of the failing tests?

You should try to push the commit again, the tests are failing on code not-related to your changes and it seems to pass on other merge requests.
Maybe @yyx990803 can help?

I relaunched tests

a readme edit would have been great. How do I set the testmatch in package.json?


Added builtin 'hasSection' to Page

While making my blog, i wanted to add a TL;DR Section to my blog posts. They all use the same layout, so i integrated the section into that layout. But I wanted the tldr section to be optional in my markdown. First i checked if the contentSection('tldr') had a specific string length, but that was neither consistent nor optimal. So i added a builtin on the Page struct, to check if the markdown has a specific contentSection.
If there already is a way to do this, this PR can be closed and i would love to learn how to do it. Also if there are still things to do before it can be merged, i will gladly add them.

Since a page contains content sections and belogs to a given site section, I'm currently using explicit wording whenever dealing with content sections.
Please change the function name to hasContentSection and I will merge. At some point it might be worth re-evaluating the nomenclature used, but at least in the meantime this explicitness helps avoiding confusion.
Thanks!

Sure, totally understandable. I already pushed the change!


Add First to api reference doc

Pull Request Description
Add First to api_reference.rst.

Looks good, can you update the changelog?


Compatibility certification request for jaxb-ri

[x] Organization Name ("Organization") and, if applicable, URL:
Eclipse Foundation
[x] Product Name, Version and download URL (if applicable):
Eclipse Implementation of JAXB 2.3.2
[x] Specification Name, Version and download URL:
Jakarta XML Binding 2.3
[x] TCK Version, digital SHA-256 fingerprint and download URL:
Jakarta XML Binding TCK 2.3.0, SHA-256: d80963c6235074536fe84292f5e31bba85027cdaee3e77bd9b0178cc1c2d7436
[x] Public URL of TCK Results Summary:
TCK results summary
[x] Any Additional Specification Certification Requirements:
None
[x] Java runtime used to run the implementation:
Oracle JDK 1.8.0_202
[x] Summary of the information for the certification environment, operating system, cloud, ...:
Linux
[x] By checking this box I acknowledge that the Organization I represent accepts the terms of the EFTL.
[x] By checking this box I attest that all TCK requirements have been met, including any compatibility rules.

I'm declaring this approved by lazy consensus.


Laravel 11.x Compatibility

This is an automated pull request from Shift to update your package code and dependencies to be compatible with Laravel 11.x.
Before merging, you need to:

Checkout the l11-compatibility branch
Review all comments for additional changes
Thoroughly test your package

If you do find an issue, please report it by commenting on this PR to help improve future automation.

:warning: Shift detected GitHub Actions which run jobs using a version matrix. Shift attempted to update your configuration for Laravel 11. However, you should review these changes to ensure the desired combination of versions are built for your package.

:alembic: Using this package? If you would like to help test these changes or believe them to be compatible, you may update your project to reference this branch.
To do so, temporarily add Shift's fork to the repositories property of your composer.json:
{
    "repositories": [
        {
            "type": "vcs",
            "url": "https://github.com/laravel-shift/laravel-soulbscription.git"
        }
    ]
}

Then update your dependency constraint to reference this branch:
{
    "require": {
        "lucasdotvin/laravel-soulbscription": "dev-l11-compatibility",
    }
}

Finally, run: composer update

Hey! Thanks for this update! I'm merging this PR right now! ❤


Notebooks can open but not be usable

Does this issue occur when all extensions are disabled?: Yes/No

VS Code Version: 1.84
OS Version: Windows 11

Steps to Reproduce:

Install Pylance extension 2023.10.53
Install Python and Jupyter extensions
Open a notebook
Define a function in one cell
Use the function in another cell
Run both cells
In the second cell goto def on the function
VS code will open a new notebook that looks just like the first one (because the URI returned by the goto def is slightly different)
Run a cell in this new notebook
It won't run
Go back to the first notebook
It won't run either.

The error shown is something like so:
19:35:14.592 [error] Notebook not found for Cell ~/test/test_pyright/test.ipynb
I believe if VS code can open the notebook editor, then it should be able to find a notebook for this editor.
I think if this was fixed, the error in https://github.com/microsoft/vscode/issues/196740 would not have been quite as bad. Instead we would have just had two notebooks open for the same file.

Thanks for creating this issue! It looks like you may be using an old version of VS Code, the latest stable release is 1.84.0. Please try upgrading to the latest version and checking whether this issue remains.
Happy Coding!

The root cause of the URI being different was Pylance removing the extra authority for notebook cells that are not also on the notebook file.

Troubleshooting this issue is like running down a rabbit hole and found multiple places where Uris are not handled properly, here is what I have found so far:

🐛  When find all references on a symbol in a notebook in Remote, Pylance 2023.10.53 returns Cell Uri whose authority was dropped. (Pylance has it fixed now)
The Peek View widget will try to open the text model for this new Cell Uri.

Since it has no authority, the Notebook Uri revived from this Cell Uri is different from current notebook document. It goes through the document open workflow.
❓ FileService loads fs provider based on Uri#scheme. It doesn't seem to care about authority, thus it loads the bytes from Remote Extension Host for the same resource.

https://github.com/microsoft/vscode/blob/540180098a63d237a10dad7a806be30af6cf298e/src/vs/platform/files/common/fileService.ts#L144-L148
@bpasero this means in FileService, a Uri with authority is seen the same as a Uri without authority. Even though technically they are different Uris?

We now have a new notebook document since we get bytes from FileService successfully from previous step.
🐛  The new notebook document has no notebookEditor opened for it, thus when the Inline Chat comparison key provider attempts to revive a notebook from the cell text model uri (as it's reading listNotebookEditors), it throws

https://github.com/microsoft/vscode/blob/540180098a63d237a10dad7a806be30af6cf298e/src/vs/workbench/contrib/inlineChat/browser/inlineChatNotebook.ts#L26-L37

Double click the reference, it will open a new notebook editor

Both notebook documents are now synced to the extension host
🐛 However Python extension doesn't understand the new notebook document without authorityERR Request textDocument/semanticTokens/range failed with message: Debug Failure. Unable to find notebook URI for vscode-notebook-cell:/Users/penlv/code/prototype/notebooks/test.ipynb#W0sdnNjb2RlLXJlbW90ZQ%3D%3D: Error: Request textDocument/semanticTokens/range failed with message: Debug Failure. Unable to find notebook URI for vscode-notebook-cell:/Users/penlv/code/prototype/notebooks/test.ipynb#W0sdnNjb2RlLXJlbW90ZQ%3D%3D
at /Users/penlv/.vscode-server-insiders/extensions/ms-python.python-2023.20.0/out/client/extension.js:2:2354913
at re (/Users/penlv/.vscode-server-insiders/extensions/ms-python.python-2023.20.0/out/client/extension.js:2:2355207)
at /Users/penlv/.vscode-server-insiders/extensions/ms-python.python-2023.20.0/out/client/extension.js:2:2349991
at Immediate.<anonymous> (/Users/penlv/.vscode-server-insiders/extensions/ms-python.python-2023.20.0/out/client/extension.js:2:2350011)
at process.processImmediate (node:internal/timers:476:21)

Close the notebook editor, VS Code now reveals the original notebook document again.
Open Kernel Picker -> Python Environments

Pick a kernel
🐛  Jupyter returns a Kernel Id back to VS Code's Kernel Picker, however the Kernel Id ends with (Interactive), which means it's a kernel created for Interactive Window.
Kernel Picker will then compare the Id with all register Notebook Controller, and couldn't find a matched controller (as all registered controllers for this notebook doesn't have (Jupyter) in their Ids) cc @DonJayamanne

I'm sure we might find more issues with Uri comparisons. However since it started from us getting a Uri which is not valid (no valid authority), how do we prevent the new document to be opened in the first place? Does it make sense to block the file opening for such Uri? @bpasero I'd love to hear your opinions.

The concept of registering a provider to the file service was first introduced in RemoteFileService (https://github.com/microsoft/vscode/commit/2aa549d5c24f74e0dccb00e98f54adc32e6b8810) and then later made more generally available in the IFileService, so adding Jo to comment as well.
I do not fully understand the issue at hand here: the file service is not really declaring that it supports providers based on authority. That is why registerProvider asks for a scheme: string and not a URI:
https://github.com/microsoft/vscode/blob/3e452bfef11522d0151fd2e884bb8bf869d7d2fa/src/vs/platform/files/common/fileService.ts#L52
In other words: file service only distinguishes providers based on scheme. If your URI uses the same scheme but different authority, it does not matter for the file service.

If your URI uses the same scheme but different authority, it does not matter for the file service.

@bpasero the problem we are seeing here is fileService thinks these two Uris (w/, w/o authority) are the same resource (thus loading the same bytes from file system correctly), but the editor service or notebook service doesn't. A more generic question to ask here is, are they the same, what's the suggestion when dealing with these two Uris in extensions?

@rebornix but the file service is not really doing anything, it just delegates to the registered file system provider for the scheme of the Uri. So it is up to the file system provider to handle a possible authority in the Uri or not.
As far as the disk file system provider goes (that I happen to own): the authority part of the Uri is taken into account when we use URI.fsPath here:
https://github.com/microsoft/vscode/blob/0bfcf3bb2ec892609ce5848ad5bd8443b12a2234/src/vs/platform/files/common/diskFileSystemProvider.ts#L217-L219
Because of:
https://github.com/microsoft/vscode/blob/0bfcf3bb2ec892609ce5848ad5bd8443b12a2234/src/vs/base/common/uri.ts#L620-L623
What file system provider is triggered here in this case?

We are only seeing this in Remote setup, so I think it's the remote file system provider?

What is an example of a URI?

When find all references on a symbol in a notebook in Remote, Pylance 2023.10.53 returns Cell Uri whose authority was dropped. (Pylance has it fixed now)

Unsure what we are investigating here? If an extension gives us an uri that is valid (form wise) but pointing to the wrong or non-existent resource then there is not much we can do here.

Final ping before closing this w/o action


JupyterHub instance won't start

Hi,
I tried opening my JupyterHub to finish the exercises, however it does not start up anymore. After waiting for a long time, I get the following ErrorMessage:

Yesterday and the days before it still worked fine and I do not know how to fix this problem...

Thank you for the report, which I'm able to reproduce, we're passing it on to our backroom team, but I'm afraid I can't guarantee an immediate fix, which may have to wait until we can reboot the system in the morning.

The department IT team have upped the maximum resources available on the JHub instance, and hopefully  everyone that needs it can now get access. Please do let us know if it's working for you too.

Yes it is working again. Thank you!


Theft and Custody

Re-flagging this as something potentially worth talking about—Art Tracks has modeled both theft and loss as ownership transfers, not as merely custody transfers—If a work is documented as stolen/and or lost, there is no functional ownership.
(This would invalidate and/or reopen issue #10.)
Not a deal-breaker, and we'd be willing to modify our model to accommodate—better to have a consistent concept across the two.

Propose to move modeling discussion to azaroth42/museum-lod and keep this set for just library issues?

👍🏼


runtime: Kubernetes' kube-proxy stuck in GC forever

Please answer these questions before submitting your issue. Thanks!
What version of Go are you using (go version)?
go1.7.4 linux/amd64
What operating system and processor architecture are you using (go env)?
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp.k8s/go-build739048657=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"

(from k8s build container)
What did you do?
Running k8s cluster
What did you expect to see?
kube-proxy process running as expected
What did you see instead?
kube-proxy gets stuck in GC code, with no goroutines being scheduled
The problem here is that the problem is quite hard to reproduce, something similar happens on some nodes from time to time, may take some days to run into the problem. We currently have a process that's in the state described here running on one of the nodes. I'm pretty sure it's related to Go runtime but in any case I'm stuck trying to find a way to debug it. Any hints on what needs to be done to find out the cause of the problem would be very appreciated. I don't want to kill the process with SIGQUIT to retrieve goroutine info, so doing this with delve instead.
The process is running in docker container (debian jessie), GOMAXPROCS=48
Goroutine info: https://gist.github.com/ivan4th/17654f6fee35a38548502de4b6f68ce4
Thread info: https://gist.github.com/ivan4th/4596664ba1f935c500250f74ade5c162
Log output from another hung kube-proxy killed with SIGQUIT: https://gist.github.com/ivan4th/5da95ebf8986c6834bca35c9b4e7895b

Accidentally posted the issue before I completed editing, will update shortly with more details, sorry

What I could get out of GDB (maybe I should try newer version?):
Program received signal SIGINT, Interrupt.
runtime.usleep () at /usr/local/go/src/runtime/sys_linux_amd64.s:95
95	/usr/local/go/src/runtime/sys_linux_amd64.s: No such file or directory.
(gdb) bt
#0  runtime.usleep () at /usr/local/go/src/runtime/sys_linux_amd64.s:95
#1  0x0000000000424f7d in runtime.getfull (~r0=0xc42002d228) at /usr/local/go/src/runtime/mgcwork.go:411
#2  0x000000000042479f in runtime.(*gcWork).get (w=0x42279d <runtime.greyobject+301>, ~r0=0) at /usr/local/go/src/runtime/mgcwork.go:199
#3  0x0000000000421e3a in runtime.gcDrain (gcw=0xc42002d228, flags=0) at /usr/local/go/src/runtime/mgcmark.go:982
#4  0x000000000041f579 in runtime.gchelper () at /usr/local/go/src/runtime/mgc.go:1819
#5  0x0000000000436a1f in runtime.stopm () at /usr/local/go/src/runtime/proc.go:1597
#6  0x00000000004377f8 in runtime.findrunnable (gp=0x43e30e <runtime.runqget+142>, inheritTime=false) at /usr/local/go/src/runtime/proc.go:2021
#7  0x000000000043836c in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2120
#8  0x00000000004386e3 in runtime.park_m (gp=0xc420461040) at /usr/local/go/src/runtime/proc.go:2183
#9  0x000000000046111b in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:240
#10 0x0000000005cace00 in runtime.work ()
#11 0x00007ffca16997f0 in ?? ()
#12 0x0000000005cacea0 in ?? ()
#13 0x00007ffca16997e0 in ?? ()
#14 0x0000000000435914 in runtime.mstart () at /usr/local/go/src/runtime/proc.go:1096
#15 0x0000000000460fc5 in runtime.rt0_go () at /usr/local/go/src/runtime/asm_amd64.s:156
#16 0x0000000000000009 in ?? ()
#17 0x00007ffca16998f8 in ?? ()
#18 0x0000000000000009 in ?? ()
#19 0x00007ffca16998f8 in ?? ()
#20 0x0000000000000000 in ?? ()
(gdb) quit

Most of kube-proxy source is here: https://github.com/kubernetes/kubernetes/tree/v1.5.2/pkg/proxy

Per-process CPU usage (just pasting this from htop, see 9th column) -- first line is for the whole process, following lines correspond to threads:
   1 root       20   0 1612M 90760 49692 S 51.9  0.0 21h29:31 /hyperkube proxy ...
   27 root       20   0 1612M 90760 49692 R  6.9  0.0  2h12:50 /hyperkube proxy ...
 4959 root       20   0 1612M 90760 49692 R  6.4  0.0  2h12:58 /hyperkube proxy ...
   14 root       20   0 1612M 90760 49692 R  6.0  0.0  2h15:43 /hyperkube proxy ...
34689 root       20   0 1612M 90760 49692 R  4.6  0.0  2h12:53 /hyperkube proxy ...
   18 root       20   0 1612M 90760 49692 R  4.1  0.0  2h13:15 /hyperkube proxy ...
   15 root       20   0 1612M 90760 49692 R  3.7  0.0  2h13:07 /hyperkube proxy ...
   17 root       20   0 1612M 90760 49692 S  0.9  0.0 18:41.65 /hyperkube proxy ...
   19 root       20   0 1612M 90760 49692 S  0.9  0.0 18:36.90 /hyperkube proxy ...
   16 root       20   0 1612M 90760 49692 S  0.9  0.0 18:44.86 /hyperkube proxy ...
 1285 root       20   0 1612M 90760 49692 S  0.9  0.0 18:38.19 /hyperkube proxy ...
 1829 root       20   0 1612M 90760 49692 S  0.9  0.0 18:43.81 /hyperkube proxy ...
   29 root       20   0 1612M 90760 49692 R  0.9  0.0 18:35.07 /hyperkube proxy ...
 4368 root       20   0 1612M 90760 49692 S  0.9  0.0 18:35.75 /hyperkube proxy ...
   20 root       20   0 1612M 90760 49692 S  0.9  0.0 18:41.37 /hyperkube proxy ...
   34 root       20   0 1612M 90760 49692 S  0.9  0.0 18:31.79 /hyperkube proxy ...
   12 root       20   0 1612M 90760 49692 S  0.9  0.0 18:40.26 /hyperkube proxy ...
   10 root       20   0 1612M 90760 49692 S  0.9  0.0 18:40.57 /hyperkube proxy ...
   13 root       20   0 1612M 90760 49692 S  0.9  0.0 18:39.82 /hyperkube proxy ...
   31 root       20   0 1612M 90760 49692 S  0.9  0.0 18:38.71 /hyperkube proxy ...
   28 root       20   0 1612M 90760 49692 S  0.9  0.0 18:38.91 /hyperkube proxy ...
   26 root       20   0 1612M 90760 49692 R  0.9  0.0 18:37.23 /hyperkube proxy ...
 1827 root       20   0 1612M 90760 49692 S  0.9  0.0 18:38.38 /hyperkube proxy ...
  733 root       20   0 1612M 90760 49692 S  0.9  0.0 18:36.00 /hyperkube proxy ...
  199 root       20   0 1612M 90760 49692 R  0.9  0.0 18:37.56 /hyperkube proxy ...
 1283 root       20   0 1612M 90760 49692 R  0.5  0.0 18:36.17 /hyperkube proxy ...
   30 root       20   0 1612M 90760 49692 S  0.5  0.0 18:41.68 /hyperkube proxy ...
47869 root       20   0 1612M 90760 49692 R  0.5  0.0 18:14.54 /hyperkube proxy ...
   25 root       20   0 1612M 90760 49692 S  0.5  0.0 18:37.63 /hyperkube proxy ...
   32 root       20   0 1612M 90760 49692 S  0.5  0.0 18:40.23 /hyperkube proxy ...
   22 root       20   0 1612M 90760 49692 S  0.5  0.0 18:38.95 /hyperkube proxy ...
    8 root       20   0 1612M 90760 49692 S  0.5  0.0 18:40.63 /hyperkube proxy ...

Can you run it with environment GOTRACEBACK=system set and then send it SIGQUIT when it gets into that state and capture the output?
That might be more useful to see Go's own output, rather than gdb's attempt.

Thanks for the very detailed report. It looks like we're in a stack-scanning deadlock.
Narrowing down to just the threads that are in scang (which happen to be exactly the threads that are high in top):
(dlv) thread 14
Switched from 13 to 14
(dlv) bt -full
 0  0x0000000000465237 in runtime.osyield
    at .usr/local/go/src/runtime/sys_linux_amd64.s:479
 1  0x000000000043500b in runtime.scang
    at .usr/local/go/src/runtime/proc.go:865
        gp = (*runtime.g)(0xc420221380)
        gcw = (*runtime.gcWork)(0xc420017c28)
        nextYield =<PHONE_NUMBER>93231
        i =<PHONE_NUMBER>79
 2  0x0000000000459e8f in runtime.markroot.func1
    at .usr/local/go/src/runtime/mgcmark.go:236
        selfScan = false
        userG = *runtime.g nil
        gp = (*runtime.g)(0xc420221380)
        gcw = (*runtime.gcWork)(0xc420017c28)

(dlv) thread 15
Switched from 14 to 15
(dlv) bt -full
 0  0x0000000000464e40 in runtime.nanotime
    at .usr/local/go/src/runtime/sys_linux_amd64.s:174
 1  0x0000000000434f66 in runtime.scang
    at .usr/local/go/src/runtime/proc.go:862
        gp = (*runtime.g)(0xc4202211e0)
        gcw = (*runtime.gcWork)(0xc420016728)
        nextYield =<PHONE_NUMBER>32703
        i =<PHONE_NUMBER>9
 2  0x0000000000459e8f in runtime.markroot.func1
    at .usr/local/go/src/runtime/mgcmark.go:236
        selfScan = false
        userG = *runtime.g nil
        gp = (*runtime.g)(0xc4202211e0)
        gcw = (*runtime.gcWork)(0xc420016728)

(dlv) thread 18
Switched from 17 to 18
(dlv) bt -full
 0  0x0000000000434885 in runtime.readgstatus
    at .usr/local/go/src/runtime/proc.go:646
        gp = (*runtime.g)(0xc420221ba0)
        ~r1 = 2
 1  0x0000000000434f7a in runtime.scang
    at .usr/local/go/src/runtime/proc.go:808
        gp = (*runtime.g)(0xc420221ba0)
        gcw = (*runtime.gcWork)(0xc42001bc28)
        nextYield =<PHONE_NUMBER>56415
        i =<PHONE_NUMBER>8
 2  0x0000000000459e8f in runtime.markroot.func1
    at .usr/local/go/src/runtime/mgcmark.go:236
        selfScan = false
        userG = *runtime.g nil
        gp = (*runtime.g)(0xc420221ba0)
        gcw = (*runtime.gcWork)(0xc42001bc28)

(dlv) thread 27
Switched from 26 to 27
(dlv) bt -full
 0  0x0000000000462d94 in runtime.procyield
    at .usr/local/go/src/runtime/asm_amd64.s:509
 1  0x0000000000434fe4 in runtime.scang
    at .usr/local/go/src/runtime/proc.go:863
        gp = (*runtime.g)(0xc420221520)
        gcw = (*runtime.gcWork)(0xc420022728)
        nextYield =<PHONE_NUMBER>65810
        i =<PHONE_NUMBER>0
 2  0x0000000000459e8f in runtime.markroot.func1
    at .usr/local/go/src/runtime/mgcmark.go:236
        selfScan = false
        userG = *runtime.g nil
        gp = (*runtime.g)(0xc420221520)
        gcw = (*runtime.gcWork)(0xc420022728)

(dlv) thread 4959
Switched from 4368 to 4959
(dlv) bt -full
 0  0x0000000000465237 in runtime.osyield
    at .usr/local/go/src/runtime/sys_linux_amd64.s:479
 1  0x000000000043500b in runtime.scang
    at .usr/local/go/src/runtime/proc.go:865
        gp = (*runtime.g)(0xc420221d40)
        gcw = (*runtime.gcWork)(0xc42001a728)
        nextYield =<PHONE_NUMBER>77361
        i =<PHONE_NUMBER>2
 2  0x0000000000459e8f in runtime.markroot.func1
    at .usr/local/go/src/runtime/mgcmark.go:236
        selfScan = false
        userG = *runtime.g nil
        gp = (*runtime.g)(0xc420221d40)
        gcw = (*runtime.gcWork)(0xc42001a728)

(dlv) thread 34689
Switched from 4959 to 34689
(dlv) bt -full
 0  0x0000000000434ff5 in runtime.scang
    at .usr/local/go/src/runtime/proc.go:807
        gp = (*runtime.g)(0xc420221a00)
        gcw = (*runtime.gcWork)(0xc420023c28)
        nextYield =<PHONE_NUMBER>75725
        i =<PHONE_NUMBER>0
 1  0x0000000000459e8f in runtime.markroot.func1
    at .usr/local/go/src/runtime/mgcmark.go:236
        selfScan = false
        userG = *runtime.g nil
        gp = (*runtime.g)(0xc420221a00)
        gcw = (*runtime.gcWork)(0xc420023c28)

It would be really useful if I could see the contents of *gp for each of these threads. Could you get that from delve or gdb?

As of GOTRACEBACK, ok, we will do this and wait for more occurrences. Just thought that I can do something with
the hung process I have currently meanwhile.
gp details - will paste in a moment.
Some observations follow. I'm not an expert in go runtime, so I may be wrong. Most threads are in gc.getfull(). Meanwhile:
(dlv) print runtime.work.nproc
32
(dlv) print runtime.work.nwait
26

6 remaining threads are in runtime.scang:
(dlv) bt
 0  0x0000000000465237 in runtime.osyield
    at .usr/local/go/src/runtime/sys_linux_amd64.s:479
 1  0x000000000043500b in runtime.scang
    at .usr/local/go/src/runtime/proc.go:865
 2  0x0000000000459e8f in runtime.markroot.func1
    at .usr/local/go/src/runtime/mgcmark.go:236
 3  0x00000000004611e9 in runtime.systemstack
    at .usr/local/go/src/runtime/asm_amd64.s:314
 4  0x0000000000420111 in runtime.markroot
    at .usr/local/go/src/runtime/mgcmark.go:241
 5  0x0000000000421cf9 in runtime.gcDrain
    at .usr/local/go/src/runtime/mgcmark.go:963
 6  0x000000000041f567 in runtime.gchelper
    at .usr/local/go/src/runtime/mgc.go:1819
 7  0x0000000000436a1a in runtime.stopm
    at .usr/local/go/src/runtime/proc.go:1597
 8  0x00000000004377f3 in runtime.findrunnable
    at .usr/local/go/src/runtime/proc.go:2021
 9  0x0000000000438367 in runtime.schedule
    at .usr/local/go/src/runtime/proc.go:2120
10  0x00000000004386de in runtime.park_m
    at .usr/local/go/src/runtime/proc.go:2183

Closer inspection reveals that gp.preemptscan && gp.preempt && gp.stackguard0 == stackPreempt condition is true in each of these places, but corresponding goroutines that must scan their stacks can't do this. You can see them in goroutine list, the look like this: ```  Goroutine 26 - User: .usr/local/go/src/runtime/mbitmap.go:388 runtime.heapBitsForObject (0x417350)

but delve can't show the stacks for them, complaining about null pointers

Will try to grab gp details now

Once you get the gp details, it would be really helpful to know their stack traces. I'm not sure why delve can't get them, but if you grab my gdbinit.py from https://github.com/aclements/my-dotfiles/blob/master/gdbinit.py, you can probably get their stacks by running btg $getg(N) where N is the goroutine ID. While you've got that loaded, it may also be helpful to have to output of print-sched (though it probably won't tell us anything you haven't already reported).

gp details: https://gist.github.com/ivan4th/d031a21b5d1043d04f4b74108b4474ff
Note that I did some c + Ctrl-cs since I captured the initial data, but all scang threads still stay in scang and getfull ones stay in getfull - this doesn't change.

Active scans:

thread
gp
goid
gstatus
target g's thread

14
0xc420221380
22
running
22

15
0xc4202211e0
21
running
14

18
0xc420221ba0
27
running
18

27
0xc420221520
23
running
26

4959
0xc420221d40
28
running
32

34689
0xc420221a00
26
running
25

All target Gs have waitreason "GC worker (idle)" and their start PCs are all the same, so these are GC workers.
It's worth noting that we're in mark termation, so these really shouldn't be in status running. STW should have moved them to runnable. We're stuck because stack scan needs to get these out of running, but we can't preempt anything during STW (because it shouldn't be running at all!), so we're just spinning around.
So, it actually could be useful to see the output of print-sched.

print-sched output is here https://gist.github.com/ivan4th/3e53dde8c1b6c3b559b6901aff176bec

btg doesn't work though :(
(gdb) btg $getg(21)
goroutine 21 stack:
thread 14 not found; stack may be incorrect (try import _ "rutime/cgo")
Python Exception <class 'gdb.MemoryError'> Cannot access memory at address 0x18:
Error occurred in Python command: Cannot access memory at address 0x18

It may be related to the environment, got some errors when attaching to the process:
(gdb) attach 1
Attaching to process 1
Reading symbols from /hyperkube...done.
warning: Missing auto-load scripts referenced in section .debug_gdb_scripts
of file /hyperkube
Use `info auto-load python-scripts [REGEXP]' to list them.
warning: .dynamic section for "/lib/x86_64-linux-gnu/libpthread.so.0" is not at the expected address (wrong library or version mismatch?)
Reading symbols from /lib/x86_64-linux-gnu/libpthread.so.0...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libpthread-2.19.so...done.
done.
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Loaded symbols for /lib/x86_64-linux-gnu/libpthread.so.0
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libdl-2.19.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libdl.so.2
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libc-2.19.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/ld-2.19.so...done.
done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib/x86_64-linux-gnu/libnss_compat.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnss_compat-2.19.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_compat.so.2
Reading symbols from /lib/x86_64-linux-gnu/libnsl.so.1...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnsl-2.19.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libnsl.so.1
Reading symbols from /lib/x86_64-linux-gnu/libnss_nis.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnss_nis-2.19.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_nis.so.2
Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnss_files-2.19.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

(yes, gdb just can't work with threads in this container... will try to fix it. meanwhile see print-sched output above)

Based on print-sched, these six goroutines that we're trying to scan are the only ones in "running":
      21 running, m 9, p 2, dedicated mark worker
         started at runtime.gcBgMarkWorker
         created by runtime.gcBgMarkStartWorkers
      22 running, m 17, p 8, dedicated mark worker
         started at runtime.gcBgMarkWorker
         created by runtime.gcBgMarkStartWorkers
      23 running, m 21, p 7, dedicated mark worker
         started at runtime.gcBgMarkWorker
         created by runtime.gcBgMarkStartWorkers
      26 running, m 20, p 3, dedicated mark worker
         started at runtime.gcBgMarkWorker
         created by runtime.gcBgMarkStartWorkers
      27 running, m 13, p 5, dedicated mark worker
         started at runtime.gcBgMarkWorker
         created by runtime.gcBgMarkStartWorkers
      28 running, m 27, p 9, idle mark worker
         started at runtime.gcBgMarkWorker
         created by runtime.gcBgMarkStartWorkers

I wouldn't totally trust the M and P numbers here, since these goroutines clearly aren't actually running, but these probably show the last M that was running each goroutine and whatever P that M was on when it was stopped for STW mark termination.
These Gs correspond to the following Ms (same order):
       9 PID 14, p 2, helpgc
      17 PID 22, p 8, helpgc
      21 PID 26, p 7, helpgc
      20 PID 25, p 3, helpgc
      13 PID 18, p 5, helpgc
      27 PID 32, p 9, helpgc

And the following Ps:
       2 gcstop, mark worker g 22
       8 gcstop, mark worker g 28
       7 gcstop, mark worker g 27
       3 gcstop, mark worker g 23
       5 gcstop, mark worker g 25
       9 gcstop, mark worker g 29

It's interesting that all of these Ps have assigned mark workers that are not the mark worker that claims it was running on that P. It could just be that the M and P numbers for each G are bogus, or this could mean something.
One good thing is that this is almost certainly not an issue in Go 1.8 because Go 1.8 doesn't scan stacks during mark termination.

@ivan4th, another thing to do in gdb on the running image:
Assuming you've still got my gdbinit.py loaded, run the python command at the GDB prompt and paste in the following:
def btfrom(g):
    pc, sp = g["sched"]["pc"], g["sched"]["sp"]
    oldsp, oldpc = gdb.parse_and_eval('$sp'), gdb.parse_and_eval('$pc')
    try:
        # TODO: This fails if we're not in the innermost frame.
        gdb.execute('set $sp = %#x' % sp)
        gdb.execute('set $pc = %#x' % pc)
        gdb.execute('backtrace')
    finally:
        gdb.execute('set $sp = %#x' % oldsp)
        gdb.execute('set $pc = %#x' % oldpc)

for goid in [21, 22, 23, 26, 27, 28]:
    print "Goroutine", goid, "sched stack:"
    btfrom(getg(n=goid))
    print

Hit Ctrl-D to run it and get back to the GDB prompt. This should dump out the stacks of these goroutines from where they actually stopped (ignoring their claim that they are in state "running").

Unfortunately I got this:
>Goroutine 21 sched stack:
Traceback (most recent call last):
  File "<string>", line 15, in <module>
  File "<string>", line 6, in btfrom
gdb.MemoryError: Cannot access memory at address 0x18
Error while executing Python code.

Actually the same error happens for each goroutine.

Yeah, looking back at your dump of the gp's, their saved SPs are all 0.
That might actually be a clue. I think the only thing that puts 0 there is returning from systemstack, which suggests that we somehow returned from systemstack on these and then didn't update their sched buffers again.

It's interesting that all of these Ps have assigned mark workers that are not the mark worker that claims it was running on that P. It could just be that the M and P numbers for each G are bogus, or this could mean something.

On further thought, I don't think this indicates anything. Looking at the P's assigned mark workers, they're all exactly in sequence, except for P 20, which would clearly have mark worker G 40 except that mark worker 40 is the one that disassociated and is driving mark termination. These are exactly the 48 mark workers, so nothing funny happened where on disassociated from a P and was replaced by another one.
@ivan4th, another potentially useful thing you can get from GDB or delve is the value of runtime.work, particularly runtime.work.rescan.list.

(dlv) print runtime.work.rescan.list
[]runtime.guintptr len: 51, cap: 1312, [842352693312,842352693728,842352694144,842352694560,842352694976,842352695392,842352695808,842352696224,842352696640,842355048448,842355049280,842355048864,842355050112,842355050528,842355051360,842355049696,842355051776,842355052192,842355052608,842355053440,842355053024,842355054272,842355054688,842355055104,842355055520,842355055936,842355050944,842355089408,842355089824,842355090656,842355090240,842355053856,842355091904,842355091072,842355092736,842355091488,842355093568,842355093984,842355093152,842355092320,842355094816,842355095232,842355095648,842355096480,842355096896,842355138560,842355096064,842355094400,842359741760,842366314528,842353073472]

struct { full uint64; empty uint64; pad0 [64]uint8; markrootNext uint32; markrootJobs uint32; nproc uint32; tstart int64; nwait uint32; ndone uint32; alldone runtime.note; nDataRoots int; nBSSRoots int; nSpanRoots int; nStackRoots int; nRescanRoots int; markrootDone bool; startSema uint32; markDoneSema uint32; bgMarkReady runtime.note; bgMarkDone uint32; mode runtime.gcMode; spans []*runtime.mspan; totaltime int64; bytesMarked uint64; initialHeapLive uint64; assistQueue struct { lock runtime.mutex; head runtime.guintptr; tail runtime.guintptr }; rescan struct { lock runtime.mutex; list []runtime.guintptr }; stwprocs int32; maxprocs int32; tSweepTerm int64; tMark int64; tMarkTerm int64; tEnd int64; pauseNS int64; pauseStart int64; heap0 uint64; heap1 uint64; heap2 uint64; heapGoal uint64 } {
        full: 0,
        empty:<PHONE_NUMBER>421147533,
        pad0: [64]uint8 [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
        markrootNext: 60,
        markrootJobs: 54,
        nproc: 32,
        tstart:<PHONE_NUMBER>33906,
        nwait: 26,
        ndone: 0,
        alldone: runtime.note {key: 0},
        nDataRoots: 0,
        nBSSRoots: 0,
        nSpanRoots: 0,
        nStackRoots: 0,
        nRescanRoots: 51,
        markrootDone: true,
        startSema: 1,
        markDoneSema: 1,
        bgMarkReady: runtime.note {key: 0},
        bgMarkDone: 0,
        mode: 0,
        spans: []*runtime.mspan len: 1977, cap: 8192, [
                *(*runtime.mspan)(0x7f4965e3e000),
                *(*runtime.mspan)(0x7f4965e3e0a0),
                *(*runtime.mspan)(0x7f4965e3e140),
                *(*runtime.mspan)(0x7f4965e3e1e0),
                *(*runtime.mspan)(0x7f4965e3e280),
                *(*runtime.mspan)(0x7f4965e3e320),
                *(*runtime.mspan)(0x7f4965e3e3c0),
                *(*runtime.mspan)(0x7f4965e3e460),
                *(*runtime.mspan)(0x7f4965e3e500),
                *(*runtime.mspan)(0x7f4965e3e5a0),
                *(*runtime.mspan)(0x7f4965e3e640),
                *(*runtime.mspan)(0x7f4965e3e6e0),
                *(*runtime.mspan)(0x7f4965e3e780),
                *(*runtime.mspan)(0x7f4965e3e820),
                *(*runtime.mspan)(0x7f4965e3e8c0),
                *(*runtime.mspan)(0x7f4965e3e960),
                *(*runtime.mspan)(0x7f4965e3ea00),
                *(*runtime.mspan)(0x7f4965e3eaa0),
                *(*runtime.mspan)(0x7f4965e3eb40),
                *(*runtime.mspan)(0x7f4965e3ebe0),
                *(*runtime.mspan)(0x7f4965e3ec80),
                *(*runtime.mspan)(0x7f4965e3ed20),
                *(*runtime.mspan)(0x7f4965e3edc0),
                *(*runtime.mspan)(0x7f4965e3ee60),
                *(*runtime.mspan)(0x7f4965e3ef00),
                *(*runtime.mspan)(0x7f4965e3efa0),
                *(*runtime.mspan)(0x7f4965e3f040),
                *(*runtime.mspan)(0x7f4965e3f0e0),
                *(*runtime.mspan)(0x7f4965e3f180),
                *(*runtime.mspan)(0x7f4965e3f220),
                *(*runtime.mspan)(0x7f4965e3f2c0),
                *(*runtime.mspan)(0x7f4965e3f360),
                *(*runtime.mspan)(0x7f4965e3f400),
                *(*runtime.mspan)(0x7f4965e3f4a0),
                *(*runtime.mspan)(0x7f4965e3f540),
                *(*runtime.mspan)(0x7f4965e3f5e0),
                *(*runtime.mspan)(0x7f4965e3f680),
                *(*runtime.mspan)(0x7f4965e3f720),
                *(*runtime.mspan)(0x7f4965e3f7c0),
                *(*runtime.mspan)(0x7f4965e3f860),
                *(*runtime.mspan)(0x7f4965e3f900),
                *(*runtime.mspan)(0x7f4965e3f9a0),
                *(*runtime.mspan)(0x7f4965e3fa40),
                *(*runtime.mspan)(0x7f4965e3fae0),
                *(*runtime.mspan)(0x7f4965e3fb80),
                *(*runtime.mspan)(0x7f4965e3fc20),
                *(*runtime.mspan)(0x7f4965e3fcc0),
                *(*runtime.mspan)(0x7f4965e3fd60),
                *(*runtime.mspan)(0x7f4965e3fe00),
                *(*runtime.mspan)(0x7f4965e3fea0),
                *(*runtime.mspan)(0x7f4965e3ff40),
                *(*runtime.mspan)(0x7f4965e3ffe0),
                *(*runtime.mspan)(0x7f4965e40080),
                *(*runtime.mspan)(0x7f4965e40120),
                *(*runtime.mspan)(0x7f4965e401c0),
                *(*runtime.mspan)(0x7f4965e40260),
                *(*runtime.mspan)(0x7f4965e40300),
                *(*runtime.mspan)(0x7f4965e403a0),
                *(*runtime.mspan)(0x7f4965e40440),
                *(*runtime.mspan)(0x7f4965e404e0),
                *(*runtime.mspan)(0x7f4965e40580),
                *(*runtime.mspan)(0x7f4965e40620),
                *(*runtime.mspan)(0x7f4965e406c0),
                *(*runtime.mspan)(0x7f4965e40760),
                ...+1913 more
        ],
        totaltime:<PHONE_NUMBER>062,
        bytesMarked: 11343072,
        initialHeapLive: 22322304,
        assistQueue: struct { lock runtime.mutex; head runtime.guintptr; tail runtime.guintptr } {
                lock: (*runtime.mutex)(0x5cacdf8),
                head: 0,
                tail: 0,},
        rescan: struct { lock runtime.mutex; list []runtime.guintptr } {
                lock: (*runtime.mutex)(0x5cace10),
                list: []runtime.guintptr len: 51, cap: 1312, [842352693312,842352693728,842352694144,842352694560,842352694976,842352695392,842352695808,842352696224,842352696640,842355048448,842355049280,842355048864,842355050112,842355050528,842355051360,842355049696,842355051776,842355052192,842355052608,842355053440,842355053024,842355054272,842355054688,842355055104,842355055520,842355055936,842355050944,842355089408,842355089824,842355090656,842355090240,842355053856,842355091904,842355091072,842355092736,842355091488,842355093568,842355093984,842355093152,842355092320,842355094816,842355095232,842355095648,842355096480,842355096896,842355138560,842355096064,842355094400,842359741760,842366314528,842353073472],},
        stwprocs: 32,
        maxprocs: 48,
        tSweepTerm:<PHONE_NUMBER>83441,
        tMark:<PHONE_NUMBER>13122,
        tMarkTerm:<PHONE_NUMBER>24180,
        tEnd:<PHONE_NUMBER>69890,
        pauseNS: 29681,
        pauseStart:<PHONE_NUMBER>24180,
        heap0: 22322304,
        heap1: 22570272,
        heap2: 11411928,
        heapGoal: 23370880,}

We had the same issue with kube-proxy at least two times.

@aclements, what's the status here? Is there something you still want to do for Go 1.9? Or do you suspect it's already been fixed?

@bradfitz, unfortunately I still have no idea what's going on here. But it may have been related to stack rescanning and 1.8 doesn't rescan stacks and 1.9 will remove the rescan mechanism entirely, so it's possible this is fixed by side-effect.
@r7vme, what version of Go was your crashing kube-proxy built with?

@aclements not sure, but kube-dns --version says Kubernetes v1.4.0-alpha.0.1055+6de83d58537f57-dirty

@r7vme @aclements on phone but #13507 should provide hints for how to find what Go version the binary was built with. (Maybe we should make a go-gettable tool for this.)

(gdb) p 'runtime.buildVersion'

Go-gettable tool: https://github.com/loeyt/go-version

I thought kubernetes is still on 1.7.6, based on the release notes for k8s
version 1.6.6 that I read last week
On Tue, 20 Jun 2017, 06:48 Damian Gryski<EMAIL_ADDRESS>wrote:

Go-gettable tool: https://github.com/loeyt/go-version
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/18925#issuecomment-309569263, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAcAx4peTc4LunhrXsKEMpmsyrjCebJks5sFt6lgaJpZM4L2pVl
.

@josharian @aclements messed up stuff. Yes, we use kube-proxy from [hyperkube v1.5.2_coreos.0].(https://quay.io/repository/coreos/hyperkube/image/90b25ac3f174580be7e1f6f42a5bc77a52984bdad7fb8c61136d90012b648733)
[root@0000dd1bb02a188c tmp]# gdb hyperkube
GNU gdb (GDB) Fedora 7.11.1-86.fc24
...
(gdb) p 'runtime.buildVersion'
$1 = 0x344629c "go1.7.4"

Based on https://github.com/kubernetes/kubernetes/issues/38228, Kubernetes 1.7 should be based on Go 1.8, which I believe does not have this bug. Kubernetes 1.7 should be out next week (June 28th).
Has anybody stress tested or run into this problem with the Kubernetes 1.7 beta?

Closing this, assuming it's fixed in Kubernetes 1.7 (Go 1.8). Please comment if not and we'll re-open.


Netmask for IPSec routed tunnels is calculated incorrectly

Important notices
Before you add a new report, we ask you kindly to acknowledge the following:

[X] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
[X] I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug
A clear and concise description of what the bug is, including last known working version (if any).
Tip: to validate your setup was working with the previous version, use opnsense-revert (https://docs.opnsense.org/manual/opnsense_tools.html#opnsense-revert)
To Reproduce
Steps to reproduce the behavior:
Create a routed ipsec tunnel. Enter values for local and remote address that are not neighbors (form a /30 network).
Then, the subnet mask is automatically calculated in weird and wildly wrong ways.
Expected behavior
There is a local and a remote address, but no subnet mask can be entered. Thus, the subnet mask should be automatically calculated correctly. Or better still, I expected a way to enter a subnet mask.
Describe alternatives you considered
Just use /30 networks, that should in theory work.
Additional context
This has been extensively discussed in the forum here: https://forum.opnsense.org/index.php?topic=24546.0 and here: https://forum.opnsense.org/index.php?topic=23901.0 . The exact place it goes wrong is here as referenced by franco in the forums: https://github.com/opnsense/core/blob/1aeb52063159df6da3546d132a765557e6a96a6d/src/etc/inc/plugins.inc.d/ipsec.inc#L1871 .
Now what I would wish to be the solution is to be able to enter the subnet mask manually, since no automatism can possibly encompass all the reasons someone might have to use a specific mask (at least, I don't see how it would decide between /30 and /31), but at least it should calculate it correctly, so it doesn't fail if the IP addresses are too far apart or use an unexpected value.
Environment
Software version used and hardware type if relevant, e.g.:
OPNsense 21.7 (amd64, OpenSSL), but probably every version since the feature was introduced.

The current calculation intends to return the smallest possible subnet where both addresses can reach each other, which is basically the only purpose of specifying the net here. If we can’t calculate it properly, a workaround might be to add a subnet selector, although this will lead to confusion by other users… it’s probably a good idea to extend the ticket with some test cases first, which work and which are (obviously) wrong.

Sounds similar (or duplicate) of https://github.com/opnsense/core/issues/4029
There's also an explanation on which cases it fails or might fail

It pretty much fails all the time, because it does not use a base 2 logarithm, but subtracts the IPs from one another. In one of the threads I referenced a subnet mask of -68 was calculated.
Besides, with IPv6 you would want to be able to decide if you want to use a /127, a /126 or a /64 as link network.
Also, this might really be a duplicate of #4029, as mimugmail mentions.

Let’s try to reason if this is only a bug (wrong net calculated) or there is a missing feature as well.

Besides, with IPv6 you would want to be able to decide if you want to use a /127, a /126 or a /64 as link network.

Which use case would require to reach other addresses then the predefined tunnel endpoint? ( hence requiring a larger net than minimum)
not saying there is an easy fix for the calculation, but if there is a fix, it might be cleaner given the existing situation.

Let’s try to reason if this is only a bug (wrong net calculated) or there is a missing feature as well.

Besides, with IPv6 you would want to be able to decide if you want to use a /127, a /126 or a /64 as link network.

Which use case would require to reach other addresses then the predefined tunnel endpoint? ( hence requiring a larger net than minimum)
not saying there is an easy fix for the calculation, but if there is a fix, it might be cleaner given the existing situation.

In IPv4 I would want to specify whether it should be a /31 or a /30, since both are perfectly valid.
Similar thing about /126 and /127. Additionally with /64 all the IPv6 mechanisms for SLAAC and link-local addresses could be used.

Don’t get me wrong, but I haven’t heard a reason why different options should be possible if the calculation is correct (which currently isn’t the case). If there are valid use cases where not only both ends of the tunnel should be accessible, that would be good to know, when trying to come up with a fix.
working around the problem by changing the scenery usually has downsides for others.
When this turns out to be only a bug, we should try to fix it first, unless there are other scenarios which we have overseen and simply won’t ever work when fixing the bug.

Routed ipsec tunnels are often used to connect to clouds, where it may not be possible to configure the netmask freely, and the client side has to comply.
Does the current interface even tell me which mask it has calculated? If not, how would I know how to configure the other side?
Also there is the possibility that the other side also chooses netmasks automatically - what if it chooses a different one?
I think automatic calculation of the netmask is just a bad idea.

If I’m not mistaken the interfaces overview should represent the configured settings.
As long as both ends are “within visible range” mask likely doesn’t matter, although you can easily test that if you have doubts.
Claiming the current input type is a bad choice likely doesn’t help identifying the issue at hand.  To be honest, I’m not sure I agree either in this case that adding a mask would be more clear for everybody (and given our current situation, it will likely lead to confusion by others)
So, the question remains, are there actual situations where the minimal calculated subnet doesn’t work (when properly calculated)

I thought that the netmask and network has to be the same one on both sides for the tunnel to work.

Well, it does work, at least between opnsense and vyos:
ipsec1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1400 	tunnel inet <IP_ADDRESS> --> <IP_ADDRESS> 	inet6 fe80::20d:b9ff:fe4e:dc84%ipsec1 prefixlen 64 scopeid 0x10 	inet <IP_ADDRESS> --> <IP_ADDRESS> netmask 0xfffffffc 	groups: ipsec 	reqid: 1 	nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
vti1: flags=209<UP,POINTOPOINT,RUNNING,NOARP>  mtu 1500         inet <IP_ADDRESS>  netmask <IP_ADDRESS>  destination <IP_ADDRESS>         tunnel   txqueuelen 1000  (IPIP Tunnel)         RX packets 8  bytes 672 (672.0 B)         RX errors 0  dropped 0  overruns 0  frame 0         TX packets 8  bytes 672 (672.0 B)         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

How about 9e44d9a1a ?
# opnsense-patch 9e44d9a1a

Cheers,
Franco

No feedback for 14 days on a patch that is hot for discussion seems strange to me. ;)

Like the crowd around someone injured, everyone thinks the other guy will help :)

FWIW, I'm going to close this and target 22.1 for this change for lack of further feedback.


mappings: change PID and role to keyword

Closes https://github.com/inveniosoftware/cookiecutter-invenio-datamodel/issues/43 and https://github.com/inveniosoftware/cookiecutter-invenio-datamodel/pull/46 (old PR in cookiecutter-datamodel)
Tested with a cookiecutter instance (v3.1). Added 13 documents and checked that queries such as ?q=id:1 gave as a result only the document with id 1 and not 1,10,11,11, etc. Same for 2 and 12, 3 and 13.

Travis is passing: https://travis-ci.org/inveniosoftware/cookiecutter-invenio-instance The first travis link is refering to an old branch. Could it be cached? If so can it be deleted?


lex and yacc emitters may get extra-files paths wrong

This is a summary of a Discord discussion (at https://discord.com/channels/571796279483564041/971831743944491008, although as a thread it probably won't survive and there's lots of other stuff in that thread - this bit should be listed as May 20, 2022).
The emitters in both the lex and yacc tools recognize certain flags, which will be passed on to the actual executed command, as requesting an additional file to be created, and grab that filename and add it to the target.  There's nothing wrong with that concept, but in doing so, they lose track of the fact that "shell paths" and "SCons paths" are not the same thing - the former are interpreted relative to the cwd of the executed command and the latter relative to the directory of the active SConscript.
The following simple subsidiary SConscript file illustrates this - src/SConscript:
env = DefaultEnvironment()
env.Append(LEXFLAGS=['--header-file=gen-src/lexer.ll.hh'])
env.CXXFile('#gen-src/lexer.ll.cc', 'lexer.ll')

Calling the CXXFile builder sets up a target for the .cc file, and the emitter spots the --header-file option and adds another target for the rest of the option, gen-src/lexer.ll.hh.  However, the latter will be interpreted relative to the directory of the SConscript, so we get this tree:
+-.
  +-SConstruct
  +-gen-src
  | +-gen-src/lexer.ll.cc
  |   +-src/lexer.ll
  |   +-/bin/flex
  +-src
    +-src/SConscript
    +-src/gen-src
    | +-src/gen-src/lexer.ll.hh
    |   +-src/lexer.ll
    |   +-/bin/flex
    +-src/lexer.ll

That is, the C++ source target is gen-src/lexer.ll.cc, while the C++ header target becomes src/gen-src/lexer.ll.hh because the SConscript is in src. For additional fun, the actual issued command is:
flex --header-file=gen-src/lexer.ll.hh -t src/lexer.ll > gen-src/lexer.ll.cc

meaning, we get SCons believing the header target is src/gen-src/lexer.ll.hh but the file was physically put in gen-src/lexer.ll.hh since the command-line flag is just passed on as it was set up.
Not sure whether the right answer is to adjust the two tools to do more figuring to get the paths consistent, or whether it's to leave the responsibility to the programmer to set the flags to be passed to lex/yacc in a way that avoids the problem. In the latter case, we probably need more documentation guidance.

Here's some from the thread on generating the flags better:

dmoody256: Try:

env.Append(LEXFLAGS=['--header-file=${File("#gen-src/lexer.ll.hh")}'])

NO_NAME: It does work for a predefined path but I have more lex files in my project. The actual line looks more like:

env.Append(LEXFLAGS=['--header-file=gen-${SOURCE.path}.hh'])

dmoody256: you can make a generator function then do all the work to resolve the string in a python function:

def lex_header_gen(env, target, source, for_signature):
    result = f'--header-file=gen-{source[0]}.hh'
    return result

env['LEX_HEADER_GEN'] = lex_header_gen
env.Append(LEXFLAGS=['$LEX_HEADER_GEN'])

NO_NAME: Wait, you can do that?!  Is this documented? How could I miss it?
dmoody256: It is an extremely powerful feature
NO_NAME: I genuinely cannot find it in the documentation.
NO_NAME: I modified the code a little and it works:

def lex_header_gen(env, target, source, for_signature):
    file = env.File(f'#gen-{source[0].path}.hh')
    result = f'--header-file={file}'
    return result

env['LEX_HEADER_GEN'] = lex_header_gen
env.Append(LEXFLAGS=['$LEX_HEADER_GEN'])

There don't seem to be any test cases that exercise the recognized "generate an extra file with a name chosen by the programmer" options, which are:
fileGenOptions = ["--header-file=", "--tables-file="]   # lex tool
fileGenOptions = ["--defines=", "--graph="]  # yacc tool

There is a related problem on Stack Overflow: https://stackoverflow.com/questions/23007461/how-to-use-scons-lex-builder-with-variant-directories
The chosen solution should work with that case too.

Seems like just having the emitter create a File() node from the extracted string (parameter to the argument), perhaps needs to be subst()'d first. Would allow specifying both relative path/to/file to current SConscript (the default for an unadorned path), and top of tree relative using #path/to/file

So:
            l = len(fileGenOption)
            if option[:l] == fileGenOption:
                # A file generating option is present, so add the
                # file name to the target list.
                fileName = option[l:].strip()
                target.append(fileName)

Does it make a difference if that's target.append(File(fileName))? Seems like SCons will just do that conversion anyway?

The difference is that you could then do #path/to/file and it should work which won't work with just a string.
Though I suppose the command string would be broken.
HRm.
Let me ponder and respond later.

Related, but not the same problem: #4168 - related because it also has to do with the use of a string extracted from the *FLAGS variable and then somehow turning that into something SCons tracks properly.

Will add a more concise explanation later, but this simple stanza seems to work out:
env.Append(LEXFLAGS=['--header-file=${TARGET.base}.hh'])

Here's my attempt at a concise analysis (written for lex, but applies to some extent to yacc/bison as well):

The default LEXCOM on Linux is: "$LEX $LEXFLAGS -t $SOURCES > $TARGET"
If LEXFLAGS contains file-output lex options, the emitter set up by the lex.py tool detects them, and adds any option-argument to the target list.
The targets (including any added by the emitter) are interpreted by SCons relative to the sconscript directory.
The actual issued commands are executed relative to the top directory, with the sources and targets adjusted by SCons.
lex just does what it's told in the arguments, so unless the contents of LEXFLAGS have been modified, we get a mismatch.

This is an even more pruned-down example - only a subdir is needed to trigger the mismatch. variantdir is a red herring, although since a variantdir always involves a subdir to build in, it does always trigger it.
$ cat SConstruct
SConscript('src/SConscript')
$ cat src/SConscript
env = Environment()
env.Append(LEXFLAGS=['--header-file=mylex.l.h'])
env.CFile('mylex.l.c', 'mylex.l')
$ scons --tree=prune
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
flex --header-file=mylex.l.h -t src/mylex.l > src/mylex.l.c
+-.
  +-SConstruct
  +-src
    +-src/SConscript
    +-src/mylex.l
    +-src/mylex.l.c
    | +-src/mylex.l
    | +-/bin/flex
    +-src/mylex.l.h
      +-src/mylex.l
      +-/bin/flex
scons: done building targets.
$ ls -R
mylex.l.h  SConstruct src

./src:
mylex.l  mylex.l.c  SConscript

So in a discord discussion (adding here, so it's captured) we mused about defining construction variables for the two lex file-generation options and the two main bison file-generation options (which don't exist in POSIX yacc, but that's another topic).  Oddly, it picks out these:
    # With --defines and --graph, the name of the file is totally defined
    # in the options.
    fileGenOptions = ["--defines=", "--graph="]

Bison documents g, --graph, but no longer documents --defines in the manpage. In the info page we see:
--defines[=file]
    Historical name for option --header before Bison 3.8.

3.8 is actually relatively new... Sept 2021, so maybe the yacc tool should look for both names?

So in a discord discussion (adding here, so it's captured) we mused about defining construction variables for the two lex file-generation options and the two main bison file-generation options (which don't exist in POSIX yacc, but that's another topic). Oddly, it picks out these:
    # With --defines and --graph, the name of the file is totally defined
    # in the options.
    fileGenOptions = ["--defines=", "--graph="]

Bison documents -g, --graph, but no longer documents --defines in the manpage. In the info page we see:
--defines[=file]
    Historical name for option --header before Bison 3.8.

3.8 is actually relatively new... Sept 2021, so maybe the yacc tool should look for both names?

Yup. The joys of change.. ;)

So to try and narrow in on a preferred solution: if new construction variables are added, just leave current behavior of sniffing in LEXFLAGS and YACCFLAGS alone, but update the doc to say the other method is preferred (and possibly, why)?

Right. The idea is to not break existing builds (and require a deprecation cycle)..
This is similar to users adding -Ipath1 to CCFLAGS instead of via CPPPATH.
There may be reasons to do that.
No need to add duplication checking.

Okay, I have a patch that implements doing this with a pair of new construction variables for each of lex and yacc, it builds on top of PR #4183.
Note that this concept is similar to one of the proposals in #3064 (look for the string "stevenknight said at 2008-10-02 10:42:18")


NFT Transfer Notifications

Fix console error when there are no transaction fees on this node.

Thanks for fixing the fees error


[REQUEST] Improve Input Handling for Multi-Input Machines (Solid Infuser Currently)

Description:
For machines with multiple required inputs, specifically the solid infuser, it would be really nice if the machine restricted the inputs of valid combinations of materials. As it stands currently, either slot will accept any valid material from the same input hopper, pipe, etc, leading to clogged machines where both slots contain only one half of the recipe. My attached screenshot below illustrates the issue. My pipe system ran short on coal dust, so the slots clogged with iron dust, removing automation and requiring my manual intervention. A potential solution is to restrict the left and right input slots to certain halves of the recipes.  For example, iron dust can only be input into the left input slot, coal dust into the right; iron ingots or diamond dust into the left, nikolite dust into the right; and so on.
Reason:
As they currently function, it's very difficult to automate the solid infuser, and whatever other multi-input machines there are or might be in the future. They can easily clog with half of the required materials when that half happens to be in overabundance at a given period.

This is already solved in dev by using two different configuration modes: Input (First Slot) and Input (Second Slot). The update should be coming out soon, I'll leave this open for now as it isn't released yet.

Implemented as of 1.9.0 which can be downloaded on curseforge.