Feature request: allow using JSDoc types inside .ts files.

I am aware of https://github.com/microsoft/TypeScript/issues/20774
Search Terms
allow using JSDoc inside TypeScript .ts files.
Suggestion
It would be AWESOME to allow types to be specified in comments inside of .ts files, just like with regular .js files.
Use Cases
I can not (do not want to) work within limitations of, current documentation solutions for TypeScript (f.e. TypeDoc, TSDoc, etc).
I'd rather use a more generic tool that understands JSDoc syntax, without regard for the source code (code agnostic), to make my documentation in a more flexible and customizable way (f.e. JSDOc with commentsOnly plugin which ignores source code, and considers only comments to be the source of truth for documentation.)
However, it'd still be nice not to duplicate types in the comments and in the source, where possible. So if I could write types in the comments (and still optionally have some types in the source as needed, especially for things that don't work in the comments), then that'd be awesome.
Examples
As shown in https://github.com/microsoft/TypeScript/issues/20774, it unfortunately does not work.
Also I don't want to switch back to .js and lose certain type features that aren't covered by the commenting features.
Checklist
My suggestion meets these guidelines:

[x] This wouldn't be a breaking change in existing TypeScript/JavaScript code
[x] This wouldn't change the runtime behavior of existing JavaScript code
[x] This could be implemented without emitting different JS based on the types of the expressions
[x] This isn't a runtime feature (e.g. library functionality, non-ECMAScript syntax with JavaScript output, etc.)
[x] This feature would agree with the rest of TypeScript's Design Goals.

For now, I'll duplicate all my types, because the flexibility and customizability that I get from handling JSDoc output in my own way is valuable. I don't like the mechanical output from TypeDoc for example, as it covers more than I need and is too complicated.

@shynome By the way! Continuing from #20774, you can use JSDoc with TypeScript files, using the commentsOnly plugin, which will ignore all the source code, and read only the comments. :)

Our stance on types from jsdoc hasn't changed since #20774. jsdoc support for Typescript sounds like a better idea to me. In that case you would not use a commentsOnly plugin, but a plugin that understands Typescript.
I'm not sure if the jsdoc maintainers are interested in supporting TS, but it is a large, and growing, chunk of JS code at this point.
For now, I'm marking this as a duplicate of #20774

@sandersn  If instead of writing
let s: string = 'asdf'

we could also write
/** @type {string} */
let s = 'asdf'

inside of a .ts file (which seems easy because the .js file support is already there), that'd be great. As far as doc generators, it would mean we wouldn't be strictly limited to those that understand TypeScript language.
Seems like this is an easy change to make considering the machinery is already implemented. Would the TypeScript team be willing to take a PR that enables this for .ts files?

Oh!!!
Also, if JSDoc types are given more support of all features, then this can also eliminate build steps if we don't have to switch to .ts to use features that don't work in JSDoc. 👍

Can this be re-opened and kept  open as a feature request?


[BUG] Auth Mode Configuration Parameter lost after updating using a magic

Describe the bug
When updating the graph notebook config using the %%graph_notebook_config or %%graph_notebook_host magics, lose the authorization parameter auth_mode in the configuration.
To Reproduce
Steps to reproduce the behavior:

Create a new notebook and verify that the auth_mode parameter exists
Update the configuration using the magics
Check the configuration and the auth_mode parameter does not exist

Upon further investigation, this was root caused to using a host not in the allowlist configuration, so it was only saving the relevant parameters.
My request here is to make it more explicit that only part of the configuration has been saved due to unneeded parameters in the request.  Something like
The provided configuration contains parameters that are not required for the selected configuration.  These parameters have not been saved.

Bonus points if we can list out the parameters.


[bugs] train, regionalisation

Description of the issue
I spotted two bugs:

regionalisation is a subclass of study factsheet, but should be (according to def) be a kind of methodology
train is equivalent to land vehicle, but should be subclass of, instead.

Ideas of solution
If you already have ideas for the solution describe them here
Workflow checklist

[ ] I am aware of the workflow for this repository

I will create a PR for fixing the bugs.


Display issue with new-bccontainer

This has been happening for a while and is totally low priority.
When creating a new container, after the "waiting for container  to be ready statement the powershell display starts doing this:

The container builds fine without issue, just is annoying.

The problem is the t in additionalparameters, which means that the container allocates a pseudo tty.

Replace -it with -i
but... - why are you adding -i (interactive)?

Thanks @freddydk - removed those and it worked as expected.  We had the -it there as part of just trying to get networking to work properly and we never thought to remove it. If it is not broken.... :)


WIP - LicensePicker component

This is an initial and WIP activity that fixes #255  , based on https://spdx.org/spdx-specification-21-web-version#h.jxpfx0ykyb60
Some improvements are still needed to the UI, as well as implement grouped expressions.

@jeffmcaffer here's a quick video of the behavior.
As i mentioned, the graphic is still ugly, and some functions are still missing, but you can start see what is the flow that we have in mind.

@jeffmcaffer did you had the chance to take a look to the behavior explained in the video?
I have some other changes ready to be applied to this one, but a feedback from you would be helpful.
Thanks

Thanks for the video. seems like a good start and we can put this in now as it is definitely an improvement over what we have today. Some questions:

How are () handled?
How does the user get to this? Is this the only experience for editing license entries?
Can we make this much smaller? more like a pop up/over than a full on, full screen dialog? Just would feel lighter weight for the user.
How will this mesh with the Suggestions experience
Hard to tell in the video, are you clicking on Add new group to get a new entry? If so, perhaps just a + icons?
Is it possible to delete a group?

BTW, did you look at the existing "query builders" like https://querybuilder.js.org/ or https://www.npmjs.com/package/react-querybuilder? Are they useful in this context?

Thanks for your feedback @jeffmcaffer , i'm going to put in place some changes to this code that i've already made.

How are () handled?
In this version wasn't possible to handle groups, i've improved the code meanwhile and i will update soon the PR with that functionality, and also sharing a new video which shows the behavior

How does the user get to this? Is this the only experience for editing license entries?
We can give to the user a choice, type in manually or use the logical builder. But in both ways i would suggest to keep the validation tool that shows to the user if the typed license is a correct one.

Can we make this much smaller? more like a pop up/over than a full on, full screen dialog? Just would feel lighter weight for the user.
Sure

How will this mesh with the Suggestions experience
The suggestion process is something that doesn't affect how a user build a new license.
I will provide a video also for that, but basically if there is a suggestion the user can select it from a list coming from the server. Then, maybe he wants to make some little changes and he can edit the license himself.

Hard to tell in the video, are you clicking on Add new group to get a new entry? If so, perhaps just a + icons?
For "plain" licenses, there is an automation: if the user select a license, and then choose a conjunction from the list, a new element appears.
To add a new group the user should do a manual activity clicking on the button.
That would build a new set of elements under the parent one.

Is it possible to delete a group?
Not yet, i'm working on editing and deleting functionalities.

BTW, did you look at the existing "query builders" like https://querybuilder.js.org/ or https://www.npmjs.com/package/react-querybuilder? Are they useful in this context?
Those are exactly my starting point on building this

@storrisi Thank you. I am looking at https://github.com/clearlydefined/website/pull/400#issuecomment-447378857 and I feel like a single text field with editing/completion assistance might be easier to use for me: fewer clicks, fewer context switch, easier to edit/delete and I can see the whole expression I am building rather than bits of it.


Cadence Profiler

Cadence Profiler for Smart Contracts / Scripts / Transactions
Grant category

[x] Developer tools / services
[x] Open source maintenance

Description
Cadence is the native programming language for developing smart contracts / scripts / transactions that run on the Flow Blockchain. The Flow Emulator displays profiling data in logs about scripts and transactions, including computationUsed and memoryEstimate (this one is currently being added by our team, https://github.com/onflow/flow-emulator/pull/366). These logs, however, are not enough to drill down and get to the source of the bottleneck that is causing execution issues, due to high execution efforts.
Hence, I would like to implement a Cadence Profiler, to enable Flow builders to build more robust smart contracts.
Problem statement
When designing smart contracts, one important concern is their performance and ability to scale. In some cases, after a period of time, the smart contract and some of its APIs, might become unwieldy due to their growth in computation & memory usage. This means that developers will not be able to run certain scripts / transactions, as they will reach the maximum execution limit. One such case can be seen here:

https://github.com/onflow/nft-catalog/issues/138
https://discord.com/channels/613813861610684416/811693600403357706/1090726294683131966

Proposed solution
Make use of the existing code that calculates the computation & memory used

https://github.com/onflow/flow-go/blob/master/fvm/environment/meter.go
https://github.com/onflow/cadence/blob/master/runtime/common/metering.go

and expose these metrics in a readable & accessible format to developers, through the Flow Emulator / Flow CLI.
Impact
Developers will be able to understand why scripts & transactions need a certain amount of computation & memory to run, and in turn they would be able to optimize their contracts.
Milestones and funding

Milestone
Deliverables
Timeline
Risks
USD proposal

1 - Emulator API for ProfilingReport
Create a new Emulator API to display a ProfilingReport including all scripts / transactions
~ 3 weeks
-
15,000

2 - Detailed ProfilingReport for contracts
Break down of computation/memory used by each statement on a contract
~ 7 weeks
-
25,000

3 - Adoption
Use the profiler on 5+ contracts
~ 4 weeks
-
1,000/contract

4 - Maintenance
Resolve and triage issues, feature requests. Write some official documentation.
3 months
-
5,000/month

Total Cost: 60,000 USD. Of course, I am open to discussion, regarding the cost and timeline :slightly_smiling_face: .
Team

Name
Role
Bio
Contact

Ardit Marku
Back-end Engineer
A seasoned back-end engineer with a focus on problem solving and designing robust software systems. Over 8 years of experience building high-traffic search engines, marketplaces, exchanges and in-house analytics solutions. Led cross-functional teams to deliver both web and mobile apps, participating in all steps of the software development lifecycle. Open source lover and advocate.
<EMAIL_ADDRESS>m_peter

I really like this proposal. It will be super useful for understanding Cadence execution.

Thank you for opening this proposal @m-Peter!
We are currently in the process of publishing more developer grants for Cadence and related tooling, guided by importance and impact for the ecosystem, and are planning to get back to reviewing this proposal afterwards. Thank you for your patience

Thanks for the heads up @turbolent :bow:
Let us know if we can be of any help, regarding the upcoming developer grants for Cadence. We would like to tackle some of the important needs of the ecosystem, and make an impact :pray:

@franklywatson Since this https://github.com/onflow/flow-emulator/issues/388 is gonna be solved soon, is there any positive sentiment from the Flow team for this grant to proceed?

I'm definitely in support of this. It would be super useful for me and many others in the ecosystem!

@joshuahannan That makes total sense, I will experiment locally, how to calculate the gas for scripts & transactions :pray:

I think this computation tracking per statement is amazing feature too. If we can integrate with tools like code-coverage, it can makes things a bit more predictable.

Yes, the end goal would to be get general execution metadata, including things like line hits, computation and memory usage. So instead of just coverage report that we have under the Emulator endpoint (http://localhost:8080/emulator/codeCoverage), we could have a unified report with execution metadata.

Yes, the end goal would to be get general execution metadata, including things like line hits, computation and memory usage. So instead of just coverage report that we have under the Emulator endpoint (http://localhost:8080/emulator/codeCoverage), we could have a unified report with execution metadata.

This is awesome! I can't wait for this to get integrated into tools.

Hey gang, hasn't been activity on this thread for over a year, wanted to check what the status on this was :)

Hey gang, hasn't been activity on this thread for over a year, wanted to check what the status on this was :)

This was actually completed to a good degree. It's bundled in flow-emulator, and can be used for gathering computation reports for scripts & transactions. Since then I've been quite busy with the EVM Gateway, and Flow EVM, so I had no capacity to make tutorials and use it in certain projects 😅

cc @j1010001 and @onflow/flow-smart-contracts for potential follow-up work (e.g. tutorial on how to optimize Cadence code)

This is an exciting feature! I'm wondering if you all have any thoughts about how to expose this in Flow CLI. For instance, it would be helpful to run a script or transaction and output a computation report (maybe behind a flag like --computation-report <OUTPUT_DIRECTORY>.

This is an exciting feature! I'm wondering if you all have any thoughts about how to expose this in Flow CLI. For instance, it would be helpful to run a script or transaction and output a computation report (maybe behind a flag like --computation-report <OUTPUT_DIRECTORY>.

@sisyphusSmiling That's a good idea, the computation reports are already available on an admin endpoint (http://localhost:8080/emulator/computationReport), where they are grouped by either script / transaction ID. We could also add some extra utility flags, for flow scripts execute & flow transactions send commands. It shouldn't be very hard to do so 🙏

Legend! Going to close this one down :)


FInding nodelets by class name

The way that nodelets are resolved is quite tricky. You can load them using the name of the class, but you don't know the name of the class until you parse the source, which we do after trying to find them. For the moment in:
What really needs to be done is to look through all the source files in info.targets to find which info.target defines the class in library.name or library.type, then add that target keyed by the type as well. This is needed because nodelets can be loaded into managers by their class name.
But probably the whole nodelet resolution architecture needs to be rethought.

There is a workaround in: https://github.com/rosqual/roswire/pull/509


Add 'Assignee' field to PullRequestEvent

GitHub webhook returns a json response with "assignee" field when a pull request is assigned or unassigned.
{
  "action": "assigned",
  "pull_request": { ... },
  "assignee": { ... },
}

It would be a good enhancement to add Assignee to PullRequestEvent, much like we already have in IssuesEvent.
I would be happy to take this one if you would consider a PR.

Hmm, I see the field documented as part of Events API (not just Webhook events only) at https://developer.github.com/v3/activity/events/types/#issuesevent, but I don’t see assignee documented at https://developer.github.com/v3/activity/events/types/#pullrequestevent.
Is the field Webhook only for PullRequestEvent? How did you confirm it exists? Or is it present in Events API payloads too, despite not being documented?

Closing issue for lack of activity/interest. Feel free to reopen.


community: update huggingface pipeline with  pipeline_kwargs  configuration

Since pipeline_kwargs cannot be updated through chain.invoke function, I added a new method to change pipeline_kwargs after model compiled.
Now we can change the generation parameter without reloading/recompiling the model.

@baskaryan Could you help to review, its critical for HF user' experience. thanks

@efriis Could help to review, thanks

Switch to a new PR https://github.com/langchain-ai/langchain/pull/21519


[RFC]: Keep a Changelog & Add FAQs in the Documentation

Motivation.
Changelog
I frequently find myself wondering what is the difference between the latest version(s) of vLLM, and the version that I currently have deployed. It seems like it would be nice to keep a simple changelog that documents features, fixes, newly-supported hardware and updates between versions so that we can easily see what has been added in recent versions -- e.g. new CLI arguments, optimizations, quantization formats, updated hardware support for features (e.g. punica -> triton kernels, expanding hardware support for multi-lora serving) patched bugs, and so forth.
A great template for this is Keep a Changelog following semver - it would be super easy to implement with Markdown in the documentation site. I think this would make vLLM's newer features much more accessible, and it would also help identify gaps in the documentation when we add something to the changelog that's not on the docs site
FAQs
There are a bunch of questions that are commonly asked over and over in the discord, including things such as:

Does vLLM support XYZ hardware/accelerator?
Does vLLM support tool use / when is tool use coming?
Does vLLM support XYZ model / model architecture?
How can I get my model to fit in a vRAM-constrained environment?
How do I get started with distributed inference?

It seems like it would be nice to have a list of rolling FAQs to refer people to, and maybe to pin in the discord, for quick reference. This could be tracked in version control in the docs site, so that we can easily make pertinent additions and deletions as necessary?
Proposed Change.

Create a simple changelog in the docs site following the Keep a Changelog structure;
a. Ask PR contributors to update this along with docs when they create a PR OR update this as part of the release process
Implement a simple FAQ in the docs site

Feedback Period.
1 week?
CC List.
@mgoin @simon-mo @WoosukKwon @petersalas @comaniac @SolitaryThinker @ywang96 @DarkLight1337 unsure who else to ask - I just pulled a list of recent contributors to the docs
Any Other Things.
No response

I tried to keep a good log here for each release. https://github.com/vllm-project/vllm/releases
FAQ page is here https://docs.vllm.ai/en/latest/serving/faq.html


workspace name not referenced

Testing #18964
expected:
Starting {Pylance/Jedi} language server for {workspace basename}
actual:

I am using the pre-release versions of the Jupyter and Python extensions

As expected since we only start 1 Pylance language server, instructions were not updated after this last minute change 😔


@web3uikit/core @web3uikit/web3 @web3uikit/icons not found

yarn add --dev  @web3uikit/core @web3uikit/web3 @web3uikit/icons
yarn add v1.22.18
info No lockfile found.
[1/4] 🔍  Resolving packages...
error An unexpected error occurred: "https://registry.yarnpkg.com/@web3uikit%2fcore: Not found".
info If you think this is a bug, please open a bug report with the information provided in "XXX.log".
info Visit https://yarnpkg.com/en/docs/cli/add for documentation about this command.

You get a similar error with npm.
Also, if you look at the NPM website, you can't find these packages either.
https://www.npmjs.com/search?q=web3uikit
The docs currently show to use this api to install web3uikit. Can we please fix?

Hey @PatrickAlphaC . We are currently experiencing some publishing issues with github actions we raised an issue here
https://github.community/t/error-changing-workflow-permissions/263648
Please install 0.1.170 for a stable legacy web3uikit version (non-monorepo-version)

Thank you @locothedev !

thank you too @locothedev 💚

hey @PatrickAlphaC can we close this one? I will be pamping like crazy on twitter where we chat when its ready for monorepo

hey @PatrickAlphaC we did it thanks to @AbhinavMV
from Discord
We were finally able to publish our monorepo 🚀 🤩
Give it a try
For all web3 components like connect wallet, Native Balance, ENSAvatar etc.....
yarn add @web3uikit/web3

For all core components like button,input etc..
yarn add @web3uikit/core

For all icons
yarn add @web3uikit/icons

closing this one now @PatrickAlphaC as we spoke on twitter, you legend, Ty for your support!

❤


upload_file returns a partially populated object

Description of the bug:
This code is from the cookbook sample:
sample_file = genai.upload_file(path="image.jpg", display_name="Sample drawing")
print(f"Uploaded file '{sample_file.display_name}' as: {sample_file.uri}")

When executed, display_name is an empty string, as we all a number of other properties (mime_type, plus timestamps, hash, etc). They are all returned correctly from get_file, and the partial objects work fine in prompts.
One fix would be to call get_file after uploading, and return that response instead.

Should be fixed in #314


Exodus

Status
Ready
Description
Update piplock file with latest eve-client 2.0.11  from pypi.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.1 out of 2 committers have signed the CLA.:white_check_mark: an-hel:x: dependabot[bot]You have signed the CLA already but the status is still pending? Let us recheck it.


[stable/logstash] Add the option to create secrets

What this PR does / why we need it:
Add the possibility to store sensitive information in secrets and mount them to container.
This could be used for plugins that require to specify file with cloud provider credentials to not put it in configmaps.
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[x] DCO signed
[x] Chart Version bumped
[x] Variables are documented in the README.md
[x] Title of the PR starts with chart name (e.g. [stable/chart])

/assign @christian-roggia

@melifaro
Would you resolve the conflict file stable/logstash/Chart.yaml please? Thank you.


Switch to super-linter for CI Linting using GitHub Actions

Describe the feature
I propose switching our CI linting setup, which includes shellcheck, markdownlint, eslint, and others, to use super-linter exclusively for GitHub Actions. super-linter is a comprehensive and centralized linter that supports multiple languages, providing a unified linting experience.
Use Case
There are some benefits of super-linter:

Unified Linting: super-linter supports a wide range of languages and linters, providing a unified linting solution for our GitHub Actions workflow.

Easy Configuration: Simplifies configuration by having a single .github/linters configuration file.

Consistent Output: Standardizes the output format for linting results across different languages and linters.

Extensibility: Supports adding or customizing linters easily, making it adaptable to our specific requirements.

Proposed Solution

Configure super-linter:

Create or update the .github/workflows configuration file to specify the linters and configurations needed.

GitHub Actions Workflows:

Update the GitHub Actions workflow files to use super-linter for linting instead of the actual implementation

Testing:

Thoroughly test the linting process with super-linter in the GitHub Actions workflow to ensure it covers all relevant languages and provides accurate results. You can test this by submitting a PR

Other Information
No response
Acknowledgements

[X] I may be able to implement this feature request
[ ] This feature might incur a breaking change

Version used
latest
Environment details (OS name and version, etc.)
any

Hi! Can I take this one??


Added build on launch

I'm not an expert on how everyone else organizes their projects and workspaces, so I'm not 100% positive this will work for every code base. Here's a summary of the changes:

launch.json and tasks.json contents now generate into .code-workspace file.

Lets you run the build command from the workspace root directory, independent of where project folders are (sub-directories, sub-sub-directories, etc.)
Launch configs generate based on the startproject specified by the user in their premake script.

Moved launch and task generation code into their own files to let vscode_workspaces access both.
Launch configs generate with a "preLaunchTask": that calls respective build task.

Might want to also generate launch configs that don't build in case user wants to run without re-building.

Hey! I don't really actively maintain this project anymore due to me moving away from VS Code, however if you're finding this useful and is willing to maintain it I'd be happy to make your fork the main repo if that's alright with you 🙂

Yeah man! I'd be happy to if that works better for you:)

Yeah man! I'd be happy to if that works better for you:)

Awesome, I'll archive this repo and update the readme to link to your fork of it 🙂


Open Images Dataset

Hello,
Open images dataset doesn't have the category I have been looking for what kind of dataset can I use in the tutorial ? For training and converting to onnx later.
Thanks.

Hi @yigittopcu, you can collect your own data following this tutorial:  https://github.com/dusty-nv/jetson-inference/blob/master/docs/pytorch-collect-detection.md
Or if you can find images of the objects, you can annotate them with CVAT tool (CVAT.org) and then export them in Pascal VOC format.

Thanks for quick reply @dusty-nv . I have one more question. I already have Pascal VOC XML dataset. Can I use it directly ? Or Do I need to convert it ?

@yigittopcu you can use it as long as the directory structure is the same as the original Pascal VOC dataset.  If not, make the directory structure the same as Pascal VOC

@dusty-nv  I did what you told me and trained with Pascal Voc dataset. I don't understand what is wrong but there is a example of the problem below .

Hi @yigittopcu, is this using jetson.inference.detectNet to run the model?  If so, you may lower the clustering threshold here:
https://github.com/dusty-nv/jetson-inference/blob/19ed62150b3e9499bad2ed6be1960dd38002bb7d/c/detectNet.h#L477
After changing this, re-run make and sudo make install

Hi @dusty-nv . This is deepstream. I changed pre-clusterthreshold to 1 from 0.2 and boxes all gone. Any value below 1 results same output.

Is there possibility this caused by training ?

Is there possibility this caused by training ?

You can test your model on the PyTorch side first using the run_ssd_example.py script.  Use one of your PyTorch checkpoints with this script, not the ONNX model.

python3 run_ssd_example.py mb1-ssd models/Hardhat/mb1-ssd-Epoch-99-Loss-1.7641232138971639.pth models/Hardhat/labels.txt ~/home/netcoz/pytorch-ssd/data/Hardhat/JPEGImages/000010_jpg.rf.ff3d356d21409f8e9fe185536c4ecad9.jpg
Got this error with run_ssd_example.py
Traceback (most recent call last):
File "run_ssd_example.py", line 50, in
image = cv2.cvtColor(orig_image, cv2.COLOR_BGR2RGB)
cv2.error: OpenCV(4.5.2) /tmp/pip-req-build-wu1ri_rx/opencv/modules/imgproc/src/color.cpp:182: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'

I believe it failed to load that image

Thanks. I will try again first thing tomorrow.

Inference time:  0.005075693130493164
Traceback (most recent call last):
  File "run_ssd_example.py", line 55, in <module>
    cv2.rectangle(orig_image, (box[0], box[1]), (box[2], box[3]), (255, 255, 0), 4)
cv2.error: OpenCV(4.5.2) :-1: error: (-5:Bad argument) in function 'rectangle'
> Overload resolution failed:
>  - Can't parse 'pt1'. Sequence item with index 0 has a wrong type
>  - Can't parse 'pt1'. Sequence item with index 0 has a wrong type
>  - Can't parse 'rec'. Expected sequence length 4, got 2
>  - Can't parse 'rec'. Expected sequence length 4, got 2


fix(contrib/drivers/mssql): support pagination for both older 2008 and newer 2012 mssql version

fix https://github.com/gogf/gf/issues/4050


Error messages make little sense and provide no recourse as to what one can do

There is very little information about the error messages returned by Sn.exe. If I try to Sn -i pkey.snk KEY_XXX I see "Object already exists" if I try to sn -pc KEY_XXX I get "Key does not exist" The behaviour does not make any sense and no recourse or advise is provided. It is a terrible tool. Whoever wrote it should be ashamed.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: c2d65a40-0ea7-09ce-33fc-39525310b05e
Version Independent ID: f6cbd1c2-7708-03aa-a935-0dbfa07e6177
Content: Sn.exe (Strong Name Tool)
Content Source: docs/framework/tools/sn-exe-strong-name-tool.md
Product: dotnet-framework
GitHub Login: @tdykstra
Microsoft Alias: dotnetcontent

@vivmishra Do you know who can help with documenting Sn.exe error messages?

Is there any design documentation about Sn.exe? Maybe Microsoft has some design documents that would help provide better error messages. I was partly joking about the authors should be ashamed. If I wrote error messages like that at my company I would be crucified.

@tdykstra can you remove the guide - ASP.NET Core web apps label from this issue? Thanks.


error: failed to compile file: null if .filter() response type is var

Description:
I tried to compile and run this file:
import ballerina/io;
public function main() {
    string currency = "USD";
    map<int> currencies = {
        USD: 318,
        EUR: 322,
        GBP: 400
    };
    var rates = currencies
        .filter(function ((string, int) pair) returns boolean {
            var (key, value) = pair;
            return key == currency;
        });
    io:println(rates);
}

Steps to reproduce:
# ballerina build test.bal
Compiling source
    test.bal
Generating executable
    ./target/test.balx
# ballerina run test.bal
error: failed to compile file: null

Affected Versions:
Ballerina 0.990.0
OS, DB, other environment details and versions:
macOS Mojave 10.14.1

Already reported


vsetvl[i] ta/tu and ma/mu flags

Is it the intention that ta/tu and ma/mu flags to vsetvl[i] are optional? If so, are they assumed to be tu and mu if omitted?
(they are absent in all cases in the 0.9 example directory)

I think longer term, people would prefer them to be mandatory in assembly code.
In the interim, to reduce software churn, it is easier for current software developers/implementers if they're optional, with tu/mu being the default.

Thanks. I think it would be useful to add your words above (especially the defaults) as a note in section 3.3.3, and perhaps to update the examples to use the preferred syntax.


<EMAIL_ADDRESS>hash check failed

Operation did not complete successfully because the file contains a virus or potentially unwanted software.
ERROR Hash check failed!
App:         extras/syncthingtray
URL:         https://github.com/Martchus/syncthingtray/releases/download/v1.4.3/syncthingtray-1.4.3-x86_64-w64-mingw32.exe.zip
First bytes:
Expected:    e52b4a6dfb17ec21d901da60402c8df1b444680eb20603d716d9cd3af8b8135d
Actual:

I tried again bypassing the cached file with -k and I still get the error from Defender. However, I later found another user having the same problem with the binaries downloaded directly from the repo, it seems a false positive with Windows Defender, so no need for further support. Thanks.


Discussion about social column in footer: sharing vs links

The "Social" column in the footer is probably intended as typical social media links. But what if instead we used those as sharing options. Thoughts?

hello the links are not present in the social column in the footer

sharing options would be good too but i think it would be also nice to keep the social media links as people get to know more about the organization and its works


multiple parentchain handlers

WIP:
Current state: Instantiate two parentchain handlers and import blocks for both parentchains, which works, but there are some minor issues:
Todo:

[ ] Add an indirect calls filter that listens for balance transfer events for the secondary chain and then we can run tests
[ ] Fix bug: The worker wants to send some extrinsics, which can't be decoded by the node, currently unclear why this happens.
[ ] Enhance docker compose to spawn a second node and test with 2 nodes in the CI
[ ] ConfirmProcessedParentchain block extrinsic should only be done for primary parentchain
[ ] Extrinsic Factory needs to be generic over additional params

Log of the error from above:
8400195c48a80a1926205e21bcd03f1737e966364d2a3c24c4fb7614ff11adf9b73e00bb8dba1bacbefcd515aaa7deb2f5ccd438ea2b9ef8fbcda58d93c56bbac410c3e55051a3458ac9ec289264471c7af4bd6e55321915fd64227138a466d2191b0800140036017ac5e3ba7222908384691ea94032a8f3a8c80a623e56d8662a994003b740e5529943539204e89f3b1aa632df5920b95e7c0ff10c01c2f1595f4b52aecdbde8da060000000000000000000000000000000000000000000000000000000000000000000000

Some more info about the error:
[2023 - 09 - 05T15: 19: 25Z DEBUG integritee_service::ocall_bridge::bridge_api]Requesting WorkerOnChain OCall API instance[2023 - 09 - 05T15: 19: 25Z DEBUG integritee_service::ocall_bridge::worker_on_chain_ocall]Enclave wants to send 1 extrinsics to parentchain: Secondary[2023 - 09 - 05T15: 19: 25Z WARN integritee_service::ocall_bridge::worker_on_chain_ocall]Createt api with client: WsRpcClient {
    url: Url {
        scheme: "ws",
        cannot_be_a_base: false,
        username: "",
        password: None,
        host: Some(Ipv4(<IP_ADDRESS>)),
        port: Some(9966),
        path: "/",
        query: None,
        fragment: None
    }
}
[worker_on_chain_ocall]Extrinsic Ok("\"0x3103840087bcdd6a1dd4b5ea932f9f667370c9c2a9445089d5af6bf528635d216f4589140028d545b9555fdfe7a63ad4ee38eec7610acce526ad18f36bf593d2e385748a4ec078bc72771edce98c7af0de5741bd579ebc28e69f725be616e66f4dc6f76e0200e8003601e3cbde669a39e678f7eeb4205dd175fb02b446e26621e899b0aad0dac3e99ff13a5f5a76f6e3cafb054f739e62d4941da4f695d0014b1a590deb49660f0827a63b0000000000000000000000000000000000000000000000000000000000000000000000\"")
[worker_on_chain_ocall]Could not send extrinsic to node: 840087bcdd6a1dd4b5ea932f9f667370c9c2a9445089d5af6bf528635d216f4589140028d545b9555fdfe7a63ad4ee38eec7610acce526ad18f36bf593d2e385748a4ec078bc72771edce98c7af0de5741bd579ebc28e69f725be616e66f4dc6f76e0200e8003601e3cbde669a39e678f7eeb4205dd175fb02b446e26621e899b0aad0dac3e99ff13a5f5a76f6e3cafb054f739e62d4941da4f695d0014b1a590deb49660f0827a63b0000000000000000000000000000000000000000000000000000000000000000000000, error: RpcClient(SerdeJson(Error("invalid type: null, expected a (both 0x-prefixed or not) hex string or byte array containing 32 bytes", line: 1, column: 4)))

I don't know what the problem is because when I decode that extrinsic to that node it works (see port and address from url matches above logs):

I could find the issue, the cause is that we have a very misleading error message. The above error is thrown, when the enclave account does not exist on chain.

I reproduced the demo successfully. will merge


mm10 genome

Dear @adamewing,
Thank you for this great tool. I would like to use methylartist  for mm10 genome. I am new in coding and  I am trying to use Mus_musculus.GRCm38.dna.primary_assembly.fa.gz but it gives error ; error when opening file GRCm38.p6.genome.fa.gz
Can you please guide me what to do in this case.
Best,
Serkan

Thanks for your help. I am really new in coding. Here is my command line;
methylartist locus -b E4.merged.sup.mod.sort.bam -i chr18:36778332-37297840 --samplepalette cubehelix -g Mus_musculus.GRCm38.91.chr.sorted.gtf.gz --labelgenes --ref GRCm38.p6.fasta.fa.gz
2023-06-30 15:16:52,539 starting methylartist with command: /exports/humgen/Serkan/anaconda3/envs/methylartist/bin/methylartist locus -b E4.merged.sup.mod.sort.bam -i chr18:36778332-37297840 --samplepalette cubehelix -g Mus_musculus.GRCm38.91.chr.sorted.gtf.gz --labelgenes --ref GRCm38.p6.fasta.fa.gz
2023-06-30 15:16:52,539 mod motif size (--motifsize) = 2 (ensure this is correct for your data)
2023-06-30 15:16:52,539 --ref and --motif are required when using --bams

Sorry for the delay, try adding --motif CG onto te end of the command line.

The other thing I might suggest is using an uncompressed reference genome that has been indexed via samtools faidx.

Thank you again for the answer and sorry for the simple questions which take your time . I have the following files below in my working directory and I ran the code again with --motif CG. But I have again some error. Do you have any other suggestion.
Files in my working directory:
E4.merged.sup.mod.sort.bam
E4.merged.sup.mod.sort.bam.bai
GRCm38.p6.genome.fa
Mus_musculus.GRCm38.91.chr.sorted.gtf.gz
Mus_musculus.GRCm38.91.chr.sorted.gtf.gz.tbi
(methylartist) $ methylartist locus -b E4.merged.sup.mod.sort.bam -i chr18:36778332-37297840 --samplepalette cubehelix -g Mus_musculus.GRCm38.91.chr.sorted.gtf.gz --labelgenes --ref GRCm38.p6.genome.fa --motif CG
2023-07-03 08:18:13,972 starting methylartist with command:
methylartist locus -b E4.merged.sup.mod.sort.bam -i chr18:36778332-37297840 --samplepalette cubehelix -g Mus_musculus.GRCm38.91.chr.sorted.gtf.gz --labelgenes --ref GRCm38.p6.genome.fa --motif CG
2023-07-03 08:18:13,972 mod motif size (--motifsize) = 2 (ensure this is correct for your data)
[W::hts_idx_load3] The index file is older than the data file: E4.merged.sup.mod.sort.bam.bai
2023-07-03 08:18:13,997 fetching mod types from E4.merged.sup.mod.sort.bam, if this takes awhile MM/ML tags may be missing.
2023-07-03 08:18:14,001 found mods: m
2023-07-03 08:18:14,005 found mods: m in bam E4.merged.sup.mod.sort.bam
2023-07-03 08:18:14,006 fetching reads from E4.merged.sup.mod.sort.bam with mod m
[W::hts_idx_load3] The index file is older than the data file: E4.merged.sup.mod.sort.bam.bai
[W::hts_idx_load3] The index file is older than the data file: E4.merged.sup.mod.sort.bam.bai
2023-07-03 08:18:15,808 region size: 519508
2023-07-03 08:18:15,808 mod space positions: 5105
2023-07-03 08:18:15,808 auto set --modspace 4
2023-07-03 08:18:15,808 auto set --smoothwindowsize 104
2023-07-03 08:18:15,828 building genes plot...
Traceback (most recent call last):
File "/anaconda3/envs/methylartist/bin/methylartist", line 5522, in
main(args)
File "/anaconda3/envs/methylartist/bin/methylartist", line 5210, in main
args.func(args)
File "/anaconda3/envs/methylartist/bin/methylartist", line 3215, in locus
genes = build_genes(gtf, chrom, elt_start, elt_end)
File "/Serkan/anaconda3/envs/methylartist/bin/methylartist", line 1656, in build_genes
for line in gtf.fetch(chrom, start, end):
File "pysam/libctabix.pyx", line 507, in pysam.libctabix.TabixFile.fetch
ValueError: could not create iterator for region 'chr18:36778333-37297840'

Hi Adam,
It worked with the genecode annotation but it is not working with other annotations; where is my mistake; do you have any suggestion for simple mm10 annotation
methylartist locus -b E4.merged.sup.mod.sort.bam -i chr18:36778332-37297840 --samplepalette cubehelix -g gencode.vM22.annotation.sorted.gtf.gz --labelgenes --ref GRCm38.p6.genome.fa --motif CG
2023-07-03 08:33:42,496 starting methylartist with command: /exports/humgen/Serkan/anaconda3/envs/methylartist/bin/methylart                                    ist locus -b E4.merged.sup.mod.sort.bam -i chr18:36778332-37297840 --samplepalette cubehelix -g gencode.vM22.annotation.sort                                    ed.gtf.gz --labelgenes --ref GRCm38.p6.genome.fa --motif CG
2023-07-03 08:33:42,497 mod motif size (--motifsize) = 2 (ensure this is correct for your data)
[W::hts_idx_load3] The index file is older than the data file: E4.merged.sup.mod.sort.bam.bai
2023-07-03 08:33:42,521 fetching mod types from E4.merged.sup.mod.sort.bam, if this takes awhile MM/ML tags may be missing.
2023-07-03 08:33:42,525 found mods: m
2023-07-03 08:33:42,530 found mods: m in bam E4.merged.sup.mod.sort.bam
2023-07-03 08:33:42,530 fetching reads from E4.merged.sup.mod.sort.bam with mod m
[W::hts_idx_load3] The index file is older than the data file: E4.merged.sup.mod.sort.bam.bai
[W::hts_idx_load3] The index file is older than the data file: E4.merged.sup.mod.sort.bam.bai
2023-07-03 08:33:44,372 region size: 519508
2023-07-03 08:33:44,373 mod space positions: 5105
2023-07-03 08:33:44,373 auto set --modspace 4
2023-07-03 08:33:44,373 auto set --smoothwindowsize 104
2023-07-03 08:33:44,395 building genes plot...
2023-07-03 08:33:44,435 gene in region: Hars
2023-07-03 08:33:44,472 gene in region: Hars2
2023-07-03 08:33:44,475 gene in region: Zmat2
2023-07-03 08:33:44,476 gene in region: Vaultrc5
2023-07-03 08:33:44,476 gene in region: Gm23424
2023-07-03 08:33:44,477 gene in region: Gm6756
2023-07-03 08:33:44,478 gene in region: Pcdha1
2023-07-03 08:33:44,479 gene in region: Gm37751
2023-07-03 08:33:44,480 gene in region: Pcdha2
2023-07-03 08:33:44,481 gene in region: Pcdha3
2023-07-03 08:33:44,481 gene in region: Pcdha4
2023-07-03 08:33:44,483 gene in region: Gm38225
2023-07-03 08:33:44,483 gene in region: Gm42416
2023-07-03 08:33:44,484 gene in region: Pcdha5
2023-07-03 08:33:44,485 gene in region: Gm37753
2023-07-03 08:33:44,486 gene in region: Pcdha6
2023-07-03 08:33:44,487 gene in region: Gm37013
2023-07-03 08:33:44,488 gene in region: Pcdha7
2023-07-03 08:33:44,489 gene in region: Gm37388
2023-07-03 08:33:44,490 gene in region: Gm36858
2023-07-03 08:33:44,491 gene in region: Pcdha8
2023-07-03 08:33:44,492 gene in region: Pcdha9
2023-07-03 08:33:44,493 gene in region: Pcdha11
2023-07-03 08:33:44,494 gene in region: Gm10545
2023-07-03 08:33:44,494 gene in region: Pcdha11
2023-07-03 08:33:44,496 gene in region: Gm18150
2023-07-03 08:33:44,497 gene in region: Pcdha12
2023-07-03 08:33:44,498 gene in region: Gm19035
2023-07-03 08:33:44,499 gene in region: Pcdhac1
2023-07-03 08:33:44,500 gene in region: Pcdhac2
2023-07-03 08:33:44,501 gene in region: Gm10544
2023-07-03 08:33:44,502 gene in region: Gm38097
2023-07-03 08:33:44,503 gene in region: Pcdhb1
2023-07-03 08:33:44,503 gene in region: Pcdhb2
2023-07-03 08:33:44,586 building read alignment plot...
[W::hts_idx_load3] The index file is older than the data file: E4.merged.sup.mod.sort.bam.bai
2023-07-03 08:33:58,651 building mod-space plot...
2023-07-03 08:33:58,931 building llr plot...
2023-07-03 08:35:02,506 building meth frac plot...
2023-07-03 08:35:29,043 chr18:36778332-37297840 (m), sample E4.merged.sup.mod.sort.m fraction masked: 0.063
2023-07-03 08:35:40,992 plot saved to E4.merged.sup.mod.sort.chr18_36778332_37297840.m.ms4.smw104.locus.meth.png

My guess is that one gtf file uses chromosome names that start with "chr" and the other doesn't? It can work either way but all inputs (.bam file, genome, annotation, and command line) need to agree with respect to this.

Thank you for your support, Adam.


cellmatch: New pass for picking out standard cells automatically

    cellmatch -lib <design> [module selection]

This pass identifies functionally equivalent counterparts between each of the
selected modules and a module from the secondary design <design>. For every such
correspondence found, a techmap rule is generated for mapping instances of the
former to instances of the latter. This techmap rule is saved in yet another
design called '$cellmatch', which is created if non-existent.

This pass restricts itself to combinational modules. Modules are functionally
equivalent as long as their truth tables are identical upto a permutation of
inputs and outputs. The supported number of inputs is limited to 6.

This is useful for writing low-level structural Verilog with some degree of portability between cell libraries.
Say you write a Kogge-Stone carry lookahead unit out of AND, AOI2 gates with the expectation that the target cell library will contain cells resembling those.
module LCU(P, G, CI, CO);
	parameter WIDTH = 4;

	(* force_downto *)
	input [WIDTH-1:0] P, G;
	input CI;

	(* force_downto *)
	output [WIDTH-1:0] CO;

	localparam STAGES = $clog2(WIDTH);

	integer i, j;
	(* force_downto *)
	reg [WIDTH-1:0] p [STAGES:0], g [STAGES:0];

	assign p[0] = P;
	assign g[0] = G | (P[0] & CI);

	generate
		genvar i, j;

		for (i = 0; i < $clog2(WIDTH); i = i + 1) begin
			for (j = 0; j < 2**i; j = j + 1) begin
				assign g[i + 1][j] = g[i][j];
				assign p[i + 1][j] = p[i][j];
			end
			for (j = 2**i; j < WIDTH; j = j + 1) begin
				AOI2 aoi2(.A(g[i][j]), .B(p[i][j]), .C(g[i][j - 2**i]), .Y(g[i + 1][j]));
				AND and_(.A(p[i][j]), .B(p[i][j - 2**i]), .Y(p[i + 1][j]));
			end
		end
	endgenerate

	assign CO = g[STAGES];
endmodule

module AND(A, B, Y);
	input wire A;
	input wire B;
	output wire Y = A & B;
endmodule

module AOI2(A, B, C, Y);
	input wire A;
	input wire B;
	input wire C;
	output wire Y = A | B & C;
endmodule

You can map to those cells with the following script (here taking a SKY130 target as an example):
read_liberty -ignore_miss_func sky130_fd_sc_hd__tt_025C_1v80.lib
aigmap
design -save gatelib
design -reset

read_verilog design.v
prep
aigmap AND AOI2
cellmatch -lib gatelib AND AOI2
techmap -map %$cellmatch_map
dump

@QuantamHD Hah, you are quick to notice the PR. :)

I think this is super cool, and really helpful to users! Thanks for working on it

This could use comb cell eval instead of limiting itself to AIGs, no?

Yes, but I leave dealing with the eval API for another day...

This could use comb cell eval instead of limiting itself to AIGs, no?

I made this change, also I updated the quoted help message above to what it is now.

Please review and consider pulling in my comments
https://github.com/widlarizer/yosys/commit/c67ccc27a256740d1e392078b1d754ed8634fd71


Iap sample v2

Using jjwt instead of auth0 jwt library. Auth0 jwt library has intermittent token verification failures for IAP on OpenJDK.

Resolves #687

Circle will fail at the moment - Ok to merge as Admin.


Any version compatiable with debian's firefox-esr ?

I'm using latest firefox-esr from Debian buster. Its About shows firefox v68.11.0esr.

You'll probably need to use something around v0.2.5, which was pushed shortly after 68 came out. It's going to be missing some refinement – I've been gradually refining the theme since I started working on it, though the last 6 months I've just been trying to keep it from breaking with each Firefox release.


refactor(bytetrack):  fix namespace and directory structure

Description
This PR puts headers in the autoware namespace.
Additional works

Align directory structure to follow the coding guidelines.
Clean unused dependencies LIST

Related links
Part of: https://github.com/autowarefoundation/autoware/issues/4569
How was this PR tested?
Tested in a local recompute environment and the TIER IV Cloud environment.
Notes for reviewers
None.
Interface changes
None.
Effects on system behavior
None.

Tested with rosbag and it worked


Project stopped building -  Could not resolve dependencies for project- No versions available for com.amazonaws:DynamoDBLocal:jar

In the last couple of days the mvn install of this project has stopped working. I generally operate within docker containers so my environment is reasonably well managed and consistent. But yesterday I started getting the below error.
I'm seeing in inside and out of docker containers, and I'm noticed others mentioning this problem.
Could it be a co-incidence that this  was reported?
[ERROR] Failed to execute goal on project dynamodb-titan100-storage-backend: Could not resolve dependencies for project com.amazonaws:dynamodb-titan100-storage-backend:jar:1.0.0: Failed to collect dependencies for [com.amazonaws:aws-java-sdk-dynamodb:jar:[<IP_ADDRESS>,2.0.0) (compile), com.amazonaws:DynamoDBLocal:jar:[<IP_ADDRESS>,2.0.0) (compile), com.thinkaurelius.titan:titan-core:jar:1.0.0 (compile), com.thinkaurelius.titan:titan-test:jar:1.0.0 (test), com.thinkaurelius.titan:titan-es:jar:1.0.0 (test), com.codahale.metrics:metrics-core:jar:3.0.1 (compile), au.com.bytecode:opencsv:jar:2.4 (compile), com.fasterxml.jackson.datatype:jackson-datatype-json-org:jar:2.5.3 (compile), org.slf4j:slf4j-log4j12:jar:1.7.5 (compile), org.apache.tinkerpop:gremlin-core:jar:3.0.1-incubating (compile), org.apache.tinkerpop:gremlin-groovy:jar:3.0.1-incubating (test), org.apache.tinkerpop:gremlin-test:jar:3.0.1-incubating (test), org.apache.tinkerpop:gremlin-console:jar:3.0.1-incubating (test)]: No versions available for com.amazonaws:DynamoDBLocal:jar:[<IP_ADDRESS>,2.0.0) within specified range -> [Help 1]
Any ideas?

Changing the repository to the one in the AWS article means changing the aws.java.sdk.version from <IP_ADDRESS> to <IP_ADDRESS>. I don't know if that's going to impact anything else?!? Whatever we can do to get this bask ASAP would be great as this is completely screwing me.

This issue appears to be fixed now. Please confirm

Yup. All fixed.  Presumably someone did something to fix it?
Thanks anyway.

Try using https to access dynamodb-local repo
https://s3-us-west-2.amazonaws.com/dynamodb-local/release

https://github.com/apivovarov/aws-dynamodb-encryption-java/pull/1


jenkins regexp triggers will not trigger for changes to various roles

Our gate jobs have a regexp trigger that looks like:
playbooks/roles/(libvirt|common|overcloud|provision|rebuild-inventory|tripleo)/.*

That will miss -- for example -- environment, and parts.  We can add those two, or consider just matching on playbooks/roles/.* instead.

resolved by https://github.com/redhat-openstack/tripleo-quickstart/commit/389d2e9af6eb3fa00fc5096246de33b71279e3a5


upgrade<EMAIL_ADDRESS>

Fix Prototype Pollution in async(>= 3.0.0, < 3.2.2): https://github.com/advisories/GHSA-fwr7-v2mv-hh25

LGTM, thanks for the PR!

This has been released in 7.2.0


Validation fails on devfile 2.0.0 with unknown ephemeral field on volume

/kind bug

Which area this bug is related to?

/area library
What versions of software are you using?
Operating System:
Fedora Linux
Go Pkg Version:
Bug Summary
Describe the bug:
When saving a devfile, the devfile library is adding an ephemeral field to the volume structure on devfile v2.0, but this field is supported starting at v2.1 only.
To Reproduce:
$ odo create nodejs
$ odo storage create vol1
$ odo describe
# works
$ odo storage create vol2
$ odo describe
 ✗  invalid devfile schema. errors :
- components.1.volume: Additional property ephemeral is not allowed

Suggestion on how to fix the bug
The problem seems very similar to https://github.com/devfile/api/issues/675

CC @anandrkskd  for PR https://github.com/openshift/odo/pull/5252

This is the boolean value omitempty issue which was resolved by @kim-tsao  in 2.2.0. Just need to port over the change from 2.2.0 to 2.1.0 as well. This won't affect any devfile usage, but will affect the api spec. assigning to Kim to take a look.


custom.css: Add "Fira Code" monospace font-family for code ligatures

This PR adds Fira Code to the list of mono-space fonts to enable the use of code ligatures.
Before:

After:

Personally I don't like the ligatures, it hides the binary from me. Can it be an option somewhere maybe?

@devoxel if you don't have Fira Code installed on your machine (which I assume you don't if you don't like it) then it won't show up


RFE firewall support? iptables/firewalld

Hello,
is there, or are you planning to add support for firewall configuration? It would be really handy for new users.

Hey,
I've been wondering about this one for a while.
I usually believe that firewall rules should sit outside of this playbook and should be managed by an infrastructure playbook.
I'm just not sure how to address that yet, the only thing I can do for now is to provide the list of ports that must be opened...
wdyt?

That would by actually great start. Document the needed ports on the readme. Because I didn't find it anywhere here, even demo video didn't mention this, so when I started the playbook, it failed on timeout, and I had to investigate what's going on. And since on clean OS firewall is enabled by default, I was hit by that.
Another step could by just to add configuration option to open ports in firewall, commented out by default. I mean in all.sample in group_vars. I think there are valid use cases for this, when deploying on internal infra, or when networking isn't very complicated..

Hum actually one thing I can potentially do is to perform some networking test in the playbook and fail if we detect that some ports are not reachable. So if it does, you know that's a firewalling issue and can act accordingly.
wdyt?

Sounds good to me, definitely better than current state.

Ok tested and valided on #469. Feel free to test, closing this :)


Add open resources in subfolders

I have a list of characters which data is contained in modular folders, each with their own scene, script, texture, etc. It would be cleaner to be able to store resource files in the corresponding folders where they are used.

Good idea, I planned recursive scanning but forgot! If there's only one custom resource per folder, should be easy - but if there are different types of custom resources, say, a CharacterStats and CharacterBehaviour in each, the table can't be built - I still need #24 sorted out.

The feature's up (9b826b5539a0abde8f10e89e68d7c221d22d6807) so should be usable for your case! But for multiple custom types, I'd like to move the discussion into #24.


[DNM(test)]-Revert "redo (ticdc): make FlushLogs in manager asynchronously (#4301)"

This reverts commit 234cee77e485773830b1cc5eaf9a552812bc369b.

What problem does this PR solve?

Issue Number: close #xxx
What is changed and how it works?
Check List
Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Code changes

Has exported function/method change
Has exported variable/fields change
Has interface methods change
Has persistent data change

Side effects

Possible performance regression
Increased code complexity
Breaking backward compatibility

Related changes

Need to cherry-pick to the release branch
Need to update the documentation
Need to update key monitor metrics in both TiCDC document and official document

Release note
Please add a release note.
If you don't think this PR needs a release note then fill it with `None`.

Codecov Report

Merging #4475 (3d6fe70) into master (9607554) will increase coverage by 0.0209%.
The diff coverage is 22.8915%.

Flag
Coverage Δ

cdc
59.9992% <50.0000%> (+0.0770%)
:arrow_up:

dm
52.0026% <21.5189%> (-0.0263%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.
@@               Coverage Diff                @@
##             master      #4475        +/-   ##
================================================
+ Coverage   55.6402%   55.6611%   +0.0209%
================================================
  Files           494        494
  Lines         61283      61348        +65
================================================
+ Hits          34098      34147        +49
- Misses        23750      23766        +16
  Partials       3435       3435

Could you add a test to prevent such issues?

/merge

Could you add a test to prevent such issues?

I will do this in another PR.


PDF rendering error

Attach (recommended) or Link to PDF file here:
Happens wit all the PDFs.
Configuration:

Web browser and its version:  Firefox 127.0
[Problem was
Operating system and its version:

RELEASE=20.1
CODENAME=ulyssa
EDITION="MATE"
DESCRIPTION="Linux Mint 20.1 Ulyssa"
DESKTOP=Gnome
TOOLKIT=GTK

PDF.js version: 4.3.8 [c419c8333]

Is a browser extension: No

Steps to reproduce the problem:

Opening any PDF with the browser.

What is the expected behavior? (add screenshot)
White background, legible fonts.
What went wrong? (add screenshot)

Link to a viewer (if hosted on a site other than mozilla.github.io/pdf.js or as Firefox/Chrome extension):

If this happens with every PDF it's most likely an issue in the browser and not in PDF.js, so it'll likely not be possible to fix it here. Could you try starting Firefox with a fresh profile, see https://support.mozilla.org/en-US/kb/refresh-firefox-reset-add-ons-and-settings, since this could be caused by e.g. browser settings that are different from the defaults? Otherwise, please try updating graphics drivers or toggling hardware acceleration as those can also be causes of rendering defects.

This happens to me too. One thing I noticed though is it starts happening after I've had Firefox running for a while. For example, yesterday I couldn't open a single PDF. Every single one just would render a black screen with blue boxes where text or input fields were. Then this morning I closed the browser and reopened, and now I can see all of those same PDFs. I'll update this comment with a screenshot once it starts happening again. I'm running 127.0 (64-bit) Linux

I had this same issue and can confirm that restarting Firefox fixed it for me.  All PDFs were having these rendering artifacts just like the screenshots above (various manuals, a paper from Arxiv, documents from my health provider).
Browser: 126.0.1 (what I first upgraded to) / 127.0 (current)
Install Type: dpkg from repositories
OS: Pop!_OS 22.04 LTS
Windowing System: X11
GNOME Version: 42.9
Graphics: NVIDIA 550.67
Is it possible that I was on an older version and restarting updated it?  I did another system upgrade and that didn't seem to harm anything.  I'll report back if it starts happening again.

You should file a bug in Firefox itself: https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=Graphics%3A Canvas2D.
It isn't related to pdf.js itself.


[flutter_tools] remove mocks, globals from golden comparator and test runner tests

Work towards https://github.com/flutter/flutter/issues/71511
Removes mocks and globals from several tests. Includes one change to how the golden comparator tears down processes

Adding @zanderso for question on correct usage of dart:io API


SCIM filtering: startwith treats _ as wildcard

Hi,
when using the SCIM API and filtering, it looks like "starts with" treats "_" as wild card.
Example: Filtering all groups using "filter=displayName sw 'GROUP_'" returns also "groups.update".
From a SCIM API point of view, I would expect "_" is treated as regular charactar. Thus, in the example above I would only expect groups with a display name starting with 'GROUP_' in the result.
Note: This issue is not on upper/lower case conversion; just on the handling of "_".
Background: The issue came up, when introducing naming conventions (prefixes like GROUP_ , ROLE_ ,...) to model logical concepts like groups and roles using SCIM Groups.

Closing due to inactivity.


[Documentation] video demo

Hello 🙂 a video/GIF about the UI/UX on different platforms would be a nice to have. Thanks for all.
Best regards.

@RichardFevrier thanks for your suggestion. Currently the UI looks a bit different than in the image of the demo, because I changed the drawing backend (from NanoVG to Skia) and some things are still not implemented. I will follow your suggestion after this is improved.
In general the framework is in a very rough state and I can't recommend using it for more than a pet project.


proper use of bullets in mark down

please add a section and update pages to use ordered lists with MD -https://guides.github.com/features/mastering-markdown/#examples

This Issue was addressed.


Update setup.py to remove broken requirements

The requirements in setup.py include pyspark, databricks and koalas. These packages are installed by default on Databricks. When running %pip install labelspark with the current setup it will install a different version of pyspark, databricks and koalas from Pypi. The package called databricks on Pypi is not the correct package and will overwrite the proper package on Databricks. This will cause import labelspark to fail with the error:

ModuleNotFoundError: No module named 'databricks.koalas'

This

Hi @msokoloff1 I think this looks good to merge. It does seem like the "databricks" requirement is already satisfied by Databricks' clusters with DBR so we're causing issues by including it in the setup.py. Thanks @mprojlb for catching it.

Considering this PR is very narrow in scope, I'll go ahead and merge this to get the fix out to people. We'll adjust the release to 0.2.1. Thanks @mprojlb for the contribution!


Console only supports zh-cn command in  0.10.13-insider Win10(Chinese-Simplified)?

VSCode Version: 0.10.13-insider
OS Version: win10

How can i  switch back to the English commands or the console can support both 'English' and 'Chinese'?When i'm coding, I have to switch the IME.

Duplicate of https://github.com/Microsoft/vscode/issues/4691, thanks for the report :)


Separate hadoop and native batch docs more

Slight adjustment to batch ingestion docs, creating a separate page for hadoop and restructuring the existing batch ingestion page to be a directory page pointing to specific task type docs

👍


feat(pre): Create Pre component

Add Pre component
Rename Code test suite variable from container to actualOutput to maintain a standard between all components tests.
Resolves #49

Awesome stuff! Thanks a lot @irangarcia 🎉🎉🎉


jdk: oraclejdk8 causing travis builds to fail

Is it possible to remove this line or specify a newer version?

resolved PR #82


Can't Find Nested Functions

Is there any particular reason why codo can't find/parse nested functions? These are completely legitimate in Coffeescript, so as such it seems like a Coffeescript documentation tool should support them.

Would it be possible to add this feature?

Because they have nothing to do with OOP. We are not supporting the functional paradigm and it is not planned.


PositionPlugin:  set default zoom level when not available

Pull Request Test Coverage Report for Build<PHONE_NUMBER>6
Details

6 of 6 (100.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.04%) to 99.67%

Totals

Change from base Build<PHONE_NUMBER>4:
0.04%

Covered Lines:
13507

Relevant Lines:
13509

💛  - Coveralls


Mobile nav enhancement

top nav should be moved to a fixed bottom bar

SIDEBAR

sidebar should open on top of main panel, not push the main panel to the side
tapping the "sidebar" icon toggles the sidebar drawer open and closed
tapping a nav item in the sidebar will close the sidebar and redirect the user to the specified node
sidebar drawer will scroll if necessary

SEARCH

search opens up in a modal with a translucent overlay
search interactions should work similar to sidebar interactions - tap a search result item to navigate to it or tap the search icon in the bottom bar to close search.

style updates (tara can do these later)

sidebar drawer has an increased font size and padding for bigger tap targets
search bar will also need a larger font size and tap targets

prototype:
https://www.figma.com/file/bGzEZo0NYHhWGJO95kwF3p/Treehouse-App?node-id=207%3A2585&t=WN6qaMSfuAkDsCyJ-1

assigning to myself so i can add mockups

@progrium will break down the implementation details, maybe split up into separate tickets

Going to use this issue for a rough pass and then minor details can be filed and handled by Dwayne.

[ ] split out nav component
[ ] split out panels component
[ ] create app/mobile component
[ ] search dialog
[ ] detect mobile

Pushed preliminary implementation. Most minor things can be fixed with css (sizing), but noticing more thought needs to be put into multiple panels, etc.


Explore converting pod restart to pod reload where possible

Instead of restarting the pod investigate and replace pod restarts with Prometheus config reload where possible.

This ticket is no longer relevant for a sidecar charm as stated. However it may still be possible to explore, if Prometheus may just be asked to reload its new pushed config instead of restarting the container.


Full-screen overlay container doesn't work when fullscreen element is within an overlay

Bug, feature request, or proposal:
I believe this might be a bug.
In this example I try to load a snackbar into a container using the viewContainerRef config.
https://stackblitz.com/edit/angular-material-tooltip-in-container
What is the expected behavior?
The snackbar should align itself to the viewContainerRef that is provided.
What is the current behavior?
But the _createOverlay function does not look at this config and always creates the overlay using the GlobalPositionStrategy instead of ConnectedPositionStrategy.
There could be another root cause, and if there is please let me know.
What are the steps to reproduce?
I have provided a stackblitz above.
What is the use-case or motivation for changing an existing behavior?

One in particular is the div has been maximized to full screen (so the window is not visible).
Therefore the snackbar is not visible.
Which versions of Angular, Material, OS, TypeScript, browsers are affected?
latest.

Looks like this doesn't work when trying to make the fullscreen element be within an overlay itself. I don't really see a good way to fix this, since the overlay container needs to be inside the fullscreen element for subsequent overlays to work.

I have a solution that will work for now.
https://angular-material-tooltip-in-container-2.stackblitz.io/
This uses custom snackbar and overlay's (thanks to this)

I've implemented the FullscreenOverlayContainer in my ngModule, and on desktop browsers, it works great! However, it does not work on Android - the menu is still shown behind the main content as it did before adding this provider. Is this a known issue and are there any workarounds to get this working on the android browser?

I am also seeing the error with overlays:
"There was a general error.Error: Failed to execute 'appendChild' on 'Node': The new child element contains the parent."  from FullscreenOverlayContainer._adjustParentForFullscreenChange .
Do we have a solution to this?

providers: [
{ provide: OverlayContainer, useClass: FullscreenOverlayContainer },
],
add this in your relevant module.ts file
it will  handle cases where your application is in full-screen mode. It ensures that overlays are appended correctly to the DOM when the application enters full-screen mode, so they remain visible.
Issue is closed 🙌


Test PR to see why Paul sees editor errors and Severin doesn't

@PaulGarewal There are almost exactly the same files changes you have in PR 232. I do not see any editor warnings.
Are you able to check out this PR's code and see what it looks like for you? This command:

@PaulGarewal There are almost exactly the same files changes you have in PR 232. I do not see any editor warnings.
Are you able to check out this PR's code and see what it looks like for you? This command:

This looks good for me, must be something I didn't catch. Will run StoryBook with this now

Test is complete. Closing this PR now.


Info needed on min. node config (at least on GKE)

gitkubed pods are unschedulable on a low config node on GKE (g1-small: 1 vCPU, 1.70 GB ).

Works like a charm on the next available config. Add information about the min. node config needed.

@dsandip Will you please add the docs/gke.md ?


Google Sheets embeds are too small (integrations)

I am working on the Integrations page for Google Sheets + Ghost, explaining how to embed a spreadsheet into a Ghost post. This is great for people who want to embed dynamic data or swipe files that their readers can export from their site.
When I published the page, the embed appears very small on the page:

This is not a usable working example of this embed and should fit to the width of the content. Can we fix this?
The embed code used is:
<iframe src="https://docs.google.com/spreadsheets/d/e/2PACX-1vQKQwfka4cgU7mnS2uAiCMKs0wkmHJQdH5sc4mp231iVKjwueBVaX3Qy8no7gTihQXr6kwJ52TLyfLt/pubhtml?widget=true&headers=false"></iframe>

Note: I've unpublished the Integrations page for now, but the content is in draft in Ghost Admin.

@kymellis The result looks the same to me with Casper (when I check the preview link), which doesn't make it right, but I think it's a general issue that we didn't tackle yet (and we should maybe think of opening the issue as well on Casper?)
I will assign this issue to @peterzimon as it's a design issue here.

@AileenCGN @peterzimon OK sounds good. I already opened an issue on Casper here for the exact same problem.
I noticed that the same thing happens in Casper with some other embeds, too. But it's not always an issue because it depends on the content being embedded.
For example:

This looks OK at a smaller size, but it would look better full-width.
This example also renders at the full-width on the live docs site: https://docs.ghost.org/integrations/github/

@peterzimon
I've now published the Backtracks integration using an OEmbed and it's not displaying quite right on the docs front-end. See here, the logo on the right is squished:

Should I create a new issue for this or is it OK in here?


Broken imports

The current version (0.1.0) installs Albumentations most recent release, which does not contain certain functions which were used. The current work around is to install from source. I.e. install the development version of Albumentations and use the cloned version of this repo.
This will need to be fixed for next update.

Packages must be imported through git for now. It works, but ideally a different solution is found in time. Maybe copying over the missing transforms that we expect to have.


Hosted Supavisor closes connection executing query with large result set

Bug report

[x] I confirm this is a bug with Supabase, not with my own application.
[x] I confirm I have searched the Docs, GitHub Discussions, and Discord.

Describe the bug
When connecting to my Supabase-hosted Supavisor using any client and executing a query that returns a sufficiently large amount of data (order of a gigabyte), Supavisor closes the connection during execution. I identified this with a real query we were running on our backend, but it's easy to reproduce with any old query, e.g. select generate_series(1, 100000000).
It fails reliably every time after about the same time limit. Other long-running queries work OK. It works fine when connecting to the instance directly, and was working fine with pgbouncer.
To Reproduce

Additional context
Supabase project ID: pxidrgkatumlvfqaxcll
Postgres version: 15.1

Thanks for this finding! We have finished a fix and it will be rolled out on production soon.


Legacy List links are not working

Legacy List links are not working

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 6545d340-7841-d22c-dcc3-85f03b72340f
Version Independent ID: 460cb2d0-4221-ce65-0912-501135463064
Content: List of Participants - Microsoft Trusted Root Program
Content Source: trusted-root-docs/Participants-list.md
Service: security
GitHub Login: @kasirota
Microsoft Alias: kasirota

@kasirota - please review and fix.


Adjust resource URL path in serve.mdx

Description
The related resources job failed when it came across this url, changing it to the format we've been using
Reviewer Notes
Make sure the nerdpack serve related resource still links to its page from the /build-apps/publish-deploy/serve guide

Closes https://github.com/newrelic/developer-website/issues/1241


How to convert UID to actual Id

My card's NFC Id is 1082.
When I scan it on https://whatwebcando.today/nfc.html on android phone it shows data = 10001082
When I use this library. NFC.GetCardUID returns 04eec4a2
I have checked chrome plugin (TagConneX) it passed 04EEC4A21C6880 to its web page.
How can I convert 04eec4a2 or 04EEC4A21C6880 to 10001082?

I came to know that NFC.GetCardUID is for universal id of NFC card.
For data we need to read blocks (readblock command). "NFC Tools" android app was helpful which displayed the hex value of data. I compared this hex value to the values returned from all blocks (loop through all blocks using this library) and found the match on block-6.
I am not sure why this value is stored at block-6.

I came to know that NFC.GetCardUID is for universal id of NFC card.
For data we need to read blocks (readblock command). NFC Tools android app was helpful which displayed the hex value of data. I compared this hex value to the values returned from all blocks (loop through all blocks using this library) and found the match on block-6.
I am not sure why this value is stored at block-6.


Multiple Plotters

Hey! Love the script, thanks for all the work you've put into it. I have 3 separate dedicated plotting machines, the plot_manager.py and kill_nc.py scripts (as well as send/rec plot scripts) are designed for a single plotter. Have any suggestions or thoughts around removing this limitation.
Initially I decided to just append the hostname to the remote_checkfile variable, this way the plotters would not kill each others netcats, but multiple netcats over a gigabit network slows down the the transfer from approx 110MB/s on a single file, to a total of 60-70MB/s across all files. I'd love to avoid that loss of speed if possible, I am already at the cusp of needing 10GBE .

Thomas - I have thought about the exact same thing and you are on the right track, I was planning on doing something like I do now with multiple harvesters where I get information from each one and make a choice about where I am going to send a plot.
Are you running just a single harvester or multiple harvesters/plotters?
If you have a single harvester and multiple plotters, we would just need to make sure that only a single plotter was transferring at the same time, otherwise, you could saturate your network, even if we used different Netcat ports which is how I was thinking of doing it.
Let me know more about your particular configuration and let me see if I can brainstorm an idea as to how to make it work.

@thomascooper  waited for @rjsears implement it, but gave. Ordered ready python script , it is using socat. i have 9 plotting machines and 1 harvester. also i have 10gbe network... making 9 plots every 11 minutes (1 on each plotting machine) and its working perfect. the only thing you need  9 disks on harvester with enough space to handle it. because copying 2 plots on 1 disk will slow things down.. 1 socat connection copying plot to 7200rpm hdd in about 11-12mins , so i need slow down plotters a little bit. i think problem will be solved when i will bound 2 10gbe NICs on harvester with 802.3ad ...

I have a single harvester and 2 plotters capable of 8TB/day. The harvester has 4GbE, plotters 1GbE each so no need to worry about network saturation.
The current version of the script checks traffic load on the NIC being used and stops if there is any, so yeah it is very much designed for a single plotter scenario. My super simple solution was to create an LXC on the farming machine and make it my "second" harvester, this way each of my plotters had a dedicated harvester to connect to.

In the end I did it a little bit differently. Using ncat I am able to copy a plot from my plotters to my harvesters in about 6 minutes. Each of my plotters has a list of harvesters that it will talk to (defined in the config file) and it will make a determination once-per-minute on which harvester to send the plot to based on an available number of plots on that particular harvester (or number of old plots to replace). In this way my harvesters almost all have the same overall number of plots on them pretty much most of the time.
Since I have 13 harvesters now and 6 dedicated plotters all doing 288 plots-per-day (madmax on Quad CPU Dell servers) I needed a large number of harvesters to receive the plots. No need to slow anything down until I run low on drive space. Additionally, many of my harvesters are also running madmax and plotting internally to themselves (see move_local_plots.py). So as you can see, it is not "very much designed for a single plotter" as I am now up to 6 dedicated plotters and another 8 not-so-dedicated plotters. Instead, each plotter has a subset of harvesters that it will talk to and within that subset, it makes a determination each time it runs as to which harvester to send the next plot to. In my opinion, this was a far more sustainable solution to having one plotter sending multiple plots at the same time, at least in my situation.
I am well over 3PiB and this setup works fantastic for me. I guess when I get more time I will try and think of a better way to do this, but what I have also found is that when my plotters are running at 100% CPU and memory utilization, I get zero benefit by trying to send multiple plots at the same time, the boxes are just not powerful enough to make plots and saturate my 10G network.

Sorry about the delay in my response. For those of us with a single harvester/NAS/farmer but multiple plotters the current system has a few major issues operating in this case.
plot_manager.py checks to see if ncat is running and if there is traffic on the local plotter, if there is not then at the start of process plot it calls a remote kill all netcats. This is problematic with multiple plotters because every plotter will check and see they are not plotting and then kill all remote transfers. This leads all plotters into a loop where they just continually kill each others netcats and never finishing a single transfer. I have fixed this problem  with a hack of adding a function to check the ncat on the server remotely and stop the plotmanager if it find it. Then I added a script to the NAS that checks its network traffic and kills all ncats if there is no traffic. This solution works for the most part, sometimes the NAS script skips a beat on traffic and kills good transfers, and  sometimes the plotters all run at the same time and start 3-4 ncat processes.
I think the best fix to this would be to have the plot_manager scripts only look the remote checkfile, and have that be 100% responsible for starting a new plot transfer. Then add a new script to the NAS with 2 functions:

Look for the checkfile, if the checkfile exists (and was created longer than 3 minutes ago) then check for a NC and transfer rate if either is not there, then killall ncat and delete the checkfile
Look for ncat if its exists, monitor traffic for 20seconds, if there is no traffic then killall  ncat and delete any checkfiles

I'll take a stab at creating this script myself and see where I endup

No worries, ok, so a couple of questions. Are you wanting to send multiple plots from multiple plotters to a single harvester at the same time? What is your network infrastructure? I did set a similar setup for a friend and just hard-coded different ncat ports and tweaked the checks a little bit. The other option is to do what you are suggesting and not worry about killing the ncat but rather pass along active transfer information with the export from the harvester that already goes to the plotter as the definitive answer to if a transfer is in progress. But that would really depend on what you are trying to accomplish in the long run.
I am absolutely certain we could figure out a way to make it work, I just really need to understand your specific use case.

The expectation is to send a single plot at a time to the NAS. I have a very similar network infrastructure to your example, I have a 10.x.x.x network (actually eno2 on each machine) for plot_manager transfers, then a 192.168.x.x network for external communication.
1x 48 Port Switch (2 VLANs) 1GBE
3x Standalone plotters (each with dual 1GBE nics)
1x NAS Server (dual 1GBE nics, HBA, also a local plotter)
3x NETAPP DS4243

OK,so just thinking out loud here but once per minute (via cron) I create a JSON export file for each harvester that has a ton of info for reporting, selecting which harvester to send plots to, etc. I could easily add a key to that file that indicates if there is a current transfer taking place. When you run .plot_manager.py, it grabs those JSON files from each harvester, or your single harvester in this case, and makes decisions based on the information contained in the file.
I could check to see if there is an existing transfer that way (driven by the harvester) and not allow a transfer in that case. In this case, transfers would be done on a first-come-first-served basis.

That sounds like a good solution in my opinion.

OK, let me play around with it for a bit. I already updated the export to include the info, now I need to figure out how to do it on the plotter side....

Ok, V0.97 I think will work for you now. You need to install it on both your harvester (first) and then your plotter. Make sure to run ./drive_manager.py on your harvester to generate the new export before running ./plot_manager.py on your plotters.
Basically, it will not allow you to send a plot to a harvester if that harvester is reporting itself to have a transfer active. Also removed the remote call to kill_nc.sh as I upgraded from nc to ncat and I don't think it has the same issues. I will have to keep an eye ion it to see.
Let me know how it works out for you.

Everything worked on individual runs and manually testing plot_manager.py gave the new error message as expected. But at some point the drive_manager.py script must have marked the export remote_transfer_active as false and let a second upload start. It has occurred multiple times since I updated

Hum, the only thing I can think would be that the drive_manager.py on the harvester does not think that there is a transfer going and hence marked it as false. Are you running drive_manager.py once per minute via cron?
I might suggest manually viewing the export file (locate in the export directory) for the harvester while a transfer is in progress and make sure it shows that a transfer is still going. If it does not, we need to troubleshoot a bit further too see why it thinks the transfer has stopped.

99% of the time when I check the export shows true as expected, but randomly it reverts to false, I have to assume its an issue with the transfer speed checking
Good one:
root@nas-1:~/plot_manager# ./drive_manager.py && cat export/nas-1_export.json Welcome to drive_manager.py Version: 0.97 (2021-09-16) check_temp_drive_utilization() started Temp Drive(s) check complete. All OK! check_dst_drive_utilization() started DST Drive(s) check complete. All OK! check_plots_available() started Plot check complete. All OK! nas_report_export() started check_for_active_remote_transfer() called Remote Transfer File does not exist, lets check for network traffic to verify.... check_network_activity() called Network Activity detected on eno2 Network traffic has been detected, a Remote Transfer is in progress. send_new_plot_notification() Started update_receive_plot() Started Replace Plots Set, will call build script for plot replacement! Checking to see if we need to fill empty drives first...... fill_empty_drives_first flag is set. Checking for empty drive space..... Found Empty Drive Space! Low Water Mark: 10 and we have 250 available Remote Transfer in Progress, will try again soon! {"server": "nas-1", ..., "remote_transfer_active": true, "trigger": "2"}
Then randomly a few minutes later (while the transfer is still active and I can still see ncat in iotop and ps, I get the following.
root@nas-1:~/plot_manager# ./drive_manager.py && cat export/nas-1_export.json Welcome to drive_manager.py Version: 0.97 (2021-09-16) check_temp_drive_utilization() started Temp Drive(s) check complete. All OK! check_dst_drive_utilization() started DST Drive(s) check complete. All OK! check_plots_available() started Plot check complete. All OK! nas_report_export() started check_for_active_remote_transfer() called Remote Transfer File does not exist, lets check for network traffic to verify.... check_network_activity() called No Network Activity detected on eno2 No Current Remote Transfers are taking place. send_new_plot_notification() Started update_receive_plot() Started Replace Plots Set, will call build script for plot replacement! Checking to see if we need to fill empty drives first...... fill_empty_drives_first flag is set. Checking for empty drive space..... Found Empty Drive Space! Low Water Mark: 10 and we have 250 available Currently Configured Plot Drive: /mnt/enclosure0/front/row5/drive20 System Selected Plot Drive:      /mnt/enclosure0/front/row5/drive20 Configured and Selected Drives Match! No changes necessary to /root/plot_manager/receive_plot.sh Plots left available on configured plotting drive: 106 {"server": "nas-1", ...m "remote_transfer_active": false, "trigger": "2"}
I tested it again right after it was true again.

Just as a bandaid I tried to run the following:
pgrep ncat |awk 'NR >= 2' | xargs -n1  kill
This looks at all ncat processes, then if finds more than one running it will kill all except the first one. This works but it leaves the first ncat averaging between 15-20M/s (instead of 110M/s which it was doing before the second started)

In order to make sure that the cron isn't just trying to start all 3 remote plotters at the same exact moment, I appliled the following changes to each of the 3 remote plotters crontab:
0-57/3 * * * * cd /root/plot_manager && /usr/bin/python3 /root/plot_manager/plot_manager.py >/dev/null 2>&1
1-58/3 * * * * cd /root/plot_manager && /usr/bin/python3 /root/plot_manager/plot_manager.py >/dev/null 2>&1
2-59/3 * * * * cd /root/plot_manager && /usr/bin/python3 /root/plot_manager/plot_manager.py >/dev/null 2>&1

This should run the script every 3 minutes but offset by 1 minute per server so that they do not cross with eachother

Maybe throw a generalized try / except around the entire check_network_activity() and if it errors just try it again. When I was running my manual script to look for duplicate ncats and kill them every now and again I would get an error trying to read the network_stats.io file, sort like if the network_check silently failed and left no file.
Here is what I did simply to get passed that issue:
while True:
  try:
    transfer_isactive = check_for_active_remote_transfer()
    break
  except FileNotFoundError as err:
    pass

This particular version is not very elegant at all and was subject to getting stuck in a loop if there was any error other than a FileNotFoundError, but it generally allowed the script a second pass which never failed 2x in row.

I woke up this morning to a hung ncat process with no transfers on going and all 3 plotters out of space so this must have happened early into the night.

OK, so I think I need to add back in the ncat kill so that there are no longer any hung ncat processes. I can also rewrite the check_network_activity() function to have better error checking to verify, as you say, that the file exists and if it does not, run the check again.
I have to think about it a little differently than I was before, so let me think about how I can rewrite it and work on it.

OK so my first step was to alter the install script and rewrite the check_network_io.sh script and have it do the heavy lifting to make certain we never not have an output file. That should eliminate the need for your change above. It might not be the perfect solution, but I think it will eliminate those weird circumstances when there is no network_stats.io file found.
I am still working on the false false when there is a network transfer in place. I cannot get it to fail on any of my systems so I am not sure what I am looking at. I am attempting to think about a way around it to make sure it reports that variable correctly!
This is the new script (created by running ./install.sh nas):
#! /bin/bash
# Script automatically created by install script on: 2021-09-17 10:48:49"
network_interface=$1

check_network_traffic(){
echo -e "Checking for network traffic on $network_interface"
if [[ -f /root/plot_manager/network_stats.io ]]; then
   echo -e "\nFound old stats file, deleting...."
   rm /root/plot_manager/network_stats.io
   /usr/bin/sar -n DEV 1 3 | egrep $network_interface > /root/plot_manager/network_stats.io
   sleep 1
else
   echo -e "\nDid not find old stats file...."
   /usr/bin/sar -n DEV 1 3 | egrep $network_interface > /root/plot_manager/network_stats.io
   sleep 1
fi

if [[ -f /root/plot_manager/network_stats.io ]]; then
   return 0
else
   return 1
fi
}

until check_network_traffic ; do : ; done

Just as a bandaid I tried to run the following:
pgrep ncat |awk 'NR >= 2' | xargs -n1 kill
This looks at all ncat processes, then if finds more than one running it will kill all except the first one. This works but it leaves the first ncat averaging between 15-20M/s (instead of 110M/s which it was doing before the second started)

Yeah, I think I have to go back to making sure ncat does not stay running. It is not supposed to stay running after receiving the EOF from the sending side, but obviously, this is not that case. I think the way you have your cron setup may help since, in theory, there should not be any ncat running after the transfer has taken place and none of the other plotters should be able to start a transfer if one is already taking place.

OK, so I did some more reading up on ncat (https://nmap.org/ncat/guide/ncat-file-transfer.html) and added a couple of options to both the send and receive scripts. The big one was the --send-only flag on the plotter. This immediately quits on EOF which should force the listener to quit as well. Give it a shot and tell me if it helps any.

OK, so in looking over the code, I decided to prevent a false false transfer thus leading to a double plot transfer, if we get a false from check_network_activity() we re-run it a second time just to verify that the false is really correct. I am sure it is not the best method, but I think it may resolve the weird false return when it should be true instead.

Sounds good, that is what i had my make-shift addition doing. So far so good with your newest commits, ill let you know if I see any issues.

Perfect, thanks! I really appreciate working with you on this stuff, helps me find errors and better ways of doing things.

No problem. At the moment everything is running smooth with only 2 personal customizations:

Changed column to * in all the globs so that I could name my NetApp DS by "row" rather than column
Offset the start time for the crontab on each of the plotters so they don't all try and start a job at the same time.

Yeah, I eventually want to redo the globs somehow so that it is easier to use other mount formats, I just haven't had time to figure it out or play with it yet.

24+ hours and no issues!

Good deal!

Found a issue as space is running low, I have a single drive left with space and only 10 plots worth. I am getting the following error with move
root@nas-1:~/plot_manager# ./move_local_plots.py
Welcome to move_local_plots.py: Version 0.97 (2021-09-16)
update_move_local_plot() Started
Traceback (most recent call last):
  File "./move_local_plots.py", line 391, in <module>
    main()
  File "./move_local_plots.py", line 384, in main
    update_move_local_plot()
  File "./move_local_plots.py", line 87, in update_move_local_plot
    internal_plot_drive_to_use = get_internal_plot_drive_to_use()[0]
IndexError: string index out of range

OK, that is very weird. I specifically address that very issue:
def get_internal_plot_drive_to_use():
    """
        Same as above but returns the next drive. This is the drive we will use for internal plots. We do
        this to make sure we are not over saturating a single drive with multiple plot copies. When you run
        out of drives, these scripts will fail.
        """
    available_drives = []
    try:
        for part in psutil.disk_partitions(all=False):
            if part.device.startswith('/dev/sd') \
                    and part.mountpoint.startswith('/mnt/enclosure') \
                    and get_drive_info('space_free_plots_by_mountpoint', part.mountpoint) >= 1 \
                    and get_drive_by_mountpoint(part.mountpoint) not in chianas.offlined_drives:
                drive = get_drive_by_mountpoint(part.mountpoint)
                available_drives.append((part.mountpoint, part.device, drive))
        return (natsorted(available_drives)[1])
    except IndexError: # We will get an IndexError when we run out of drive space
        # If we have no more drive space available on a drive not already being use to store local plot AND we are using pools and have elected to
        # replace non-pool plots, fall back to returning pool plot internal replacement drive:
        if chianas.pools and chianas.replace_non_pool_plots: # Sanity check, must have pools and replace_non__pool_plots set to true in config file.
            log.debug('CAUTION: No additional internal drives are available for use! Since you have replace_non_pools_plots set,')
            log.debug('we are going to return the next available local plot drives with old plots to replace.')
            return chiaplots.local_plot_drive
        else:
            # If we have no more drives left, fall back to the only drive left on the system with space available
            log.debug('CAUTION: No additional internal drives are available for use! Defaulting to using the next available drive with space available.')
            log.debug('This can cause contention on the drive bus and slow down all transfers, internal and external. It is recommended that you resolve')
            log.debug('this issue is you are able.')
            notify('Drive Overlap!', 'Internal and External plotting drives now overlap! Suggest fixing to prevent drive bus contention and slow transfers. If you have selected plot replacement, we will attempt to convert to replacement now.')
            return get_plot_drive_to_use()

Specifically this:
except IndexError: # We will get an IndexError when we run out of drive space

Can you check get_internal_plot_drive_to_use() in drive_manager.py and make sure that code is in there...?

Sorry about the delay, yes that code is there. And the error went away the moment I added a new hard drive and the space adjusted. Once I get near low space again in the next 10 or 15 days, I will test and reach out. I am also going to start a different thread to discuss an odd speed issue I am seeing.

Once or twice a day I still catch two NCATs running, even with the changes to remote_transfer.

OK, thanks for the update, I am just not certain what is causing that, but let me rebuild the kill script for the ncat, I will call it immediately after the transfer is complete to try and prevent it from wacking a transfer in progress.