Single-copy orthologous genes identified is 0

Hi Jacob
I want to use OrthoSNAP identify single-copy orthologous genes nested within larger multi-copy gene families. But  I have a question: Single-copy orthologous genes identified is 0. following is my orthogroup tree. I think this orthgroup should have multiple subsets of  single-copy orthologous genes. I'm not sure if I had a problem with my data preparation. The fasta file is in FASTA format
test.faa.zip
(species0|01G00667-RA:0.5279117043,(((species4|01G04915-RA:0.3408345014,((((((((((((((((((species0|02G00068-RA:0.2833922109,species0|03G01048-RA:0.2202978172)87:0.0438419260,species0|06G05308-RA:0.0342599303)89:0.0234458576,((species0|08G00253-RA:0.0300651474,species0|01G06890-RA:0.1020280754)93:0.0297929956,species0|05G03661-RA:0.0292380589)81:0.0003640542)68:0.0148926744,(species0|10G05798-RA:0.0100306324,species0|10G05809-RA:0.0099046340)100:0.0425698431)65:0.0239536424,species0|03G03143-RA:0.0037587110)64:0.0901618950,((species0|06G04570-RA:0.2009566336,(species0|09G00242-RA:0.0276064627,species0|09G00252-RA:0.0000010000)98:0.0444388887)95:0.0272777858,species0|02G00069-RA:0.0184186553)92:0.0247560096)65:0.0684129707,(species0|03G02980-RA:0.0311118160,species0|01G01632-RA:0.0081907706)48:0.0068875751)30:0.0000022508,species0|06G06430-RA:0.0171662576)47:0.0824734516,species0|08G01583-RA:0.0582321366)74:0.0698105372,(((((((((species0|06G04577-RA:0.1029693336,species0|08G01179-RA:0.0058083040)65:0.0037224542,species0|06G06159-RA:0.0062634697)18:0.0000010000,species0|02G02446-RA:0.0031141986)9:0.0000010000,species0|02G04676-RA:0.0000010000)13:0.0000010000,species0|05G06142-RA:0.0031219956)34:0.0000023735,species0|05G00002-RA:0.0000010000)66:0.0176742730,species0|05G02195-RA:0.0000010000)67:0.0270298851,species0|07G00780-RA:0.0111600524)93:0.0810354911,species0|05G06683-RA:0.0094745401)56:0.0033010953)75:0.0139675175,species3|04G02997-RA:0.0479693279)54:0.0070020470,((((species0|UnG00061-RA:0.1019755829,((species0|05G04281-RA:0.0100712598,species0|UnG01151-RA:0.0000010000)63:0.0000010000,species0|08G00680-RA:0.0033416155)97:0.0075728491)76:0.0047777879,species0|07G05284-RA:0.0205874047)34:0.0000022133,((((species0|11G02225-RA:0.0000010000,species0|11G02226-RA:0.0034223709)100:0.0206299805,species0|UnG00947-RA:0.0101420749)53:0.0000024204,species0|05G06682-RA:0.0050443888)54:0.0000010000,species0|03G03240-RA:0.0151959726)73:0.0025157288)62:0.0224812810,species0|03G01047-RA:0.0274408894)31:0.0000444198)61:0.0209780864,(species3|09G02646-RA:0.0263452359,species0|06G04032-RA:0.0190750798)96:0.0027947135)44:0.0000024207,species0|07G03150-RA:0.0339043024)65:0.0126331495,(((((species3|03G03823-RA:0.1104765017,species3|07G00184-RA:0.0282350932)68:0.0021200555,species3|UnG04073-RA:0.0334773575)63:0.0049922998,(((species0|05G00255-RA:0.0717264427,species3|01G02394-RA:0.2180804066)93:0.0232292895,species3|07G03562-RA:0.0328082244)12:0.0000022505,species3|02G01645-RA:0.0456063875)9:0.0000024289)15:0.0000028490,species3|08G02640-RA:0.1005701721)72:0.0080842576,species0|09G00240-RA:0.0608569853)76:0.0151201690)61:0.0085221777,species2|09G00394-RA:0.1317027250)66:0.0083495796,species0|03G03342-RA:0.0000010000)67:0.0112977953,((((((species2|11G04129-RA:0.1107995970,species2|11G04120-RA:0.0000010000)74:0.0000024273,species2|11G04119-RA:0.0000010000)81:0.0000027880,(species2|11G04138-RA:0.0000022138,species2|11G04146-RA:0.0164280934)100:0.0500509920)100:0.0318182024,species2|07G02618-RA:0.0105404848)96:0.0180265075,(((species2|11G03854-RA:0.0286863546,(((species2|06G01852-RA:0.0708707538,species2|11G00501-RA:0.0364722483)87:0.0123542412,species2|11G02708-RA:0.0300289749)56:0.0000027713,species2|05G01547-RA:0.0300265612)70:0.0000024696)90:0.0257159554,species2|04G04476-RA:0.0156910819)52:0.0000026998,species2|05G02497-RA:0.0301598075)72:0.0000024693)98:0.0200837784,((((species2|10G02227-RA:0.3108728301,species2|05G03233-RA:0.0000026898)97:0.0464486734,species2|10G04334-RA:0.0489527240)91:0.0170403849,(species2|02G02902-RA:0.0602162416,(species2|06G02624-RA:0.0000010000,species2|06G02625-RA:0.0000010000)96:0.0018256185)99:0.0294009743)17:0.0000023342,((species2|08G02827-RA:0.0711202157,species2|01G04476-RA:0.0898182083)97:0.0348380375,species2|01G05041-RA:0.0233416284)49:0.0000027600)23:0.0000028939)100:0.1365387571)66:0.0777546471)61:0.1920394558,((species1|04G01317-RA:0.1160132618,species1|02G00465-RA:0.1715145619)99:0.1552966767,species1|11G02518-RA:0.1294596501)94:0.1533573617)61:0.0288696434,species1|05G02443-RA:0.3565432133)78:0.0900322084,(species3|07G04113-RA:0.4880719806,(((((species0|09G00325-RA:0.0689512854,species0|09G00326-RA:0.0829119289)100:0.3091446375,species0|05G03566-RA:0.3080354823)82:0.0870903929,(species0|08G05223-RA:0.0376953437,species0|05G03094-RA:0.0244950842)62:0.0000025151)82:0.0439946415,species0|01G06839-RA:0.0354282611)47:0.0322824766,species3|05G04758-RA:0.0950203399)36:0.0000024398)98:0.2712607015);

test.faa.zip
The fasta file is in FASTA format

Hi Xiao Xu,
After looking at the phylogeny, I don't see any instances where OrthoSNAP should be finding a subgroup of single-copy orthologous genes (or SNAP-OGs). To test the behavior of OrthoSNAP, I ran your files but changed the occupancy threshold to have at least two sequences. As expected, SNAP-OGs were identified. Based on the tree topology, are you expecting specific SNAP-OGs?
Assuming no SNAP-OGs being the correct output, there are some clear cases wherein this could happen. For example, some multi-copy gene families have unclear evolutionary histories that make highly confident SNAP-OG detection difficult. Unclear evolutionary histories may be driven by complex patterns of duplication and loss or a lack of information, making tree inference challenging.
Hope this helps!
All the best,
Jacob

Hi! Jacob
Thank you very much for your answer, which helped me a lot. I still have a few questions I would like to ask: my ultimate goal is to test whether two different multicopy orthgroups exist coevolution using PhyKIT. Therefore, I would like to use orthosnap to identify single-copy orthologous genes first. If one has many subsets of multicopy orthgroups, which subset should I choose for coevolution identification? In addition, like in my data, there are five species, but genes of only two species are identified in the subset. Can this incomplete SNAP-OGs represent its multi-copy orthgroups

Hi Xiaoxu,
Thank you so much for adopting the software we have developed. Perhaps others may also be of use to you. A complete list of software our team has engineered can be viewed here: https://jlsteenwyk.com/software.html.
If you are coevolutionary calculations with five species, which is relatively few, I would recommend using the occupancy argument when running OrthoSNAP; specifically, set this parameter to five. In doing so, OrthoSNAP will only report SNAP-OGs with full taxon occupancy (N=5).
To directly address your question, if multiple SNAP-OGs are identified in a larger multi-copy gene family, use them all instead of just one. This can substantially increase the gene space examined.
Please feel free to comment with any other questions you may have. I will close the ticket since the main issue has been addressed.
Thank you again for choosing to use OrthoSNAP and PhyKIT.
best,
Jacob


Upgrade Smart Answers to Ruby 2.2.4

This PR bumps the version for Ruby from 2.2.3 to 2.2.4, which fixes CVE-2015-7551 of Ruby's Fiddle and DL modules, neither which Smart Answer uses directly.
Stage
https://warm-shelf-8029.herokuapp.com/am-i-getting-minimum-wage

Looks like CI isn't setup with 2.2.4 yet

It looks as if Ruby 2.2.4 was added to ci-puppet in alphagov/ci-puppet#375 which has been merged. I'm going to try kicking the CI build for this PR and see whether it now passes.

The CI build passed, so I'm now going to rebase this against master and force-push it ready for merging.

The CI build has now passed, so I assume that Ruby 2.2.4 is now available on production.
I've reviewed the changes and they look good to me. The branch is up-to-date with master and ready for merging.

Ok, let's give this a try


Consolidate bosh-packages and bosh-io into cloudfoundry github org

This PR is the first step in consolidating the bosh-packages and bosh-io github organizations into the main cloudfoundry org.
After this PR gets approved and merged I would like to propose that repo creation is temporary enable for the cloudfoundry organization so that foundational infrastructure approvers who are also admin in the to be moved repo's can do so at their own pace (there will be some ci fixes that need to be done while moving).
The to-be-moved repos will also be renamed to clarify their scope (for example bosh-io/web will become cloudfoundry/bosh-io-web).

Adding @cloudfoundry/toc as reviewers for the ask to temporary enable repo creation for the cloudfoundry org to satisfy the below requirement:
To transfer a repository that you own to an organization, you must have permission to create a repository in the target organization.

source: https://docs.github.com/en/repositories/creating-and-managing-repositories/transferring-a-repository

How long is it reasonable for us to leave org creation temporarily enabled so that the FI approvers can move these repos in coordination with their CI fixes? 2 weeks?

This PR should be merged after all repo's have been moved. Will convert to draft so it does not show up in the github project board.

The bosh-package-cf-cli-release repo should probably live under the app-runtime-interfaces working group. Not sure if they also want the bosh-package-java-release repo too. I know foundation infrastructure doesn't run any pipelines for that, but I'm not sure if anybody does or not.

@Gerg, @a-b what is your opinion on moving the bosh-package-cf-cli-release to the ARI WG as @jpalermo commented in the previous comment?


Fix #134: Ensure Discord is destroyed before IoThread

Fix #134: IoThreadHolder edge case
If the static fields are destroyed in the wrong order then a deadlock can occur when calling Discord_Shutdown. This aligns the creation and deletion of IoThread with Discord. Ideally IoThread should be a member of connection but that would require moving the definition of it to one of the headers.
This fixes the shutdown issue we were getting in OpenRCT2.
https://github.com/OpenRCT2/OpenRCT2/pull/7431

To provide a bit more information: the deadlock mentioned here is one of the ways the issue can exhibit itself and it will very much depend on used toolchain and system.
The real issue we're dealing with here is the static initialisation order fiasco, whereby user of discord-rpc library cannot guarantee Discord_Shutdown gets called only before the IoThreadHolder::~IoThreadHolder.

I believe I also identified a race condition on Discord_Shutdown causing std::thread::join to be called twice resulting in:
libc++abi.dylib: terminating with uncaught exception of type std::__1::system_error: thread::join failed: Invalid argument

This could be resolved by only allowing the thread to be stopped once.

Thank you very much for this fix; incredibly helpful. Tagging @snowyote to review!

Sorry for the delay—happy to merge this to unblock folks.


【FR】Integration with Hover Editor plugin

Currently if we want to edit the notes displayed in the hierarchical-backlinks plugin, we need to click on link of the note to open it for editing.
Yet，the orginal backlinks panel is supported by the Hover Editor plugin, which can pop up an editing window for the corresponding note at the location of the note link, so
that we can quickly edit the notes in the backlinks.
FYI， the following image is the effect of displaying the note editing window with the Hover Editor plugin.

If it is possible, please consider making the hierarchical-backlinks plugin support the Hover Editor plugin as well! Thank you!

I'll look into this. Typically its the other plugin that needs to provide integration with Hierarchical Backlinks. I'll look into submitting a PR to Hover Editor to support Hierarchical Backlinks


Add support for user types in IOs

This PR adds support for using user defined types in module's IO list

Tested-by: Sean Anderson<EMAIL_ADDRESS>

Tested this PR with my workflow. Works great for my needs (typedefs and typedef structs) and found no bugs so far. I'm not an experienced developer though, so take this with a grain of salt.

@zachjs Could you take a look at this PR please?

I squashed the commit history and rebased on top of current master.

I squashed the commit history and rebased on top of current master.

@zachjs can you please check ?
This Verilog file now fail loading https://github.com/YosysHQ/yosys-tests/blob/master/frontends/read_verilog/top_dpi.v
"wire logic" is valid construct but it now fails parsing.

@zachjs can you please check ?
This Verilog file now fail loading https://github.com/YosysHQ/yosys-tests/blob/master/frontends/read_verilog/top_dpi.v
"wire logic" is valid construct but it now fails parsing.

@mmicko I've opened a PR with a fix for that issue.
While I've begun running make ystests locally for my larger changes, it seems this feedback is missing in the public CI pipeline. Would we consider adding this to avoid missing such issues in the future?

@mmicko I've opened a PR with a fix for that issue.
While I've begun running make ystests locally for my larger changes, it seems this feedback is missing in the public CI pipeline. Would we consider adding this to avoid missing such issues in the future?

Thanks @zachjs, it takes a while to run all those tests so for now it is not possible to run them for each commit.

Thanks @zachjs, it takes a while to run all those tests so for now it is not possible to run them for each commit.


Build Python 3.9 Wheels

When trying to install into python3.9 it will not install a version later than 0.2.2. I am not certain but I believe that this is because wheels are only build for versions 3.6, 3.7 and 3.8. Would it be possible to add wheels for the 3.9 version?

Hi, sorry about that! That is indeed the issue.
I'll build for Python 3.9 and release.

Unfortunately onnxruntime does not yet support Python 3.9: https://github.com/microsoft/onnxruntime/issues/6125 and https://github.com/microsoft/onnxruntime/issues/5697.
They say:

The next release will be in late Feb. or early March.

So until then NNSplit can unfortunately not support Python 3.9. If you really want to use NNSplit with 3.9 I just manually built a wheel for Linux:
Uploading nnsplit-0.5.2_post0-cp39-cp39-manylinux2014_x86_64.zip…
(rename it from .zip to .whl, GH issues doesn't let me upload wheels)
You'll have to also install the dev wheels for onnxruntime in that case: https://test.pypi.org/project/ort-nightly/#files
Also, to avoid confusion like in the original issue, I yanked version 0.2.2 and downwards of NNSplit since that was an old Python-only version that is not comparable to the current package.

onnxruntime now has Python 3.9 wheels! So it was easy to fix now. As of Release 0.5.7, NNSplit supports Python 3.9.


"The 'cache' field on 'RequestInitializerDict' is not implemented" in live cloudflare worker

Getting The 'cache' field on 'RequestInitializerDict' is not implemented. when trying to run this code in a worker (live). It does work fine in locally though. Whats wrong? 🤔
import { Redis } from '@upstash/redis/cloudflare'

const cfModule = {
  async fetch(request: Request, env: Env) {
    const asd = await Redis.fromEnv(env).lrange('asd', 0, 100)
    return new Response(JSON.stringify(asd), {
      headers: { 'Content-Type': 'application/json' },
    })
  },
}

Thanks, I'll fix it. In the meantime, can you try  an older version, 1.19.1 should work

1.19.1 works! Thanks! 🙌

I can confirm that this bug exists only on 1.20.2 . After reverting dependabot PR to 1.20.1 it works as expected.

I believe this commit introduced the "bug".
However, I believe its only relevant on cloudflare since they do not support the cache field in fetch but other platforms do.
We might track this issue and see if they add support for the cache field....

Sorry about that, our ci broke and didn't catch this before releasing
Fixed it in v1.20.3


fix fix stackoverflow bug in routine load

Proposed changes
When executing routine load job, there may encounter StackOverflowException.
This is because the expr in column setting list will be analyze for each routine load sub task,
and there is a self-reference bug that may cause endless loop when analyzing expr.
The following columns expr list may trigger this bug:
columns(col1, col2,
col2=null_or_empty(col2),
col1=null_or_empty(col2))

This fix is verified by user, but I can't add regression test for this case, because I can't submit a routine load job
in our regression test, and this bug can only be triggered in routine load.
Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall


[SQL] Adding AAD-only Support for SQL Managed Instances and Servers

Adding support for a new feature to az cli. Azure Active Directory only allows for only AD users to be admins not traditional username/password combinations. This feature has been added to Azure Python SDK and is now being supported in az cli
Using either a managed instance or server with a ad admin run:
az sql server ad-only-auth enable
az sql server ad-only-auth disable
az sql server ad-only-auth get
az sql mi ad-only-auth enable
az sql mi ad-only-auth disable
az sql mi ad-only-auth get
to enable, disable and show AAD only authentication.

Looks good.

SQL

Sure. I'll write them up.

Tests were added

Style updates

@Juliehzl I don't think that error is caused by me. according to the logs the error comes from loading ai-examples which I haven't touched. Please give guidance on how to proceed.

@Juliehzl I don't think that error is caused by me. according to the logs the error comes from loading ai-examples extension which I haven't touched. Please give guidance on how to proceed.

Yes. We could fix the CI error.

Changes were made @Juliehzl


Automatic Workload Creation

is there a way for kueue to make workloads for kubeflow jobs? I know for batch job kueue automatically creates the workload but how to do it with other jobs is not clarified in the documentation. just adding the label to the manifest doesn't seem to be creating the workload automatically then creating the job.

Hi we have a couple of guides relating  to kubeflow jobs here, please make sure you follow the administer cluster quotas guide before creating the kubefflow jobs.

I have done all the steps of the guide several times, but the problem is that for any job that isn't a Batch/v1 job the workload doesn't get created.
I looked into the controller_manager_config.yaml as well and the name is already added ( I want to deploy kubeflow and xgboost). Is there a step besides the guide that I have to do?
because it works perfectly well with Batch jobs just not any other job.
any help would be very much appreciated

Can you check the logs of the kueue's controller manager?

I found the issue to be a misconfiguration on my part in the helm installation using crossplane.
I managed to fix it.  Thanks @trasc for responding and trying to help me out!

@m-mamdouhi would you mind sharing what was the misconfiguration?

it was one (or both) of these two things not sure which:

the naming of the cluster to be created by crossplane was too long so the workloads weren't being created.
I had not correctly added the name of a custom workload configuration to the external integration framework


Block/Disable upstream kernel module builds

For the new module version (2.5.0 or 3.0) kernel module driver builds should be disabled/blocked via the block list.

https://github.com/stackrox/collector/pull/1117


Ensemble doesn't need to be made in OAL on instantiation

We don't need to actually make an ensemble in the online active learning until we're certain we're going to need. For example, if the first point is the same as the training data, we should just use the data and make/update the ensemble as points are added.
https://github.com/ulissigroup/al_mlp/blob/buildfix/al_mlp/online_learner.py#L79

Addressed in #9


Error when build the pip package for Tensorflow 1.8

Have I written custom code (as opposed to using a stock example script
provided in TensorFlow):No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):Linux Ubuntu 18.04
TensorFlow installed from (source or binary):Source
TensorFlow version (use command below):1.8
Python version:2.7.17/3.6.9
Bazel version (if compiling from source):0.10.0
GCC/Compiler version (if compiling from source):Tried 6.5, 5.5
CUDA/cuDNN version: 9.0/7.6.5
GPU model and memory:GTX 1080

ERROR: /home/yuyangleng/tensorflow/tensorflow/contrib/framework/BUILD:101:1: output 'tensorflow/contrib/framework/_objs/python/ops/_variable_ops_gpu/tensorflow/contrib/framework/kernels/zero_initializer_op_gpu.cu.pic.o' was not created
ERROR: /home/yuyangleng/tensorflow/tensorflow/contrib/framework/BUILD:101:1: not all outputs were created or valid
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 79.510s, Critical Path: 9.19s
FAILED: Build did NOT complete successfully

@YuyangLeng,
TensorFlow 1.x is not actively supported. Could you please try building TensorFlow 2.x and check if you are facing the same issue.
Also, make sure you have all the required dependencies installed as per the tested build configurations. Thanks!


istio: fix debounce metrics

Please provide a description of this PR:
The debounce metric should be emitted when we set free=false, and reset debouncedEvents =0, because that's when the next startDebounce starts counting.

@yingzhuivy: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

lint_istio
d95594630106c9a6b0f49acfbd42fec45c464dc6
link
true
/test lint

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


Falls the cosmos client into the "Improper Instantiation antipattern"?

Just stumbled across this helpful article Performance antipatterns for cloud applications.
There is a section called Improper Instantiation which contains (and least for me) well known problems with HttpClient (by the way, here the documentation is wrong, cause it should be solved by using the IHttpClientFactory and not a static instance of HttpClient) and some other common classes.
But to my surprise there is also the Microsoft.Azure.Documents.Client.DocumentClient listed that is also known as V2. Due to the fact, that I already upgraded to the CosmosClient (aka V3), I would like to know, if this client has the same problems and should be managed through some kind of factory that holds a pool of clients for re-usage or if these problems are solved internally and within a asp.net core rest api controller we can safely create a new CosmosClient in an transient lifetime for each request?
Unfortunately I couldn't find any documentation about these things nor any extension method for IServiceCollection that could maybe answer this question. So, is it okay to create and dispose a CosmosClient for each request or should we manage some client pool or even a single lifetime instance, that probably gets multiple calls in parallel?

No, you should use a singleton pattern. The SDK won't scale with creating a new instance for each request. The SDK has caches that it warms up on the first requests which costs more overhead. It's listed in the performance documentation.


notify the background worker if the latest to-expire key is dropped

ahh, my bad. The condition I tried to target is already handled a couple of lines above.


A Typo in the documentation

Dear,
I think there is a mistake in the documentation which is:
// Revoking the 'general.read' permission to the current user
$this->simple_auth->grantPermission('general.read');
I think you meant:
// Revoking the 'general.read' permission to the current user
$this->simple_auth->revokePermission('general.read');
Please confirm.

Thanks for reporting that typo. It will be fixed in the next update :+1:


PR: updated UI

see issue #37 for list of updates

once again, i forgot to update the code the in bin/ so closing this PR


Soundcloud downloading 6 times JSON metadata for each tracks

Checklist

[X] I'm reporting a broken site
[X] I've verified that I'm running yt-dlp version 2<IP_ADDRESS>. (update instructions)
[X] I've checked that all provided URLs are alive and playable in a browser
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue
[ ] I've read about sharing account credentials and I'm willing to share it if required

Region
USA
Description
For each tracks, it does download the metadata 6 times, which slow down significantly while using "--sleep-requests"
Seems that is does many unnecessary requests, which quickly trigger their limit rate.
Verbose log
[debug] Command-line config: ['-o', '%(title)s.%(ext)s', '--add-header', 'OAuth', '--sleep-requests', '2', 'https://soundcloud.com/hippiesabotage/sets/floating-palace', '-Uv']
'path'
CMD.EXE was started with the above path as the current directory.
UNC paths are not supported.  Defaulting to Windows directory.
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, err utf-8, pref cp1252
[debug] yt-dlp version 2<IP_ADDRESS>
[debug] Python version 3.9.6 (CPython 64bit) - Windows-10-10.0.19043-SP0
[debug] exe versions: ffmpeg 2021-09-08-git-5e7e2e5031-full_build-www.gyan.dev (setts), ffprobe 2021-09-08-git-5e7e2e5031-full_build-www.gyan.dev
[debug] Optional libraries: Cryptodome, mutagen, sqlite, websockets
[debug] Proxy map: {}
yt-dlp is up to date (2<IP_ADDRESS>)
[debug] [soundcloud:set] Extracting URL: https://soundcloud.com/hippiesabotage/sets/floating-palace
[soundcloud:set] hippiesabotage/sets/floating-palace: Downloading JSON metadata
[download] Downloading playlist: FLOATING PALACE
[soundcloud:set] playlist FLOATING PALACE: Collected 14 videos; downloading 14 of them
[download] Downloading video 1 of 14
[debug] [soundcloud] Extracting URL: https://soundcloud.com/hippiesabotage/01-crypticcaptions-4424-ref-15oct2021
[soundcloud] Sleeping 2.0 seconds ...
[soundcloud] hippiesabotage/01-crypticcaptions-4424-ref-15oct2021: Downloading info JSON
[soundcloud] Sleeping 2.0 seconds ...
[soundcloud]<PHONE_NUMBER>: Downloading JSON metadata
[soundcloud] Sleeping 2.0 seconds ...
[soundcloud]<PHONE_NUMBER>: Downloading JSON metadata
[soundcloud] Sleeping 2.0 seconds ...
[soundcloud]<PHONE_NUMBER>: Downloading JSON metadata
[soundcloud] Sleeping 2.0 seconds ...
[soundcloud]<PHONE_NUMBER>: Downloading JSON metadata
[soundcloud] Sleeping 2.0 seconds ...
[soundcloud]<PHONE_NUMBER>: Downloading JSON metadata
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info]<PHONE_NUMBER>: Downloading 1 format(s): http_aac_256

For each tracks, it does download the metadata 6 time

That's just the default message.
If you are getting rate limited, then try increasing the time between requests.
def _download_json_handle(
            self, url_or_request, video_id, note='Downloading JSON metadata',
            errnote='Unable to download JSON metadata', transform_source=None,
            fatal=True, encoding=None, data=None, headers={}, query={},
            expected_status=None)

Also how is this a broken site issue? From your logs it seems like it's downloading just fine.

If you are getting rate limited, then try increasing the time between requests.

Was implying that I use a rate limit because of that

That's just the default message.

So the default message is supposed to show up 6 times per tracks and actually way 6 times x amount of seconds ?
I don't get this with other websites

Not every site works the same way.
Make a PR if u think you can drop some requests.

Not every site works the same way.

Well, check other users logs in other issues and you will see that they do not have this issue.
I am reporting a bug here, if your answer is DIY, then I don't see where this is constructive behavior

I tested this and i see your issue now.
But i can't reproduce this. Did u strip out information from the Command-line config: line?
Here's what i get
C:\Users\Ashish\Desktop> yt-dlp -v https://soundcloud.com/hippiesabotage/sets/floating-palace
[debug] Command-line config: ['-v', 'https://soundcloud.com/hippiesabotage/sets/floating-palace']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, err utf-8, pref cp1252
[debug] yt-dlp version 2<IP_ADDRESS>
[debug] Python version 3.8.8 (CPython 64bit) - Windows-10-10.0.19041-SP0
[debug] exe versions: ffmpeg 3.0.1 (needs_adtstoasc), ffprobe 3.0.1
[debug] Optional libraries: Cryptodome, keyring, mutagen, sqlite, websockets
[debug] Proxy map: {}
[debug] [soundcloud:set] Extracting URL: https://soundcloud.com/hippiesabotage/sets/floating-palace
[soundcloud:set] hippiesabotage/sets/floating-palace: Downloading JSON metadata
[download] Downloading playlist: FLOATING PALACE
[soundcloud:set] playlist FLOATING PALACE: Collected 14 videos; downloading 14 of them
[download] Downloading video 1 of 14
[debug] [soundcloud] Extracting URL: https://soundcloud.com/hippiesabotage/01-crypticcaptions-4424-ref-15oct2021
[soundcloud] hippiesabotage/01-crypticcaptions-4424-ref-15oct2021: Downloading info JSON
[soundcloud]<PHONE_NUMBER>: Downloading JSON metadata
[soundcloud]<PHONE_NUMBER>: Downloading JSON metadata
[soundcloud]<PHONE_NUMBER>: Downloading JSON metadata
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info]<PHONE_NUMBER>: Downloading 1 format(s): http_mp3_128
[debug] Invoking downloader on "https://cf-media.sndcdn.com/lVqAHpBvh7SB.128.mp3?Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiKjovL2NmLW1lZGlhLnNuZGNkbi5jb20vbFZxQUhwQnZoN1NCLjEyOC5tcDMqIiwiQ29uZGl0aW9uIjp7IkRhdGVMZXNzVGhhbiI6eyJBV1M6RXBvY2hUaW1lIjoxNjM3MDYzMTk0fX19XX0_&Signature=ET2OPdGpLURsWTZhD6WfoJbX5dJOSJP7ImYNO~5L0i7ZQjJQN3EXJK4Z0FZHWWACz-TTvKqAA717-LQVOUtQVLbcQOCa6aDTcEyTGHEfPYUzQmWkHfFAsAcmZOpYkHDF9rbbNEALsFyrhB1n2ALZjuufrMPzDgkcnu0DgogPb~fT2bYLdBfhIdhO5eXzwoPplpoviGAIXjJvZLdynCp0UEDzOuFheMrcyaH15ptvtNCiu2cqXT7ZFFuDFpxJFcCJZkz6uK9rIc7D6DrMcB1gJSDFrqPd7s-8lNHzP6ZaVLDvas74tM9OzaEYK7onHzvMx2MutvLYEjpRpfCBirTFXw__&Key-Pair-Id=APKAI6TU7MMXM5DG6EPQ"
[download] 1. CRYPTIC CAPTIONS [1156240180].mp3 has already been downloaded
[download] 100% of 4.50MiB[download] Downloading video 2 of 14
[debug] [soundcloud] Extracting URL: https://soundcloud.com/hippiesabotage/02-safeharbor-4424-ref-15oct2021
[soundcloud] hippiesabotage/02-safeharbor-4424-ref-15oct2021: Downloading info JSON

ERROR: Interrupted by user

Notice how [soundcloud]<PHONE_NUMBER>: Downloading JSON metadata is only thrice for me

The json is downloaded once for the video metadata and once for each format. Since you passed oauth, u have more formats available than most. Hence why you get the message more times. Other than chaging the message to be more clear, there is nothing to be done

The json is downloaded once for the video metadata and once for each format. Since you passed oauth, u have more formats available than most. Hence why you get the message more times. Other than chaging the message to be more clear, there is nothing to be done

Thank you for the clarification, makes more sense
Is it possible to reduce that if best or specific format is specified ?

But i can't reproduce this. Did u strip out information from the Command-line config: line?

I did not. But I think the previous message would explain why there is a difference between your result and mine.

likely not. We need to get the data of all formats to find which is the best one in the first place. Perhaps an extractor arg can be added to limit the formats extracted, but (unlike sites like funimation where we know the language/version ahead of time), we know very little about the format before actually downloading the JSONs. So not sure how useful that will be

One thing I noticed in the different audio format is that the original soundtrack (uploaded by the author) sits on top of the format list.
Would that mean that it would be possible to grab the first one if looking for best quality ?

It should be at the bottom (best). If you can share account details, I can check

Sorry, but that is not something I am willing to do. Perhaps do you have an other solution so I can help you ?

You can do --write-pages and give me all the .dump files generated. But I am not sure if there is any sensitive account-related info in these files. Also consider opening a proper issue for this so that some developer who already has an account would be able to have a look

Sorry for the late reply.
Do you have suggestions for the title and content for making a new issue ?
In the meanwhile I will have a look into .dump file see if there are sensitive information in it
Kudos for maintaining the project by the way

Something like [SoundCloud] Wrong format order should do. Title is honestly not very important. Just make sure you provide the verbose log and attach the dump files (preferably from the same run as the logs)


azure event hubs "latestEventPosition" not working as intended.

@azure/event-hubs:
5.3.1:
Ubuntu 20.04:
[x] nodejs

14.13.1

[ ] browser

N/A

[ ] typescript

4.0.3

Is the bug related to documentation in

[ ] README.md
[ ] source code documentation
[* ] SDK API docs on https://docs.microsoft.com

Describe the bug
I wrote a mocha test to simple subscribe and publish on a single partition to an eventhub. consumer.subscribe documentation indicates we provide for the first argument an object with 2 functions keyed as "processEvents" and "processError". The second argument is an options object which can include startPosition. This allows for a further object, but can be simplified by the "latestEventPosition" and "earliestEventPosition" imports from '@azure/event-hubs'. when using earliestEventPosition and publishing i get what is expected - all events. when I use latestEventPosition i never get an event published after the method call. Reading the code i found i could provide startPosition with an options object and when providing {enqueuedOn:new Date()} events published after the method call trigger processEvents.
To Reproduce
Steps to reproduce the behavior:

See bug description.

Expected behavior
I expect that when I provide "latestEventPosition"to the startPosition option parameter that my subscription will only process events made after the method call.
Screenshots
N/A
Additional context
This may be a server time sync issue and not a code issue. Regardless I'd have thought that event time is measured globally - probably with a epoch timestamp created from GMT/UTC.

@verikono
Thanks for logging this issue. Can you share the code for your test?
Are you calling eventHubProducerClient.sendBatch() right after calling eventHubConsumerClient.subscribe()?
Under the hood, subscribe is non-blocking while it starts a loop to receive messages, so if you call sendBatch directly after calling subscribe, you'll have sent the events before subscribe has had a chance to begin receiving messages. This would be why when you set the startPosition using the enqueuedTime, you get the expected results.
For the purposes of writing a test, there are a couple things you could do.
Wait for processEvents to be invoked before sending the event
You can wait for processEvents to be invoked with an empty list of events before sending from your producer. This would ensure that the receiver is actively receiving events. To speed up your test, you can also change the maxWaitTimeInSeconds to a lower value. This won't be affected by clockSkew like enqueuedTime potentially could.
We have a test that essentially does this here.
Set starting position to the last sequence number in each partition
This is another option that's more resilient than using enqueuedTime since it takes clockSkew out of the equation.
We have an example of how to get the starting position across all partitions in our tests here as well.

Thanks for you hasty response @chradek!
I tried a 10 second delay   between eventHubConsumerClient.subscribe() and eventHubProducerClient.sendBatch() (using setTimeout) to allow any transactions to finalise.
I'll put my test battery in a repo at some point today.
Thank you so much for your assistance :)

I apologise for the delayed response to this. Work pulling me left and right at this time.
I've put up a repo for this issue here:
https://github.com/verikono/eventhubs-test
It's a simple mocha test that you can run which shows either my misunderstanding of something or the issue i'm trying to highlight here.
You'll need to create a .env file based off the .env-dev file in the repo root. I left the .vscode configuration in the repo in case it's of use to you.
lines 137 and 138 are of interest. The test will pass if 138 is used but fail if 137 is used. Line 121 is also of interest as typescript believes consumer.subscribes() argument signature is wrong.
Thank you so much for your help here :+1:

@verikono
I'm not seeing a delay between when you call subscribe and when you call sendBatch (createBatch will cause a slight delay but not a very long one.)
So, I believe you're still seeing the issue where you send an event before subscribe has had a chance to begin reading events. You could try putting a timed delay in, or wait for processEvents to be invoked once before moving on to your next it() in your tests.
To do the latter, you could do something like this where you pass done into your it() and then invoke it when you're ready to move on:
it(`Instantiate the consumer subscription for latestEventPosition`, (done) => {
  let isDoneCalled = false;
  subscription = consumer.subscribe(
    {
      processEvents: async ( events, context ) => {
        if(!events.length && !isDoneCalled) {
          isDoneCalled = true;
          // Now the next `it()` can be ran.
          done();
          return;
        }
        if(events.length > 1) throw new Error(`Expected exactly 1 event, but got ${events.length}`);
          const event = events[0]
          context.updateCheckpoint(event);
          received_events.push(event);
      },
      processError: err => {
        errors.push(err);
      }
    },
    {
      startPosition: latestEventPosition,
      maxWaitTimeInSeconds: 1 // Lower the max wait time so we don't wait a minute before ending this test.
    }
  );
});

With regards to the incorrect signature on line 121, this is because all of the process* functions are meant to be async (return a Promise.) They'll technically still work if you don't make them async since you can await a function that doesn't return a promise, but since your processEvents function doesn't return a promise that's why TypeScript is complaining.


Error: Transaction query alread complete, run with DEBUG=knex:tx for more info

Describe the bug
Have you every received an error like this and can you tell me where to look / enable DEBUG=knex.ts?
The error comes when executing multiple times (quickly) the query:
return this.em.findOne(Availability, {
      user: user,
      isAdHocAvailability: true,
      endTime: null,
      startTime: { $lte: new Date() },
    });

The error does not pop up all the time, however, when testing with many devices (Android Emulator, iOS Simulator, actual iPhone Device, Websive) all accessing the API (NestJS with MikroORM and Apollo GraphQL), this pops up.
It could be a problem with the Identity Map still, where I haven't found a way to solve this properly with GraphQL Execution Context with NestJS, MikroORM and Postgres (a very popular stack for backend).
Stack trace
Other error, {"message":"Transaction query already complete, run with DEBUG=knex:tx for more info","locations":[{"line":7,"column":5}],"path":["findOrCreateAdHocAvailability","lastLocation"],"extensions":{"code":"INTERNAL_SERVER_ERROR","exception":{"name":"DriverException","stacktrace":["DriverException: Transaction query already complete, run with DEBUG=knex:tx for more info","    at PostgreSqlExceptionConverter.convertException (backend/node_modules/@mikro-orm/core/platforms/ExceptionConverter.js:8:16)","    at PostgreSqlExceptionConverter.convertException (/backend/node_modules/@mikro-orm/postgresql/PostgreSqlExceptionConverter.js:40:22)","    at PostgreSqlDriver.convertException (/backend/node_modules/@mikro-orm/core/drivers/DatabaseDriver.js:194:54)","    at /backend/node_modules/@mikro-orm/core/drivers/DatabaseDriver.js:198:24","    at processTicksAndRejections (internal/process/task_queues.js:93:5)","    at async PostgreSqlDriver.find (/backend/node_modules/@mikro-orm/knex/AbstractSqlDriver.js:46:24)","    at async PostgreSqlDriver.findOne (/backend/node_modules/@mikro-orm/knex/AbstractSqlDriver.js:60:21)","    at async SqlEntityManager.findOne (/backend/node_modules/@mikro-orm/core/EntityManager.js:237:22)","    at async AvailabilityService.getLastLocationForAvailability (/backend/dist/modules/availability/availability.service.js:79:38)","    at async target (/backend/node_modules/@nestjs/core/helpers/external-context-creator.js:76:28)","","previous Error: Transaction query already complete, run with DEBUG=knex:tx for more info","    at completedError (/backend/node_modules/knex/lib/transaction.js:358:9)","    at /backend/node_modules/knex/lib/transaction.js:324:24","    at new Promise (<anonymous>)","    at Client_PG.trxClient.query (/backend/node_modules/knex/lib/transaction.js:320:12)","    at Runner.query (/backend/node_modules/knex/lib/runner.js:151:36)","    at /backend/node_modules/knex/lib/runner.js:40:23","    at /backend/node_modules/knex/lib/runner.js:277:24","    at processTicksAndRejections (internal/process/task_queues.js:93:5)","    at async PostgreSqlConnection.executeQuery (/backend/node_modules/@mikro-orm/core/connections/Connection.js:56:25)","    at async PostgreSqlConnection.execute (/backend/node_modules/@mikro-orm/knex/AbstractSqlConnection.js:105:21)"]}}}
at src/globals/index.ts:171:14 in onError$argument_0
at node_modules/@apollo/client/link/error/error.cjs.js:11:40 in utilities.Observable$argument_0
at node_modules/zen-observable/lib/Observable.js:135:15 in notifySubscription
at node_modules/zen-observable/lib/Observable.js:179:20 in onNotify
at node_modules/zen-observable/lib/Observable.js:235:14 in next
at node_modules/@apollo/client/link/retry/retry.cjs.js:41:25 in RetryableOperation
at node_modules/zen-observable/lib/Observable.js:135:15 in notifySubscription
at node_modules/zen-observable/lib/Observable.js:179:20 in onNotify
at node_modules/zen-observable/lib/Observable.js:235:14 in next
at [native code]:null in next
at [native code]:null in forEach
at http://<IP_ADDRESS>:19001/node_modules/expo/AppEntry.bundle?platform=android&dev=true&hot=false&minify=false:237346:34 in <unknown>
at [native code]:null in forEach
at node_modules/@apollo/client/link/batch/batch.cjs.js:86:31 in batchedObservable.subscribe$argument_0.next
at node_modules/zen-observable/lib/Observable.js:135:15 in notifySubscription
at node_modules/zen-observable/lib/Observable.js:179:20 in onNotify
at node_modules/zen-observable/lib/Observable.js:235:14 in next
at node_modules/@apollo/client/link/batch-http/batch-http.cjs.js:73:23 in batchHandler
at [native code]:null in flushedQueue
at [native code]:null in invokeCallbackAndReturnFlushedQueue

To Reproduce
Steps to reproduce the behavior:

...
...
...

Expected behavior
A clear and concise description of what you expected to happen.
Additional context
Add any other context about the problem here.
Versions

Dependency
Version

node
?

typescript
?

mikro-orm
4.5.5

postgres

Have you every received an error like this and can you tell me where to look / enable DEBUG=knex.ts?

Just add DEBUG=knex:tx environment variable, e.g. you can run the script with DEBUG=knex:tx prefix:
DEBUG=knex:tx yarn start

Similar issue was reported recently when doing this from inside flush events, but this does not seem to be related. The issue is here: #1824

The error comes when executing multiple times (quickly) the query:

You should not use single EM instance in parallel. Also seeing findOrCreateAdHocAvailability in your stack trace, that is even forbidden, not just suggested, you can't flush in parallel. Also you definitely need to use explicit transaction for that, otherwise the select query won't use the same transaction. Something like this:
const availability = await em.transactional(async inner => {
  let a = await inner.findOne(Availability, { ... });

  if (!a) {
    a = new Availability(...);
    await inner.persistAndFlush(a);
  }

  return a;
});

The transactional callback gets a fork in its parameter (here called inner). It will contain all the entities from parent EM too, but ideally it should be clear/empty.
But seeing this is triggered by a find call (which won't create implicit transaction on its own), I guess you are probably already doing something like that?

It could be a problem with the Identity Map still, where I haven't found a way to solve this properly with GraphQL Execution Context with NestJS, MikroORM and Postgres (a very popular stack for backend).

Have you seen this?
https://mikro-orm.io/docs/usage-with-nestjs#request-scoping-when-using-graphql

Thanks for the quick reply.
This look quite complicated. Why do I have to use em.transactional... for every query? Can't this be handled by MikroORM if (as I understand) this would be required for every query and do I have to change every query in my application to match the above example? Or did I misunderstand?
Looking at other frameworks, one can simply do Entity.findFirst({column: "value"} and done, running perfectly under thousands of concurrent requests.
If I use EntityRepository, would that be simpler?
I believe if you would provide a NestJS example with some typical queries that would run under high concurrency it would help a lot. The app I'm working on needs to go into production soon and has to run without errors or concurrency problems.
Thank you so much for an elaboration.

This look quite complicated. Why do I have to use em.transactional... for every query? Can't this be handled by MikroORM if (as I understand) this would be required for every query and do I have to change every query in my application to match the above example? Or did I misunderstand?

I am talking only about the find-or-create kind of action (= upserts). You need the explicit tx because you need the select query to be executed inside it, otherwise there is no guarantee at the database level.

If I use EntityRepository, would that be simpler?

Nope, that's just a thin layer on top of EM, nothing is different with repositories.

I am talking only about the find-or-create kind of action (= upserts). You need the explicit tx because you need the select query to be executed inside it, otherwise there is no guarantee at the database level.
Thanks, understood now, that's actually an important thing :)

I am still receiving this error:
[Nest] 23794   - 31/05/2021, 19:11:36   [MikroORM] [query] insert into "availability" ("created_at", "end_time", "id", "is_ad_hoc_availability", "start_time", "updated_at", "user_id") values ('2021-05-31T17:11:36.263Z', NULL, 'b2210507-f813-4bad-aeac-6ec1f7fca68a', true, '2021-05-31T17:11:36.263Z', '2021-05-31T17:11:36.263Z', 'c59620ed-37a5-452f-9b3d-d2ff979322ec') returning "id"

(node:23794) UnhandledPromiseRejectionWarning: DriverException: Transaction query already complete, run with DEBUG=knex:tx for more info

ValidationError: You cannot call em.flush() from inside lifecycle hook handlers

here is the code:

const availability = await this.em.transactional(async (inner) => {
      let availability: Availability | null = await this.em.findOne(
        Availability,
        {
          user: userContext.user,
          isAdHocAvailability: true,
          endTime: null,
          startTime: { $lte: new Date() },
        },
      );

      if (!availability) {
        availability = new Availability({
          user: userContext.user,
          isAdHocAvailability: true,
          startTime: new Date(),
        });
      } else {
        availability.updatedAt = new Date();
      }

      inner.persistAndFlush(availability);
      return availability;
    });

Any query after this can still be executed and will not cause problems with the above code?

Okay, I forgot to put await in front of inner.persistAndFlush(availability);. With await inner.persistAndFlush(availability); it seems to be working.

Ok good, closing then.


Added Logentries Driver

Reopening https://github.com/docker/docker/pull/22699 because I removed my fork and github lost track of it.
Also made the changes asked by @thaJeztah .
Docs were made on PR https://github.com/docker/docker.github.io/pull/244

bash completion is perfect :+1:

Thanks @albers @sdurrheimer !

Thanks all!

ping @caarlos0 can you take a look at bsphere/le_go#14 regarding licensing for le_go?
Thanks!

@cpuguy83 indeed there is no license... I'm not the owner of the lib though, maybe @bsphere can help us here :)

@cpuguy83 @caarlos0 just added a LICENSE file to that repo (MIT).
let me know if you have any issues with such license

❤  thanks!


Limits and best practices for network between kubelet and apiserver

Where is the documentation that have details on the limitation or requirement running control plane node and a worker node far apart like in separate cloud regions? Or public (control plane) and on-premises (worker nodes)
Not talking about the distance between etcd nodes this is cover in etcd tuning, I'm referring about distance/connectivity between kubelet and api server.
I tried searching but only thing close was this page and it doesn’t mention it https://kubernetes.io/docs/setup/best-practices/multiple-zones/

/retitle Document limits and best practices for network between kubelet and API server
/language en
/kind feature

/sig docs


Support multiple outputs and multiple out connections for a layer

Current implementation limits all the outputs of a layer to be sent to the output nodes. However, that is not always the desired case. Consider the example below:
In -> Layer A -> out1, out2
out1 -> Layer B
out2 -> Layer C

This kind of graph is not supported as of now and must be supported.
Further, there is inconsistency between the routing of the nodes output. In split layer, each outputs is passed to individual layer, but in multiout layer, all the inputs are passed to all the layers. This also needs to be resolved.

:octocat: cibot: Thank you for posting issue #1706. The person in charge will reply soon.

There are two solutions:

Routing Layer
Routing layer will take n inputs and route these n inputs to n layers. As routing layer will only perform routing of the tensors, both forwarding and backwarding will always be no-op and routing layer can always work in-place.
Routing layer will not support routing an input to multiple layers (this must be done with multiout layer). The inputs of the routing layer can be from 1 layer or multiple layers.

Update the graph
This requires updating the graph nodes and connections to allow multiple tensors to flow over a connection. This implies that numInputConnections <= numInputTensors and will require changes in the core of NetworkGraph and LayerNode.

Let's consider a case where
A outputs a1, a2
B outputs b1
C needs a1, b1
D needs a2

naive router idea

[layer A]
type=will_have_two_outputs

[layer B]
type=will_have_one_output

[router]
type=router
input_layers=layer A, layer B
routed_to=layer C, layer D, layer C

[layer C]
type=need_two_inputs
input_layers=router

[layer D]
type=need_one_inputs
input_layers=router

In this case, we need to propagate routed_to to layer node, because graph needs to query which implies that this has to be global property for everyone, which means it will be able to do like below.
Anyway, IMO, the takeaway here is that the connection information can't stay inside a layer, so that even if we make a router layer it won't be any time saver.

router idea with global property

So below idea should be valid, and more concise.
[layer A]
type=will_have_two_outputs
route_to=layer_c, layer_d

[layer B]
type=will_have_one_output
route_to=layer_c

[layer C]
type=need_two_inputs
input_layers=layer A, layer B

[layer D]
type=need_one_inputs
input_layers=layer A

I think this option is likely to make some mistake as there is a two pairs should match (route_to / input_layers)

routing from input_layers directly.

[layer A]
type=will_have_two_outputs

[layer B]
type=will_have_one_output

[layer C]
type=need_two_inputs
input_layers=layer A:0, layer B:0

[layer D]
type=need_one_inputs
input_layers=layer A:1

with this setup it will be more concise and more readable.

I think the keywords need to be changed later for the case 3. %s/input_layers/ inputs/
and I wonder if we need to set the type even. We can realize with the info from children's inputs.

As discussed offline, the user-interface can be multiple and the one described at  https://github.com/nnstreamer/nntrainer/issues/1706#issuecomment-969839564 looks good.
There needs to be fixed semantics about the number of tensors that can flow over each edge between nodes (layers) in the graph. For now, we have fixed it to be 1 tensor per edge in the graph.

workout plan -

Multiout before setting output
Add & parse index information


Refactoring

This is a huge refactoring coming in. Now from #49 I was able to follow the basics of ArteJS usage so it's being easier to refactor every API feature by TDD.
Status:

[x] Core (it does exist)
[x] Core Options
[x] Core methods
[x] Custom Events
[x] Commands for plain text editor
[ ] Commands for rich text editor (will probably require some range selection skills)
[ ] Toolbar
[ ] Toolbar buttons
[ ] Plugins
[ ] Documentation review

Some features were changed as a proposal:

[ ] Arte is now an exposed class, so you can create your arte object like this: var arte = new Arte( element, { options } );, it will return the created elemtent as arte.element and the given element container as arte.container;

This way allow us to use methods from a stored variable, like arte.bold();, arte.destroy(), and arte.triggerEvent("foo"), e.g.
All the commands should be called from this stored object, there're not longer Arte("bold").

[ ] the instantiated arte is no longer attached to the element's data, avoiding a big memory issue
[ ] events namespace were replaced from on to arte- to flag them as proper Arte events, some of them are still connected to the Arte editor common events, e.g.: click on the element triggers arte-click on the arte object.
[ ] rangy will probably be excluded from the external dependencies

I can't say yet how smaller or faster the code is, but it is being successful so far and it's already compatible with any jQuery version from 1.7.2 to 1.11.x.

I've got a few questions about removing the rangy dependency. Is this worth the trouble it may cause / what are the main reasons to avoid using rangy. I was wondering what your reasons were for wanting to remove this dependency (is it mainly a performance thing?) I know rangy is used all over the place to help with selections in Arte (there's a whole folder of rangy-extensions) and am not really sure what the implications of removing it and doing this all ourselves may be. I believe we do some fairly complex operations with selections and am a little concerned about whether or not its worth it to try and port all of it. - For example - https://github.com/vistaprint/ArteJS/blob/master/src/editor/lib/rangy-extensions/rangy-extensions.js

To add to @gshenar's point, it might be worthwhile to build a range manipulation abstraction layer that will call rangy functions, so that anytime a component has to do any range operations it goes through this layer and does not have to call rangy directly. That way if we do decide to remove rangy, we only have to remove it in one place and the range manipulation API stays intact.


NIFI-2540: Exclude from templates the parts of property descriptors t…

…hat are not necessary. Also ensure that Property Descriptors are not completely removed from Controller Services.

I'll be reviewing it, primarily due to the recent deterministic template work where some scrubbing cause issues.

Reviewing.

Looks pretty straight forward. Validated that it did not break the correctness of template import/export. Merging


Psql bump

This PR ads support for psql 15+
version 15 and 16 have been added to the testing matrix
I had to introduce a new parameter to postgresql_users
The changes should be backward compatible
The changes are NOT backward compatible
i cherry-picked the commit from #33 and fix warning introduced
i will fix the other warnings in a follow-up PR to simplify the review of the required changes to support psql 15+
cc @technics3 mlukasik-dev
see https://www.postgresql.org/docs/release/15.0/ for background

The changes are related to the public schema.
The failure without the suggested changes
TASK [create test tables publicdb] *********************************************
  failed: [postgresql-15-u2204] (item=create table if not exists regular (text text primary key);) => {"ansible_loop_var": "item", "changed": true, "cmd": ["env", "PGPASSWORD=alice123", "psql", "-h", "localhost", "-U", "alice", "publicdb", "-c", "create table if not exists regular (text text primary key);"], "delta": "0:00:00.058269", "end": "2024-02-15 15:48:06.778143", "item": "create table if not exists regular (text text primary key);", "msg": "non-zero return code", "rc": 1, "start": "2024-02-15 15:48:06.719874", "stderr": "ERROR:  permission denied for schema public\nLINE 1: create table if not exists regular (text text primary key);\n                                   ^", "stderr_lines": ["ERROR:  permission denied for schema public", "LINE 1: create table if not exists regular (text text primary key);", "                                   ^"], "stdout": "", "stdout_lines": []}

Basically
With Postgresql 15, the default permissions for a public schema have been modified. Therefore, the correct permissions will have to be granted to the alice for Postgresql 15.
Since the alice user is the one to create the database tables we need to have
GRANT ALL ON SCHEMA public TO alice; and not USAGE as it is currently in the role but at the same time we do not have to grant to other users

Setting restrict to true for publicdb creation leads to the same error ERROR:  permission denied for schema public

With Postgresql 15, the default permissions for a public schema have been modified. Therefore, the correct permissions will have to be granted to alice for Postgresql 15.

I think my concern is about the validity of the scenario tested by Molecule. In all the use cases I am aware of, tables should be created by the database owner e.g . alice/secretdb test. Is there a use case where we want a non-owner to be granted creation privileges on the public schema?

Setting restrict to true for publicdb creation leads to the same error ERROR:  permission denied for schema public

Exactly. So the way I see it, the second part of the restrict variable has no effect on PSQL 15+. I see two paths:

we update this role to maintain the current behavior. With the current proposal, I think we need additional explanation on  the privileges variable. Additionally, we should link to CVE-2018-1058 since this reverts a CVE mitigation path.
we update this role to comply with the mitigation CVE-2018-1058 and match the PSQL 15+ changesi.e. we restrict privileges on the public schema across the board except for non database owners, non-superusers

Discussed with @jburel earlier today, the new proposed behavior makes sense to me and is inline with the breaking privileges changes made in PSQL 15+ but it would be useful to have at least another review from the OME team
The latest changes update the default to restrict: True but the Molecule tests are no longer testing restrict: False use case. Given the behavior would be PSQL version dependent, would we consider dropping restrict fully and make all databases restricted? Or keep restrict: False for backwards-compatibility?
The README should be updated accordingly

Considering that it is a security breach fixed in newer version of psql, I think it will make more sense to restrict all databases and drop the restrict flag

@pwalczysko @khaledk2 What do you think?

@pwalczysko @khaledk2 What do you think?

What will this change do with an already deployed psql database if I run the playbook with the postgresql role including this change on such already deployed psql DB ?
It says in the header of this PR The changes are NOT backward compatible.

Test 1:
on omero-ci-upgrade testing server RHEL 9, with the postgres-14 installed and running by ansible with the postgresql role from this branch, run:

the playbook from https://github.com/openmicroscopy/management_tools/pull/1710 with the postgresql role installed from this branch and the postgres version set as 15. Result: the postgres-15 service does not start, cannot find anything in the journalctl or systemd status.
delete everything you can (/opt/omero, stop and disable the postgres and omero-server services, rerun the playbook as per Ad 1. above. Result: fails at the stage install omero

TASK [ome.omero_server : omero server | install omero] ********************************************
[WARNING]: Module remote_tmp /opt/omero/server/.ansible/tmp did not exist and was created with a
mode of 0700, this may cause issues when running as another user. To avoid this, create the
remote_tmp dir with the correct permissions manually
fatal: [<IP_ADDRESS>]: FAILED! => {"changed": true, "cmd": ["/opt/omero/server/venv3/bin/omego", "install", "--release", "5.6.10", "--sym", "OMERO.server", "--ice", "3.6", "--no-start", "--no-web", "--ignoreconfig", "--omerocli", "/opt/omero/server/venv3/bin/omero", "-qq", "--dbhost", "localhost", "--dbuser", "omero", "--dbname", "omero", "--dbpass", "omero", "--managedb", "--rootpass", "omero"], "delta": "0:00:26.107814", "end": "2024-02-20 13:55:41.233835", "msg": "non-zero return code", "rc": 1, "start": "2024-02-20 13:55:15.126021", "stderr": "OMERO.py version:\nOMERO.server version:\n5.6.10-ice36\n2024-02-20 13:55:37,583 [omego.extern] ERROR Failed [0.037 s]\n2024-02-20 13:55:37,584 [    omego.db] ERROR Non-zero return code\ncommand: psql -v ON_ERROR_STOP=on -d omero -h localhost -U omero -w -A -t -c SELECT currentversion, currentpatch FROM dbpatch ORDER BY id DESC LIMIT 1\nreturn code: 1\nstdout: b''\nstderr: b'ERROR:  relation \"dbpatch\" does not exist\\nLINE 1: SELECT currentversion, currentpatch FROM dbpatch ORDER BY id...\\n                                                 ^\\n'\n2024-02-20 13:55:41,195 [omego.extern] ERROR Failed [0.034 s]\nTraceback (most recent call last):\n  File \"/opt/omero/server/venv3/bin/omego\", line 33, in <module>\n    sys.exit(load_entry_point('omego==0.7.0', 'console_scripts', 'omego')())\n  File \"/opt/omero/server/venv3/lib64/python3.9/site-packages/omego/main.py\", line 48, in entry_point\n    main(\"omego\", items=[\n  File \"/opt/omero/server/venv3/lib64/python3.9/site-packages/yaclifw/framework.py\", line 188, in main\n    ns.func(ns)\n  File \"/opt/omero/server/venv3/lib64/python3.9/site-packages/omego/upgrade.py\", line 534, in __call__\n    UnixInstall(self.NAME, args)\n  File \"/opt/omero/server/venv3/lib64/python3.9/site-packages/omego/upgrade.py\", line 78, in __init__\n    self.handle_database()\n  File \"/opt/omero/server/venv3/lib64/python3.9/site-packages/omego/upgrade.py\", line 290, in handle_database\n    db.init()\n  File \"/opt/omero/server/venv3/lib64/python3.9/site-packages/omego/db.py\", line 121, in init\n    self.psql('-f', omerosql)\n  File \"/opt/omero/server/venv3/lib64/python3.9/site-packages/omego/db.py\", line 276, in psql\n    stdout, stderr = external.run('psql', args, capturestd=True, env=env)\n  File \"/opt/omero/server/venv3/lib64/python3.9/site-packages/omego/external.py\", line 82, in run\n    raise RunException(\nomego.external.RunException: Non-zero return code\ncommand: psql -v ON_ERROR_STOP=on -d omero -h localhost -U omero -w -A -t -f omero-20240220-135537-584658.sql\nreturn code: 3\nstdout: b'BEGIN\\n'\nstderr: b'psql:omero-20240220-135537-584658.sql:27: ERROR:  permission denied for schema public\\n'", "stderr_lines": ["OMERO.py version:", "OMERO.server version:", "5.6.10-ice36", "2024-02-20 13:55:37,583 [omego.extern] ERROR Failed [0.037 s]", "2024-02-20 13:55:37,584 [    omego.db] ERROR Non-zero return code", "command: psql -v ON_ERROR_STOP=on -d omero -h localhost -U omero -w -A -t -c SELECT currentversion, currentpatch FROM dbpatch ORDER BY id DESC LIMIT 1", "return code: 1", "stdout: b''", "stderr: b'ERROR:  relation \"dbpatch\" does not exist\\nLINE 1: SELECT currentversion, currentpatch FROM dbpatch ORDER BY id...\\n                                                 ^\\n'", "2024-02-20 13:55:41,195 [omego.extern] ERROR Failed [0.034 s]", "Traceback (most recent call last):", "  File \"/opt/omero/server/venv3/bin/omego\", line 33, in <module>", "    sys.exit(load_entry_point('omego==0.7.0', 'console_scripts', 'omego')())", "  File \"/opt/omero/server/venv3/lib64/python3.9/site-packages/omego/main.py\", line 48, in entry_point", "    main(\"omego\", items=[", "  File \"/opt/omero/server/venv3/lib64/python3.9/site-packages/yaclifw/framework.py\", line 188, in main", "    ns.func(ns)", "  File \"/opt/omero/server/venv3/lib64/python3.9/site-packages/omego/upgrade.py\", line 534, in __call__", "    UnixInstall(self.NAME, args)", "  File \"/opt/omero/server/venv3/lib64/python3.9/site-packages/omego/upgrade.py\", line 78, in __init__", "    self.handle_database()", "  File \"/opt/omero/server/venv3/lib64/python3.9/site-packages/omego/upgrade.py\", line 290, in handle_database", "    db.init()", "  File \"/opt/omero/server/venv3/lib64/python3.9/site-packages/omego/db.py\", line 121, in init", "    self.psql('-f', omerosql)", "  File \"/opt/omero/server/venv3/lib64/python3.9/site-packages/omego/db.py\", line 276, in psql", "    stdout, stderr = external.run('psql', args, capturestd=True, env=env)", "  File \"/opt/omero/server/venv3/lib64/python3.9/site-packages/omego/external.py\", line 82, in run", "    raise RunException(", "omego.external.RunException: Non-zero return code", "command: psql -v ON_ERROR_STOP=on -d omero -h localhost -U omero -w -A -t -f omero-20240220-135537-584658.sql", "return code: 3", "stdout: b'BEGIN\\n'", "stderr: b'psql:omero-20240220-135537-584658.sql:27: ERROR:  permission denied for schema public\\n'"], "stdout": "5.18.0", "stdout_lines": ["5.18.0"]

I think the main point from the above is that the following line from the output strongly suggests that this change will break the install of OMERO via ansible playbooks even on a clean system where the postgres-14,15 and OMERO.server is not installed. I would think that the OMERO.server role is trying to execute psql cmds during install where the schema used is public - see below from the output of the ansible cmd.
psql -v ON_ERROR_STOP=on -d omero -h localhost -U omero -w -A -t -f omero-20240220-135537-584658.sql\nreturn code: 3\nstdout: b'BEGIN\\n'\nstderr: b'psql:omero-20240220-135537-584658.sql:27: ERROR:  permission denied for schema public\\n'

I can see that the tests here are not trying to install OMERO server which would be using this role - it seems that the problem here is indeed https://github.com/ome/ansible-role-postgresql/pull/39#issuecomment-1954287302

The error means that the user omero does not have the correct permissions. I think that should be fixed at the level of the playbook and not the role

I do not see any permissions setting possibilities inside the playbook.
Also note that the DBs created by the playbook are owned by user postgres.
    - role: ome.postgresql
      postgresql_databases:
      - name: omero
      postgresql_users:
      - user: omero
        password: xxx
        databases: [omero]

    - role: ome.postgresql
      postgresql_databases:
      - name: omero
        owner: omero
      postgresql_users:
      - user: omero
        password: xxx
        databases: [omero]

i.e. makes omero the owner of the omero db

    - role: ome.postgresql
      postgresql_databases:
      - name: omero
        owner: omero
      postgresql_users:
      - user: omero
        password: xxx
        databases: [omero]

i.e. makes omero the owner of the omero db

But see below: the psql -l output is saying that the owner is still postgres.
bash-5.1$ psql -l
                                  List of databases
   Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
 omero     | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =Tc/postgres         +
           |          |          |             |             | postgres=CTc/postgres+
           |          |          |             |             | omero=c/postgres
 postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
           |          |          |             |             | postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
           |          |          |             |             | postgres=CTc/postgres
(4 rows)

I made an adjustment to the previous command

I made an adjustment to the previous command

This failed with
TASK [postgresql : postgres | grant connect privileges] *******************************************
fatal: [<IP_ADDRESS>]: FAILED! => {"msg": "could not find 'databases' key in iterated item '{'user': 'omero', 'password': 'omero'}'"}

@sbesson is here and suggesting a test where I have a postgres 14 installed and then I just bump to 15 (ignoring this PR). Doing just that to verify.

I can install postgres 15 and server successfully.
But, in case there was a pre-existing running server with postgres 14, and I do following steps (stop and remove the postgres 14 service, remove the 14 folder of /var/lib/pgsql/, uninstall postgres, the OMERO.server will not restart, see below for Blitz log error:
[root@ome-ci-upgrade pgsql]# systemctl stop omero-server
[root@ome-ci-upgrade pgsql]# sudo -u postgres -s
bash-5.1$
bash-5.1$
bash-5.1$ psql -l
                              	List of databases
   Name	|  Owner   | Encoding |   Collate   |	Ctype	|   Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
 omero 	| postgres | UTF8 	| en_US.UTF-8 | en_US.UTF-8 | =Tc/postgres     	+
       	|      	|      	|         	|         	| postgres=CTc/postgres+
       	|      	|      	|         	|         	| omero=c/postgres
 postgres  | postgres | UTF8 	| en_US.UTF-8 | en_US.UTF-8 |
 template0 | postgres | UTF8 	| en_US.UTF-8 | en_US.UTF-8 | =c/postgres      	+
       	|      	|      	|         	|         	| postgres=CTc/postgres
 template1 | postgres | UTF8 	| en_US.UTF-8 | en_US.UTF-8 | =c/postgres      	+
       	|      	|      	|         	|         	| postgres=CTc/postgres
(4 rows)

bash-5.1$ pg_dump -Fc -f ome-learning-20240221-test.db.dump omero
bash-5.1$ pwd
/var/lib/pgsql
bash-5.1$ ls
14  ome-learning-20240221-test.db.dump
bash-5.1$ ls -lah
total 960K
drwx------.  4 postgres postgres  153 Feb 21 09:05 .
drwxr-xr-x. 56 root 	root 	4.0K Dec  4 18:22 ..
drwx------.  4 postgres postgres   51 Feb 21 08:41 14
drwxr-xr-x.  3 postgres postgres   17 Oct 19 14:12 .ansible
-rw-------.  1 postgres postgres 8.1K Feb 20 17:01 .bash_history
-rwx------.  1 postgres postgres  266 Feb 21 08:40 .bash_profile
-rw-r--r--.  1 postgres postgres 929K Feb 21 09:05 ome-learning-20240221-test.db.dump
-rw-------.  1 postgres postgres 2.1K Feb 20 17:00 .psql_history
-rw-------.  1 postgres postgres  781 Jan  8 10:03 .viminfo
bash-5.1$ mv ome-learning-20240221-test.db.dump
mv: missing destination file operand after 'ome-learning-20240221-test.db.dump'
Try 'mv --help' for more information.
bash-5.1$ mv ome-learning-20240221-test.db.dump /tmp/
[root@ome-ci-upgrade pgsql]# systemctl stop postgresql-14.service
[root@ome-ci-upgrade pgsql]#
[root@ome-ci-upgrade pgsql]#
[root@ome-ci-upgrade pgsql]# systemctl disable postgresql-14.service
Removed "/etc/systemd/system/multi-user.target.wants/postgresql-14.service".

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'currentDetails' defined in class path resource [ome/services/sec-primitives.xml]: Cannot resolve reference to bean 'roles' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dbPatchCheck' defined in class path resource [ome/services/startup.xml]: Invocation of init method failed; nested exception is ome.conditions.InternalException:
***************************************************************************************
Error connecting to database table dbpatch. You may need to bootstrap.
See https://docs.openmicroscopy.org/latest/omero/sysadmins/server-upgrade.html
***************************************************************************************
    at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359)
    at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108)
    at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648)
    at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:145)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1198)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1100)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:511)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
    at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
    at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:105)
    at ome.system.OmeroContext.<init>(OmeroContext.java:97)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:142)
    ... 27 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dbPatchCheck' defined in class path resource [ome/services/startup.xml]: Invocation of init method failed; nested exception is ome.conditions.InternalException:
***************************************************************************************
Error connecting to database table dbpatch. You may need to bootstrap.
See https://docs.openmicroscopy.org/latest/omero/sysadmins/server-upgrade.html
***************************************************************************************
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1631)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:297)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:372)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1178)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1072)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:511)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
    at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
    ... 49 common frames omitted
Caused by: ome.conditions.InternalException:
...

I can deploy with this role on
postgres 15 and 16 as upgrade from postgres 14 on the RHEL 9 testing server.


Fix magic methods called directly on dataobject

Hi!
Been using this phpstan extension for a couple of days now, it's being very much appreciated!
Today I noticed a small problem, in that I was instantiating a DataObject class and called some setters on it. And phpstan was complaining about calling undefined methods on it, even though this extension was installed which was supposed to fix this.
The problem was, that this extension only checked on subclasses of DataObject and not on instances of DataObject itself.
This PR fixes this.
I also took the liberty to ignore the /vendor/ directory in the .gitignore file which was missing.
Awesome work with those captainhook scripts btw! First time I've been using them in practice even though I told myself to look into those after seeing your very nice talk on MM19NL 😉

Thanks for your contribution! Good catch :)

I usually have set-up a global .gitignore file, thus I tend to forget that other developers don't have that. Thanks for that minor improvement.

Thanks for merging so quickly!


[Summer of Code] Support switch role for broker

What is the purpose of the change
We want unified log replication, using rocketmq's original HaService instead of dledger mode.
Previously, I have done the following job with @RongtongJin

Add statemachine mode for dledger: https://github.com/openmessaging/dledger/pull/128
Embed a strongly consistent controller based on dledger in name-srv: https://github.com/apache/rocketmq/pull/4195
Add a new HaService -- AutoSwitchHAService, which use new log replicating protocol to support switch role in haService level.: https://github.com/apache/rocketmq/pull/4236

This pr is the last pr: connecting the interface of Dledger-controller at the Broker level, so that the Broker has the ability of master-slave switching
Noted that this Pr is still in development and not fully completed.
Brief changelog
The architecture is shown in the figure below. The pr is a link between the 'DledgerController' and 'AutoSwitchHASerivce' through a new component - ReplicasManager, it connects to the api of the controller and notifies the lower AutoSwitchHASerivce.

Verifying this change
Follow this checklist to help us incorporate your contribution quickly and easily. Notice, it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR.

[x] Make sure there is a Github issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
[x] Format the pull request title like [ISSUE #123] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
[x] Run mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle to make sure basic checks pass. Run mvn clean install -DskipITs to make sure unit-test pass. Run mvn clean test-compile failsafe:integration-test  to make sure integration-test pass.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Codecov Report

Merging #4272 (c41aefc) into 5.0.0-beta-dledger-controller (ce534ee) will decrease coverage by 0.28%.
The diff coverage is 23.34%.

@@                         Coverage Diff                         @@
##             5.0.0-beta-dledger-controller    #4272      +/-   ##
===================================================================
- Coverage                            43.73%   43.45%   -0.29%
- Complexity                            6272     6274       +2
===================================================================
  Files                                  827      829       +2
  Lines                                58610    58937     +327
  Branches                              8003     8046      +43
===================================================================
- Hits                                 25634    25610      -24
- Misses                               29670    30008     +338
- Partials                              3306     3319      +13

Impacted Files
Coverage Δ

...a/org/apache/rocketmq/broker/BrokerController.java
45.20% <0.00%> (-3.92%)
:arrow_down:

.../rocketmq/broker/hacontroller/ReplicasManager.java
0.00% <0.00%> (ø)

...org/apache/rocketmq/broker/out/BrokerOuterAPI.java
19.86% <0.00%> (-1.71%)
:arrow_down:

...he/rocketmq/common/protocol/body/SyncStateSet.java
0.00% <0.00%> (ø)

...srv/controller/AlterSyncStateSetRequestHeader.java
0.00% <0.00%> (ø)

...rv/controller/AlterSyncStateSetResponseHeader.java
0.00% <ø> (ø)

...amesrv/controller/BrokerRegisterRequestHeader.java
0.00% <0.00%> (ø)

...mesrv/controller/BrokerRegisterResponseHeader.java
0.00% <0.00%> (ø)

...r/namesrv/controller/ElectMasterRequestHeader.java
0.00% <0.00%> (ø)

.../namesrv/controller/GetMetaDataResponseHeader.java
0.00% <0.00%> (ø)

... and 35 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ce534ee...c41aefc. Read the comment docs.

Coverage decreased (-0.3%) to 47.653% when pulling c41aefc4ce92c8de51fc10f6ff66c7b4cb499665 on hzh0425:feature/role-switch into ce534eea149a422299d474faac15c73871dc7ba0 on apache:5.0.0-beta-dledger-controller.


net: lib: sockets: yield on blocking mbedtls_handshake

mbedtls handshake blocks the net_offload:ed driver
from receiving data and passing it up to mbedtls.
As such mbedtls does not receive the server's
Server Hello
Certificate
Server Hello Done
sequence.
Signed-off-by: Emil Hammarstrom<EMAIL_ADDRESS>

What networking driver are you using? Following the discussion on slack, it seemed that the offloaded driver was misbehaving (i. e. not blocking on a send() call as expected from the API). I'm not convinced about putting workarounds at the TLS layer.

@rlubos the socket does block if a timeout is supplied.
Could this be related? During TLS over TCP setup this behavior is applied,
		/* TODO For simplicity, TLS handshake blocks the socket
		 * even for non-blocking socket.
		 */
		ret = tls_mbedtls_handshake(ctx, true);

And that is problematic in the case that the driver is busy for a while and returns -EAGAIN on sendto,
	ret = k_sem_take(&sock->sem_tx_pkt, ktimeout);
	if (ret == -EBUSY) {
		return -EAGAIN;
	}
	if (ret == -EAGAIN) {
		LOG_WRN("Send failed, tx not ready");
		return -ENOBUFS;
	}

k_sem_take with timeout == 0 results in ret == -EBUSY and thus -EAGAIN.
Per documentation the driver is returning -EAGAIN on a K_NO_WAIT that would be blocking due to waiting for a resource.
Does this make sense?

Could this be related? During TLS over TCP setup this behavior is applied,
		/* TODO For simplicity, TLS handshake blocks the socket
		 * even for non-blocking socket.
		 */
		ret = tls_mbedtls_handshake(ctx, true);

The TLS layer uses blocking socket for the handshake, and that means a properly implemented TCP/UDP socket shall not return EAGAIN in that case (as it can only by returned by a non-blocking socket call).

k_sem_take with timeout == 0 results in ret == -EBUSY and thus -EAGAIN.
Per documentation the driver is returning -EAGAIN on a K_NO_WAIT that would be blocking due to waiting for a resource.

That seems fine, but now the question is why is the offloaded driver provided with K_NO_WAIT, while it should be a blocking socket call. I did some research and I think I see the problem now.
The net_context layer operates on kernel timeout structure (k_timeout_t), but the net_offload API uses integers for timeouts. Therefore, there's a conversion in the net_offload API calls:
https://github.com/zephyrproject-rtos/zephyr/blob/master/include/net/net_offload.h#L377
This brought my attention, as typically there are special cases to handle K_FOREVER and K_NO_WAIT, yet here we convert everything as is. I did some trials on qemu, and conversion from K_FOREVER -> int -> k_timeout_t using this function gave K_NO_WAIT as a result (which is weird but I assume these converters are not supposed to use constants like K_FOREVER.
I guess that a proper fix would be to sort out the net_offfload API functions and fix the conversions in there (possibly with some helper function that converts constants separately, there are a few places where it's done this way). What do you think @jukkar, does it make sense?

This brought my attention, as typically there are special cases to handle K_FOREVER and K_NO_WAIT, yet here we convert everything as is. I did some trials on qemu, and conversion from K_FOREVER -> int -> k_timeout_t using this function gave K_NO_WAIT as a result (which is weird but I assume these converters are not supposed to use constants like K_FOREVER.

Yes, as you noticed, the K_FOREVER cannot be converted directly to k_timeout_t

I guess that a proper fix would be to sort out the net_offfload API functions and fix the conversions in there (possibly with some helper function that converts constants separately, there are a few places where it's done this way). What do you think @jukkar, does it make sense?

Earlier when I did the various k_timeout_t conversions, I left the net_offload API functions to use the ms timeout values. It did not seem that useful to convert them to use k_timeout_t as all the offloaded drivers would need changing too which looked quite pointless.

@jukkar @rlubos how should this be resolved? we are currently using this patch on our fork meanwhile

Well, I don't have a device using net_offload layer to verify, but I guess some helper function like this should be used instead of direct k_ticks_to_ms_floor64(timeout.ticks) calls in the net_offload.h (assuming -1 is interpreted as an infinite timeout):
static int32_t timeout_to_int32(k_timeout_t timeout)
{
	if (K_TIMEOUT_EQ(timeout, K_NO_WAIT)) {
		return 0;
	} else if (K_TIMEOUT_EQ(timeout, K_FOREVER)) {
		return -1
	} else {
		return k_ticks_to_ms_floor32(timeout.ticks);
	}
}

And respective offloaded drivers shall use reverse conversion if they wish to use k_timeout_t in their implementations. Also note that the offloaded driver maximum timeout would be limited to INT32_MAX given the type representing timeout in the net_offload API.

@rlubos Okay. I will investigate this later :tm: (most likely next week)

See https://github.com/zephyrproject-rtos/zephyr/pull/29579


fix: PR#4 – Gemfile from RalphHightower/ralphhightower.githup.io

/opt/hostedtoolcache/Ruby/3.3.4/x64/lib/ruby/3.3.0/psych/parser.rb:62:in _native_parse': (/home/runner/work/blog/blog/_config.yml): did not find expected key while parsing a block mapping at line 3 column 3 (Psych::SyntaxError) 	from /opt/hostedtoolcache/Ruby/3.3.4/x64/lib/ruby/3.3.0/psych/parser.rb:62:in parse'
from /home/runner/work/blog/blog/vendor/bundle/ruby/3.3.0/gems/safe_yaml-1.0.5/lib/safe_yaml/load.rb:143:in load' 	from /home/runner/work/blog/blog/vendor/bundle/ruby/3.3.0/gems/safe_yaml-1.0.5/lib/safe_yaml/load.rb:157:in block in load_file'
from /home/runner/work/blog/blog/vendor/bundle/ruby/3.3.0/gems/safe_yaml-1.0.5/lib/safe_yaml/load.rb:157:in open' 	from /home/runner/work/blog/blog/vendor/bundle/ruby/3.3.0/gems/safe_yaml-1.0.5/lib/safe_yaml/load.rb:157:in load_file'
from /home/runner/work/blog/blog/vendor/bundle/ruby/3.3.0/gems/jekyll-3.10.0/lib/jekyll/configuration.rb:143:in safe_load_file' 	from /home/runner/work/blog/blog/vendor/bundle/ruby/3.3.0/gems/jekyll-3.10.0/lib/jekyll/configuration.rb:180:in read_config_file'
from /home/runner/work/blog/blog/vendor/bundle/ruby/3.3.0/gems/jekyll-3.10.0/lib/jekyll/configuration.rb:207:in `block in read_config_files'
jekyll 3.10.0 | Error:  (/home/runner/work/blog/blog/_config.yml): did not find expected key while parsing a block mapping at line 3 column 3


Scraping only given profiles

I am curious if I can scrape only a given set of profiles through this?
rootProfiles in config.json contains the URLs of from where to scrape and not the actual profile urls. Is there a provision to do that?

Read the README file of the project, it says:

The profiles will be stored on the directory configured at config.json as individual file per profile. If you want to do something else (as saving on a database), just rewrite the src/saveProfile.js function.

@abhij89 That is not what I my question says. Allow me to phrase it in a more descriptive manner.
I want the crawler to just go to a pre-defined set of URLs, that's it. Right now the crawler gets profiles from related, etc.
I don't want it to do that. I just want profile data for a given list of LinkedIn profile URLs which I can provide it.


Fix ZNR icon color in news

Our default icon color is black, but the font here is white. I think the real way to do these symbols if we use the more often would be to make a font, but for now this is much easier.

As an alternative, we could just delete the image from the news if we want to avoid the inline style variable.

I don't have a problem with inline style for one-of, specially here since it's only temporary.


test Add more p0 test case for cte

Proposed changes
Add more test cases to cover cte
Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall

run buildall

run buildall

run buildall

run buildall


Unassigned input by default

Currently the buttons come preassigned with button 1, since all of the dinput gamepads that i have don't have a '13' button to assign to the home button i have been doing a workaround using axis 5 which isn't used in any of them, please make the default config unassigned/nil

I think this use-case is covered by the README.md file. If you do not want to use any input, but a static value instead, you can set the minimum and maximum values manually.
For example if you do not want to use the DPad at all, set both minimum and maximum values to 0.
Unassigned would not work properly with axes though. The program cannot know if you want it to be centered or positioned to a fix value. I think, that you need to give a static value, is more useful in the long run.
Tell me if your issue can be resolved with this method.

Think the case where the gamepad doesn't have all the necessary buttons/axes
you are recommending a workaround, setting the button to 'Axis 1' 0 0 works, but a special state of unassigned/disabled is a more elegant solution

Static value option will be added to the dropdown menu. If it is selected, instead of minimum and maximum, a single value can be set.

Added to v1.8


[22.11] github-runner: make ldd and ldconfig paths absolute on Linux

Description of changes

Combined backport of #204072 and #205294
Closes #205400
Things done

Built on platform(s)

[x] x86_64-linux
[x] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

@aanderse can you take a look at this one when you have the time?

Sorry for the delay!