Arguments field - no docs

[Enter feedback here]
Would be nice to see an example of what goes in that arguments field, how it is formatted etc

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 07e70665-a794-82fd-287b-65a747488efa
Version Independent ID: 0c34f5c3-bc80-061d-cf7a-f30c224ce3b1
Content: Bash task - Azure Pipelines
Content Source: docs/pipelines/tasks/utility/bash.md
Product: devops
Technology: devops-cicd-tasks
GitHub Login: @steved0x
Microsoft Alias: sdanie

@garrypas Thanks for your suggestion. While we look into it, consider these resources:

Azure DevOps Services
Azure DevOps on Stack Overflow


Add support cgroup config and systemd cgroups

@runcom PTAL

The library is too old in our Dockerfile (debian/ubuntu) and doesn't have function that I am using :/

And this is where the fun begins. :wink:

@cyphar I added own macro to get past that :wink:

LGTM tests fine on my end :)


Beamline-specific fields in web form

@vkuznet: when multiple schemas are loaded, the web form always displays the union of all fields for all schemas, no matter which schema is selected in the Beamlines dropdown. However, each schema defines fields that do not appear in other schemas (i.e. are unique to each beamline). Can the web form display only the fields that are defined for each schema?

Werner, I'm not sure I understand. When I select different schemas I see different fields, e.g. in common schema there is HolderLabel but it does exists in ID4B schema. If the issue you describe still persist please provide screenshots and particular fields.

Thanks @vkuznet, but HolderLabel shows exactly the problem. HolderLabel is meaningful only for the ID4B beamline, so it is only defined in ID4B.json, not common.json or ID1A3.json:
$ ls *.json
beamline1.json  beamline2.json  common.json  ID1A3.json  ID4B.json
$ grep HolderLabel *.json
ID4B.json:        "key": "HolderLabel",

So I am expecting HolderLabel only appears when selecting ID4B on the web form, not common or ID1A3. Each beamline should be able to define its own fields without affecting the other beamlines.
(Reason: let's pretend there is a field called MyTestField defined in ID1A3.json and ID4B.json, but NOT in common.json. And MyTestField is optional in ID1A3 but required in ID4B. If I select common on the web form, how will it display MyTestField -- will it be optional or required?)

@wernersun , the form is fixed now, please check the server. I'm closing this issue, but do not hesitate to re-open if you'll find that it is not fixed. In this case please again provide details/screenshot that I can work on. I deployed the fix on chess node.


Autogeneration of CRDs.

Before merging:

[ ] merge #1239 which is a prerequisite
[ ] change base branch to master

What this PR does / why we need it:
With this change, the YAML files in config/crds/ are auto-generated from the structs in pkg/apis/, which in turn become the single source of truth for both Go code and CRDs.

add controller-gen command to the generate target
change annotations to relax validation somewhat, to match current
usage patterns, and in some cases fix tests to conform to newly added
validation
re-generate files. The changes to CRDs fall in the the following
categories:

descriptions copied from the comments in structs (the vast majority of
this change)
validation tree expanded deeper (i.e. actual allowed properties are
listed rather than just saying that "this is an object") and the
required properties brought up to date with the structs
some fields added explicitly with the same value as they have by
default when omitted (singular, listKind, versions)
added controller-gen.kubebuilder.io/version annotation
the controller-tools.k8s.io: "1.0" label removed from the Test*
types
some missing properties added in the Test* types (commands,
kindContainers, kindNodeCache)

Fixes #862
Signed-off-by: Marcin Owsiany<EMAIL_ADDRESS>

While you're touching this: The CRDs are currently residing in:
config/crds/kudo.dev_instances.yaml
for example. Would it be a bit deal to move them to
config/crds/v1beta1/kudo.dev_instances.yaml
?
At some point we're going to have multiple CRD versions in the code base, it would be nice to have this already prepared. If it's a big change, i'll put it on my list for later.

@ANeumann82 I think multiple versions will go into a single YAML.

@porridge Right, you're correct. Forgot about that. In that case, never mind :)


MINIFI-516: Added bootstrap option to override processors ssl references to use m…

…inifi parent ssl instead
When using a ingestor to deploy flows to MiNiFi, processors using SSL contexts don't start up as the sensitive properties are not passed along in the NiFi templates.
A user should be able to set a boolean in the bootstrap.conf to tell MiNiFi to ignore any custom SSL contexts in the config.yml, and repoint to the parent SSL called "SSL-Context-Service" instead.
Thank you for submitting a contribution to Apache NiFi - MiNiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[x] Does your PR title start with MINIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[x] Has your PR been rebased against the latest commit within the target branch (typically master)?

[x] Is your initial contribution a single, squashed commit?

For code changes:

[x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi-minifi folder?
[x] Have you written or updated unit tests to verify your changes?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under minifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under minifi-assembly?

For documentation related changes:

[x] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

@gruselglatz - This commit should fix the SSL issues we spoke about in #167

I've read your commit. So when i understand it correctly, i set nifi.minifi.flow.use.parent.ssl=true and enter my security properties in the bootstrap and it will propagate to my config.yml.
Or do I miss something? Because I am not so sure what the "MiNiFi parent SSL controller" is.

Yup, you're right. Setting the nifi.minifi.flow.use.parent.ssl=true will use the Security Properties in the config.yml for all processor SSL contexts instead of custom ones from the C2/template xml.

@GCHQ-NiFi , nice, thank you!


feat(split-button): add new component (VIV-786)

The outlined hover doesn't look good now:

Looks great!
Aside of one issue:
Outline button hover state looks kind of off,
but only when you hover the more actions part.

Looks great!
Aside of one issue: Outline button hover state looks kind of off, but only when you hover the more actions part.

@AyalaBu @rinaok
I fixed the hover issue :)


Implement OpenSearch

Enables quick searches like this:

Nice! Django-opensearch is GPL-licenced though, which is incompatible with Wikipendium's Apache 2 licence :/ Is there an easy way to implement this without using Django-opensearch?

Yeah, it is easy to hard-code the XML instead of generating it dynamically. Shall I do that instead?

From: Sigve Sebastian<EMAIL_ADDRESS>Sent: 29.11.2015 20:12
To<EMAIL_ADDRESS>Cc<EMAIL_ADDRESS>Subject: Re: [wikipendium.no] Implement OpenSearch (#415)
Nice! Django-opensearch is GPL-licenced though, which is incompatible with Wikipendium's Apache 2 licence :/ Is there an easy way to implement this without using Django-opensearch?

Reply to this email directly or view it on GitHub:
https://github.com/stianjensen/wikipendium.no/pull/415#issuecomment-160453898

Shall I do that instead?

If you like :3

Done. New pull request here:
https://github.com/stianjensen/wikipendium.no/pull/417


Improve YAML string creation cross-platform (Issue #3)

When passing --args to a "dbt run-operation" macro as a YAML string, Command Prompt (Windows) requires a different quoting syntax than Bash (Mac/Linux) does.
While testing the new quoting scheme, it became apparent that in certain situations, quotes and/or other special characters need to be escaped, so I added this escaping.
Updated tests/test_dbt_project/profiles.yml to work with most recent version of dbt-sqlite (0.1.2)
Updated tests accordingly

please merge @jenniferczhan


Visual bug with period facet

Probably due to empty string in period name

It is a data issue after all, we are working on that

Cleared data from empty values


Update it.yml - parse error on line 321 (323)

Error while building Italian version of framework. There is a wrong formatting in line 321 (ex. 323).

I've also fixed this string at https://www.transifex.com/silverstripe/silverstripe-framework/translate/#it/master/72668585?key=URLDESCRIPTION
Can you please make any translation strings in transifex; Just ping me if you need to be added as a reviewer.
It's ok to make a PR as long as it's changed there first. :)

Hy, yes please, add me as a reviewer. I may be have some more translations strings to fix/add.
My Transifex username is @petar.simic

Hi @hdpero I've added you to our Italian team. Check your email for invitation.

@tractorcow could you also add me to Croatian team? Ty

All done (I think!).


Added MEnu Calculator

Your repository has merge conflicts. Google how to solve them! Then after solving the conflicts re-send your Pull Requests!


notebooks: code blocks don't publish

Create a notebook
Create a new note
Add some text and a code block
Publish
Observe code block is gone
Edit note
Observe code block isn't there either

I tried to add my substack troubleshooting doc (can find it in tlon public) and it's impossible to get it to publish the code blocks.

https://martiancomputing.substack.com/p/urbit-troubleshooting-basics
https://zurbit.io/apps/groups/groups/~nibset-napwyn/tlon/channels/diary/~nibset-napwyn/winter-updates/note/170141184506052471484675538997779365888

It did eventually work - but took a few tries saving it, it still doesn't show up consistently in the edit menu.
It may be a time to load issue, but something here is broken.

Also when I set one of them to bash it's not broken with this:

Well now it appears to be working, pretty painful to use though.

Spoke too soon, some other things:

Whenever I save it seems to remove one of them?
When I go to edit they're never present

Attempting to repro on local, no luck with any of the aforementioned. Incidentally, found another issue (the . getting auto-inserted after the bartis in the hoon gate definition).
creating a post with text and a code snippet; editing and adding more
https://user-images.githubusercontent.com/16504501/216005065-eb10c656-93da-4b8b-9a86-75edd3d817f8.mov
publishing a bash script
https://user-images.githubusercontent.com/16504501/216006159-38bf8205-119c-4417-b393-032c26e92749.mov
pasting snippets from the Troubleshooting blog post
https://user-images.githubusercontent.com/16504501/216007313-e7d07d53-73de-4a4d-8bfc-49df243cf9ea.mov

@jamesacklin FYI unassigned myself and moved this ticket to Awaiting Info in light of the findings in the previous comment. TLDR, could be related to #1733

Cool, thank you

My exact repro steps:
Go to Tlon public group
Create notebook in 2023 Winter Updates
Copy and paste substack post
Publish
Observe missing code blocks
Attempt to fix
Observe described issues


Exception when running example splits: Failure "Hotkeys stream exited unexpectedly"

I've successfully built flitter but when I try to run flitter splits.scm with the example, I see the timer as it's shown in the screenshot for just a moment then get the following error:
Uncaught exception:

  (Failure "Hotkeys stream exited unexpectedly")

Raised at file "src/event_loop.ml", line 169, characters 4-241
Re-raised at file "src/core/lwt.ml", line 2987, characters 20-29
Called from file "src/unix/lwt_main.ml", line 26, characters 8-18
Called from file "bin/main.ml", line 1, characters 9-23

I ran opam list and lwt is the latest version, 4.1.0. Also using opam 2.0.0 and ocaml 4.07.0 but on debian 9 stable. I installed opam manually from the unstable repo.

That was the problem, thanks. I had Anaconda python in my path rather than the system's python so it couldn't find pynput. Simply had to remove it from my path and flitter works as intended.

Great, glad to hear!


Client does not properly handle the time format string that contains the timezone string

Reopen of legitimate issue : https://github.com/spotify/docker-client/issues/1123#issue-389028598

Can confirm. This is happening on every client version for a docker env where the timezone is anything except UTC .
Docker client version : 8.15.1


Removing property definitions from main specification text?

Now that #445 is merged, should we remove the property definitions from optimade.rst? It does not seem ideal that we have two set of "competing" definitions that could diverge going forward. Technically it is of course possible to insert a link in that part of the document to what we consider the single-point-of-truth of property definitions. However, that cannot be https://schemas.optimade.org/, since we need to be able to edit and reivew edits of the property definitions while we work.
In discussion with @ml-evs, one idea is that we actually check into the repo a generated markdown document for the "OPTIMADE Standard" from the property definition framework that as a single document contains all the property definitions in the same style as before.
(If we do this, we probably want to automatically validate that it is up to date as a commit hook)

I agree with the removal of actual property definitions. AFAIK, they do not contain anything which is not in #445, and there should be a single source of truth.

In discussion with @ml-evs, one idea is that we actually check into the repo a generated markdown document for the "OPTIMADE Standard" from the property definition framework that as a single document contains all the property definitions in the same style as before.

Sounds great - a human-readable description generated from the property definition framework makes a lot of sense.

This is however blocked by the pending PRs introducing or modifying already existing properties. I would suggest resolving them first, or reworking them into PRs modifying property definitions directly.


Count hashes in database

Hello!
It would be cool if there was an endpoint to see the amount of hashes in the database

Hi!
I bet it would. Currently there are 1,283,316 lines in the table. Most of them where generated on init with generateStrings.py (Or another version I might have ran back then).
My biggest worry is that showing that number would incentivise users to increase it, eventually making me clear out the table. In other projects I had to deal with automated scripts users used to game the system (see 0qp0.com, and I wouldn't want to have to deal with that here as well.
I would consider having some statistics manually added to the html. I should look into the database and see how many lines were auto generated by myself and how many added manually by users. That would be interesting. Will think about it 👍

👍


Swift 3.0

This builds on the existing pull request #10 (also thereby issue #9) and updates the dependencies to the latest RxSwift as well as fixing the example app (there is now a swift 3 RxDataSources)
I've also bumped the version number in the podspec and updated the readme.

Hi @samscam @sergdort @serluca Thanks for your work on this repo. I updated the project to recommended settings. Everything built and ran fine, so I merged @samscam's PR. However, now I'm unable to update CocoaPods trunk, because lint is failing:
=== CLEAN TARGET RxSwift OF PROJECT Pods WITH CONFIGURATION Release ===

Check dependencies
“Use Legacy Swift Language Version” (SWIFT_VERSION) is required to be configured correctly for targets which use Swift. Use the [Edit > Convert > To Current Swift Syntax…] menu to choose a Swift version or use the Build Settings editor to configure the build setting directly.
“Use Legacy Swift Language Version” (SWIFT_VERSION) is required to be configured correctly for targets which use Swift. Use the [Edit > Convert > To Current Swift Syntax…] menu to choose a Swift version or use the Build Settings editor to configure the build setting directly.

** CLEAN FAILED **

The following build commands failed:
	Check dependencies
(1 failure)
=== BUILD TARGET RxSwift OF PROJECT Pods WITH CONFIGURATION Release ===

Check dependencies
“Use Legacy Swift Language Version” (SWIFT_VERSION) is required to be configured correctly for targets which use Swift. Use the [Edit > Convert > To Current Swift Syntax…] menu to choose a Swift version or use the Build Settings editor to configure the build setting directly.
“Use Legacy Swift Language Version” (SWIFT_VERSION) is required to be configured correctly for targets which use Swift. Use the [Edit > Convert > To Current Swift Syntax…] menu to choose a Swift version or use the Build Settings editor to configure the build setting directly.

** BUILD FAILED **

The following build commands failed:
	Check dependencies
(1 failure)
 -> RxCoreData (0.3.0-rc.1)
    - ERROR | [iOS] xcodebuild: Returned an unsuccessful exit code.

Analyzed 1 podspec.

[!] The spec did not pass validation, due to 1 error.

Changing the build setting to use legacy does not resolve the issue, and running the converter generates several refactors:

Any suggestions? Thanks!

Hi @scotteg
just do echo "3.0" > .swift-version and try again, then it should work

@serluca Thanks for the suggestion, but that did not resolve the issue. Still getting the same error.

@scotteg what version of cocoapods are you using?

@serluca I was running 1.0.0. So, I updated to 1.0.1 and tried again. Same error.

How do things look for getting off of the '~> .rc' versions of RxSwift dependencies? All my other Rx pods now depend on the '~> 3.0.0' releases. So i can't run my side in swift 3 until.. i've tried every combination of the various pods. And looks like this is the only "sane" solution
So I hope dev is going well :)

@chrisjrex I submitted PR #16 last night which should sort that :)

I was going to do one myself next week if no one had gotten around to it yet. So thank you :)


Fix failing unit test

Problem: Intermittently one of our unit tests was failing.
Solution: My hunch as to why it was failing is because of the parallel nature would occasionally cause an object to be in the wrong state for a specific test. So that object is now being defined for every test instead of globally.
Closes #2625
Checklist
Before creating a PR, run through this checklist and mark each as complete.

[x] I have read the CONTRIBUTING doc
[ ] I have added tests that prove my fix is effective or that my feature works
[x] I have checked that all unit tests pass after adding my changes
[ ] I have updated necessary documentation
[x] I have rebased my branch onto main
[x] I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

Release notes
If this PR introduces a change that affects users and needs to be mentioned in the release notes,
please add a brief note that summarizes the change.

Nice!


Test-SecretVault does not store sign-in token in env var if timeout exceeded

Scenario

Use Test-SecretVault to trigger authentication
Sign-in token gets put in env variable
Do stuff - after 30 minutes the token expires
Use Test-SecretVault to trigger authentication again
Token is not retrieved & refreshed in env var

# Suggest following syntax for statement 39
$token = & op signin $accountName --raw

Relates to #2


DirectX 12 command buffer not closed when enable API validation.

Description
D3D12 validation layer show Reset fails because the command list was not closed. and some resource's barrier cannot show correctly in Resource Inspector, like Texture98 in rdc file.
Texture 98 firstly used in EID 330, but Resource Inspector show it used in EID 342.

This validation error will not show when I run my client without renderdoc.
Steps to reproduce
In My Engine, I use this way to track dx12's resource state.
firstly I dispatch N command buffer to record parallely in worker threads. after all N command buffers record task finished, then I do resource barriers which resource are used between k-1 and k command buffers at k - 1 command buffer. these resource barrier are recorded in main thread.
at last I close all N command buffers in main thread.
I have send rdc file to<EMAIL_ADDRESS>email title is "bugreport: DirectX 12 command buffer not closed when enable API validation.".
Environment

RenderDoc version: v1.28 stable
Operating System: Windows 10
Graphics API: DirectX 12

I didn't receive any email. Looking at my server logs it seems like the message may have been rejected for being too large. Did you try to attach a large rdc file as an attachment instead of uploading it to a fileshare and sending a link to it?

very thanks, and I upload it here. https://github.com/MioMioReimu/renderdoc_rdc/blob/main/dx12_cmd_buffer_not_closed.rdc

Thanks for sharing that. I don't see anything that would be wrong there, and since the validation errors are captured from the running program (and because of D3D12's limitations they are all in the same single execute) I can't tell exactly what's causing it. I suspect this may be a duplicate of #3042 though that only reproduced for me when capturing the first frame.
Can you test again with the latest nightly build? If the problem still occurs can you share your program so that I can reproduce the problem at capture time?

Thanks. it works, latest nightly build fix all. validation error and resource inspector error not show any more.

OK, it sounds like you were reporting two different bugs? but if they're both fixed then that's good.


Attempting to fix login issues to uwhackweeks staging hub

ref: https://2i2c.freshdesk.com/a/tickets/104

Tighten up auth scopes on uwhackweeks

I wasn't sure if the mix of orgs and teams was confusing things
so I restricted us just to teams.

Regenerate client secrets for uwhackweeks staging github oauth app

Since the issue was isolated to staging, not prod, I tried
regenerating the client secret for the github oauth app and
revoked all active user tokens to force people to log back in
again in case that helped the situation.

Probably worth noting that this brings it more in-line with how the pangeo hubs are configured:
https://github.com/2i2c-org/infrastructure/blob/87ee7de3de3903e62e8f196b1dad63aab0c8fb8a/config/clusters/pangeo-hubs/common.values.yaml#L42-L59

So, because I revoked the tokens and forced the auth flow again Scott noticed that the GitHub OAuth App had not been granted permission to the ICESat org and that is why Authentication was failing. This PR is probably not strictly required but I'll leave it open in case @2i2c-org/tech-team feel it is useful.

I believe this has been deployed already, yes? If so we should merge it in right?

@choldgraf yes it has been deployed, but the actual issue ended up being outside our config. So if this was not merged and we reverted to the old config, we don't lose anything.

Sounds good - I will leave that decision up to others, I don't have strong preferences either way, but I think we should either close this PR or merge it soon :-)

uwhackweeks always fails in ci/cd atm so I will redeploy manually too


SSH host names for files in history

Hi,
I like using the recent folders menu to navigate through my projects, specially in conjunction with a plethora of remote machines. For folders this works well, because the ssh config name is attached to the name of the folder in the menu:

Oddly, for files this is not done.

In this example, I'd like to have a quick overview of the .zshrc files on different machines, but can't distinguish between them. It would be consistent to also have their ssh host names attached like for the folders.

Which quick pick is this? Can you tell me how you get to this quick pick just so we're on the same page?

File: Open Recent if thats what you mean.

I imagine this has something to do with the LabelService:
https://github.com/microsoft/vscode/blob/afd3f0abf02675155a01770b7f45fcf3bb9202d6/src/vs/workbench/browser/actions/windowActions.ts#L192-L215
The workspace gets this mark that it came from a specific remote but files don't.

/duplicate https://github.com/microsoft/vscode/issues/143096


enable telemetry, docs port

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Thanks for remembering this! If we don't pass -w1, Materialize will use 3 workers by default (which is unnecessary in this case).


Invisible text in Text()

The same issue as #214. App builded on kotlin 1.4.20, 0.2.0-build132. On PC Win10 x64 work fine, on notebook Win7 x64 all Text() contains invisible text, it selectable, can copy/paste but can't see. I try to change kotlin and compose versions like in #214 but all text become unselectable or just gone. Tryed latest versions, all the same.
on 1.4.31, 0.4.0-build173

Task :run
Failed to create DirectX12 device.
Failed to create Skia Direct3D context!
Falling back to OPENGL rendering...

We need to know what is your machine, OS and videocard. Also will export SKIKO_RENDER_API=SOFTWARE fix the problem?

Operating System: Windows 7 Home Basic 64-bit (6.1, Build 7601) Service Pack 1 (7601.win7sp1_ldr.170913-0600)
System Manufacturer: ASUSTeK Computer Inc.
System Model: K53SJ
Processor: Intel(R) Core(TM) i3-2310M CPU @ 2.10GHz (4 CPUs), ~2.1GHz
DirectX Version: DirectX 11

SKIKO_RENDER fix it on 0.4.0. Think it's may be conflict between integrated and discrete video cards.

Please check the following ticket on YouTrack for follow-ups to this issue. GitHub issues will be closed in the coming weeks.


ERROR ~ .nextflow/plr/cdcf0bbe-717d-4d9a-a858-72cf72178f9b/nf-validation-1.1.3: Function not implemented

Have you checked the docs?

[X] nf-core website: troubleshooting
[X] nf-core modules documentation

Description of the bug
I'm running a test run
nextflow run nf-core/rnaseq -r 3.14.0 -profile test,docker --outdir ~/Descargas/prueba_rnaseq

And I get the following error:
Launching `https://github.com/nf-core/rnaseq` [sick_pauling] DSL2 - revision: b89fac3265 [3.14.0]
ERROR ~ .nextflow/plr/cdcf0bbe-717d-4d9a-a858-72cf72178f9b/nf-validation-1.1.3: Function not implemented

 -- Check '.nextflow.log' file for details

This error occurs with other nf-core pipelines and versions.
We have reinstalled nextflow (from conda), and cleared nextflow downloaded pipelines with rm -rf ~/.nextflow/assets.
Command used and terminal output
No response
Relevant files
nextflow.log
System information
Nextflow version: 23.10.1
Hardware: Desktop
Executor: local
Container engine: Docker
OS: Linux Mint

The error doesn't appear now after rebooting.


Refactoring & getTreeInfo speedup

stack build && stack exec nitta -- -p=8080 -t=fx32.32 examples/sin_ident/sin_ident.lua
Performance of time http GET localhost:8080/treeInfo:

Before optimisation:
Executed in   19.20 secs      fish           external

After predicate rewriting (https://github.com/ryukzak/nitta/commit/f262ad3e5dbfd3f665eb543145f57763b734e2a8):
Executed in    2.02 secs      fish           external

After Force strictness (https://github.com/ryukzak/nitta/commit/c80ae1ad83d951907f335b126d4c2fb9d57b54ae):
Executed in    1.93 secs      fish           external

After extraction isLeaf and isComplete to tree data (https://github.com/ryukzak/nitta/commit/8020ccbb194958c218ad35c7db130419eaa44fe9):
Executed in    1.86 secs      fish           external

close #153

This was the problem isComplete and isLeaf in ML branch

If I get it right, in the ViewHelper module we can now take raw data from Tree with isLeaf and isComplete from NITTA.Synthesis.Types, so it shouldn't be a problem, no this kind of cyclic dependency is reintroduced.


[BUG] The /<index>/_upgrade API can produce segments larger than max segment size

Describe the bug
I was talking with @mch2 yesterday, saying "Hey -- we should add a dedicated API, or maybe just a flag to force_merge, to rewrite old segments using the latest codec after doing an upgrade." I'm glad I talked with him before creating an issue for it, because he told me that it already exists. Great!
I was curious about how it works, so I dove through the call to InternalEngine.forceMerge(), where it passes the upgrade = true parameter, which it sets on the OpenSearchMergePolicy, and then this logic gets called to do the actual segment rewriting. Cool.
Except, it's not cool at all, because, if there's nothing to upgrade (because all the segments have already been (re)written using the latest codec), it falls back to the underlying merge policy's force merge logic, which will compute merges with unbounded max segment size.
Overall, the real problem is OpenSearchTieredMergePolicy and that annoying forceMergePolicy that ignores max segment size. While we need it in order to maintain the existing behavior when merging down to a single segment (e.g. before hot-to-warm migration), IMO we should not use forceMergePolicy unless we're explicitly trying to merge down to a small number of segments. Specifically, I would say that we should only use forceMergePolicy if total_shard_size / maxSegmentCount > maxSegmentSizeMB -- that is, whenever we can merge down to maxSegmentCount segments without exceeding the max segment size, then we should honor the max segment size.
Related component
Indexing
To Reproduce

Create an index with 1 shard, 0 replicas (for simplicity), called my-index, with a merge policy that sets max segment size to 500MB.
Index 5GB of data.
Run POST /my-index/_upgrade
Run GET /_cat/segments
Maybe repeat steps 3 and 4 a few times.
Notice that (eventually), a segment with size greater than 500MB gets written.

Expected behavior
I would expect max segment size to be honored except when I force merge to a small number of segments.
Additional Details
Plugins
N/A
Screenshots
N/A
Host/Environment (please complete the following information):

OS: My brain reading through code
Version: Way too old

Additional context
N/A

Whenever we can merge down to maxSegmentCount segments without exceeding the max segment size, then we should honor the max segment size.

Makes sense to me meaning max segment count would be honored over max size.  This would cover any force_merge not just on upgrade calls right?
Also this api does not appear to be documented at index or cluster lvl at https://opensearch.org/docs/latest/api-reference/index-apis/index/.

[Triage - attendees 1 2 3 4]
@msfroh Thanks for filing.


API changes to SM regarding prediction and changes to docs

:white_check_mark: Build SMT 0.1.51 completed (commit https://github.com/SMTorg/SMT/commit/47d2a7b2cb by @)

:white_check_mark: Build SMT 0.1.52 completed (commit https://github.com/SMTorg/SMT/commit/962946c45f by @)


lora-gateway-bridge didn`t  emit join-accept  message

I  tried to perform OTTA using this.
However it seems the gateway successfully sent join-request message to NS and AS agreed
here are some logs  ,
journalctl -u lora-app-server -u loraserver -u lora-gateway-bridge

Nov 20 20:18:22 lora-gateway-bridge[30219]: time="2017-11-20T20:18:22+08:00" level=info msg="gateway: received udp packet from gateway" addr=### protocol_version=2 type=P
Nov 20 20:18:22 lora-gateway-bridge[30219]: time="2017-11-20T20:18:22+08:00" level=info msg="gateway: rxpk packet received" addr=###data="ABERERERERERExERERERERE0DfFWPM0
Nov 20 20:18:22 lora-gateway-bridge[30219]: time="2017-11-20T20:18:22+08:00" level=info msg="backend: publishing packet" topic="gateway/0000000000000000/rx"
Nov 20 20:18:22 lora-gateway-bridge[30219]: time="2017-11-20T20:18:22+08:00" level=info msg="gateway: sending udp packet to gateway" addr=### protocol_version=2 type=Push
Nov 20 20:18:22 loraserver[30214]: time="2017-11-20T20:18:22+08:00" level=info msg="backend/gateway: rx packet received"
Nov 20 20:18:23 loraserver[30214]: time="2017-11-20T20:18:23+08:00" level=info msg="packet(s) collected" dev_eui=1111111111111113 gw_count=1 gw_macs=0000000000000000 mtype=JoinRequest
Nov 20 20:18:23 lora-app-server[30213]: time="2017-11-20T20:18:23+08:00" level=info msg="node updated" dev_eui=1111111111111113
Nov 20 20:18:23 lora-app-server[30213]: time="2017-11-20T20:18:23+08:00" level=info msg="join-request accepted" app_eui=1111111111111111 application_name=ludeng dev_addr=073703f6 dev_eu
Nov 20 20:18:23 lora-app-server[30213]: time="2017-11-20T20:18:23+08:00" level=info msg="handler/mqtt: publishing join notification" topic="application/2/node/1111111111111113/join"
Nov 20 20:18:23 loraserver[30214]: time="2017-11-20T20:18:23+08:00" level=info msg="node-session saved" dev_addr=073703f6 dev_eui=1111111111111113
Nov 20 20:18:23 loraserver[30214]: time="2017-11-20T20:18:23+08:00" level=info msg="backend/gateway: publishing tx packet" topic="gateway/0000000000000000/tx"

tcpdump udp port 1700
20:18:22.851293 IP gateway >server.1700: UDP, length 197
20:18:22.851843 IP server > gateway: UDP, length 4

according to tcpdump there is only ACK message , no  join-accept message。

LoRa Gateway Bridge doesn't know anything about the LoRaWAN protocol, so it doesn't know about join-accepts. It only forwards LoRa payloads (which so happens to be a join-request / join-accept). I don't understand your question related to the ACK message, but I think you are referring to the UDP ACK, see also PROTOCOL.txt.

Sorry for the confusion.
ACK means PUSH_ACK in PROTOCOL.txt, it`s a 4-bytes packet .
My problem is that  the  procedure of OTAA is like this:
node --(join request)--> gateway -> LoRa Gateway Bridge(send PUSH_ACK to gateway immediately and then publish MQTT message) -> loraserver -> lora-app-server--(join-accept)--> loraserver -> MQTT -> LoRa Gateway Bridge -> gateway.
so the gateway should receive 2 packets if everything went right,one is  PUSH_ACK , the other is join-accept, however , according to tcpdump there is only ACK message , no join-accept message。

Are you running the LoRa Gateway Bridge on the gateway itself? If not, could you give that a try?

thank you for your time, I`ll try running the LoRa Gateway Bridge on the gateway

Perfect, please let us know if that solved your issue (I expect it will), so we can close the issue afterwards.

After reading  the codes and logs, I find out that the LoRa Gateway Bridge didnt subscribe to the tx channel, and thus it wasnt able to get data published by the Loraserver, let along sending data to gateway.
And I found that only by handling PULL_DATA first will the LoRa Gateway Bridge subscribe to the tx channel, and the gateway didt send PULL_DATA first , thats the problem, and I fixed it.
the PROTOCOL.txt did tell us that:
The gateway must periodically send PULL_DATA packets to be sure the network
route stays open for the server to be used at any time.

I should have read it carefully, my bad.
thanks again for your time

How did you handle the PULL_DATA, did you change something in the Lora-gateway-bridge or you wrote some script for that. I have a setup like this:
concentrator--(spi)-->Packet forwarder--(udp)-->Lora-gateway-bridge--(mqtt/pub)-->Mosquitto--(mqtt/sub)-->myScript"
i want to transmit some data to my node, so where should i handle the PULL_DATA ?

I would also like to know where to handle the PULL_DATA

The whole idea of the LoRa Gateway Bridge is that you don't need to implement the UDP protocol yourself / deal with PULL_DATA. Simply use the MQTT topics as documented here: https://docs.loraserver.io/lora-gateway-bridge/use/data/. LoRa Gateway Bridge will handle the UDP protocol specifics for your.

Do you have any documentation regarding the hosted version for loraserver?

If you mean the server hosted by Kyrio, the documentation hosted at https://docs.loraserver.io/ applies to Kyrio too. For other questions it is better to contact Kyrio directly :-)


getting econnreset error everytime I input something

Hi,
After upgrading to 0.4.x, I get the econnreset error as following for whatever i say in interactive mode.
Hello! How may I assist you today?Error('read ECONNRESET\n=========================== logs ===========================\n→ POST https://chat.openai.com/backend-api/conversation/gen_title/3f2b4db9-29a5-483b-8590-39714cf743e6\n  user-agent:

And another half of a page.
Can you advise? It's really annoying.

That error message is not enough for me to go on, and I cannot reproduce.

Closing due to inactivity.


⚠ Bofrost.at has degraded performance

In 71c3c85, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 813 ms

Resolved: Bofrost.at performance has improved in 4ecf0c7.


Special character section incorrectly explains how en.microsoft analyzer will analyze special characters like € and ❤

The article's "Special Character' section incorrectly says that the en.microsoft analyzer will tokenize special characters like € and ❤.

Analyzers that will tokenize special characters include the "whitespace" analyzer, which takes into consideration any character sequences separated by whitespaces as tokens (so the "❤" string would be considered a token). Also, a language analyzer like the Microsoft English analyzer ("en.microsoft"), would take the "€" string as a token.

This is not correct. Running the analyzer:
{
"text": "hotel ❤ heart",
"analyzer": "en.microsoft"
}
Produces tokens hotel and heart only.
"tokens": [
    {
        "token": "hotel",
        "startOffset": 0,
        "endOffset": 5,
        "position": 0
    },
    {
        "token": "heart",
        "startOffset": 8,
        "endOffset": 13,
        "position": 1
    }
]

While the whitespace analyzer produces three tokens as expected.
"tokens": [
    {
        "token": "hotel",
        "startOffset": 0,
        "endOffset": 5,
        "position": 0
    },
    {
        "token": "❤",
        "startOffset": 6,
        "endOffset": 7,
        "position": 1
    },
    {
        "token": "heart",
        "startOffset": 8,
        "endOffset": 13,
        "position": 2
    }
]

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: e1c2c641-bd55-a7aa-b747-bfb75671a7cc
Version Independent ID: 616f0123-6306-f0b3-806b-30c1fbfe702a
Content: Simple query syntax - Azure Cognitive Search
Content Source: articles/search/query-simple-syntax.md
Service: cognitive-search
GitHub Login: @BevLoh
Microsoft Alias: beloh

@danglund
Thanks for your feedback! We will investigate and update as appropriate.

Hi @danglund
Thanks for your contribution. Please add your ideas in below link, so our production team can review it and update the same.
Ideas · Community (azure.com)

@danglund, I read it as the whitespace analyzer will parse the emoji and the language analyzer will parse the currency symbol, but  it doesn't say that the language analyzer will parse the emoji.  It's not trying to say that the whitespace analyzer and language analyzers do the same thing.  I'll make that clear in a doc update today.
#please-close

@danglund  Here's the update:  https://learn.microsoft.com/en-us/azure/search/query-simple-syntax#special-characters   Let me know if you have more feedback??


Temporal fix: Quarkus native mode throws a weird error of constructor

not found
Related to #111
It will disable native mode on generated docker images until we understand why it fails.

Codecov Report

Merging #122 (a1efc17) into main (be92a08) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##               main     #122   +/-   ##
=========================================
  Coverage     80.15%   80.15%
  Complexity      146      146
=========================================
  Files            40       40
  Lines          1940     1940
  Branches        306      306
=========================================
  Hits           1555     1555
  Misses          273      273
  Partials        112      112

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update be92a08...a1efc17. Read the comment docs.


Backend Dashboard-Admin-App - Sprint 1

Create all the required entities, routes, requests for the admin dashboard.

I will work on this @Honeyloveet

@philipomoigui When you create a pull request comment on what you have done in detail.

Am working on this


[Bug]: Add program to channel perf regression after external ID introduction

Contact Details
No response
What happened?
Getting an Error 504 when saving channels with lots of programs. Some times it does end up saving even though it throws the error and some times it does not. Not sure what the amount of programs causes the issue but working on one now that is 376 programs and is about 50/50 on whether it saves or errors out. It does throw the 504 error each time though.
Version
0.1.0
What browser did you experience the the problem on?
No response
What operating system are you using?
No response
Relevant log output
Not seeing any relevant logs or errors.

Let me know if this is helpful.
Console.txt

Hmmm, nothing sticks out there. Was hoping for some more specific error details. 504 is actually pretty interesting... I think the parsing issues actually would manifest as 500s. 504 replies that the server is just taking too long, not something I've seen before. Let me see if I can reproduce on my build with a lot of programs.

If you watch your browser network tab while it's saving - how long is that POST request taking?
I just executed a channel update for about 750 channels and it took 8 seconds on the server.

I'll have to check this later. After testing out the backups again, I can't seem to keep the container running. See #544.

Wow! Just ran this under my fresh install and it was 1.7 mins on that event. It did end up saving the programs but was only about half of the number you processed in seconds...hmmm.

Wooof yea, that is quite slow. I haven't experienced anything even close to that on my local machine, which is pretty modest. What type of hardware are you running Tunarr on, out of curiosity?

I'm going to see if there is some work we can push to the background on this path. Usually the DB queries / inserts are very fast. This path does 3 things:

Use the Plex metadata sent from the frontend to create or update programs in the DB
Use the Plex metadata sent from the frontend to create or update  program "external ids" in the DB
For tv shows / music tracks, we will follow parent/grandparent IDs by querying Plex to create the hierarchies in our DB (albums/artists + seasons/shows). This could potentially be slow.

One thing I can do is add some performance debug logging on this path and have you try this and post the results.

Yeah it's not a hardware issue. Running on Dual Xeon's with 64GB of RAM.
Generally things have been very quick to query. Could it possibly be the path to Plex? I am currently using an external IP address Plex URL rather than a local one. This worked better previously with DizqueTV. I could play around with that on my end if it could possibly be.

Ah yep, that's definitely not the problem. It could very well be a network issue ... if you have the http log level enabled you should see latency measurements for all outgoing plex queries. There will be quite a few of them when upserting programs.
I just created a fresh database to ensure that we were doing all inserts and not skipping any work. Some preliminary results:
@tunarr/server:dev: 2024-06-18T20:14:31.555Z [debug]: findAndUpdateProgramRelations took 2656.43 (success)
@tunarr/server:dev: 2024-06-18T20:14:31.660Z [debug]: upsertContentPrograms to 17285.341 millis. 771 upsertedPrograms

Some more measurements on an even larger set:
@tunarr/server:dev: 2024-06-18T20:24:18.665Z [debug]: Upserting 1907 programs
@tunarr/server:dev: 2024-06-18T20:24:20.842Z [debug]: upsertProgramsDB took 2177.057 (success)                                @tunarr/server:dev: 2024-06-18T20:24:53.915Z [debug]: upsertProgramExternalIds took 33058.136 (success)
@tunarr/server:dev: 2024-06-18T20:25:01.477Z [debug]: findAndUpdateProgramRelations took 7560.623 (success)
@tunarr/server:dev: 2024-06-18T20:25:01.712Z [debug]: upsertContentPrograms to 43786.908 millis. 1907 upsertedPrograms
@tunarr/server:dev: 2024-06-18T20:25:02.216Z [debug] <dao/channelDb.ts>: createNewLineup took 44290.791 (success)
@tunarr/server:dev: 2024-06-18T20:25:02.787Z [debug] <dao/channelDb.ts>: updateChannel took 571.503 (success)
@tunarr/server:dev: 2024-06-18T20:25:02.852Z [info] <tasks/Task.ts>: Task ReconcileProgramDurationsTask ran in 2.69 ms {"task":"ReconcileProgramDurationsTask"}
@tunarr/server:dev: 2024-06-18T20:25:02.852Z [debug] <dao/channelDb.ts>: saveLineup took 64.576 (success)

Note that some of these measurements contain the others... upserting external IDs is clearly a problem here.

After the browser timeout, If I just wait long enough, 3-5 mins, the upserting does complete and the channel programming is saved. Not seeing times anywhere close to yours but let me know if I can provide anything further to help. Will be curious to see if these improvements on your end, help on mine.

Definitely. I probably won't have a chance today but I'd be curious to get a build out with the perf trace logs and have you run the insert to see if the slow areas are the same. I have an inkling they are.
I have two things in progress:

Instead of serially writing all of the external ids, which we have to do because of a current limitation in the ORM we're using with upserts and conditional indexes, we just do a read-before-write and calculate the differences in memory. Not ideal but I think it's ok in this situation
Moving program grouping (hierarchy) building to a background task triggered on upsert. This is something I've been meaning to do anyway

In theory the biggest wins should be with the first change

Sorry for the delay here. I have some improvements locally but I've fallen down a bit of a rabbit hole with some other perf issues I've noticed with super large channels.
At this point the change is quite large, so I think I may make it the first change to go into the dev branch (the new source of the edge tag soon)

Ok, I have a ton of changes in #552. I'll merge it into the dev branch once it's built and then kick off a build. I've kept the timing code in there at debug log level, so if it's still unbearably slow you can post your measurements here.

I am unable to manually pull the dev branch on Unraid. Getting an image not found error. Should @nwithan8 update the template to include the dev branch as well? Not sure if there is another step in between or if it's on my end. Happy to keep testing out those changes once I can grab that image.

Ah! My bad. I thought you were referring to a separate new dev build above. I’ll run some test channels on the newest edge update then.

Definitely seeing big improvements but still nowhere near your times. Just built a ~700 program channel in 48 sec and no timeout. Improvement but still not sure what's up on my end. Hopefully someone else can give you some more details that can help continue to improve things. At this point it's useable for me. Will keep playing around with things.
2024-06-26T15:53:27.035Z [debug]: loadLineup took 0.042 (success)
2024-06-26T15:53:27.036Z [debug]: select channel took 0.195 (success)
2024-06-26T15:53:27.639Z [debug]: gather programs took 602.192 (success)
2024-06-26T15:53:27.724Z [debug]: program convert took 5.234 (success)
2024-06-26T15:53:27.800Z [debug]: build condensed lineup took 75.664 (success)
2024-06-26T15:54:18.520Z [debug]: loadLineup took 0.026 (success)
2024-06-26T15:54:18.521Z [debug]: select channel took 0.188 (success)
2024-06-26T15:54:19.215Z [debug]: gather programs took 692.92 (success)
2024-06-26T15:54:19.302Z [debug]: program convert took 4.941 (success)
2024-06-26T15:54:19.371Z [debug]: build condensed lineup took 68.232 (success)
2024-06-26T15:54:21.063Z [debug]: loadLineup took 0.018 (success)
2024-06-26T15:54:21.063Z [debug]: select channel took 0.144 (success)
2024-06-26T15:54:21.611Z [debug]: gather programs took 546.957 (success)
2024-06-26T15:54:21.689Z [debug]: program convert took 8.661 (success)
2024-06-26T15:54:21.760Z [debug]: build condensed lineup took 69.509 (success)

Happy to hear there is improvement, but still wanna figure out what is going on! I noticed that times on my workstation (NVMe drive) were much speedier than my laptop (SSD), interestingly. Naturally I imagine sqlite would be faster on a faster drive...
On another branch I'm messing around with ditching our ORM for some queries, since there is a non-trivial overhead when select tons of entities. Since the query builder ditches a lot of the niceties of the ORM (entity managerment, etc) we'd naturally see a significant improvement in some areas. If I have success there, I may also work to substitute this "direct" access on the more critical paths that tend to deal with a ton of data (lineups).
Those timings look like they are for selecting the lineup. Let me know if you can grab timings for creating a channel. The create path should have timings like "upsertProgramsDB" and "upsert external ids"

Brand new channel with 621 programs
2024-06-26T18:24:31.341Z [debug]: Push event: Started building tv-guide at = 2024-06-26T13:24:31-05:00
2024-06-26T18:24:50.043Z [debug]: loadLineup took 0.028 (success)
2024-06-26T18:24:50.044Z [debug]: select channel took 0.162 (success)
2024-06-26T18:24:50.045Z [debug]: gather programs took 0.013 (success)
2024-06-26T18:24:50.045Z [debug]: program convert took 0.068 (success)
2024-06-26T18:24:50.049Z [debug]: build condensed lineup took 3.706 (success)
2024-06-26T18:24:50.050Z [debug]: Generated and wrote xmltv file in 69.905ms
2024-06-26T18:24:50.051Z [debug]: Push event: XMLTV updated at server time = 2024-06-26T18:24:50.051Z
2024-06-26T18:24:50.052Z [debug]: Built TV Guide for a day in 164.062 millis
2024-06-26T18:24:50.052Z [info]: XMLTV Updated at 6/26/2024, 1:24:50 PM {"task":"update-xmltv"}
2024-06-26T18:24:50.060Z [debug]: Task UpdateXmlTvTask ran in 18720.82 ms {"task":"update-xmltv"}
2024-06-26T18:26:23.098Z [debug]: loadLineup took 0.034 (success)
2024-06-26T18:26:23.099Z [debug]: select channel took 0.171 (success)
2024-06-26T18:26:23.099Z [debug]: gather programs took 0.013 (success)
2024-06-26T18:26:23.099Z [debug]: program convert took 0.011 (success)
2024-06-26T18:26:23.103Z [debug]: build condensed lineup took 3.278 (success)
2024-06-26T18:27:02.291Z [debug]: Upserting 621 programs
2024-06-26T18:27:05.166Z [debug]: upsertProgramsDB took 2874.152 (success)
2024-06-26T18:27:17.219Z [debug]: upsert external ids took 12042.536 (success)
2024-06-26T18:27:17.373Z [debug]: Took 13.05523799918592 ms to schedule tasks
2024-06-26T18:27:17.374Z [debug]: upsertContentPrograms to 15241.823 millis. 621 upsertedPrograms
2024-06-26T18:27:17.428Z [debug]: createNewLineup took 15296.722 (success)
2024-06-26T18:27:17.976Z [debug]: updateChannel took 547.411 (success)
2024-06-26T18:27:18.028Z [debug]: saveLineup took 50.994 (success)
2024-06-26T18:27:18.029Z [debug]: loadLineup took 0.03 (success)
2024-06-26T18:27:18.030Z [debug]: select channel took 0.25 (success)
2024-06-26T18:27:18.579Z [debug]: gather programs took 547.582 (success)
2024-06-26T18:27:18.640Z [debug]: program convert took 3.721 (success)
2024-06-26T18:27:18.688Z [debug]: build condensed lineup took 47.866 (success)
2024-06-26T18:27:18.690Z [debug]: build fresh lineup took 660.135 (success)
2024-06-26T18:27:30.997Z [debug]: Task ReconcileProgramDurationsTask ran in 12269.68 ms {"task":"ReconcileProgramDurationsTask"}
2024-06-26T18:27:31.001Z [debug]: Push event: Started building tv-guide at = 2024-06-26T13:27:31-05:00
2024-06-26T18:27:56.252Z [debug]: loadLineup took 0.019 (success)
2024-06-26T18:27:56.253Z [debug]: select channel took 0.151 (success)
2024-06-26T18:27:56.680Z [debug]: gather programs took 426.245 (success)
2024-06-26T18:27:59.758Z [debug]: program convert took 3.894 (success)
2024-06-26T18:28:01.435Z [debug]: build condensed lineup took 1676.025 (success)
2024-06-26T18:28:02.692Z [debug]: Generated and wrote xmltv file in 58.995ms
2024-06-26T18:28:02.693Z [debug]: Push event: XMLTV updated at server time = 2024-06-26T18:28:02.693Z
2024-06-26T18:28:02.694Z [debug]: Built TV Guide for a day in 6991.173 millis
2024-06-26T18:28:02.695Z [info]: XMLTV Updated at 6/26/2024, 1:28:02 PM {"task":"update-xmltv"}
2024-06-26T18:28:02.702Z [debug]: Task UpdateXmlTvTask ran in 44673.19 ms {"task":"update-xmltv"}

Upon creating a second new channel I ran into this error. This was just on channel creation.
2024-06-26T18:29:21.618Z [debug]: select channel took 0.159 (success)
2024-06-26T18:29:22.088Z [debug]: gather programs took 468.761 (success)
2024-06-26T18:29:22.158Z [debug]: program convert took 4.887 (success)
2024-06-26T18:29:22.219Z [debug]: build condensed lineup took 60.147 (success)
2024-06-26T18:29:47.293Z [debug]: Push event: Started building tv-guide at = 2024-06-26T13:29:47-05:00
2024-06-26T18:30:09.668Z [debug]: Task CleanupSessionsTask ran in 0.2 ms
2024-06-26T18:30:09.672Z [error]: Plex request error: socket hang up
err: {
"message": "socket hang up: socket hang up",
"name": "Error",
"stack":
Error: socket hang up
at AxiosError.from (file:///tunarr/server/build/bundle.js:241511:14)
at RedirectableRequest.handleRequestError (file:///tunarr/server/build/bundle.js:243258:34)
at RedirectableRequest.emit (node:events:519:28)
at eventHandlers. (file:///tunarr/server/build/bundle.js:121942:28)
at ClientRequest.emit (node:events:519:28)
at TLSSocket.socketOnEnd (node:_http_client:524:9)
at TLSSocket.emit (node:events:531:35)
at endReadableNT (node:internal/streams/readable:1696:12)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
caused by: Error: socket hang up
at TLSSocket.socketOnEnd (node:_http_client:524:23)
at TLSSocket.emit (node:events:531:35)
at endReadableNT (node:internal/streams/readable:1696:12)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
"config": {
"transitional": {
"silentJSONParsing": true,
"forcedJSONParsing": true,
"clarifyTimeoutError": false
},
"adapter": [
"xhr",
"http"
],
"transformRequest": [
null
],
"transformResponse": [
null
],
"timeout": 0,
"xsrfCookieName": "XSRF-TOKEN",
"xsrfHeaderName": "X-XSRF-TOKEN",
"maxContentLength": -1,
"maxBodyLength": -1,
"env": {},
"headers": {
"Accept": "application/json",
"X-Plex-Device": "Tunarr",
"X-Plex-Device-Name": "Tunarr",
"X-Plex-Product": "Tunarr",
"X-Plex-Version": "0.1",
"X-Plex-Client-Identifier": "p86cy1w47clco3ro8t92nfy1",
"X-Plex-Platform": "Chrome",
"X-Plex-Platform-Version": "80.0",
"X-Plex-Token": "xriinDdAtxsjfmutGycy",
"User-Agent": "axios/1.6.0",
"Accept-Encoding": "gzip, compress, deflate, br"
},
"baseURL": "https://67-4-135-239.f7b9c382cdb54276a8d140eef0f50f4a.plex.direct:32400",
"method": "get",
"url": "/library/metadata/16994",
"metadata": {
"startTime":<PHONE_NUMBER>659
}
},
"code": "ECONNRESET",
"status": null
}
2024-06-26T18:30:09.673Z [error]: Error querying Plex for item 16994
2024-06-26T18:30:09.683Z [error]: Plex request error: socket hang up
err: {
"message": "socket hang up: socket hang up",
"name": "Error",
"stack":
Error: socket hang up
at AxiosError.from (file:///tunarr/server/build/bundle.js:241511:14)
at RedirectableRequest.handleRequestError (file:///tunarr/server/build/bundle.js:243258:34)
at RedirectableRequest.emit (node:events:519:28)
at eventHandlers. (file:///tunarr/server/build/bundle.js:121942:28)
at ClientRequest.emit (node:events:519:28)
at TLSSocket.socketOnEnd (node:_http_client:524:9)
at TLSSocket.emit (node:events:531:35)
at endReadableNT (node:internal/streams/readable:1696:12)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
caused by: Error: socket hang up
at TLSSocket.socketOnEnd (node:_http_client:524:23)
at TLSSocket.emit (node:events:531:35)
at endReadableNT (node:internal/streams/readable:1696:12)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
"config": {
"transitional": {
"silentJSONParsing": true,
"forcedJSONParsing": true,
"clarifyTimeoutError": false
},
"adapter": [
"xhr",
"http"
],
"transformRequest": [
null
],
"transformResponse": [
null
],
"timeout": 0,
"xsrfCookieName": "XSRF-TOKEN",
"xsrfHeaderName": "X-XSRF-TOKEN",
"maxContentLength": -1,
"maxBodyLength": -1,
"env": {},
"headers": {
"Accept": "application/json",
"X-Plex-Device": "Tunarr",
"X-Plex-Device-Name": "Tunarr",
"X-Plex-Product": "Tunarr",
"X-Plex-Version": "0.1",
"X-Plex-Client-Identifier": "p86cy1w47clco3ro8t92nfy1",
"X-Plex-Platform": "Chrome",
"X-Plex-Platform-Version": "80.0",
"X-Plex-Token": "xriinDdAtxsjfmutGycy",
"User-Agent": "axios/1.6.0",
"Accept-Encoding": "gzip, compress, deflate, br"
},
"baseURL": "https://67-4-135-239.f7b9c382cdb54276a8d140eef0f50f4a.plex.direct:32400",
"method": "get",
"url": "/library/metadata/16991",
"metadata": {
"startTime":<PHONE_NUMBER>657
}
},
"code": "ECONNRESET",
"status": null
}
2024-06-26T18:30:09.684Z [error]: Error querying Plex for item 16991
2024-06-26T18:30:09.890Z [debug]: loadLineup took 0.018 (success)
2024-06-26T18:30:09.891Z [debug]: select channel took 0.151 (success)
2024-06-26T18:30:09.891Z [debug]: gather programs took 0.011 (success)
2024-06-26T18:30:09.892Z [debug]: program convert took 0.012 (success)
2024-06-26T18:30:09.897Z [debug]: build condensed lineup took 5.729 (success)
2024-06-26T18:30:10.274Z [debug]: Generated and wrote xmltv file in 59.844ms
2024-06-26T18:30:10.274Z [debug]: Push event: XMLTV updated at server time = 2024-06-26T18:30:10.274Z
2024-06-26T18:30:10.275Z [debug]: Built TV Guide for a day in 639.81 millis
2024-06-26T18:30:10.276Z [info]: XMLTV Updated at 6/26/2024, 1:30:10 PM {"task":"update-xmltv"}
2024-06-26T18:30:10.283Z [debug]: Task UpdateXmlTvTask ran in 22993.02 ms {"task":"update-xmltv"}

And timings on finishing the second channel with 3x the programs.
2024-06-26T18:32:49.380Z [debug]: loadLineup took 0.033 (success)
2024-06-26T18:32:49.381Z [debug]: select channel took 0.207 (success)
2024-06-26T18:32:49.381Z [debug]: gather programs took 0.013 (success)
2024-06-26T18:32:49.381Z [debug]: program convert took 0.008 (success)
2024-06-26T18:32:49.385Z [debug]: build condensed lineup took 3.427 (success)
2024-06-26T18:33:13.069Z [debug]: loadLineup took 0.049 (success)
2024-06-26T18:33:13.070Z [debug]: select channel took 0.235 (success)
2024-06-26T18:33:13.071Z [debug]: gather programs took 0.018 (success)
2024-06-26T18:33:13.071Z [debug]: program convert took 0.012 (success)
2024-06-26T18:33:13.075Z [debug]: build condensed lineup took 3.598 (success)
2024-06-26T18:33:25.425Z [debug]: Upserting 1763 programs
2024-06-26T18:33:32.584Z [debug]: upsertProgramsDB took 7159.396 (success)
2024-06-26T18:34:04.811Z [debug]: upsert external ids took 32215.493 (success)
2024-06-26T18:34:05.038Z [debug]: Took 35.61136299930513 ms to schedule tasks
2024-06-26T18:34:05.039Z [debug]: upsertContentPrograms to 42726.981 millis. 1763 upsertedPrograms
2024-06-26T18:34:05.734Z [debug]: createNewLineup took 43421.652 (success)
2024-06-26T18:34:07.011Z [debug]: updateChannel took 1276.41 (success)
2024-06-26T18:34:07.094Z [debug]: saveLineup took 82.336 (success)
2024-06-26T18:34:07.094Z [debug]: loadLineup took 0.028 (success)
2024-06-26T18:34:07.095Z [debug]: select channel took 0.237 (success)
2024-06-26T18:34:08.306Z [debug]: gather programs took 1208.097 (success)
2024-06-26T18:34:13.648Z [debug]: program convert took 14.311 (success)
2024-06-26T18:34:14.335Z [debug]: build condensed lineup took 686.74 (success)
2024-06-26T18:34:14.337Z [debug]: build fresh lineup took 7242.226 (success)
2024-06-26T18:34:24.699Z [debug]: Task ReconcileProgramDurationsTask ran in 10247.05 ms {"task":"ReconcileProgramDurationsTask"}
2024-06-26T18:34:24.700Z [debug]: Push event: Started building tv-guide at = 2024-06-26T13:34:24-05:00
2024-06-26T18:34:53.015Z [debug]: loadLineup took 0.022 (success)
2024-06-26T18:34:53.016Z [debug]: select channel took 0.157 (success)
2024-06-26T18:34:54.270Z [debug]: gather programs took 1252.81 (success)
2024-06-26T18:34:55.010Z [debug]: program convert took 10.344 (success)
2024-06-26T18:34:55.741Z [debug]: build condensed lineup took 730.039 (success)
2024-06-26T18:34:56.865Z [debug]: Generated and wrote xmltv file in 53.83ms
2024-06-26T18:34:56.866Z [debug]: Push event: XMLTV updated at server time = 2024-06-26T18:34:56.866Z
2024-06-26T18:34:56.867Z [debug]: Built TV Guide for a day in 5006.636 millis
2024-06-26T18:34:56.867Z [info]: XMLTV Updated at 6/26/2024, 1:34:56 PM {"task":"update-xmltv"}
2024-06-26T18:34:56.874Z [debug]: Task UpdateXmlTvTask ran in 49779.95 ms {"task":"update-xmltv"}

I've seen the socket hangups from time to time - still working those out.  Did the creation actually error our there?
As for the timings... it's slow where I expected, but still way slower than I anticipated. The external IDs insertion is still really bad. For the synchronous portion there we are doing batches of 100 inserts (split between two different types of inserts) over 2x the number of programs being inserted (2 IDs per program). So a 2k channel program will generate 4k IDs (one plex rating key and one plex GUID), and thus 80 upserts (4k / 50).
For comparison, I have some timings on my local dev server when saving about 2k items:
@tunarr/server:dev: 2024-06-26T18:40:02.657Z [debug] <dao/channelDb.ts>: loadLineup took 0.066 (success)
@tunarr/server:dev: 2024-06-26T18:40:02.658Z [debug] <dao/channelDb.ts>: select channel took 0.122 (success)
@tunarr/server:dev: 2024-06-26T18:40:02.658Z [debug] <dao/channelDb.ts>: gather programs took 0.009 (success)
@tunarr/server:dev: 2024-06-26T18:40:02.658Z [debug] <dao/channelDb.ts>: program convert took 0.003 (success)
@tunarr/server:dev: 2024-06-26T18:40:02.659Z [debug] <dao/channelDb.ts>: build condensed lineup took 1.147 (success)
@tunarr/server:dev: 2024-06-26T18:40:08.450Z [debug]: Upserting 1903 programs
@tunarr/server:dev: 2024-06-26T18:40:09.020Z [debug]: upsertProgramsDB took 569.075 (success)
@tunarr/server:dev: 2024-06-26T18:40:09.980Z [debug]: upsert external ids took 929.312 (success)
@tunarr/server:dev: 2024-06-26T18:40:10.039Z [debug]: Took 23.4046129998751 ms to schedule tasks
@tunarr/server:dev: 2024-06-26T18:40:10.039Z [debug]: upsertContentPrograms to 2804.298 millis. 1903 upsertedPrograms
@tunarr/server:dev: 2024-06-26T18:40:10.553Z [debug] <dao/channelDb.ts>: createNewLineup took 3317.72 (success)
@tunarr/server:dev: 2024-06-26T18:40:11.018Z [debug] <dao/channelDb.ts>: updateChannel took 465.316 (success)
@tunarr/server:dev: 2024-06-26T18:40:11.101Z [debug] <dao/channelDb.ts>: saveLineup took 82.756 (success)
@tunarr/server:dev: 2024-06-26T18:40:11.102Z [debug] <dao/channelDb.ts>: loadLineup took 0.064 (success)
@tunarr/server:dev: 2024-06-26T18:40:11.102Z [debug] <dao/channelDb.ts>: select channel took 0.188 (success)
@tunarr/server:dev: 2024-06-26T18:40:11.752Z [debug] <dao/channelDb.ts>: gather programs took 648.971 (success)
@tunarr/server:dev: 2024-06-26T18:40:13.032Z [debug] <dao/channelDb.ts>: program convert took 5.921 (success)
@tunarr/server:dev: 2024-06-26T18:40:13.540Z [debug] <dao/channelDb.ts>: build condensed lineup took 507.844 (success)
@tunarr/server:dev: 2024-06-26T18:40:13.542Z [debug]: build fresh lineup took 2440.485 (success)

Pretty wild that my external ID upsert takes less than a second here. I will keep digging around.
I'll also try and reproduce the socket hangup... I haven't seen that on my local machine since I merged in the perf changes. Is your Plex server remote by any chance?

Plex and Tunarr are on the same machine.
Tunarr is connected to Plex via an external IP. This was a mirror of my dizque setup. Could look at changing this but think you previously said it shouldn't matter.
I am accessing Tunarr in my browser via a reverse proxy with NPM (nginxproxymanager).

Yea was just asking in reference to the socket hangups. Trying to figure out a common cause for those. That shouldn't have a bearing on the performance (maybe we should start a separate thread for the socket hangup issue)

I'm able to reproduce the slowness for inserting massive amounts of programs on my test deploy on my own server (spinning rust). Going to try a few things and see if I can get this to be a bit quicker

Oh wait - this is my bad - I think i'm running an older version, haha. Let me do a fresh deploy.

OK - some timings from my server. For context this is Tunarr running in Docker, nested in an LXC. Storage is old disks...
2024-06-26T19:12:37.658Z [debug]: Upserting 2211 programs
2024-06-26T19:12:43.552Z [debug]: upsertProgramsDB took 5893.365 (success)
2024-06-26T19:12:53.556Z [debug]: upsert external ids took 9815.377 (success)
2024-06-26T19:12:54.340Z [debug]: Took 30.310330000007525 ms to schedule tasks
2024-06-26T19:12:54.341Z [debug]: upsertContentPrograms to 18875.183 millis. 2211 upsertedPrograms
2024-06-26T19:12:55.981Z [debug]: createNewLineup took 20516.025 (success)
2024-06-26T19:12:57.519Z [debug]: updateChannel took 1537.802 (success)

So I see some slowness here too. Going to test out some of my theories.
I also got a single errant socket hang up while doing the save. Will look into that too

Have some more improvements here: https://github.com/chrisbenincasa/tunarr/pull/593

Seeing significant improvements...including the warnings as those are new on my end.
2024-06-27T20:28:22.109Z [debug]: loadLineup took 0.127 (success)
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
2024-06-27T20:28:22.112Z [debug]: select channel took 2.065 (success)
2024-06-27T20:28:22.118Z [debug]: direct took 5.043 (success)
2024-06-27T20:28:22.184Z [debug]: eids took 65.382 (success)
2024-06-27T20:28:22.185Z [debug]: program convert took 0.176 (success)
2024-06-27T20:28:22.190Z [debug]: build condensed lineup took 5.274 (success)
2024-06-27T20:29:34.031Z [info]: Remove event channel.
2024-06-27T20:29:35.963Z [debug]: loadLineup took 0.09 (success)
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
2024-06-27T20:29:35.965Z [debug]: select channel took 1.521 (success)
2024-06-27T20:29:35.968Z [debug]: direct took 2.107 (success)
2024-06-27T20:29:35.977Z [debug]: eids took 9.046 (success)
2024-06-27T20:29:35.978Z [debug]: program convert took 0.005 (success)
2024-06-27T20:29:35.980Z [debug]: build condensed lineup took 2.477 (success)
2024-06-27T20:29:49.981Z [debug]: Upserting 2407 programs
2024-06-27T20:29:51.143Z [debug]: upsertProgramsDB took 1161.421 (success)
2024-06-27T20:30:03.372Z [debug]: upsert external ids took 12209.293 (success)
2024-06-27T20:30:03.515Z [debug]: Took 26.294632000033744 ms to schedule tasks
2024-06-27T20:30:03.516Z [debug]: upsertContentPrograms to 15028.99 millis. 2407 upsertedPrograms
2024-06-27T20:30:03.528Z [debug]: createNewLineup took 15041.815 (success)
2024-06-27T20:30:05.451Z [debug]: updateChannel took 1922.2 (success)
2024-06-27T20:30:05.568Z [debug]: Running task CleanupSessionsTask
2024-06-27T20:30:05.569Z [debug]: Task CleanupSessionsTask ran in 0.28 ms
2024-06-27T20:30:05.590Z [debug]: saveLineup took 138.689 (success)
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
2024-06-27T20:30:05.591Z [debug]: Running task UpdateXmlTvTask {"task":"update-xmltv"}
2024-06-27T20:30:05.592Z [debug]: loadLineup took 0.1 (success)
2024-06-27T20:30:05.593Z [debug]: select channel took 0.318 (success)
2024-06-27T20:30:05.721Z [debug]: direct took 127.485 (success)
2024-06-27T20:30:05.912Z [debug]: eids took 190.352 (success)
2024-06-27T20:30:05.953Z [debug]: program convert took 29.897 (success)
2024-06-27T20:30:05.962Z [debug]: build condensed lineup took 7.867 (success)
2024-06-27T20:30:05.968Z [debug]: build fresh lineup took 376.408 (success)

Ahhhh I think I may have left those accidentally. I will remove them
shortly but they shouldn’t cause any harm.
On Thu, Jun 27, 2024 at 4:35 PM josway @.***> wrote:

Seeing significant improvements...including the warnings as those are new
on my end.
2024-06-27T20:28:22.109Z [debug]: loadLineup took 0.127 (success)
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
2024-06-27T20:28:22.112Z [debug]: select channel took 2.065 (success)
2024-06-27T20:28:22.118Z [debug]: direct took 5.043 (success)
2024-06-27T20:28:22.184Z [debug]: eids took 65.382 (success)
2024-06-27T20:28:22.185Z [debug]: program convert took 0.176 (success)
2024-06-27T20:28:22.190Z [debug]: build condensed lineup took 5.274
(success)
2024-06-27T20:29:34.031Z [info]: Remove event channel.
2024-06-27T20:29:35.963Z [debug]: loadLineup took 0.09 (success)
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
2024-06-27T20:29:35.965Z [debug]: select channel took 1.521 (success)
2024-06-27T20:29:35.968Z [debug]: direct took 2.107 (success)
2024-06-27T20:29:35.977Z [debug]: eids took 9.046 (success)
2024-06-27T20:29:35.978Z [debug]: program convert took 0.005 (success)
2024-06-27T20:29:35.980Z [debug]: build condensed lineup took 2.477
(success)
2024-06-27T20:29:49.981Z [debug]: Upserting 2407 programs
2024-06-27T20:29:51.143Z [debug]: upsertProgramsDB took 1161.421 (success)
2024-06-27T20:30:03.372Z [debug]: upsert external ids took 12209.293
(success)
2024-06-27T20:30:03.515Z [debug]: Took 26.294632000033744 ms to schedule
tasks
2024-06-27T20:30:03.516Z [debug]: upsertContentPrograms to 15028.99
millis. 2407 upsertedPrograms
2024-06-27T20:30:03.528Z [debug]: createNewLineup took 15041.815 (success)
2024-06-27T20:30:05.451Z [debug]: updateChannel took 1922.2 (success)
2024-06-27T20:30:05.568Z [debug]: Running task CleanupSessionsTask
2024-06-27T20:30:05.569Z [debug]: Task CleanupSessionsTask ran in 0.28 ms
2024-06-27T20:30:05.590Z [debug]: saveLineup took 138.689 (success)
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
(node:1) Warning: No such label 'req' for console.timeLog()
2024-06-27T20:30:05.591Z [debug]: Running task UpdateXmlTvTask
{"task":"update-xmltv"}
2024-06-27T20:30:05.592Z [debug]: loadLineup took 0.1 (success)
2024-06-27T20:30:05.593Z [debug]: select channel took 0.318 (success)
2024-06-27T20:30:05.721Z [debug]: direct took 127.485 (success)
2024-06-27T20:30:05.912Z [debug]: eids took 190.352 (success)
2024-06-27T20:30:05.953Z [debug]: program convert took 29.897 (success)
2024-06-27T20:30:05.962Z [debug]: build condensed lineup took 7.867
(success)
2024-06-27T20:30:05.968Z [debug]: build fresh lineup took 376.408 (success)
—
Reply to this email directly, view it on GitHub
https://github.com/chrisbenincasa/tunarr/issues/543#issuecomment-2195623292,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAMQRXYOMXSDDHRDQMX4BM3ZJRZPHAVCNFSM6AAAAABJQKGK5WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJVGYZDGMRZGI
.
You are receiving this because you were assigned.Message ID:
@.***>

Logs removed in #594
I think I will close this one out now and open some more targeted issues around performance


Create discord utils to get objects

Description
Add custom functions to improve how we retrieve specific objects from discord.
Type of change

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation update

How Has This Been Tested?
Added unit tests and tested manually in test server.
Checklist:

[x] I have commented my code, particularly in hard-to-understand areas
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works

Current dependencies on/for this PR:

develop

PR #74   👈

PR #75

PR #76

This stack of pull requests is managed by Graphite.


Feature request : Highlight 'user code' in traceback interpreter output

Expected Behavior
In debug mode, a very helpful call trace page comes up when an exception is thrown. Thanks for that!
A quick improvement would be to highlight the blocks which represent user's code. I believe this would give a decent speed boost when recognizing where errors come from. I think this could be as simple as giving a light red-tint on any code boxes which comes from the file defined in the FLASK_APP variable.
This would be unobtrusive, and would help speed up use for many users.
e.g. we could highlight this line:

Alternatively, the call trace could automatically collapse the 'app runner' code which the user is likely uninterested in.
Apologies if this place is for bugs only (not features)

FLASK_APP rarely points to just a single file, so this would require to consider the whole package containing whatever FLASK_APP points to as user code.
However, the debugger is part of Werkzeug which does not even know about flask or FLASK_APP.

A better example screenshot :

OK. I presumed it wouldn't be trivial considering that flask is so far developed. Anyway, seems like this isn't the place to discuss/consider whether a feature is useful. Apologies for distraction

Feature requests do belong into the issue tracker - it's just that this would be better suited in pallets/werkzeug instead of pallets/flask


Somoene hacked me How do i remove the injection?

How do i remove injection

https://github.com/Blank-c/Blank-Grabber/issues/142


fix dependency handling for functions

there are complex logic to handle object dependencies in GetDependencies
function. This logic starts from non-internal dependencies (e.g. with
namespaces) and enrich them by up to two levels of internal dependencies.
E.g. some function depends on composite type and this type depends
on a user table. But sometimes it's not enough, e.g. in case of
function argument is an array of user composite types. In this case
dependency chain is function-array_of_type-type-relation:
 classid  |     objid      |  refclassid  |   refobjid   | deptype
----------+----------------+--------------+--------------+---------
 pg_type  | some_table     | pg_class     | some_table   | i
 pg_type  | some_table[]   | pg_type      | some_table   | i
 pg_class | some_table     | pg_namespace | public       | n
 pg_proc  | dependant_func | pg_namespace | public       | n
 pg_proc  | dependant_func | pg_type      | some_table[] | n

This patch append handling for one more dependency level. I've
intentionally avoided recursive queries here because there are problems
with recursive query support in the old gpdb versions. This code part
has branches even for gpdb 4. Also, I couldn't reproduce longer chain of
internal dependencies.
The previous approaches (#57) has banch of issues:

poor performance that caused by EXIST clause that can't be rewritten to
semi-join by planners (26+ hours for the prod schema)
relying on object order in list without fixing dependency handling

The high-level e2e test was replaced with more precise integration test
The patch must be merged by rebase as two separate commits.

Moreover, the previous patch place both table and function in the same tier as far as the dependency wasn't detected:
- schema: public
  name: table_provides_type
  objecttype: TABLE
  tier:
  - 1
  - 3
- schema: public
  name: func_depends_on_row_type(arg public.table_provides_type[])
  objecttype: FUNCTION
  tier:
  - 1
  - 5

As a result, we still may get fail in case of parallel metadata restore. With the give patch, we get the next output in TOC-file:
- schema: public
  name: table_provides_type
  objecttype: TABLE
  tier:
  - 1
  - 3
- schema: public
  name: func_depends_on_row_type(arg public.table_provides_type[])
  objecttype: FUNCTION
  tier:
  - 2
  - 3


装上MOD没有反应

Hold down ctrl key while construction.
https://user-images.githubusercontent.com/33700/123503242-acee5c80-d68c-11eb-9b67-c4af8da65953.mp4

cool, but what does it mean well it says 'It does not support blueprint.'?

does it mean it will not be able to be copied by blue print or can't be place?

It does not support blueprint/copy.
It will pasted in default short lengths.


onDrop callback not returning event

The drop property is documented as follows:
onDrop: (elemParams: { x: number, y: number, e: Event }) => void
However, as you can see in the demo provided (https://strml.github.io/react-grid-layout/examples/15-drag-from-outside.html), it does not return the event property. However, it returns the width and height.
Therefore it's not possible to retrieve data from the dropped element using getData.

This issue is affecting my application as well. The current implementation only seems to support the narrow use case defined in the examples, instead of an extensible solution that allows you to drag and drop any node from a palette onto the canvas.

@flashtheman see my PR. I believe this will solve the use case. Can you confirm?

This issue is now fixed, thanks @Jacob-Grafenstein great job.


Test types called Array generate invalid code

Describe the bug
If you have a type called Array, then try and generate tests, it will not compile. We need a using alias to be generated.

Won't fix this - fixing it would require creating a 'test' compilation context before emitting, which would be quite a performance hit for multiple file generation. For the instances where it does occur, it's simple enough for the user to rectify.


Sync eng/common directory with azure-sdk-tools for PR 2321

Sync eng/common directory with azure-sdk-tools for PR https://github.com/Azure/azure-sdk-tools/pull/2321  See eng/common workflow

Build failure originated with a timeout, unrelated to the changes being made here.

/check-enforcer override


Improvement: Add ability to modify prompt for summarization

As a user, I would like to modify the prompt for summarization, and occasionally have the summarization done in a non-standard or not normal characterization (Have it summarized by Darth Vader)
The prompt should be definable as a CLI arg, or as a modifiable text box within the Gradio UI.

Custom prompt box added to UI.
Will add cli arg for specifying custom prompt text from text file.


V0.2 james book multiple slot

Ready for merge

Coverage decreased (-0.8%) to 88.913% when pulling fce70bc0ab17be3754827bdb8c59fb4fb58f025a on V0.2-james-book-multiple-slot into 9fd5a293ab8178ca99c430d92cce98777026b7e6 on V0.2.


How to remove tabnine recommendations

How to remove,  e.g. useless completion like this?

PS: couldn't find the right place to ask (StackOverflow doesn't get any replies either).
OS: macOS
Application: VS Code
gz#9644

Hi!
In VSCode, you should be able to disable "comments" suggestions
by adding the following configuration  to your vscode settings.json file:
"editor.quickSuggestions": {
 "other": true,
 "comments": false,
 "strings": true
 },

Does it solve your issue?

unfortunately not..

That should work for VSCode in general.. I looked online for other options and found this https://github.com/microsoft/vscode/issues/9504 , maybe it can be helpful.
Anyway, You can try disabling Tabnine in lines that contain specific strings.
You need to define Disable_line_regex in Tabnine plugin settings

I'm having similar issue here.
It's really annoying when you need to write long comments/strings and need to keep pressing escape before enter. 😥
Tried the solution above but not working

Every time I'm writing a python function, just after typing ":" I receive this recommendation "\n". I really don't know why and it's very annoying. It would be awesome to be able to tag specific recommendations to don't appear anymore.

Every time I'm writing a python function, just after typing "):" I receive this recommendation "\n". I really don't know why and it's very annoying. It would be awesome to be able to tag specific recommendations to don't appear anymore.
This is an example: def function(param1, param2): "\n"

VS Code was suggesting suggestions for React in non-React projects. Below option resolved my issue.
Settings >  Suggestions > Word Based Suggestions Mode > select "current document" from dropdown.

There are Tabnine suggestions above each line of code.
suggestions: tabnine:test | explain | document | ask
Can I turn this off?

abnine:test | explain | document | ask

Did you find a way?

Guys i know how to turn off the suggestions above each line code in Vs Code.
i found this website that explains: https://support.tabnine.com/hc/en-us/articles/20951637600273-Enable-Disable-Code-Lens-in-VS-Code-and-JetBrains
But is simples, is only goes to your configurations of Vs Code and search for Code lens and check/uncheck the box.

Guys i know how to turn off the suggestions above each line code in Vs Code.
i found this website that explains: https://support.tabnine.com/hc/en-us/articles/20951637600273-Enable-Disable-Code-Lens-in-VS-Code-and-JetBrains
But is simples, is only goes to your configurations of Vs Code and search for Code lens and check/uncheck the box.

Yes, this works in PhpStorm (Jetbrains)! I had already unchecked that box ... but you have to reopen the file you are working in... otherwise the suggestions stay visible 🤔


Adding preflight documentation

Codecov Report
Base: 77.55% // Head: 77.55% // No change to project coverage :thumbsup:

Coverage data is based on head (ce74838) compared to base (00d1995).
Patch has no changes to coverable lines.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #229   +/-   ##
=======================================
  Coverage   77.55%   77.55%
=======================================
  Files          27       27
  Lines        2736     2736
=======================================
  Hits         2122     2122
  Misses        507      507
  Partials      107      107

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

I think it worth adding a Preflight CR example to the doc.

/lgtm


attribute:'torch.device' object has no attribute 'device'

How can I resolve this?
I used cuda 11.8+ torch 2.0.1 + python 3.9 and this problem still exists.(my environment)

I know that some people uesd detectreon2-v0.2.1 before,but there was no amp function.

请问解决了吗，我也是这个问题

wrapper最后一行改了
---Original---
From: @.>
Date: Tue, Jun 4, 2024 17:08 PM
To: @.>;
Cc<EMAIL_ADDRESS>Subject: Re: [ChenhongyiYang/QueryDet-PyTorch] attribute:'torch.device' objecthas no attribute 'device' (Issue #90)
请问解决了吗，我也是这个问题
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

能说一下wrapper的最后一行改成什么了吗
------------------ 原始邮件 ------------------
发件人:                                                                                "ChenhongyiYang/QueryDet-PyTorch"                                                                                @.>;
发送时间: 2024年6月4日(星期二) 晚上10:28
@.>;
<EMAIL_ADDRESS>主题: Re: [ChenhongyiYang/QueryDet-PyTorch] attribute:'torch.device' object has no attribute 'device' (Issue #90)
wrapper最后一行改了
---Original---
From: @.>
Date: Tue, Jun 4, 2024 17:08 PM
To: @.>;
Cc: @.@.>;
Subject: Re: [ChenhongyiYang/QueryDet-PyTorch] attribute:'torch.device' objecthas no attribute 'device' (Issue #90)
请问解决了吗，我也是这个问题
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.>
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.Message ID: @.>

我改了最后一行之后的训练时长就很长了，请问有更好的办法吗

训练时长跟你的参数设置有关，跟这个地方貌似关系不大。
---Original---
From: @.>
Date: Tue, Jun 11, 2024 10:45 AM
To: @.>;
Cc<EMAIL_ADDRESS>Subject: Re: [ChenhongyiYang/QueryDet-PyTorch] attribute:'torch.device' objecthas no attribute 'device' (Issue #90)
我改了最后一行之后的训练时长就很长了，请问有更好的办法吗
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

请问解决了吗，我也是这个问题

请问解决了吗，我也是这个问题

请问解决了吗，我也是这个问题


publish typescript definitions

fixes #48

Codecov Report

Merging #83 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master     #83   +/-   ##
======================================
  Coverage    84.3%   84.3%
======================================
  Files           1       1
  Lines         172     172
======================================
  Hits          145     145
  Misses         27      27

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 96d340e...a598df1. Read the comment docs.

Pull Request Test Coverage Report for Build 141

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 70.976%

Totals

Change from base Build 140:
0.0%

Covered Lines:
145

Relevant Lines:
172

💛  - Coveralls

@amir-arad thx!


Flame breaks other CustomPainter's paint() methods?

I've created some CustomPainter and set shouldRepaint to false. So it shouldn't repaint my CustomPainter forever, right? But it does.
Only when I remove game.widget from my scene then all works fine as expected. What does Flame-engine change in terms of Render? Or maybe I do something wrong?
class MyPainter extends CustomPainter {

  final colors = [Colors.red, Colors.green, Colors.yellow, Colors.blue, Colors.pink];
  final random = new Random();

  @override
  void paint(Canvas canvas, Size size) {
    var index = random.nextInt(colors.length);
    canvas.drawRect(
      new Rect.fromLTRB(0.0, 0.0, size.width, size.height),
      new Paint()..color = colors[index]..style = PaintingStyle.fill,
    );
    print("==> repaint()");
  }

  @override bool shouldRepaint(CustomPainter old) => false;
  @override bool shouldRebuildSemantics(CustomPainter old) => false;
}

class MyApp extends StatelessWidget {
  //const MyApp({ Key key }) : super(key: key);

  Future<Widget> loadGame() async {
    final size = await Flame.util.initialDimensions();
    final game = MyGame(size);
    return game.widget;
    // return Container(); //<-- When I use THIS line instead of the previous one - CustomPainter works as expected.
  }

  @override
  Widget build(BuildContext context) {

    return MaterialApp(
      debugShowCheckedModeBanner: false,
      title: 'Some Title',
      theme: ThemeData(
        primarySwatch: Colors.blue,
        visualDensity: VisualDensity.adaptivePlatformDensity,
      ),
      home: Stack(
        children: <Widget>[

          Positioned(
            left: 0,
            top: 0,
            width: 720,
            height: 1280,
            child: FutureBuilder(
              future: loadGame(),
              builder: (context, snapshot) {
                if (snapshot.hasData) {
                  return snapshot.data;
                } else {
                  return CircularProgressIndicator();
                }
              }
            ),
          ),

          Positioned(
            left: 200,
            top: 200,
            width: 500,
            height: 300,
            child: CustomPaint(
              size: Size(500, 300),
              painter: MyPainter(),
              //willChange: false,
              //isComplex: false,
            ),
          ),

        ],
      ),
    );
  }
}

To discover the root cause of this we would need to dig a lot, but CustomPainter can be repainted even if the shouldRepaint method returns false:
From: https://api.flutter.dev/flutter/rendering/CustomPainter/shouldRepaint.html

It's possible that the paint method will get called even if shouldRepaint returns false (e.g. if an ancestor or descendant needed to be repainted).

My guess is that since we use a Ticker and marksRepaint to create the render loop and that may be triggering the CustomPainter to repaint.
Also, by a quick look on your code, it seems that the game is under the CustomPaint, because of the use of the stack,I don't fond it weird that another paint happens when the game loads.
Does the paint on CustomPainter happens all the time? Or only once?

@ProgrammingLife any news on this?

Closing due inactivity


Through the interface

After installing via HACS it did not appear on the interface making the path:

Navigate to Settings > Devices & Services and then click Add Integration
Search for Sodexo

Você reiniciou seu HA conforme exigido pelo HACS?

Yes


HOT integration

After integrating HOT with Platform we found out that the single-threaded implementation of HOT does not work on multiple CPUs. During the benchmarking, it was revealed that HOT uses a memory pool to allocate the memory to nodes. But as it is declared static, memory was getting corrupted when the benchmark used 10 CPUs. HOT works well when the benchmark was done on a single CPU.


Accept the non-null ID type (!ID) as an ID type

Thanks for having done all the heavy lifting to get this started!


Exporter, export empty dicts

Empty dicts are exported in json
(export extras is unchecked)
For example, any primitive have the following extras: {}, extensions: {}
There are 2 main problems:

file weight, export unneeded information
some importer have some issue with empty dictionary, even if this seems to be OK regarding json specification.

Seems other nodes (mesh, asset, accessor... do not generate {} if extras is empty
We currently have this issue in Airbus D&S, so we need to use old exporter for now.

(Seems to have to have the same issue for extensions dict, for some nodes (for example buffers)


Mirrored Network: Local DNS resolution does not work

Windows Version
Microsoft Windows [Version 10.0.22621.3155]
WSL Version
<IP_ADDRESS>
Are you using WSL 1 or WSL 2?

[X] WSL 2
[ ] WSL 1

Kernel Version
<IP_ADDRESS>
Distro Version
Ubuntu 22.04
Other Software
No response
Repro Steps

Update networkingMode to NAT in .wslconfig
Run wsl --shutdown
Start a WSL2 shell (Ubuntu in my case)
Ping a local machine on the network with its DNS name (rutzberry in my case)
Ping works, resolved local IP address ✅
Update networkingMode to mirrored in .wslconfig
Run wsl --shutdown
Start a WSL2 shell (Ubuntu in my case)
Ping a local machine on the network with its DNS name (rutzberry in my case)

Expected Behavior
Ping does not work anymore and outputs the error ping: rutzberry: Name or service not known
Note: Pinging the actual local IP address does work.
Actual Behavior
Ping should work, just as with networkingMode=NAT.
Diagnostic Logs
https://1drv.ms/u/s!AsTaR5LHyZP9tMJB_nJZr7sGE7XfSA?e=wsNDrt

I am in the same boat where some flows seem to work in "mirrored", but DNS seems to fail by default.  I also enabled dnsTunneling and no change.
Scenarios tested w "mirrored"

host example.com -> <IP_ADDRESS>:53 connection refused
host example.com <IP_ADDRESS> -> resolves via known good DNS
ping <IP_ADDRESS> -> ICMP seems to be fine
curl https://<IP_ADDRESS> -> works fine bypassing dns

I did do a netsh trace and no DNS packets when using the <IP_ADDRESS> proxy were captured, but ones with the known good server were fine.  I checked firewall logs and not seeing any DROPs.  Any recommendations on debugging if one of our other network controls might be stepping on this specific proxying feature?

I'm in the same situation.
wsl and windows , cannot access applications through <IP_ADDRESS>,Only through Windows ip

Same issue for me, guys, maybe you found some solution?

Hi. Can you please collect networking logs by following the instructions below?
https://github.com/microsoft/WSL/blob/master/CONTRIBUTING.md#collect-wsl-logs-for-networking-issues


Adapt code to latest djunits version

Classes like AbstractDoubleScalar and AbstractFloatScalar do not exist anymore. Code has to be adapted to align with the changes in dunits v5.1.2. Code therefore does not compile when upgrading to djunits v5.2.1.

These dependencies on non-existent classes in djunits also exist in dsol-swing and dsol-web (input parameters).

The class AbstractDoubleScalar can be replaced by DoubleScalar, and AbstractFloatScalar can be replaced by FloatScalar.

generic scalars have been updated to class structure in new djunits version.


Provide  actionable error message in RegexpTagger

The RegexpTagger error messages are obscure when a regexp has an error.
This is because re messages may be obscure too.
To help with tracing the cause or regex errors, we now catch regex
compilation errors when a RegexpTagger is contrsuctor and raise a new
exception that contains the required diagnostic information with the
error message, the faulty regex and its corresponding tag.
The RegexpTagger.decode_json_obj has been streamlined also to avoid code
duplication and use the default constructor instead.
Signed-off-by: Philippe Ombredanne<EMAIL_ADDRESS>

Thanks @pombredanne!

@stevenbird thank you ++ for the quick review and merge!