Add dataset upgrade button in UI

Q0 and Q1 datasets should have a button on their page linking to a notebook in the user's workspace showing how to upgrade it to Q1 and Q2, respectively.

@achtsnits how can we do this? Right now I have to open my workspace, go to the EOTDL notebook catalog and search for the notebook (03 q1 datasets, for example). Can we have a direct link to the notebooks in the catalog so we can redirect from the UI ?

In the mean time I am using the "search" funcionality, with links like this "https://notebooks.api.eotdl.com/?search=notebook_name"

so you mean to directly open a page like this
https://hub.api.eotdl.com/services/eoxhub-gateway/eotdl/notebook-view/notebooks/03_q1_datasets.ipynb

Yes, thank. Can I use that url structure?

is there a way to make it easily configurable so that you don't have to release?
in general I'm fine with the url so we can keep that stable

is there a way to make it easily configurable so that you don't have to release?
Let's talk about this on Discord.
I am closing this issue with the solution provided to directly open the notebook in the workspace.

We were checking the approach to the datasets quality enhancement and we'd like to investigate with you an approach (on top of the provided notebook) to streamline the procedure and possibly link it to a remuneration mechanism to stimulate users contribution. Let's coordinate via mail to set up a call next week to discuss about it, thank you! @earthpulse @juansensio


gcc: fix build on x86_64-darwin

Description of changes
https://github.com/NixOS/nixpkgs/pull/241951#issuecomment-1625726447
Looks like as from cctools-llvm is not a perfect drop-in replacement? 🤔
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[x] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

@ofborg build gcc gcc.passthru.tests

/nix/store/jq3vqh8g07ssnigy7vj7iwv1gfcki5vr-cctools-port-973.0.1/libexec/as/x86_64/as: this system assembler is deprecated. Please migrate to the clang integrated assembler (`as -q').
error: building of '/nix/store/w6y2ncrffs960gmvw1n2cwr6ww2yln3v-gcc-12.3.0.drv!info,lib,man,out' from .drv file timed out after 3600 seconds

Yeah, gcc and gfortran each took two hours to build on my machine (Rosetta on M1).

As part of the Darwin stdenv rework, the default assembler was switched to the clang one on x86_64-darwin. Before switching it back just for GCC, I’d like to look into what’s happening to see if it should be done in cctools-llvm instead (conditionally for older versions of LLVM). I didn’t encounter this issue in my testing, so the failure is a bit surprising.

I have a hunch what it might be. I’m testing a build with a clang 13 stdenv (and thus a clang 13 integrated assembler in cctools-llvm) to see if it also fails.

I confirm that this does fix the build of gcc and gfortran on my x86_64-darwin (no emulation).


 For example,
Run bash .github/workflows/frontier/submit.sh .github/workflows/frontier/test.sh
Submitted batch job 2070639
Error: The operation was canceled.

and
Run cat test-gpu.out
cat: test-gpu.out: No such file or directory
Error: Process completed with exit code

from https://github.com/MFlowCode/MFC/actions/runs/9748333690/job/26903120742
I'm sure there is a bash or slurm command that would trigger a proper stop error code.

I guess the problem is
  self:
    name: Self Hosted
    if: github.repository == 'MFlowCode/MFC' && needs.file-changes.outputs.checkall == 'true'
    needs: file-changes
    continue-on-error: true

in the test.yml file?

These people seem somewhat confused by the situation. Continue-on-error should be able to be enabled without giving a green check to failed jobs: https://github.com/orgs/community/discussions/15452


ISSUE-465 replace ClientBuilder with JerseyClientBuilder

replace ClientBuilder with JerseyClientBuilder to avoid collisions in case of multiple implementations

Looks good to me, nice find and simple fix.


Inconsistent JSX rendering of

Describe the bug
Special characters such as   render differently if enclosed in an intrinsic tag (eg div) versus a component.
To Reproduce
Per this playground example:
const CustomDiv = props =>
;

function App() {
  return (
    <>

One Two

      <CustomDiv>One Two</CustomDiv>
    </>
  );
}

...renders...
One Two
One Two

Expected behavior
One Two
One Two

Additional context
These characters can be handy as a lazy way to add space before or after text.  For example to space text from an icon inside a button.  We've been migrating components from React and suddenly started noticing these showing up on our pages.

I thought this would be straight forward but it isn't. See: https://stackoverflow.com/questions/37909134/nbsp-jsx-not-working
The problem is we use methods textContent to dynamically set text. This is both performant and handles escaping. But character codes like that will not work. innerHTML works but that is dangerous. What I can do is resolve the encoding so that it works for top-level static JSXText. However anything dynamically inserted should not use those and just use the characters directly.
For example:
// this does what is expected:
<CustomDiv>One Two</CustomDiv>
// this does not
<CustomDiv>One{" "}Two</CustomDiv>
// do this instead
<CustomDiv>One{" "}Two</CustomDiv>

Thanks Ryan.
Coincidentally I was just doing some performance tests swapping out textContent for innerHtml on line 336 in dom-expressions/src/client.js. and yes, it was about 5 to 6 times slower.
It would be great if there was a way to handle to top-level case as that would cover of pretty much all the times we'd run into this.

Technically it's released but you might need to blow out your node modules/locks.. babel-plugin-jsx-dom-expressions 0.29.3  used by babel-preset-solid has the fix, I will bump up the preset when I do my next Solid release.

and when is the next release planned?


Enable easy use of mermaid.js in Jupyter on the research delight hub

Context
2i2c's Partnerships Team wishes to be able to include mermaid.js diagrams in Jupyter notebooks authored on the research delight hub.
Proposal
Enable easy usage of mermaid diagrams on the research delight hub.
Updates and actions
No response

FYI @damianavila


Add support for retry options (#12)

Added configuration options:

--retry
--retry-delay
--retry-max-time
--retry-connrefused

@ecampidoglio I will create new commit with your proposals. I agree with all of them. Just one explanation: I thought that simple ToString() method called on integer formats that integer number with NumberFormatInfo.NumberGroupSeparator, so 1000 could be formatted as 1 000 (thousand separator). That was the reason why I used Invariant() method and also number 1000 in tests. But I was totally wrong. Sorry fot that.

@ecampidoglio All requested changes are commited.

Merged in 1bb6a24.


pgn reader corner case

[Result "1/2-1/2"]

1. e4 Nc6 2. Nf3 d5 3. exd5 Qxd5 4. Nc3 Qd8 5. Nb1 Qd5 6. Ng1 Qd8 7. Nf3 Qd5 8. Nc3 Qa5 1/2-1/2

when specified plies=16  , cutechess would only parse it to 7. Nf3 while fastchess parses the full 8 moves. the reason cutechess stops prematurely is because 7. Qd5 is a 3fold repetition draw. should we do the same thing for fast-chess?

note that in general (e.g. for analysis) you'd like to be able to go past a 3-fold, it is just for the book lines this might make some sense.

given that we stop at an illegal move https://github.com/Disservin/fast-chess/pull/642, this makes sense to also respect if the game is over and stop just right before


Release v1.0.9

I'm an Ubuntu package maintainer and am looking to upgrade the Ubuntu package to get the latest improvements in for Oracular.
It would help if the latest commit could be made into a release since the last release was a while ago.

Also I am curious if there is any defined release cadence?

Hi,
Unfortunately the Ubuntu Agent has been forked from this repository for a while now. There is a good amount of work that'll be required in order to realign the two agents (for one they use different swap file names, and changing that for whichever OS will require new code to handle the potential existence of an unexpected swap file).
The most recent commit is required for Ubuntu ARM, but has been added here to keep the two repositories from forking further. If you are grabbing this and other improvements made to this agent I would recommend careful testing afterwards. (For example, the testing done on Ubuntu was using the existing Ubuntu agent, with the commit patched in - details more testing details here).
I will still create a new release with the latest commits (Release v1.0.9).
There is no defined release cadence, changes to the agent are limited and there will be a new release with any new feature or bug fix. (It was a miss on this latest commit to not create the new release, thanks for catching it!)

Hi Seth! Thanks for the fast response, and thanks for creating the release.

Unfortunately the Ubuntu Agent has been forked from this repository for a while now.

That's exactly why I am doing the work to update it :)
Ideally I'd like the Ubuntu agent and this one to have no deltas, but your example with the change of /swap to /swap-hibinit is a good example of a delta that would be hard to squash at this point.
As a heads up - I'll see if I can integrate some Ubuntu deltas during my merge this cycle so we don't diverge too much.

Also re: testing, I hate doing things manually, so I test with a script that does everything for me from the commandline for initial testing and let CI/CD take over the testing after.
You might be interested in the script I've made for my personal use (although I'm sure you might have your own, I just love sharing). You can find it here if you are interested with an example of how I use it in the README https://github.com/mitchdz/mitch-awstool

Fix merged in this pull request. Please let me know the timeline for getting the updates into the released version of the agent (specifically Jammy and Focal and this commit that is required for Ubuntu ARM Hibernation support).

Hi, I am closing out this issue as the fix has been merged in the above pull request and the main commit has been merged into canonicals repo. Feel free to reopen if there any more questions/asks.


Optionally preserve whitespace in test cases

Related to #3
What do you think about something like this? This preserves the existing behavior as a default, but adds an option to use a particular delimiter (I picked + but don't feel strongly about it) to signal that you'd like to avoid trimming whitespace at the start/end of the "code" block in the test cases.
Sample test

++++++

file,format
requires,terminal
new,line

++++++

(source_file
  ...
)

Added some tests and marked this ready-for-review.
One thing that occurs to me: maybe it would be better to still trim leading whitespace, while preserving trailing newlines? Add another option? Or should that be the default behavior?
I'm not really sure what's the most ergonomic. There's many file dialects that expect a terminating newline. I'm not aware of any off the top of my head that require leading whitespace.

Thanks for trying to tackle this, and for adding tests!
You have a reasonable approach, but I feel like it might be overly complicated. My default is to do whatever tree-sitter does since that is the original inspiration for pest-test. Looking at the tree-sitter test parser, they trim all the leading whitespace from the source code (as you suggest) and the trailing newline. I think this is the right way to do it - then you can control the trailing whitespace by how many empty lines you leave between the end of the code and the separator line.
If you want to take a stab at that approach I'd be grateful, otherwise I can start from your PR and working on implementing it when I have a chance.

Closing in favor of #6


build: Update the commit message for Requirements upgrade action

Commit message parameter doesn't add a new line with \n so removing that from the commit message.

Change made in PR https://github.com/edx/edx-repo-health/pull/134 so closing this PR.


With GNU C++, line 85 of hpx/config/version.hpp causes link failure when linking application

When I build HPX with GNU C++ and then try to build an application with it, I get unresolved symbol errors coming from line 85 of hpx/config/version.hpp. Everything works when I comment this line out.

I assume (you have not shown the error) this linker error happens whenever HPX was compiled against a different version of Boost than you're trying to use in your application.

Here is the error, from trying to compile a hello world program (same error with octotiger):
CMakeFiles/octotiger_exe.dir/src/main.cpp.o: In function (anonymous namespace)::check_hpx_version()': /work/dmarce1/debug/include/hpx/config/version.hpp:85: undefined reference to  hpx::hpx_check_boost_version_105700'
collect2: error: ld returned 1 exit status
I looked at CMakeCache.txt in the hpx build directory, and it appears to have the correct boost. Its the only boost I've got on the system right now, except the default boost (1.41) which I've never been able to compile against. I will redo the compilation from scratch to make sure.

I get the same error when redoing from scratch, however, I have -D_GLIBCXX_DEBUG enabled. That may have something to do with it. I will recompile without it.

That error means that you are trying to build octotiger with a different boost version than you have used to build HPX. -D_GLIBCXX_DEBUG is unrelated to that.


[FEATURE] Expose metrics of TOP10 app bytes in one shuffle server

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Search before asking

[X] I have searched in the issues and found no similar issues.

Describe the feature
It's better to expose metrics of TOP10 app bytes in one shuffle server, especially when reaching the high watermark of disk.
Motivation
No response
Describe the solution
No response
Additional context
No response
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

cc @jerqi @kaijchen Do you have some suggestion on this?

What can the metrics tell us? Which problems do you want to solve?

What can the metrics tell us? Which problems do you want to solve?

When reaching the high watermark, we could kill the max data size  app to keep other apps stability

I think it's ok to include this metrics if it doen't introduce too much overhead.

I think it's ok to include this metrics if it doen't introduce noticeable overhead.

I think it won’t. The aggregated data of single app used bytes has been collected in current codebase, we just need to expose

Given this is a starter issue, do you mind sharing some key code pointers to help understand the workflow of the shuffle server that collects then observes the app bytes?
I assume it cannot be limited to only TOP 10 since the metrics will change and there always a need to sort?

@zuston could you give me some feedback on my WIP PR to see if I am on the right direction: https://github.com/apache/incubator-uniffle/pull/387


Issues with PA100k.py and AttrDataset.py

Hi! I am trying out SequencePAR in this repository. I followed the steps for preprocessing of PA100k data, however i encountered some problems when I run the training script.
In the dataset_info OrderedDict there is no .attribute feature, .neg_attr_word, .expand_pos_attr_words, .expand_neg_attr_words, may I know what should I do to replace them for PA100k?

So i tried to replace it with .attr_vector and got this error. May I know what is .attribute so that I can replace it correctly?

[neg_attr_word, .expand_pos_attr_words, .expand_neg_attr_words] is useless, you can just comment them out

.attribute refers to a list of attributes, i.e. attr_words

Hello! Thanks for the help. I have tried replacing .attribute with .attr_words. It works, however, I am unable to train the model due to cuda out of memory issues.
Is there any other way that I am able to try sequencePAR via inference??

One can try to use ViT-B to replace the backbone or fp16, which are not implemented in our submission and which will somewhat impair the model performance.


Please Implement Referrer Bypassing?

Description
When I visit key.thundermods.com it redirects me to an ad site if I'm not coming from the specific clicksfly site or whatever, and was wondering if you can implemenet a feature to turn off/on referrer bypass?? I want to bypass this, as recently you have no way of successfully accessing the site. so i can't get the key for my program
Steps to reproduce issue
Open the App, it autoredirects (via referrer script) to ad site.
Details

Are you nativefying a public website? Yes
Feature request? Have you looked at nativefier --help to see if an existing option could fit your needs? Yes
Full nativefier command used to build your app: cd desktop nativefier https://key.thundermods.com
Version of Nativefier (run nativefier --version): v7.6.9
Version of node.js (run node --version): v11.4.0
OS: Windows 7
Error message / stack trace (if any):

<Error message / Stack trace here>

Hi! Thanks for the feedback, but there isn't much we can do with this little information. This project is a hobby with only one maintainer at the wheel, looking at issues and merging the occasional Pull Request during spare time.

Feature requests: do not expect new features (see manpower comment above) and do not expect Nativefier to fill every niche feature provided by a full-blown browser. Still convinced you want something? Build it. If you don't have the resources to build it and are sure you want to file an issue, then take the time to write a concise & clear title / description; I'll leave the issue open and maybe someday someone will take a stab at it.
Bugs reports: do not expect "customer support"-style interaction; we need you to investigate as much as possible before opening an issue: try to narrow it as much as possible, provide the minimum details requested by the issue template, offer intuitions, a reproducible public website & scenario, etc: help us help you 🙂, contributors are not wizards and do not read your mind. Of course, the best you can provide is code, PR very welcome.

With that, closing, but will re-open if you find the time to bring details. Cheers!
Text inspired by uBlock's CONTRIBUTING.md.


Limit WireLib.TriggerInput to avoid stack overflow

Now, same as WireLib.TriggerOutput, WireLib.TriggerInput has a limit of being called 4 times a tick (for each input) to avoid stack overflows.
This fixes #2415

Sounds like this could break some stuff. Why not just fix whatever's outputting too much.

We could try this out on CFC for a week or so. We get a lot of E2 activity.
Would that be helpful?

That would be nice, I think it's fine as it is right now, just need to make sure it doesn't break stuff like Sparky said (which I don't think it would, why would any wire components rely on being executed 4+ times in a single tick? Maybe digiscreens? But you can't even trigger outputs more than 4 times a tick so I dunno how inputs would be triggered faster intentionally.)

I think it looks good too. I'll use this branch on CFC for a few days and let you know if I see any errors that might be related 👍

We've been running this for about three days now. ~2,400 E2s were spawned in that time, and the only wiremod error I've seen is the holoemitter bug that I fixed in another PR - so I'd say this PR is good to go 👍

Yeah I was thinking it's fine, although the issue would likely be with niche wire components that might somehow rely on this behavior. Going to merge anyway, thank you

Right - E2s was the only metric I could get, haha. I figure it's a decent measure of how much use wiremod generally saw during that period.
No problem


High to Very High CPU usage for the first 30 seconds.

Please see the attached screen shot. I have noticed that the CPU usage of the ionic v1 iOS app is remain High to Very high for the first 30 seconds. Is there a way to make sure it will stay low as soon as it's idle? Because it will cause other usage of the app become much slower for the first 30 seconds.

Greetings @MrBokeh!
My sensors indicate that you need to update your issue through our custom issue form. We are now requiring all issues to be submitted this way, to ensure that we have all of the information necessary to fix them as quickly as possible.
Click Here To Update Your Issue
I will have no choice but to close this issue if it is not resubmitted through the form. Please fill out the rest of the form, so that I may use my friendly robot powers to assist you.
Thank you!


[release/v1.5] Migrate from buildah to buildx

What this PR does / why we need it:
This is combined backport of #2807 and #2808 to the release/v1.5 branch.
What type of PR is this?
/kind cleanup
Does this PR introduce a user-facing change? Then add your Release Note here:
Use buildx instead of Buildah to create multi-architecture KubeOne container images.

Documentation:
NONE

/approve
/lgtm


AVRO-2639: add option to write Optional only when fields are nullable

Make sure you have checked all steps below.
Jira

[x] My PR addresses the following issues and references them in the PR title. For example, "AVRO-1234: My Avro PR"

AVRO-2639
In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.

Tests

[x] My PR adds the following unit tests OR does not need testing for this extremely good reason:

testPojoWithOptionalOnlyWhenNullableCreatedTurnedOn

testPojoWithOptionalOnlyWhenNullableCreatedTurnedOnAndGettersReturnOptionalTurnedOff

Commits

[x] My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters (not including Jira issue reference)
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Documentation

[x] In case of new functionality, my PR adds documentation that describes how to use it.

All the public functions and the classes in the PR contain Javadoc that explain what it does

This looks like it implements the same functionality as #817, but that implementation's velocity template seemed a bit cleaner at first glance.
Regardless, is there something blocking this or the other PR? I don't mind helping out.

Would be nice to have an update on this important feature

Thanks for the PR, I like this one. Would be much nicer if we could use the primitive types, as @kan-izh suggests.


travis: remove "gimme.local" script

This patch removes a "gimme.local" script used within the Travis CI
in order to install the correct version of the Go compiler for aarch64
architecture as the version for Travis was broken.
An issue was fixed in the Travis CI itself, therefore the copy of
"gimme.local" is no longer required within rkt source tree. Updated
the configuration of Travis CI accordingly.
Fixes #3555

ok to test

Despite the version of gimme is 1.0.0, it installs Go amd64 compiler.
An output from the arm64 job https://travis-ci.org/coreos/rkt/jobs/194940626:
$ export TARGET=arm64
...
$ gimme version
v1.0.0
$ go version
go version go1.5.4 linux/amd64

The patch indeed in the updated version of gimme: https://github.com/travis-ci/gimme/compare/v0.2.4...v1.0.0

Oh, it's my mistake, it should export an appropriate environment variable as well:
-  - TARGET=amd64
-  - TARGET=arm64
+  - TARGET=amd64 GIMME_ARCH=amd64
+  - TARGET=arm64 GIMME_ARCH=arm64

Finally make things work.

Both Jenkins jobs failed on TestAppUserGroup test, which is unlikely caused by my changes.

Jenkins failures are the usual flakes. Semaphore for some reason didn't pick it up. Do you mind re-pushing?


ytdlp scrape comments and replies no longer work , whats the fixed code now?

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

[X] I'm asking a question and not reporting a bug or requesting a feature
[X] I've looked through the README
[X] I've verified that I have updated yt-dlp to nightly or master (update instructions)
[X] I've searched known issues and the bugtracker for similar questions including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue

Please make sure the question is worded well enough to be understood
import json
import yt_dlp
import os
import re
def sanitize_filename(name, max_length=100):
"""
Sanitize the filename by removing disallowed characters and optionally truncating if necessary.
"""
name = re.sub(r'[\/*?:"<>|]', '', name)
name = name.replace(' ', '_')
if len(name) > max_length:
return name[:max_length]
return name
def download_comments_info_json(url: str, top_comments: str = 'all', replies: str = 'all'):
"""
Fetches comments (and replies, if specified) for the given video URL.
"""
max_comments = ['all'] if top_comments == 'all' else [top_comments, replies]
ydl_opts = {
'getcomments': True,
'skip_download': True,
'writesubtitles': True,
'writeautomaticsub': True,
'extractor_args': {
'youtube': {
'max_comments': max_comments,
'comment_sort': 'top',
}
},
}
with yt_dlp.YoutubeDL(ydl_opts) as ydl:
    info = ydl.extract_info(url, download=False)
    comments_info = {'comments': info.get('comments', [])}
    video_title = info.get('title', 'video')
    return comments_info, video_title

def main(urls_file, base_directory):
"""
Reads video URLs from a file, downloads comments and replies for each, and saves them as JSON files named after the video titles.
"""
# Ensure the base directory exists
if not os.path.exists(base_directory):
os.makedirs(base_directory)
with open(urls_file, 'r') as file:
    urls = file.readlines()

for url in urls:
    url = url.strip()
    comments_info, video_title = download_comments_info_json(url)

    # Sanitize and prepare the filename
    filename = sanitize_filename(video_title) + ".json"
    output_path = os.path.join(base_directory, filename)

    # Save comments information to a JSON file
    with open(output_path, 'wt') as fh:
        json.dump(comments_info, fh, indent=2)
    print(f"Comments and replies saved successfully as {output_path}")

Paths
urls_file = 'C:\Users\v1\Documents\sample.txt'
base_directory = 'C:\Users\v1\Documents\saampleout'
Execute the main function
main(urls_file, base_directory)
this is the result C:\Users\v1\PycharmProjects\pythonProject.venv\Scripts\python.exe C:\Users\v1\PycharmProjects\pythonProject\comme.py
[youtube] Extracting URL: https://www.youtube.com/watch?v=YNechg4MqXI
[youtube] YNechg4MqXI: Downloading webpage
[youtube] YNechg4MqXI: Downloading ios player API JSON
[youtube] YNechg4MqXI: Downloading android player API JSON
WARNING: [youtube] Skipping player responses from android clients (got player responses for video "aQvGIIdgFDM" instead of "YNechg4MqXI")
[youtube] YNechg4MqXI: Downloading player 7ee5b648
[youtube] YNechg4MqXI: Downloading m3u8 information
[info] YNechg4MqXI: Downloading subtitles: en-uYU-mmqFLq8
[youtube] Downloading comment section API JSON
[youtube] Downloading ~211 comments
[youtube] Sorting comments by newest first
[youtube] Downloading comment API JSON page 1 (0/~211)
[youtube] Downloading comment API JSON page 2 (0/~211)
[youtube] Downloading comment API JSON page 3 (0/~211)
[youtube] Downloading comment API JSON page 4 (0/~211)
[youtube] Downloading comment API JSON page 5 (0/~211)
[youtube] Downloading comment API JSON page 6 (0/~211)
[youtube] Downloading comment API JSON page 7 (0/~211)
[youtube] Extracted 0 comments
Comments and replies saved successfully as C:\Users\v1\Documents\saampleout\George_Washington_statue_draped_in_Palestinian_flag_on_DC_campus.json
[youtube] Extracting URL: https://www.youtube.com/watch?v=Ip9I9tUZU1c
[youtube] Ip9I9tUZU1c: Downloading webpage
[youtube] Ip9I9tUZU1c: Downloading ios player API JSON
[youtube] Ip9I9tUZU1c: Downloading android player API JSON
WARNING: [youtube] Skipping player responses from android clients (got player responses for video "aQvGIIdgFDM" instead of "Ip9I9tUZU1c")
[youtube] Ip9I9tUZU1c: Downloading m3u8 information
[info] Ip9I9tUZU1c: Downloading subtitles: en-uYU-mmqFLq8
[youtube] Downloading comment section API JSON
[youtube] Downloading ~802 comments
[youtube] Sorting comments by newest first
[youtube] Downloading comment API JSON page 1 (0/~802)
[youtube] Downloading comment API JSON page 2 (0/~802)
[youtube] Downloading comment API JSON page 3 (0/~802)
[youtube] Downloading comment API JSON page 4 (0/~802)
[youtube] Downloading comment API JSON page 5 (0/~802)
[youtube] Downloading comment API JSON page 6 (0/~802)
[youtube] Downloading comment API JSON page 7 (0/~802)
[youtube] Downloading comment API JSON page 8 (0/~802)
[youtube] Downloading comment API JSON page 9 (0/~802)
[youtube] Downloading comment API JSON page 10 (0/~802)
[youtube] Downloading comment API JSON page 11 (0/~802)
[youtube] Downloading comment API JSON page 12 (0/~802)
[youtube] Downloading comment API JSON page 13 (0/~802)
[youtube] Downloading comment API JSON page 14 (0/~802)
[youtube] Downloading comment API JSON page 15 (0/~802)
[youtube] Downloading comment API JSON page 16 (0/~802)
[youtube] Downloading comment API JSON page 17 (0/~802)
[youtube] Downloading comment API JSON page 18 (0/~802)
[youtube] Downloading comment API JSON page 19 (0/~802)
[youtube] Downloading comment API JSON page 20 (0/~802)
[youtube] Downloading comment API JSON page 21 (0/~802)
[youtube] Downloading comment API JSON page 22 (0/~802)
[youtube] Downloading comment API JSON page 23 (0/~802)
[youtube] Extracted 0 comments
Comments and replies saved successfully as C:\Users\v1\Documents\saampleout\Columbia_protesters_smash_windows,hang'intifada'_banner.json
Process finished with exit code 0
it extracted 0 please help whats the updated code?
Provide verbose output that clearly demonstrates the problem

[ ] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[ ] If using API, add 'verbose': True to YoutubeDL params instead
[ ] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
No response

here the whole code in txt the txt uyou will attach to path contains yt video link and paath where the json files will be saved
code comments replies.txt

https://github.com/yt-dlp/yt-dlp/pull/9775

@pukkandan is there a new code? diodnt quite understood the last reply in https://github.com/yt-dlp/yt-dlp/pull/9775

The issue is being worked on. Subscribe to the linked PR to be notified when it's merged


Devtooling 286: Fixed Consistency Checker failing test

Did you commit your changes for the unit test fix? The test is still failing here


Translated text keep new lines ?

Hey,
Is there a way to keep new lines /  in the translated text?
Or did i miss something?

Keeping new lines depends on translator engine. As I know, Google Translate usually keeps them, but Deepl not. That's not an responsibility of core engine for now.

@janvarev Thanks for the reply, as i was testing it i found out that if you use the offline translator we can replace newlines with 's and that will keep the text nice with the offline engine!.
So for anyone who is facing this issue just regex new lines and replace them with  then if you do not want  but new lines reverse the regex the translator will keep the .


Add Network framework and Streamerbot support

This PR adds a complete Network framework with connection support to Streamerbot. For now, this is done simply by reading/writing every few seconds to inbound and outbound files (Json). Other connection types may become available in the future as they become officially (and properly) integrated into Bizhawk itself.
The initial goal of this feature is to allow Twitch chat to communicate with the Tracker via chat commands, or by redeeming a channel reward to change something about the tracker, like picking a starter ball or changing the Tracker theme.
Setup
To user first needs to setup the Network through the Steam Connect screen found in the Streaming settings menu. This setup process will be outlined in a wiki page which I'll built out later. Once setup, the tracker will automatically connect when new games are loaded, and disconnect when the tracker resets/closes down.

Tracker Wiki: Stream Connect Guide

Network Framework Details

The Network class handles setting up connections, sending/receiving from the connection, and closing down the connections when necessary.
The RequestHandler class manages incoming "Requests", either sent from Streamerbot or a Tracker extension, processes those requests, and creates "Responses" to send back. A Request needs a matching "Event" to determine how to process it and fulfill (complete).
The EventHandler class contains all known "Events" that can be triggered. The type of events are chat commands and channel rewards, as well as internal game triggers which are currently only available through custom Tracker extensions. There are also core events with are managed here, such as Start, Update, and Stop events for establishing and syncing Network communication.
To properly communicate with Streamerbot, C# code is required on be run on Streamerbot itself. This includes event entry points, such as chat commands and channel redeems, as well as all the other internal processes needed for sending requests and receiving responses.
For external developers, I've added support to trigger on in-game conditions and send notifications to Streamerbot. A developer can build a Tracker extension which adds a new Game Event, then use Tracker code to determine when to trigger the event, which will send a request/response to Streamerbot. Then the developer can define their own Streamerbot client triggers to receive these notifications and respond accordingly, perhaps with a fun sound effect or game overlay update. Developing this will be explained in more detail on the Stream Connect wiki page.

Connection Modes

Text (Json) - Fully supported
WebSockets - Not properly supported by Bizhawk
HTTP - Not properly supported by Bizhawk

New Core Files

/network/Network.lua
/network/RequestHandler.lua
/network/EventHandler.lua
/screens/StreamConnectOverlay.lua
/Json.lua (from a 3rd-party source)
/Debug/Tracker-StreamerBot.cs (for streamerbot only)

Other Notes

I accidentally added the Utils.isNilOrEmpty() helper function in this PR instead of the dev branch, which is why so many files are affected by this PR. Whoops!
Ignore the Tracker version update, this PR will release with a new major version change anyway.
Currently there are no Language resources for most of the new texts that were added. I'll add this in after any other code changes need to be made.
I still want a "Delay Once" button for the pick ball redeems, that appears on the random ball picker. This is helpful for cases where a favorite or something pops up, and the user's ball redeem needs to be temporarily skipped.
I still want to add a !trainers chat command for looking up what trainers the player still has left to fight. Having figured out the logic for this one yet. Will likely delay for a future patch release.
There are a handful of other features and support for customizations, so let me know if something feels a bit off.

How To Test

Install Streamerbot and log into the application
On Stream Connect, click "Get Code" and follow instructions for importing it
Then set the connection data folder through the tracker to the data folder found in Streamerbot.
Click Connect!

I'm interested in a better way to explain how to connect purely through the Tracker UI if at all possible. As of now, I plan to have instructions on the wiki. I think only step 3 is the hard part to explain that I'm stuck on.

I've completed the Stream Connect Guide wiki page if you want to check it out. The developers section is has a few links that won't work yet until this feature is officially released.

Finished adding in the resource files for this feature, as well as fixed a tiny bug with HMs displaying as TMs in chat output.


Renders incorrectly on terminal Vim 7.4, t_Co=256, set termguicolors

environment

vim: 7.4 (2013 Aug 10, compiled Jul 19 2016 15:42:57)
vim-airline: Aug 29th 2016
OS: Mac OS
if you are using terminal:
terminal: iTerm and Hyperterm
$TERM variable: xterm-256color
color configuration (:set t_Co?): 256

actual behavior
Renders incorrectly
expected behavior
Renders correctly.
Here's what I have in my .vimrc:
set t_Co=256

" From https://github.com/rakr/vim-one
"Use 24-bit (true-color) mode in Vim/Neovim when outside tmux.
"If you're using tmux version 2.2 or later, you can remove the outermost $TMUX check and use tmux's 24-bit color support
"(see < http://sunaku.github.io/tmux-24bit-color.html#usage > for more information.)
if (empty($TMUX))
  if (has("nvim"))
  "For Neovim 0.1.3 and 0.1.4 < https://github.com/neovim/neovim/pull/2198 >
  let $NVIM_TUI_ENABLE_TRUE_COLOR=1
  endif
  "For Neovim > 0.1.5 and Vim > patch 7.4.1799 < https://github.com/vim/vim/commit/61be73bb0f965a895bfb064ea3e55476ac175162 >
  "Based on Vim patch 7.4.1770 (`guicolors` option) < https://github.com/vim/vim/commit/8a633e3427b47286869aa4b96f2bfc1fe65b25cd >
  " < https://github.com/neovim/neovim/wiki/Following-HEAD#20160511 >
  if (has("termguicolors"))
    set termguicolors
  endif
endif

Here's what Vim 7.4 looks like:

Here's what neoVim looks like.

Why does neovim work, but Vim does not?

Am 2016-08-29 12:01, schrieb Jordan W:

ENVIRONMENT

vim: 7.4 (2013 Aug 10, compiled Jul 19 2016 15:42:57)
vim-airline: Aug 29th 2016
OS: Mac OS if you are using terminal:
terminal: iTerm and Hyperterm
$TERM variable: xterm-256color
color configuration (:set t_Co?): 256

ACTUAL BEHAVIOR
Renders incorrectly
EXPECTED BEHAVIOR
Renders correctly.
Here's what I have in my .vimrc:
set t_Co=256
" From https://github.com/rakr/vim-one
"Use 24-bit (true-color) mode in Vim/Neovim when outside tmux.
"If you're using tmux version 2.2 or later, you can remove the
outermost $TMUX check and use tmux's 24-bit color support
"(see < http://sunaku.github.io/tmux-24bit-color.html#usage > for more
information.)
if (empty($TMUX))
if (has("nvim"))
"For Neovim 0.1.3 and 0.1.4 <
https://github.com/neovim/neovim/pull/2198 >
let $NVIM_TUI_ENABLE_TRUE_COLOR=1
endif
"For Neovim > 0.1.5 and Vim > patch 7.4.1799 <
https://github.com/vim/vim/commit/61be73bb0f965a895bfb064ea3e55476ac175162

"Based on Vim patch 7.4.1770 (guicolors option) <
https://github.com/vim/vim/commit/8a633e3427b47286869aa4b96f2bfc1fe65b25cd

" < https://github.com/neovim/neovim/wiki/Following-HEAD#20160511 >
if (has("termguicolors"))
set termguicolors
endif
endif
Here's what Vim 7.4 looks like:
[1]
Here's what neoVim looks like.
[2]
Why does neovim work, but Vim does not?

That is most likely a Vim bug and nothing vim-airline can influence.

What leads you to believe this?

Am 2016-08-29 12:07, schrieb Jordan W:

What leads you to believe this?

There are already some Vim issues about termguicolors difference.
Unfortunately, nobody seems to be able to reproduce/fix the issue so
far.
Also this is about a feature, which is a relative recent addition to Vim
and vim-airline hasn't changed in that area for a while.

I see termguicolors support in Airline, so I was curious if someone could confirm that at one point it was working.

Am 2016-08-29 23:50, schrieb Jordan W:

I see termguicolors support in Airline, so I was curious if someone
could confirm that at one point it was working.

What is your Vim version? Also is this reproducable on linux? I don't
have a mac to test.

using the same theme, mine renders correctly on Mac OS X El Capitan using
following vim on newest iTerm2
VIM - Vi IMproved 7.4 (2013 Aug 10, compiled Aug 21 2016 15:01:09)
MacOS X (unix) version
Included patches: 1-2235
Compiled by Homebrew
Huge version without GUI.  Features included (+) or not (-):
+acl             +file_in_path    +mouse_sgr       +tag_old_static
+arabic          +find_in_path    -mouse_sysmouse  -tag_any_white
+autocmd         +float           +mouse_urxvt     -tcl
-balloon_eval    +folding         +mouse_xterm     +termguicolors
-browse          -footer          +multi_byte      +terminfo
++builtin_terms  +fork()          +multi_lang      +termresponse
+byte_offset     -gettext         -mzscheme        +textobjects
+channel         -hangul_input    +netbeans_intg   +timers
+cindent         +iconv           +num64           +title
-clientserver    +insert_expand   +packages        -toolbar
+clipboard       +job             +path_extra      +user_commands
+cmdline_compl   +jumplist        +perl            +vertsplit
+cmdline_hist    +keymap          +persistent_undo +virtualedit
+cmdline_info    +lambda          +postscript      +visual
+comments        +langmap         +printer         +visualextra
+conceal         +libcall         +profile         +viminfo
+cryptv          +linebreak       +python          +vreplace
+cscope          +lispindent      -python3         +wildignore
+cursorbind      +listcmds        +quickfix        +wildmenu
+cursorshape     +localmap        +reltime         +windows
+dialog_con      -lua             +rightleft       +writebackup
+diff            +menu            +ruby            -X11
+digraphs        +mksession       +scrollbind      -xfontset
-dnd             +modify_fname    +signs           -xim
-ebcdic          +mouse           +smartindent     -xpm
+emacs_tags      -mouseshape      +startuptime     -xsmp
+eval            +mouse_dec       +statusline      -xterm_clipboard
+ex_extra        -mouse_gpm       -sun_workshop    -xterm_save
+extra_search    -mouse_jsbterm   +syntax
+farsi           +mouse_netterm   +tag_binary
On Tue, Aug 30, 2016 at 8:54 AM, Christian Brabandt <
<EMAIL_ADDRESS>wrote:

Am 2016-08-29 23:50, schrieb Jordan W:

I see termguicolors support in Airline, so I was curious if someone
could confirm that at one point it was working.

What is your Vim version? Also is this reproducable on linux? I don't
have a mac to test.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/vim-airline/vim-airline/issues/1244#issuecomment-243446128,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABaFwu7Ez-6WZP_SxIBiJonNmm0irmy5ks5qlDYpgaJpZM4JvU3D
.

--
rd

i am closing this, as this does not seem to be a airline issue

I do have the exact same issue...

And now that I just upgraded to the latest neovim, the errors happens there as well! This must be something to do with recent versions of vim/neovim that cause the issue with Airline.

Is it possible to reopen this issue?

Am 2016-10-11 08:34, schrieb Yves Chevallier:

Is it possible to reopen this issue?

As far as i understand the problem, it's not an airline issue, but
either a vim or neovim bug.

@chrisbra I tried other statusline plugins which use the same font glyphs etc, and the problem did not occur there. It appears to be airline, on the very newest vims.

considering, that this only happens on :set termguicolors hints, that this is no problem with airline.

https://github.com/vim/vim/issues/1157

So, for some reason when I set the following in my .vimrc, airline then works:
" set Vim-specific sequences for RGB colors
let &t_8f = "\<Esc>[38;2;%lu;%lu;%lum"
let &t_8b = "\<Esc>[48;2;%lu;%lu;%lum"

I got that snippet from https://github.com/vim/vim/issues/993
I don't know what's going on, or if I've somehow disabled termguicolors by enabling those settings. My suspicion is that some of the built in airline themes aren't updated to support termguicolors.

Following chrisbra comment, the issue was in fact an airline issue. It is now solved in ef3746d and vim-airline now works in true-colors with cygwin and mintty.


perf: do not lowercase returned values

We're sacrificing a lot of performance just to lowercase the string values, which is not ideal as it's not always necessary. Here are some numbers demonstrating the difference in performance:
With .toLowerCase():
string literal     x 31,742,089 ops/sec ±6.85% (70 runs sampled)
array literal      x 12,508,545 ops/sec ±3.86% (67 runs sampled)
boolean literal    x 33,502,011 ops/sec ±3.58% (66 runs sampled)
object literal     x 1,178,855 ops/sec ±2.06% (79 runs sampled)
object from null   x 3,356,249 ops/sec ±2.03% (81 runs sampled)
regex literal      x 2,738,477 ops/sec ±1.95% (80 runs sampled)
number literal     x 30,119,855 ops/sec ±2.04% (78 runs sampled)
promise            x 7,260,140 ops/sec ±1.79% (81 runs sampled)
null               x 34,793,675 ops/sec ±1.58% (84 runs sampled)
undefined          x 31,537,228 ops/sec ±1.77% (81 runs sampled)
function           x 35,571,426 ops/sec ±1.69% (82 runs sampled)
buffer             x 5,468,033 ops/sec ±2.19% (83 runs sampled)
date               x 4,402,505 ops/sec ±1.47% (83 runs sampled)
error              x 932,833 ops/sec ±2.16% (80 runs sampled)
map                x 7,298,659 ops/sec ±1.64% (81 runs sampled)
regex constructor  x 3,042,918 ops/sec ±2.28% (80 runs sampled)
set                x 6,075,985 ops/sec ±1.91% (82 runs sampled)
string constructor x 1,077,682 ops/sec ±2.32% (82 runs sampled)
weakmap            x 5,949,120 ops/sec ±1.99% (83 runs sampled)
weakset            x 5,574,794 ops/sec ±1.89% (78 runs sampled)
arguments          x 1,065,249 ops/sec ±1.37% (83 runs sampled)
arrow function     x 28,776,605 ops/sec ±1.61% (81 runs sampled)
generator function x 34,581,601 ops/sec ±2.11% (79 runs sampled)
Float64Array       x 5,306,129 ops/sec ±1.64% (83 runs sampled)
Float32Array       x 5,635,134 ops/sec ±1.37% (84 runs sampled)
Uint32Array        x 5,584,889 ops/sec ±1.72% (83 runs sampled)
Uint16Array        x 5,024,561 ops/sec ±1.40% (85 runs sampled)
Uint8Array         x 5,032,024 ops/sec ±1.93% (81 runs sampled)
Int32Array         x 6,261,825 ops/sec ±2.21% (78 runs sampled)
Int16Array         x 6,380,182 ops/sec ±1.68% (81 runs sampled)
Int8Array          x 6,156,981 ops/sec ±2.27% (78 runs sampled)
Uint8ClampedArray  x 5,911,482 ops/sec ±1.85% (80 runs sampled)
DataView           x 7,308,534 ops/sec ±1.83% (78 runs sampled)

Without .toLowerCase():
string literal     x 47,522,814 ops/sec ±1.93% (80 runs sampled)
array literal      x 18,729,054 ops/sec ±1.88% (81 runs sampled)
boolean literal    x 33,037,749 ops/sec ±2.17% (83 runs sampled)
object literal     x 1,536,749 ops/sec ±2.99% (80 runs sampled)
object from null   x 3,148,007 ops/sec ±2.39% (83 runs sampled)
regex literal      x 3,373,384 ops/sec ±1.82% (82 runs sampled)
number literal     x 33,096,650 ops/sec ±1.43% (85 runs sampled)
promise            x 13,416,347 ops/sec ±1.62% (80 runs sampled)
null               x 32,376,000 ops/sec ±1.70% (82 runs sampled)
undefined          x 30,734,992 ops/sec ±1.60% (82 runs sampled)
function           x 35,044,726 ops/sec ±1.87% (81 runs sampled)
buffer             x 10,831,858 ops/sec ±1.93% (82 runs sampled)
date               x 4,511,245 ops/sec ±1.99% (81 runs sampled)
error              x 836,481 ops/sec ±2.31% (79 runs sampled)
map                x 14,289,403 ops/sec ±1.72% (83 runs sampled)
regex constructor  x 3,764,635 ops/sec ±2.69% (78 runs sampled)
set                x 12,494,254 ops/sec ±1.95% (82 runs sampled)
string constructor x 1,282,794 ops/sec ±1.92% (80 runs sampled)
weakmap            x 15,418,935 ops/sec ±1.84% (79 runs sampled)
weakset            x 14,768,545 ops/sec ±1.60% (82 runs sampled)
arguments          x 993,064 ops/sec ±1.88% (81 runs sampled)
arrow function     x 30,511,620 ops/sec ±2.07% (79 runs sampled)
generator function x 35,487,241 ops/sec ±1.53% (81 runs sampled)
Float64Array       x 9,666,668 ops/sec ±1.75% (79 runs sampled)
Float32Array       x 9,286,265 ops/sec ±1.66% (81 runs sampled)
Uint32Array        x 9,352,490 ops/sec ±1.37% (82 runs sampled)
Uint16Array        x 8,404,961 ops/sec ±2.03% (80 runs sampled)
Uint8Array         x 8,635,152 ops/sec ±1.81% (80 runs sampled)
Int32Array         x 10,569,543 ops/sec ±1.67% (82 runs sampled)
Int16Array         x 9,444,358 ops/sec ±1.53% (80 runs sampled)
Int8Array          x 7,235,174 ops/sec ±2.43% (76 runs sampled)
Uint8ClampedArray  x 8,055,645 ops/sec ±1.67% (82 runs sampled)
DataView           x 15,014,475 ops/sec ±2.08% (79 runs sampled)

BREAKING CHANGE:
All strings will no longer be lowercase. If you want them in lowercase,
simply add .toLowerCase() to the return value.
@meeber @lucasfcosta this should be released as v3.0.0 due to BREAKING CHANGE: in the commit.

LGTM! I agree with that, it should be a user's responsibility to sanitize the output of this module.
Also, when committing a breaking change you just gotta add BREAKING CHANGE: <description> to the commit right?

@lucasfcosta yup indeed. I've added that to the commit, quite far down. IMO it's not the most ideal as its easy to forget/mess up and release a patch/feature version instead. I suppose we'll just have to now start scrutinising commit messages too.

@meeber care to review?

At dinner but will review in about an hour :D

LGTM


fix(components): popover closing issue in autocomplete with open modal

closes: #2475
closes: #2082
closes: #1987
closes: #2143
closes: #2247
closes: #2510

📝 Description

revised shouldCloseOnInteractOutside in FreeSoloPopover
some people suggested to set allowsCustomValue to true. This could temporarily make the popover close because it makes the inputRef.current not to blur so that Autocomplete won't be re-rendered with isOpen = true. However, this won't blur the input if the selection is empty (i.e. it'll stay focused).

https://github.com/nextui-org/nextui/blob/2894aecca1a2ef0dfb3066b9b8df24ce48c99dae/packages/components/autocomplete/src/use-autocomplete.ts#L287-L291
⛳ Current behavior (updates)
The popover can't be closed with open modal. See the above issue #2475 for the demo.
🚀 New behavior
nextui-pr2494-demo.webm
💣 Is this a breaking change (Yes/No):

No
📝 Additional Information

Summary by CodeRabbit

Bug Fixes

Fixed an issue in the "@nextui-org/popover" package where the popover would incorrectly close when using autocomplete with an open modal.

Still broken. Are there any workarounds?
https://github.com/nextui-org/nextui/assets/8882611/607610a8-94cb-43ff-b150-57c7017c2e26
browsers:
Edge: Version 122.0.2365.113 (Official build) stable app, extended channel (64-bit)
Chrome: Version 122.0.6261.156 (Official Build) extended (64-bit)
package.json:
"dependencies": {
    "@heroicons/react": "^2.1.3",
    "@nextui-org/react": "^2.2.10",
    "@reduxjs/toolkit": "^2.2.3",
    "firebase": "^10.10.0",
    "framer-motion": "^11.0.24",
    "next-themes": "^0.3.0",
    "react": "^18.2.0",
    "react-dom": "^18.2.0",
    "react-redux": "^9.1.0",
    "react-router-dom": "^6.22.3",
    "react-use-wizard": "^2.3.0"
  },
  "devDependencies": {
    "@types/react": "^18.2.74",
    "@types/react-dom": "^18.2.23",
    "@typescript-eslint/eslint-plugin": "^7.5.0",
    "@typescript-eslint/parser": "^7.5.0",
    "@vitejs/plugin-react": "^4.2.1",
    "autoprefixer": "^10.4.19",
    "eslint": "^8.57.0",
    "eslint-plugin-react-hooks": "^4.6.0",
    "eslint-plugin-react-refresh": "^0.4.6",
    "openapi-typescript": "^6.7.5",
    "postcss": "^8.4.38",
    "tailwindcss": "^3.4.3",
    "typescript": "^5.4.3",
    "vite": "^5.2.7"
  }

@Tosters We haven't released that yet - it should be in the next release.

@wingkwong when is the next release?

@Hervinho target on April 15. See here.

I updated to the latest version and it still seems broken


Added fix to detect EOF error on queries and retry with plain text

https://github.com/lxc/lxd/issues/11388
Added fix to detect EOF error on queries and retry HTTP request with plain text encoding.

@jeffreyxie0615 you need a quick rebase on your branch to get rid of that Merge commit and also edit your commit message to include the required Signed-off-by: Firstname Lastname <email> line (see CONTRIBUTING.md).

jenkins: test this please


exporting spectral match results to excel

mzmine 3.9.0
macOS Sonoma 14.1.2 (23B92)
When I view 'Spectral DB search results' I see the results I want. For publication, I need to replot the mirror plot, so I'm trying to export the mirror plot data. When I export data to excel, the excel table is completely empty. Any suggestions?
<img width="844" alt="Screenshot 2024-01-11 at 2 18 13 PM" src="https://github.com/mzmine/mzmine3/assets/133689959/59e0

c7ed-f0a5-485f-83bd-aca3911f1c32">

I would export the graphics to PDF and then open in illustrator.
You can also export all matches as graphics in one go using an export module.

Thanks for your reply! Yes, that does work as a backup. I did try to replicate everything on a windows machine and get the same results. Just a blank excel file when I export as excel file.
I assume the export normally works? Should I perhaps try an earlier version of mzmine?
For some reason when I import the PDF in illustrator, information is lost so I have to use EPS which is much less easy to work with
It would be a great feature to be able to export the top spectral hit data for every metabolite.

It would be a great feature to be able to export the top spectral hit data for every metabolite.

You can use the "Feature list methods -> Feature list export -> Graphics -> Annotations export" for that.


Support tearing down the network

It would be good if the scripts could also support the cleanup of the network.  Particularly interesting for CI/CD use cases.  Ideally for both Docker and IBM Blockchain Platform.

The standard Ansible way for this is to have a state parameter, and perform the right action based on the current state and the desired state (e.g. present, absent, stopped).

Correct.  However, this has to be added to the playbook and needs to make its way to the tasks.
Something like:
      - &Org2
        msp:
          id: "Org2MSP"
          admin:
            identity: "org2Admin"
            secret: "org2Adminpw"
          ibp:
            display_name: "Org2 MSP"
          state: absent

which results in the deletion of the MSP and all underlying infra? (CA, Peers, etc...)?

@ldesrosi yep, that's what I was thinking of. I think we need the ability to have it fine grained as well, so users can delete individual components if they want to.

The PR #69 adds a top-level state property that allows a user to teardown the entire network. We need to expand this so that the state property is available on:

Individual organizations
Individual components (CAs, peers, orderers)

It may not make sense to offer a state on a channel, as these cannot be removed. The same applies for smart contracts, as these cannot be undeployed.


integration: Upstash Chat Store

Description
This PR adds Upstash Chat Store integration that works over HTTP. It uses Upstash Redis under the hood.
New Package?
Did I fill in the tool.llamahub section in the pyproject.toml and provide a detailed README.md for my new integration or package?

[x] Yes
[ ] No

Version Bump?
Did I bump the version in the pyproject.toml file of the package I am updating? (Except for the llama-index-core package)

[x] Yes
[ ] No

Type of Change
Please delete options that are not relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

[x] Added new unit/integration tests
[ ] Added new notebook (that tests end-to-end)
[ ] I stared at the code and made sure it makes sense

Suggested Checklist:

[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have added Google Colab support for the newly added notebooks.
[ ] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[x] I ran make format; make lint to appease the lint gods

@fahreddinozcan the tests seem to be failing, can you take a look?

Closing this to update with a new PR.


Proposed Recipe for GridRad-Severe

Source Dataset

GridRad is a gridded mosaic of Level II NEXRAD radar data across the CONUS, which is a very useful observational analysis product for studies of convective weather and precipitation. While the hourly fields of GridRad are certainly something that it would be useful to have a Pangeo Forge recipe for as well (I could write that up at a later date), my current research needs the high-temporal-frequency GridRad-Severe dataset. So, I figured if I'm munging the netCDFs to zarr anyway to have easier sample selection across times, I mind as well try it out with Pangeo Forge and maybe let other people benefit as well!

Link to the website / online documentation for the data: http://gridrad.org/data.html and https://rda.ucar.edu/datasets/ds841.6/
The file format (e.g. netCDF, csv): netCDF with non-CF-compliant compression-by-gathering
How are the source files organized? (e.g. one file per day): one file per analysis time (every 5 minutes) in directories separated by year and then 24-hr periods starting at 12 UTC ("convective day")
How are the source files accessed (e.g. FTP): HTTP, see belowdef make_url(time):
    volume = (time - pd.Timedelta(12, "H")).floor("D")
    volume_str = volume.strftime("%Y%m%d")
    time_str = time.strftime("%Y%m%dT%H%M%SZ")
    return (
        "https://rda.ucar.edu/data/OS/ds841.6/volumes/"
        f"{volume_str[:4]}/{volume_str}/nexrad_3d_v4_2_{time_str}.nc"
    )

Any special steps required to access the data (e.g. password required)

Session auth (username and password) with NCAR RDA (this appears to cause issues with fsspec in my initial attempts, however)

Dataset size...the raw files total 20.14 TB, and given the "dense-ification" needed to have alignment in the full zarr store, I think the final output would be lucky to be below 30 TB (given good compressor choice). Given the practical limits of resource availability, this may not be something that can run in full in the existing Bakery?
Transformation / Alignment / Merging

There's some pretty substantial process_input transformation needed given the input files' differing domains (luckily, all are aligned subsets of the same parent):
import numpy as np
import sparse
import xarray as xr

Latitude_full = np.arange(24, 50, 1/48) + (1/96)
Longitude_full = np.arange(-125 % 360, -66 % 360, 1/48) + (1/96)

keep_attrs = {
    'Description', 'Analysis_time_window', 'Algorithm_version', 'Algorithm_description',
    'Processing_Software', 'NOAA_toolkit_root', 'Grid_scheme', 'Data_source',
    'Data_source_URL', 'Authors', 'Project_sponsor', 'Project_name'
}
new_attrs = {
    'Edited_by': 'Pangeo Forge',
    'Storage_scheme': 'zarr, blosc zstd compressed',
    'Name': 'GridRad Severe - Dense Zarr for Pangeo Forge'
}

def gridrad_to_full_dense(ds, filename):
    # Some oft-used quantities
    nz = ds.Altitude.shape[0]
    ny_source = ds.Latitude.shape[0]
    ny_out = Latitude_full.shape[0]
    nx_source = ds.Longitude.shape[0]
    nx_out = Longitude_full.shape[0]

    # Prep the output dataset, ready for data var looping
    new_ds_template = {
        'data_vars': {},
        'coords': {
            'time': ds['time'].variable,
            'Altitude': ds['Altitude'].variable,
            'Latitude': xr.Variable(('Latitude',), Latitude_full, attrs=ds['Latitude'].attrs),
            'Longitude': xr.Variable(('Longitude',), Longitude_full, attrs=ds['Longitude'].attrs),
        },
        'attrs': {
            **new_attrs,
            **{k: v for k, v in ds.attrs.items() if k in keep_attrs}
        }
    }
    # We don't have full grids (usually), so the starting point in the full range can be used
    # as an adjustment index (end only matters for 3D Nrad* padding, and is computed below
    # when used)
    Latitude_offset = np.argmin(np.abs(Latitude_full - ds.Latitude.values[0]))
    Longitude_offset = np.argmin(np.abs(Longitude_full - ds.Longitude.values[0]))

    # Nradobs and Nradecho are already on 3D, so we just need boundary padding
    for nrad_name in ('Nradobs', 'Nradecho'):
        nrad_new = np.zeros((nz, ny_out, nx_out), dtype=ds[nrad_name].dtype)
        nrad_new[
            :,
            Latitude_offset:(Latitude_offset + ny_source),
            Longitude_offset:(Longitude_offset + nx_source)
        ] = ds[nrad_name].values
        new_ds_template['data_vars'][nrad_name] = xr.Variable(
            ds[nrad_name].dims, nrad_new, ds[nrad_name].attrs
        )

    # All others are "compressed by gathering", so we need to use sparse.COO to re-orient in
    # full dense 3D space
    source_index = ds['index'].values
    index_coords_out = np.stack(
        [
            np.mod(np.floor_divide(source_index, ny_source * nx_source), nz),
            np.mod(np.floor_divide(source_index, nx_source), ny_source) + Latitude_offset,
            np.mod(source_index, nx_source) + Longitude_offset
        ],
        axis=0
    )
    for varname in ds.data_vars:
        if varname != 'index' and 'Index' in ds[varname].dims:
            new_ds_template['data_vars'][varname] = xr.Variable(
                ('Altitude', 'Latitude', 'Longitude'),
                sparse.COO(
                    index_coords_out,
                    ds[varname].values,
                    shape=(nz, ny_out, nx_out),
                    fill_value=ds[varname].encoding.get('missing_value', np.nan)
                ).todense(),
                ds[varname].attrs
            )

    return xr.Dataset(**new_ds_template)

However, after this transformation, then they simply need to be concatenated along time!
Output Dataset

Output as zarr; only chunked in time and Altitude (fully contiguous in Latitude and Longitude). Because of the very high sparsity of the data, a good compressor choice will be critical. In similar radar mosaic applications, I've stuck with blosc zstd by default, but I haven't explored if that's optimal or not.

I almost was able to get through the tutorial and be able to submit a PR for this, but https://github.com/fsspec/filesystem_spec/discussions/918 is a blocker. Plus, the practical considerations of dataset size are probably a hindrance too!

Thanks for this contribution, @jthielen!
I've posed a possible solution to the fsspec question in https://github.com/fsspec/filesystem_spec/discussions/918#discussioncomment-2325001. This approach has worked for me to authenticate with other HTTP servers, so probably worth a try, though of course is not guaranteed to be the solution here.

Based on the help received in https://github.com/fsspec/filesystem_spec/discussions/918 (key takeaway: use aiohttp and client_kwargs, not requests), I was able to successfully test a local pruned recipe! Demo notebook here: https://nbviewer.org/gist/jthielen/d010706fc681c4f5b991666cd7dd7e69

Awesome! So now running this on Pangeo Forge Cloud is only blocked by our not yet having a mechanism for recipe contributors to securely pass secrets to the bakery. @rabernat, are you still thinking https://github.com/pangeo-forge/roadmap/pull/36 is the right way to go?


use refined fork with less Typeable; +Refined1

Rationale:

I really don't think I should be Showing as much as I do into errors
I am throwing around way too many Typeable contexts where I feel I shouldn't need to - the refined lib has Typeable contexts for all the logical predicates!...
Refined1 would give me nicer behaviour and less boilerplate, and feels useful in general. (And I see the pattern everywhere.)

Irrelevant after moving to my refined rewrite, rerefined.


Cannot make it to work with go modules

Hello,
quick note: I'm go newbie and I've never encountered similar error with other go packages.
I'm trying to use this package using go modules (go.mod). Unfortunately, it seems like only color.go file is visible. Here are my project files:
go.mod:
module goasmf

go 1.12

require github.com/dennwc/dom v0.3.0

main.go:
package goasmf

import (
    "fmt"
    "github.com/dennwc/dom"
)

func main() {
    println("Testing WebAsm!")
    var color dom.Color = "red"
    dom.GetWindow()
}

go build output:
tomek$ go build
# goasmf
./main.go:13:2: undefined: dom.GetWindow

Working without modules and downloading dependency using go get -t github.com/dennwc/dom works fine, though

Hi @Kerrex!
The problem is not with Go modules, but rather with go build. This package expects to be built for WebAssembly and won't work for other architectures (except special packages like ./net/ws). Use GOOS and GOARCH flags for the build command to set the correct architecture:
GOOS=js GOARCH=wasm go build

Thanks @dennwc!
Now it makes sense why linting and autosuggesting in VSCode didn't work :) After adding flags to settings.json everything seems to work fine.


Data Formatting is not used for generated DML commands

System information:

Operating system (distribution) and version
Windows 10
DBeaver version
7.1.2
Additional extensions

Connection specification:

Database name and version
Exasol, Generic JDBC
Driver name
exasol
Do you use tunnels or proxies (SSH, SOCKS, etc)?
no

Describe the problem you're observing:
When configuring timestamp format for a data source this affects only the display format not the format for generated sql.
Steps to reproduce, if exist:
For example:

Display for the select:

Generated Insert:

Excepted outcome
Use the display format for DML generation or an option to be able to use the display format for DML generation.

thanks for suggestion

acutally this is duplicate to #9331

DBeaver uses native date/time format for SQL scripts. Because generally UI format has nothing to do with it.
For Generic driver native format can be set in the driver settings:

For non-generic drivers native format must be provided by value handler. See examples:

https://github.com/dbeaver/dbeaver/blob/devel/plugins/org.jkiss.dbeaver.ext.oracle/src/org/jkiss/dbeaver/ext/oracle/data/OracleTimestampValueHandler.java
https://github.com/dbeaver/dbeaver/blob/devel/plugins/org.jkiss.dbeaver.ext.db2/src/org/jkiss/dbeaver/ext/db2/data/DB2ValueHandlerProvider.java

Probably we need something similar for Exasol

Tried to implement this for Exasol but the definied format has to be read from the database configuration as it depends on the settings and no connection context ist available in the value handler.
So basically if someone changed the default Oracle timestamp format using NLS_TIMESTAMP_FORMAT  dbeaver would stop working with that instance on genrated SQLs

Also tried now for a generic connection:

Then the sql is generated without single quotes around the timestamp values:

Hi Team,
I have installed 7.1.5 and still having same issue with date Format. When we are running query then result displayed in correct format as requested DD-MMM-YYYY but when we try to use Generate SQL feature for Insert/Update then still timestamp coming DD-MM-YYYY format but we need into DD-MMM-YYYY as this ticket was raised for same. Can you take a look and help me out.
Results of Insert Command
VALUES('22', 1, '092121', 'UV0', 'H0609', 804, 0, '2019-01-01 00:00:00.000', '2019-12-31 00:00:00.000', 6000, '2019-09-25 01:47:35.050'
Expected Results :--
VALUES('22', 1, '092121', 'UV0', 'H0609', 804, 0, '01-JAN-2019 00:00:00.000', '12-DEC-2019 00:00:00.000', 6000, '25-SEP-2019 01:47:35.050'

@tati-kru  @Sargul  Kindly take a look into for my comment.

Hi @tati-kru @serge-rider  @Sargul @LonwoLonwo  @kseniiaguzeeva  - Can we include this feature in upcoming release as its really important for us to use this and our whole team is looking forward for this change. So please try to have as earliest to ready for users.  If you have any question about my ask please refer my question above for timestamp format issue in Generate SQL Feature. Thanks in Advance!

Ok, we will see what we can do...

@sovittyagi that database you use?

@sovittyagi that database you use?

Hi @LonwoLonwo  we are using Oracle RDB DataBase through JDBC Connection and we have mass Insert/Update requirement in Database so we found an option Generate SQL statement for Insert/Update but when we are using it its giving timestamp in '2019-12-31 00:00:00.000' this format but we are looking into '12-DEC-2019 00:00:00.000' format. Without this dd-MMM-yyyy format we can perform Update/Insert query in our databse.

May be fixed in #9696

Fixed.
Now SQL exporter uses native date/time format when possible. It means: it fetches timestamp values in server-provided format so there is no need to mess with NLS_* variables.
Generally it should work for all databases unless they have some tricky timestamp represetnation (like Oracle with its TIMESTAMP prefix). In that case databases-specific timestamp value handler has to be patched.

Hi - In which Dbeaver Version this fixed we can see and do we invoke any
additional settings to see required timestamp in SQL Generator feature like
24-SEP-2020.
Kindly advise the steps as appropriate. Thanks in advance for your help.
Thanks
Sovit
On Sat, 26 Sep, 2020, 1:23 pm Serge Rider<EMAIL_ADDRESS>wrote:

Fixed.
Now SQL exporter uses native date/time format when possible. It means: it
fetches timestamp values in server-provided format so there is no need to
mess with NLS_* variables.
Generally it should work for all databases unless they have some tricky
timestamp represetnation (like Oracle with its TIMESTAMP prefix). In that
case databases-specific timestamp value handler has to be patched.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dbeaver/dbeaver/issues/9293#issuecomment-699452998,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AQLJ5RQPRSJN4AAGWILUYFTSHWMWTANCNFSM4O6CMR2A
.

In the next one (7.2.2), see milestone information.
You could check the Early Access version at https://dbeaver.io/files/ea

@serge-rider  On given link for early access version i can see only 7.2.1 folder there. When i can check the 7.2.2 version to verify the changes for Time Format issue. Please Advise!

doesn't work for generated statements

"Generate SQL" command doesn't perform any server-side queries, it uses data already read in the data viewer.
In order to have native date/time formatting in the "Generate SQL" dialog you need to enable option Data Formatting->Use native date/time format.

verified

@serge-rider @uslss @tati-kru  Hi All , I have installed version 7.2.2 but not able to find option to have time format in INSERT query generated by Generate SQL feature in desired format 05-OCT-2020. Can you please help me out with steps how i can get this timestamp in INSERT query.

Hi All,
Can you please provide steps to have timestamp in 01-Jan-2020 format in
Insert/update query using SQL Generator feature.
I am not able to find in version 7.2.2.
Thanks
Sovit
On Wed, 30 Sep, 2020, 2:11 pm uslss<EMAIL_ADDRESS>wrote:

verified
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dbeaver/dbeaver/issues/9293#issuecomment-701250907,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AQLJ5RUI6TTIXJSNKWQP2PTSILVMNANCNFSM4O6CMR2A
.

Reopened.
Generate SQL use native format in any case

Yes i need the format like 15-Feb-2006 in SQL query but i can see in your
screenshot also its 2006-02-15 which is not required.
On Tue, 6 Oct, 2020, 11:15 pm uslss<EMAIL_ADDRESS>wrote:

Reopened.
Generate SQL use native format in any case
[image: image]
https://user-images.githubusercontent.com/31996417/95240304-dde3e400-0814-11eb-992c-5045a0ead636.png
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dbeaver/dbeaver/issues/9293#issuecomment-704440620,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AQLJ5RQFNMFDXXMRW6SA7ZTSJNJUXANCNFSM4O6CMR2A
.

@serge-rider @uslss  Can you please make sure in version 7.2.3 this issue will resolve as we are waiting since long time for this feature. Many Thanks!

Fixed.
When native format disabled we will use UI format

@serge-rider  Can we test this latest update before wait till next release. If so please let us know so we can test and confirm results are as expected. Thanks in Advance!

We will publish Early Access version later this week (probably at Friday).

@sovittyagi actual EA is available for now https://dbeaver.io/files/ea/

@uslss  Can you please try in your machine this feature. I am getting error while trying to install this EA version. Not sure what is the issue but not able to install this version.

Using native/non-native date/time formatting for SQL exporter was implemented in the scope of this ticket. So if you need to Generate Insert Into statement e.g. with a custom timestamp, you could use Export - SQL

The same feature for Generated SQL will be implemented in the scope of #10020 .


Refactor to use QueryManager

What does this PR do?

Refs #5715
Refactor the rethinkdb integration so that it usesQueryManager instead of the ad-hoc document_db helpers.
Motivation

At the time of #5717 I introduced the document_db helpers because I was not familiar with QueryManager enough to realize it could be used for RethinkDB as well.
Switching to QueryManager improves maintainability:

QueryManager is well-known by the team, while document_db was ad-hoc. (It took me a while to understand again how it worked, and I am the one who implemented it. 😄)
LOC reduced 2x.

Additional Notes

There are two small tweaks needed to QueryManager before getting this over the line:

[ ] ~Allow query to be a callable.~ — Actually not necessary, I found a workaround using importlib in this PR, but we may still want to consider it if the use case comes up again in the future?
[ ] Allow "type": "tag_list" columns. => https://github.com/DataDog/integrations-core/pull/8147

Review checklist (to be filled by reviewers)

[ ] Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
[ ] PR title must be written as a CHANGELOG entry (see why)
[ ] Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
[ ] PR must have changelog/ and integration/ labels attached

Also fyi, I would like to do your suggestion and remove https://github.com/DataDog/integrations-core/blob/256c3e759103f3cbf29d61778b1c94e1391e52d8/datadog_checks_base/datadog_checks/base/utils/db/query.py#L47-L48


Support FHIR extensions

The SQL-on-FHIR spec supports FHIR extensions as first-class citizens. We should make sure extensions definitions can easily be added to the pipeline config and generated in the output.

Just wanted to get a quick check-in on the progress here.  I saw that there was https://github.com/google/fhir-data-pipes/pull/497 was fixed, but wasn't sure if that meant this overall issue is closed out.

No, this turned out to be far larger than just #497. I have spent many hours debugging the extension issues of R4 and finally have some potential fix. I'll write more details when I send the first PR.

So the PR to fix the Bunsen part of this is #562. That PR will close this bug as the main remaining part will be tracked under  #560. There is also a minor unit-test improvement (#558) and a more fundamental fix for a Bunsen issue (#559) which we can deal with later.


Gap between navbar and the top of the window (see through gap)

The same issue exits on Hydra website from where I actually found the bug

I don't see this, please report your full env config

Mind sending the screen grab in dark mode of the documentation with an image behind ?  Fyi my env is as follows:
Windows 10 Version 21H1, Windows 10 Home Single Language OS, Chrome Version 91.0.4472.77, Microsoft Edge
Version 91.0.864.41
I've tried in private mode on both the browsers.

Couldn't reproduce this using Chrome 91/Edge 91 on Windows 10. Are you sure this isn't affected by some extensions or something else? Perhaps something is added to the site markup, this can be checked through dev tools (screenshot with this is highly appreciated). Could you please record video with reproduce this bug?

I'll try to find out what extension might cause it I checked the dev tools the moment I was the gap the css seemed proper, anyway I'll try with other method and get back asap

I checked with another machine it's perfect.


Gini coeffs Python vs R

In R I calculate gene coefficients separately, in Python Erick does it on the outer product
IMO, let's keep it as it is for now. Ultimately, the interpretation yields the same conclusions.

If they are not in the quick starts I would modify the R version later to match the python implementation

Not in quickstarts, I agree to leave as they are for this iteration - I can adapt R code to be able to combine both source & target.

We agreed this is likely too minor.


Integrate Teller with Popular Windows Package Managers

Feature Request
Is your feature request related to a problem? Please describe.
While the current manual installation process for Teller on Windows is functional, incorporating package managers like Chocolatey and winget could offer users a more seamless and integrated experience.
Describe the solution you'd like
I would like to request the inclusion of a Chocolatey, winget, and Scoop package for Teller. This would allow users to easily install and update Teller using simple commands such as:
Chocolatey:

Installation: choco install teller
Update: choco upgrade teller

winget:

Installation: winget install tellerops.teller
Update: winget upgrade tellerops.teller

Scoop:

Installation: scoop install teller
Update: scoop update teller

Offering these installation methods not only provides a smoother experience for users but also ensures that they always have access to the latest stable version.
Describe alternatives you've considered
Manual Installation: While manual installation works, it's not as streamlined or convenient as using a package manager. This method also doesn't provide automatic updates.

Thanks for this request -- i would love for this to come from contribution, as I'm no longer a Windows expert. Closing this for now, but if you want to submit a PR we'd happily look at it.


Allow HALT status to be sent to controller when in RESTART state

if an edge app upgrade is done while an EVE update is done,
the app status state stays in RESTARTING although EVE
will restart and therefore halt the app instance; therefore
the app status should go to HALT in this case.

Could you please explain why you decided to split the inactivation function?
Also, please test the snapshot functionality. This code path could be critical for the volume snap creation as we wait for the app to be deactivated during the purge. I can share some details in DM on how to perform it.

The places in the code I'm worrying about:

https://github.com/lf-edge/eve/blob/5fa2f8717812531d6d9b40fc2324ce400d8cc2be/pkg/pillar/cmd/zedmanager/updatestatus.go#L131
https://github.com/lf-edge/eve/blob/5fa2f8717812531d6d9b40fc2324ce400d8cc2be/pkg/pillar/cmd/zedmanager/updatestatus.go#L61
And here is why:
https://github.com/lf-edge/eve/blob/a93b85069a10d11113d159cb6873e822cf6606f5/pkg/pillar/docs/app-snapshot.md#L226
https://github.com/lf-edge/eve/blob/a93b85069a10d11113d159cb6873e822cf6606f5/pkg/pillar/docs/app-snapshot.md#L408
https://github.com/lf-edge/eve/blob/a93b85069a10d11113d159cb6873e822cf6606f5/pkg/pillar/docs/app-snapshot.md#L539

TL;DR:  The code of the deactivation detection during the snap operations is pretty tricky, and I figured out the places where the wait should happen empirically. So it's pretty fragile.
I'll run the tests for snap manually.

We ran a simple Snap/Rollback/Delete test on an ext4-based app, and it looks working.

Also, what is so special about PURGING? You still keep PURGING in your check.
The original commit:
commit 6bf39d2b4453f841cff799bd63292ee191a5c627
Author: eriknordmark<EMAIL_ADDRESS>Date:   Tue Sep 25 00:01:38 2018 -0700

    do not clobber RESTARTING and PURGING states

introduced the following:
+       switch status.State {
+       case types.RESTARTING, types.PURGING:
+               // Leave unchanged
+       default:
+               status.State = minState
+       }

for these callbacks: doInstall(), doPrepare(), doActivate(), but not for the doInactivateHalt().
and then this commit followed:
commit b24e1c2cae3e8b96934b57f1527862ec00ae4d06
Author: eriknordmark<EMAIL_ADDRESS>Date:   Mon Oct 15 16:53:43 2018 -0700

    update from DomainStatus even if Pending

which covers the doInactivateHalt() with the same hunk.
@eriknordmark do we have a special meaning for the RESTARTING,PURGING in the doInactivateHalt()?

Do you understand what is a new state to which we transit from RESTARTING? I do not see how we can switch to HALTED from the doUpdate(), rather than these lines:
https://github.com/lf-edge/eve/blob/c2ef64380d526f60869d9c6bbb57cbeb3053f11d/pkg/pillar/cmd/zedmanager/updatestatus.go#L216
	if !effectiveActivate {
		if status.Activated || status.ActivateInprogress {
			c := doInactivateHalt(ctx, config, status)
			changed = changed || c
		}
		// Activated and ActivateInprogress flags may be changed during doInactivateHalt call
		if !status.Activated && !status.ActivateInprogress {
			// Since we are not activating we set the state to
			// HALTED to indicate it is not running since it
			// might have been halted before the device was rebooted
			if status.State == types.INSTALLED || status.State == types.START_DELAYED {
				status.State = types.HALTED
				changed = true
			}
		}

so we should set INSTALLED or START_DELAYED in the doInactivateHalt().
The other branch where we can transit to HALTED is doCleanup() called from the doInactivate(), but this is not our case I assume.

doUpdate calls doInactivateHalt, which calls doInactivateHaltFromDomainStatus and this sets the status.(*types.AppInstanceStatus).State to what is in ds.(*types.DomainStatus).State) .
Does this answer your question?

Does this answer your question?

Not quite. In order to see halted on the cloud you should set it somewhere, right? I see only two places in the code (specified in the comment below). But we should not take any of this paths. So I assume HALTED comes from the domainmgr and we update the applicationstatus with what comes from the domainmgr (now you removed RESTARTING, so can be updated with any state).
But! Take a look here:
https://github.com/lf-edge/eve/blob/c2ef64380d526f60869d9c6bbb57cbeb3053f11d/pkg/pillar/cmd/zedmanager/updatestatus.go#L716
		// Check that we delay a not yet active VM or a VM in the bring-up state after restarting/purging
		if !status.Activated || status.RestartInprogress == types.BringUp || status.PurgeInprogress == types.BringUp {
			// If we try to activate it for the first time - mark is with the corresponding state
			if status.State != types.START_DELAYED {
				status.State = types.START_DELAYED
				return true
			}
			// if the VM is already in the START_DELAYED state - just return from the doActivate now
			return changed
		}

This is piece of code, which should eventually delay the start if we are in restart or purge.
Then  eventually this code should do the final state transit to HALTED:
https://github.com/lf-edge/eve/blob/c2ef64380d526f60869d9c6bbb57cbeb3053f11d/pkg/pillar/cmd/zedmanager/updatestatus.go#L216
	if !effectiveActivate {
		if status.Activated || status.ActivateInprogress {
			c := doInactivateHalt(ctx, config, status)
			changed = changed || c
		}
		// Activated and ActivateInprogress flags may be changed during doInactivateHalt call
		if !status.Activated && !status.ActivateInprogress {
			// Since we are not activating we set the state to
			// HALTED to indicate it is not running since it
			// might have been halted before the device was rebooted
			if status.State == types.INSTALLED || status.State == types.START_DELAYED {
				status.State = types.HALTED
				changed = true
			}
		}

Either this two chunks are for something different, or they never worked as expected and you simply bypass them for only RESTARTING case (PURGING seems also should be covered). That's why my question: what exact state comes from domainmgr, which you set to the applicationstatus?

@eriknordmark do we have a special meaning for the RESTARTING,PURGING in the doInactivateHalt()?

They are special in that they are pending operations and not really state like the others. But we need to track that those operations are pending somewhere. I think that can be refactored so that we set HALTING, HALTED, BOOTING in State (hence report those to the controller) and we track the pending restart/purge in some separate field in AppInstanceStatus.
[Sorry, I haven't looked at the diffs in this PR yet - just got back from vacation.)

@eriknordmark do we have a special meaning for the RESTARTING,PURGING in the doInactivateHalt()?

They are special in that they are pending operations and not really state like the others. But we need to track that those operations are pending somewhere. I think that can be refactored so that we set HALTING, HALTED, BOOTING in State (hence report those to the controller) and we track the pending restart/purge in some separate field in AppInstanceStatus. [Sorry, I haven't looked at the diffs in this PR yet - just got back from vacation.)

We already have the following fields:
status.RestartInProgress == (NotInprogress | BringUp | BringDown)
status.PurgeInProgress == (NotInprogress | BringUp | BringDown)

So this information should not be lost.

So this information should not be lost.

Can you all check whether there is any code which looks at the app instance status State field and does something different it is RESTARTING or PURGING (apart from the lines which avoid overwriting those values).
Or does all of the logic inside zedmanager look at RestartInProgress and PurgeInProgress? If so we can just stop using the RESTARTING and PURGING states - the reported state will then become RUNNING -> HALTING -> HALTED -> BOOTING -> RUNNING when someone does a restart or a purge.
"Ask not what you can add but what you can remove".

Can you all check whether there is any code which looks at the app instance status State field and does something different it is RESTARTING or PURGING (apart from the lines which avoid overwriting those values). Or does all of the logic inside zedmanager look at RestartInProgress and PurgeInProgress? If so we can just stop using the RESTARTING and PURGING states - the reported state will then become RUNNING -> HALTING -> HALTED -> BOOTING -> RUNNING when someone does a restart or a purge.
"Ask not what you can add but what you can remove".

Do you mean:

Only in this special case when app upgrade and eve node update are done at the same time? Then it should not use RESTARTING?
In general RESTARTING should not be used?
In this case what should we report to the controller when we do a restart?

RESTARTING is used (including reads):
 │  types.go:47
 │  eve/pkg/pillar/cmd/zedagent/parseconfig.go:222
 │  eve/pkg/pillar/cmd/zedmanager/updatestatus.go:806
 │  eve/pkg/pillar/cmd/zedmanager/updatestatus.go:1178
 │  eve/pkg/pillar/types/types.go:101
 │  eve/pkg/pillar/types/types.go:173

The IMO only interesting check is in parseconfig.go:222 in countRunningApps.
same for PURGING.

2. In general RESTARTING should not be used?
In this case what should we report to the controller when we do a restart?

Yes, that is what I mean.
The user asked to see HALTED, which means the underlying behavior from domainmgr will report RUNNING -> HALTING -> HALTED -> BOOTING -> RUNNING as the restart proceeds. Same for purging.

Yes, that is what I mean.
The user asked to see HALTED, which means the underlying behavior from domainmgr will report RUNNING -> HALTING -> HALTED -> BOOTING -> RUNNING as the restart proceeds. Same for purging.

Hmm, I tried, but I don't think this is the way forward:
diff --git a/pkg/pillar/cmd/zedagent/handlemetrics.go b/pkg/pillar/cmd/zedagent/handlemetrics.go
index 2704d45582..d20c6c9b64 100644
--- a/pkg/pillar/cmd/zedagent/handlemetrics.go
+++ b/pkg/pillar/cmd/zedagent/handlemetrics.go
@@ -1034,10 +1034,50 @@ func encodeProxyStatus(proxyConfig *types.ProxyConfig) *info.ProxyStatus {
        return status
 }

+var ignoreSendingOutFollowingRestarts map[string]struct{}
+
+func init() {
+       ignoreSendingOutFollowingRestarts = make(map[string]struct{})
+}
+
+func PublishAppInfoToZedCloud(ctx *zedagentContext, uuid string,
+       aiStatus *types.AppInstanceStatus,
+       aa *types.AssignableAdapters, iteration int, dest destinationBitset) {
+       if aiStatus.State != types.RESTARTING {
+               publishAppInfoToZedCloud(ctx, uuid, aiStatus, aa, iteration, dest)
+               delete(ignoreSendingOutFollowingRestarts, uuid)
+               return
+       }
+
+       _, found := ignoreSendingOutFollowingRestarts[uuid]
+       if found {
+               return
+       }
+
+       for _, state := range []types.SwState{types.HALTING, types.HALTED, types.BOOTING} {
+               aiStatusCopy := *aiStatus
+               aiStatusCopy.State = state
+
+               publishAppInfoToZedCloud(ctx, uuid, &aiStatusCopy, aa, iteration, dest)
+
+               // do not overwrite states, but send them out first!
+               if dest&ControllerDest != 0 {
+                       deferredCtx := zedcloudCtx.DeferredEventCtx
+                       deferredCtx.HandleDeferredNow()
+               }
+               locConfig := ctx.getconfigCtx.sideController.locConfig
+               if dest&LOCDest != 0 && locConfig != nil {
+                       zedcloudCtx.DeferredPeriodicCtx.HandleDeferredNow()
+               }
+
+               ignoreSendingOutFollowingRestarts[uuid] = struct{}{}
+       }
+}
+
 // This function is called per change, hence needs to try over all management ports
 // When aiStatus is nil it means a delete and we send a message
 // containing only the UUID to inform zedcloud about the delete.
-func PublishAppInfoToZedCloud(ctx *zedagentContext, uuid string,
+func publishAppInfoToZedCloud(ctx *zedagentContext, uuid string,
        aiStatus *types.AppInstanceStatus,
        aa *types.AssignableAdapters, iteration int, dest destinationBitset) {
        log.Functionf("PublishAppInfoToZedCloud uuid %s", uuid)
diff --git a/pkg/pillar/zedcloud/deferred.go b/pkg/pillar/zedcloud/deferred.go
index 5e2610c0c0..34039a82d2 100644
--- a/pkg/pillar/zedcloud/deferred.go
+++ b/pkg/pillar/zedcloud/deferred.go
@@ -132,6 +132,10 @@ func (ctx *DeferredContext) processQueueTask(ps *pubsub.PubSub,
        }
 }

+func (ctx *DeferredContext) HandleDeferredNow() {
+       ctx.handleDeferred()
+}
+

It basically checks if the state is RESTARTING and then sends out (and immediately flushes) the states HALTING, HALTED, BOOTING.
The only way that RESTARTING can be set in domainStatus.State (afaik) is in zedmanager.go under
  1224      if config.RestartCmd.Counter != oldConfig.RestartCmd.Counter ||
  1225          config.LocalRestartCmd.Counter != oldConfig.LocalRestartCmd.Counter {

The only way that RESTARTING can be set in domainStatus.State (afaik) is in zedmanager.go under
  1224      if config.RestartCmd.Counter != oldConfig.RestartCmd.Counter ||
  1225          config.LocalRestartCmd.Counter != oldConfig.LocalRestartCmd.Counter {

@rouming pointed out that all this is not necessary and that I can just remove the
 1238              status.State = types.RESTARTING

I will retest the PR manually... I wanna be sure it does not break my stuff.

I've tested what I wanted, and it looks fine...
But we should also announce this change loud, I believe:

The test team may have some tests that expect the states as they are now.
As it's seen in the UI in the events flow, we should inform users about the change. Users may get used to the current events flow and expect this to remain the same. In the worst case, it can break some of their automation.

I've tested what I wanted, and it looks fine...
But we should also announce this change loud, I believe:

The test team may have some tests that expect the states as they are now.
As it's seen in the UI in the events flow, we should inform users about the change. Users may get used to the current events flow and expect this to remain the same. In the worst case, it can break some of their automation.

In addition to testing and annoucing, please check the docs we have in the eve repo and check with the ZEDEDA commercial controller documentation if there are things which need to be updated there. (We don't have documentation at this level of detail for Adam/Eden so no need to check there.)

In addition to testing and annoucing, please check the docs we have in the eve repo and check with the ZEDEDA commercial controller documentation if there are things which need to be updated there. (We don't have documentation at this level of detail for Adam/Eden so no need to check there.)

I did not find anything regarding this yet (one thing I still have to check for the commercial controller).
I found that some tests/eden rely on RESTARTING or PURGING state, so I changed that to HALTING (see top commit), but I am not sure about the impact; the log files of the failed eden tests don't complain about those two states.

I found that some tests/eden rely on RESTARTING or PURGING state, so I changed that to HALTING (see top commit), but I am not sure about the impact; the log files of the failed eden tests don't complain about those two states.

The changes to eden tests make sense, but need to go as a PR to the eden repo.

I found that some tests/eden rely on RESTARTING or PURGING state, so I changed that to HALTING (see top commit), but I am not sure about the impact; the log files of the failed eden tests don't complain about those two states.

The changes to eden tests make sense, but need to go as a PR to the eden repo.

PR is here: https://github.com/lf-edge/eden/pull/970/files - I tried it out locally with one of the tests and it fixes it


VDK DAGs example dag-job fails to run

Describe the bug
When all the orchestrated jobs are created and deployed as well as the DAG job, vdk run dag-job fails with an error.
Steps To Reproduce
Steps to reproduce the behavior:

Follow the steps in the example to create and deploy all jobs.
Run the job: vdk run dag-job (If the dag-job already exists in the cloud, the job might fail due to missing keytab so make sure to download its keytab using  vdk download-key -n dag-job -t <TEAM_NAME> -p <PATH_TO_EXAMPLE_DIR>)
See error: [VDK] dag-job [ERROR] vdk.internal.builtin_plugins.r            errors.py:438  log_and_rethrow [id:e5cf6db7-88fe-4fca-8ab5-04e089457818-1682425978-960bf]- An error in data job code  occurred. The error should be resolved by User.  Here are the details: WHAT HAPPENED : Data Job step dag_job.py completed with error. WHY IT HAPPENED : An exception occurred, exception message was: (404) Reason:  HTTP response headers: ... HTTP response body: {"what":"The Data Job 'ingest-job-table-one' does not exist.","why":"The Data Job must be existing.","consequences":"The Data Job will not be started.","countermeasures":"Provide a Data Job that already exists or create a new one and try again."} CONSEQUENCES : I will not process the remaining steps (if any), and this Data Job execution will be marked as failed. COUNTERMEASURES : See exception and fix the root cause, so that the exception does not appear anymore.

Expected behavior
DAG job should be able to recognize the deployed jobs that are part of the graph. Then the DAG would be able to be executed and return the Success log as expected in the example.
Version  (please complete the following information):

OS: macOS Ventura v13.3.1
Version

Version: 0.3.765384785
Build details: RELEASE_VERSION=0.3.765384785, BUILD_DATE=Thu Feb  2 13:10:29 UTC 2023, BUILD_MACHINE_INFO=Linux runner-cvrszkxw-project-28359933-concurrent-0zl9pr 5.4.209-116.363.amzn2.x86_64 #1 SMP Wed Aug 10 21:19:18 UTC 2022 x86_64 GNU/Linux, GITLAB_CI_JOB_ID=3703508297, GIT_COMMIT_SHA=70719cfa0dfb590f607d08a4abdb6c645d74a69d, GIT_BRANCH=main
Python version: 3.7.16 64bit (/Users/ysalambashev/opt/anaconda3/envs/vdk3.7/bin/python)

Installed plugins:
vdk-server (from package vdk-server, version 0.1.715017056)
vdk-trino (from package vdk-trino, version 0.4.703555598)
sc-config-prod (from package sc-config-prod, version 0.0.6422538)
vdk-sqlite (from package vdk-sqlite, version 0.1.730902357)
meta-jobs (from package vdk-meta-jobs, version 0.1.836044868)
vdk-control-service-properties (from package vdk-plugin-control-cli, version 0.1.715017056)
vdk-execution-skip (from package vdk-plugin-control-cli, version 0.1.715017056)
vdk-plugin-control-cli (from package vdk-plugin-control-cli, version 0.1.715017056)
vdk-ingest-http (from package vdk-ingest-http, version 0.2.715017056)
vdk-ingest-file (from package vdk-ingest-file, version 0.1.715017056)
vdk-kerberos-auth (from package vdk-kerberos-auth, version 0.3.730902357)
vdk-logging-format (from package vdk-logging-format, version 0.1.715017056)```

The problem here is related to the team name. The team_name in each orchestrated job in the JOBS dictionary was not changed the same way it was changed in the config.ini files and the commands ("-t TEAM_NAME"). That's why when the DAG job is run, it is unable to find job "ingest-job-table-one" with a different team.
The team name should be the same everywhere. If you want to set a default value for a variable such as team name and not worry about changing it everywhere one by one, you can do so using the command vdk set-default.
Check vdk set-default --help for more information.