Retrieve infodict of entries matching archive through API

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I remove or skip any mandatory* field

Checklist

[X] I'm requesting a feature unrelated to a specific site
[X] I've looked through the README
[X] I've verified that I'm running yt-dlp version 2022.11.11 (update instructions) or later (specify commit)
[X] I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue

Provide a description that is worded well enough to be understood
I'm providing a list of URLs that may or may not lead to duplicate videos and would like to identify which videos are connected to which URLs that have already been downloaded (using the download_archive and, potentially, break_on_existing). However, I have had some difficulty programmatically identifying which video in the archive is the culprit for a given URL. Currently, I believe, the only way to do so monitor debug messages via a logger which seems like overkill.
It seems like you could add the extractor and id to the ExistingVideoReached exception. I think only here and here.
I could be missing something, but I tried to use the available hooks, but at least in the extract_info method, there isn't any info_dict data available yet. Thank you for your help!
Provide verbose output that clearly demonstrates the problem

[ ] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[ ] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
No response

I don't understand the use-case. Could you elaborate?

Of course. I send a list of urls. One url is to a video already downloaded. I would like to know which video; how best can I link that url to the previously downloaded video?
Im trying to get the previously stored info_dict and filename and associate it with the new URL. For example, if I have a list of urls that includes both direct to YouTube links and links to Tweets, YT-DLP will kindly extract the linked video in a Tweet and download it, but if I happen to have already downloaded it, I can get an error to raise, but I am having trouble figuring out the best way to link that URL to the previously downloaded video since the error doesn’t say which video.

Perhaps I could do a call to in_download_archive with only a url? I’m not sure exactly at what point you determine the extractor and ID, but it expects the info_dict.

Even if the Exception gives the archive id, you may still not be able to match it with the URL.
If all links are single videos, you can do this:
from yt_dlp import YoutubeDL
from yt_dlp.utils import ExistingVideoReached

with YoutubeDL({'download_archive': 'archive.txt', 'break_on_existing': True}) as ydl:
    for url in ['test:youtube', 'test:twitter']:
        try:
            ydl.download(url)
        except ExistingVideoReached:
            print(f'Already downloaded: {url}')

But I see that there is currently no easy way to do this for playlists. An implementation of https://github.com/yt-dlp/yt-dlp/pull/5633#issuecomment-1332195717 may help

Thanks for the quick response. That's actually exactly what I am doing so I indeed know the URL. What I'm interested in knowing is what video is associated with that URL that's already in the archive.
I'd love to do in pseudo code
from yt_dlp import YoutubeDL
from yt_dlp.utils import ExistingVideoReached

with YoutubeDL({'download_archive': 'archive.txt', 'break_on_existing': True}) as ydl:
    for url in ['test:youtube', 'test:twitter']:
        try:
           info = ydl.extract_info(url)
           dict_with_video_id_keys_and_info_dicts[info.get('id')+info.get('extractor')] = info  # this just so I can get it later if the video is already downloaded
           storage_dictionary[url]['info_dict'] = info
        except ExistingVideoReached as e:
            video_id_in_archive = str(e) # extract it here
            print(f'Already downloaded: {video_id_in_archive} with {url}')
            storage_dictionary[url]['info_dict'] = dict_with_video_id_keys_and_info_dicts[video_id_in_archive]

I have tried using progress_hooks and postprocessor_hooks, but they do not seem to be called when using info = ydl.extract_info(url) prior to the archive check. I'd be happy to extract the video_id_in_archive in anyway at all. I don't even have to use the exception, as I could do result = ydl.download(url) and check result. One way I could resolve this given the current infrastructure is with a second request:
with YoutubeDL({'download_archive': 'archive.txt', 'break_on_existing': True}) as ydl:
    for url in ['test:youtube', 'test:twitter']:
        try:
            ydl.download(url)
        except ExistingVideoReached:
            info = ydl2.extract_info(url, download=False) # this doesn't actually work; I need a new YoutubeDL object without `download_archive` or `break_on_existing`

Regardless, this seems like unnecessary overhead since I already have the info dict from the first time I downloaded the video.
You make a good point about playlist/channels. I haven't been downloading them, but, if I were, I actually don't really want to break in that case. That's an edge for me that I'm willing to ignore, but perhaps there is a better solution. If a hook was called prior to the archive check, I could use that to get the id and extractor (that's how I extract info for multiple videos per URL currently). Another option would be to overwrite my logger and catch your to_screen debug messages and look for the archive related ones, but, again, that seems unnecessary overhead.
Here's my current work in progress if you want to see what I'm really doing (but it involves a lot of extra as it exists inside a much larger framework).
Also made a PR fork, but I haven't had a chance to read all your contribution guidelines. And I won't be able to test it until sometime next week.

What I'm interested in knowing is what video is associated with that URL that's already in the archive.

gotcha. Now I finally understand your exact requirement

If a hook was called prior to the archive check,

I have tried using progress_hooks and postprocessor_hooks, but they do not seem to be called when using info = ydl.extract_info(url) prior to the archive check

In order to detect duplicates as soon as possible, the check is performed multiple times, some of them before any hook/pre-processor and before the full infodict is constructed. Youtube is one example that is highly optimized.

One way I could resolve this given the current infrastructure is with a second request:

Regardless, this seems like unnecessary overhead since I already have the info dict from the first time I downloaded the video.

Another option would be to overwrite my logger and catch your to_screen debug messages and look for the archive related ones, but, again, that seems unnecessary overhead.

I agree. These are hacky workarounds

Also made a PR fork, but I haven't had a chance to read all your contribution guidelines. And I won't be able to test it until sometime next week.

This is quite hacky too. You can only get the archive id and not any of the other metadata in the infodict. I'd prefer to implement a more general solution

I'm not sure to grasp the whole idea behind this very issue, but I feel my concern is very similar: Retrieving the info-dict of the playlist's items even if these are already inside the download-archive.

When using the cli --flat-playlist  --dump-json on a <playlist URL> happily retrieve the JSON.
If using a download-archive (and item where found inside it), then it won't output anything
With the cli, we could workaround this by using --dump-single-json
but using the API, there is no way (that I know), that calling extract_info(<playlist-url>) would retrieve the entries

(I hope I'm not too off-topic and this could rather help finding a unified solution to this class of shortcomings)

...  knowing is what video is associated with that URL that's already in the archive.

The archive records video IDs per extractor. Multiple URLs may resolve to the same entry, and also a single URL may resolve to multiple entries, eg if the extractor has changed how video IDs are constructed. The normal case when using a download-archive is that yt-dlp (or yt-dl) resolves each URL and skips processing the URL as soon as it knows that the URL matches an entry in the archive (which as above may be very soon depending on the site).
OP's problem is that there's no explicit hook into this decision, although various hacks as above can try to provide one.
To retrieve the info-json unconditionally, just don't mention any archive; the page will have to be extracted anyway to populate the result. If the concern is that the list of URLs for which the info-json is requested may include the same video ID/extractor pair several times, use a scratch archive.
In OP's case the problem is to get just the info-json if the URL is (matched) in the archive and both to get the info-json and download the video if it isn't. Apparently no existing parameter combinations cater for this.

Would this work?
with (
    YoutubeDL({'download_archive': 'archive.txt', 'break_on_existing': True}) as dl_ydl,
    YoutubeDL() as info_ydl,
):

    for url in ['test:youtube', 'test:twitter']:
        # Optimization that yt-dlp performs. Remove it to get full info instead
        for key, ie in dl_ydl._ies.items():
            if ie.suitable(url):
                info = {'id': ie.get_temp_id(url), 'ie_key': key, 'original_url': url}
                if info['id'] and dl_ydl.in_download_archive(info):
                    return info
                break
        else:
            key = None

        info = info_ydl.extract_info(url, download=False, ie_key=key)
        try:
            dl_ydl.process_ie_result(info)
        except ExistingVideoReached:
            return info


Susbscription dispose when undock the viewer parent container.

When the component is set on an Doking system like RadDocking on telerik, when the pane is undock from the main windows the log stop working, this is caused by the _Subscription disposition of the viewer.
I know that this is an expected behavior because we want to stop listen for new logs when the windows is closed, or unloaded, but the issue cause that the viewer dons't start to listen again whe the new windows set or even when the current windows loads again, and since the control dons't has any recovery strategy to start listen again, it fall in a useless state.
To solve this issue we should separete the log subscription login on two main public method:
StartListen();
This method should start listen the log by subscribing to the cache, and should be called in the current _OnLoaded even handler in order to keep the same functionality and start Listen as soon as the windows loads.
StopListen();
This method should stop the listen subscription for new logs as happen now in the _OnUnloaded event handler, and also should be called from the same hadler to keep the functionality.
Whit this 2 methods the user can start again to listen for new logs in such scenarios like the docking windows or any future scenario that require Start/Stop listen for logs.
Even the pause functionality could be promoted to use this method in order to improve performance.
I open this discussion, @dojo90  if you agree I could start a PR

I think this is a special case that I have not yet considered or does not occur with me.
I appreciate it if you want to improve the code and start a PR. 😀💪


Measure performance in logs

Just thinking whether it would be good to store these metrics in a file. We could name it -.txt. I think you will want to compare different models and/or devices and if they are just printed to the console then you will have to copy them manually somewhere to compare. Thoughts?

Yes, I thought about store it in a file. My only concern was that storing it into a file can affect the processor performance for the file access. But I can implement a short version and test it.

Just thinking whether it would be good to store these metrics in a file. We could name it -.txt. I think you will want to compare different models and/or devices and if they are just printed to the console then you will have to copy them manually somewhere to compare. Thoughts?

Yes, I thought about store it in a file. My only concern was that storing it into a file can affect the processor performance for the file access. But I can implement a short version and test it.

Just thinking whether it would be good to store these metrics in a file. We could name it -.txt. I think you will want to compare different models and/or devices and if they are just printed to the console then you will have to copy them manually somewhere to compare. Thoughts?

Yes, I thought about store it in a file. My only concern was that storing it into a file can affect the processor performance for the file access. But I can implement a short version and test it.

I added now the possibility of storing the performance metrics logs into a CSV file.

Just thinking whether it would be good to store these metrics in a file. We could name it -.txt. I think you will want to compare different models and/or devices and if they are just printed to the console then you will have to copy them manually somewhere to compare. Thoughts?

Yes, I thought about store it in a file. My only concern was that storing it into a file can affect the processor performance for the file access. But I can implement a short version and test it.

I added now the possibility of storing the performance metrics logs into a CSV file.


Exact starts-with should take precidence

To replicate:

Open a tree containing files cox.rb and context.rb
cmd-p and type "cox".
Fuzzy finder lists context.rb above  cox.rb

Perhaps not wrong but certainly surprising.

+1 Exact searches sometimes don't match

:+1:

:+1: Large project with a lot of files, exact matches are often buried by results that have several of the search characters but spread far apart. This makes fuzzy finder almost less useful than if it weren't fuzzy at all.

I would argue that it's quite frequent to prefer matching at the end of the
file path than anywhere at the begining.
I mean the file name it almost always the only part of the path that i can
recall
2015-10-05 17:11 GMT-04:00 gdaolewe<EMAIL_ADDRESS>
[image: :+1:] Large project with a lot of files, exact matches are often
buried by results that have several of the search characters but spread far
apart. This makes fuzzy finder almost less useful than if it weren't fuzzy
at all.
—
Reply to this email directly or view it on GitHub
https://github.com/atom/fuzzy-finder/issues/41#issuecomment-145669703.

--
Benjamin DREUX

Related to #98 no doubt.
@izuzak this feels buggy to me.  Not sure I agree with the enhancement label.

Yes I want job_spec.rb...  that's what I'm typing ain't it?  :)

@broson Yes, this is the exact case. Because of this, a simple starts-with search would be better like 80% of the time.

@benzen I agree that it prefer matching the filename rather than the path, but it should prefer matching towards the beginning of the filename itself. If the matching is done on the full path + filename as one string, I think that's quite wrong.
I don't know how many I speak for, but I almost never type part of the path to find a file; the only reason I would is this exact scenario, where typing the filename exactly puts the desired file way down in the suggestions and I have to start typing the path to disambiguate.

Maybe I badly explained myself, but I totally agree with @gdaolewe


Online deployment bugs

I'm compiling a list of bugs in our online deployment that I've found while looking at our MDC results. There's been only two so far, but I imagine others will be found.

The check_refractory function checks whether the current time is at least refractory_period seconds after the previous detection time, but it ought to compare the new detection time to the previous detection time. Otherwise, if event submission takes a while for some reason, we can have situations like the below, where we submit two events that are essentially at the same time (taken from /home/aframe/dev/o3_mdc/events/log/deploy_2024-09-18T02:30:00.log)

2024-09-17 20:30:28,736 - root - DEBUG - Reading frames from timestamp<PHONE_NUMBER>
2024-09-17 20:30:28,766 - root - DEBUG - Read successful
2024-09-17 20:30:28,985 - root - INFO - Detected event with detection statistic>=3.342
2024-09-17 20:30:29,970 - root - INFO - Event coalescence time found to be<PHONE_NUMBER>.250 with FAR 9.457e-05 Hz
2024-09-17 20:30:29,981 - root - INFO - Submitting trigger to file event_1410665416.json
2024-09-17 20:30:29,984 - scitokens - INFO - Validating SciToken with jti: https://cilogon.org/oauth2/328592bce2ee7ac5745c4fc89082da03?type=accessToken&ts=1726626606482&version=v2.0&lifetime=10800000
2024-09-17 20:30:30,273 - urllib3.connectionpool - DEBUG - https://gracedb-playground.ligo.org:443 "POST /api/events/ HTTP/1.1" 201 577
2024-09-17 20:30:34,210 - scitokens - INFO - Validating SciToken with jti: https://cilogon.org/oauth2/328592bce2ee7ac5745c4fc89082da03?type=accessToken&ts=1726626606482&version=v2.0&lifetime=10800000
2024-09-17 20:30:34,488 - urllib3.connectionpool - DEBUG - https://gracedb-playground.ligo.org:443 "POST /api/events/G2229021/log/ HTTP/1.1" 201 397
2024-09-17 20:30:34,689 - scitokens - INFO - Validating SciToken with jti: https://cilogon.org/oauth2/328592bce2ee7ac5745c4fc89082da03?type=accessToken&ts=1726626606482&version=v2.0&lifetime=10800000
2024-09-17 20:30:34,868 - urllib3.connectionpool - DEBUG - https://gracedb-playground.ligo.org:443 "POST /api/events/G2229021/log/ HTTP/1.1" 201 414
2024-09-17 20:30:45,068 - scitokens - INFO - Validating SciToken with jti: https://cilogon.org/oauth2/328592bce2ee7ac5745c4fc89082da03?type=accessToken&ts=1726626606482&version=v2.0&lifetime=10800000
2024-09-17 20:30:45,269 - urllib3.connectionpool - DEBUG - https://gracedb-playground.ligo.org:443 "POST /api/events/G2229021/log/ HTTP/1.1" 201 411
2024-09-17 20:30:45,269 - root - DEBUG - Reading frames from timestamp<PHONE_NUMBER>
2024-09-17 20:30:45,291 - root - DEBUG - Read successful
2024-09-17 20:30:45,524 - root - INFO - Detected event with detection statistic>=3.342
2024-09-17 20:30:45,791 - root - INFO - Event coalescence time found to be<PHONE_NUMBER>.252 with FAR 9.462e-05 Hz
2024-09-17 20:30:45,792 - root - INFO - Submitting trigger to file event_1410665416.json
2024-09-17 20:30:45,797 - scitokens - INFO - Validating SciToken with jti: https://cilogon.org/oauth2/328592bce2ee7ac5745c4fc89082da03?type=accessToken&ts=1726626606482&version=v2.0&lifetime=10800000
2024-09-17 20:30:46,067 - urllib3.connectionpool - DEBUG - https://gracedb-playground.ligo.org:443 "POST /api/events/ HTTP/1.1" 201 584
2024-09-17 20:30:50,285 - scitokens - INFO - Validating SciToken with jti: https://cilogon.org/oauth2/328592bce2ee7ac5745c4fc89082da03?type=accessToken&ts=1726626606482&version=v2.0&lifetime=10800000
2024-09-17 20:30:50,570 - urllib3.connectionpool - DEBUG - https://gracedb-playground.ligo.org:443 "POST /api/events/G2229022/log/ HTTP/1.1" 201 397
2024-09-17 20:30:50,806 - scitokens - INFO - Validating SciToken with jti: https://cilogon.org/oauth2/328592bce2ee7ac5745c4fc89082da03?type=accessToken&ts=1726626606482&version=v2.0&lifetime=10800000
2024-09-17 20:30:50,979 - urllib3.connectionpool - DEBUG - https://gracedb-playground.ligo.org:443 "POST /api/events/G2229022/log/ HTTP/1.1" 201 414
2024-09-17 20:31:02,815 - scitokens - INFO - Validating SciToken with jti: https://cilogon.org/oauth2/328592bce2ee7ac5745c4fc89082da03?type=accessToken&ts=1726626606482&version=v2.0&lifetime=10800000
2024-09-17 20:31:02,982 - urllib3.connectionpool - DEBUG - https://gracedb-playground.ligo.org:443 "POST /api/events/G2229022/log/ HTTP/1.1" 201 411
2024-09-17 20:31:02,983 - root - DEBUG - Reading frames from timestamp<PHONE_NUMBER>
2024-09-17 20:31:03,006 - root - DEBUG - Read successful
2024-09-17 20:31:03,167 - root - DEBUG - Reading frames from timestamp<PHONE_NUMBER>

Somehow, the reset_t0 function can reset to a time prior to the frame that failed. Coupled to this, we don't reset the snapshotter after missing a frame file. This means that it's possible for an event to be detected, a frame to be missed, t0 reset to before the event, and the event to be detected again. For example, in /home/aframe/dev/o3_mdc/events/log/deploy_2024-09-18T11:25:49.log:

2024-09-18 07:45:23,248 - root - DEBUG - Reading frames from timestamp<PHONE_NUMBER>
2024-09-18 07:45:23,264 - root - DEBUG - Read successful
2024-09-18 07:45:23,356 - root - INFO - Detected event with detection statistic>=3.081
2024-09-18 07:45:24,997 - root - INFO - Event coalescence time found to be<PHONE_NUMBER>.787 with FAR 1.381e-04 Hz
2024-09-18 07:45:25,010 - root - INFO - Submitting trigger to file event_1410705924.json
2024-09-18 07:45:35,069 - scitokens - INFO - Validating SciToken with jti: https://cilogon.org/oauth2/f3f4a63ecccd6b73b9d7410279ffa5?type=accessToken&ts=1726669802473&version=v2.0&lifetime=10800000
2024-09-18 07:45:35,070 - urllib3.connectionpool - DEBUG - Resetting dropped connection: gracedb-playground.ligo.org
2024-09-18 07:45:35,512 - urllib3.connectionpool - DEBUG - https://gracedb-playground.ligo.org:443 "POST /api/events/ HTTP/1.1" 201 583
2024-09-18 07:45:37,525 - scitokens - INFO - Validating SciToken with jti: https://cilogon.org/oauth2/f3f4a63ecccd6b73b9d7410279ffa5?type=accessToken&ts=1726669802473&version=v2.0&lifetime=10800000
2024-09-18 07:45:37,803 - urllib3.connectionpool - DEBUG - https://gracedb-playground.ligo.org:443 "POST /api/events/G2230630/log/ HTTP/1.1" 201 397
2024-09-18 07:45:37,997 - scitokens - INFO - Validating SciToken with jti: https://cilogon.org/oauth2/f3f4a63ecccd6b73b9d7410279ffa5?type=accessToken&ts=1726669802473&version=v2.0&lifetime=10800000
2024-09-18 07:45:38,187 - urllib3.connectionpool - DEBUG - https://gracedb-playground.ligo.org:443 "POST /api/events/G2230630/log/ HTTP/1.1" 201 414
2024-09-18 07:45:46,703 - scitokens - INFO - Validating SciToken with jti: https://cilogon.org/oauth2/f3f4a63ecccd6b73b9d7410279ffa5?type=accessToken&ts=1726669802473&version=v2.0&lifetime=10800000
2024-09-18 07:45:46,888 - urllib3.connectionpool - DEBUG - https://gracedb-playground.ligo.org:443 "POST /api/events/G2230630/log/ HTTP/1.1" 201 411
2024-09-18 07:45:46,888 - root - DEBUG - Reading frames from timestamp<PHONE_NUMBER>
2024-09-18 07:45:46,908 - root - DEBUG - Read successful
2024-09-18 07:45:46,985 - root - DEBUG - Reading frames from timestamp<PHONE_NUMBER>
2024-09-18 07:45:47,002 - root - DEBUG - Read successful
2024-09-18 07:45:47,086 - root - DEBUG - Reading frames from timestamp<PHONE_NUMBER>
2024-09-18 07:45:57,096 - root - WARNING - Couldn't find frame file /dev/shm/kafka/L1_O3ReplayMDC/L-L1_O3ReplayMDC_llhoft-1410705930-1.gwf after 10s
2024-09-18 07:45:57,097 - root - WARNING - Missing frame files after timestep<PHONE_NUMBER>, resetting states
2024-09-18 07:45:57,106 - root - INFO - Resetting timestamp to<PHONE_NUMBER>
2024-09-18 07:45:57,106 - root - DEBUG - Reading frames from timestamp<PHONE_NUMBER>
2024-09-18 07:45:57,125 - root - DEBUG - Read successful
2024-09-18 07:45:57,125 - root - DEBUG - Reading frames from timestamp<PHONE_NUMBER>
2024-09-18 07:45:57,142 - root - DEBUG - Read successful
2024-09-18 07:45:57,143 - root - DEBUG - Reading frames from timestamp<PHONE_NUMBER>
2024-09-18 07:45:57,160 - root - DEBUG - Read successful
2024-09-18 07:45:57,253 - root - DEBUG - Reading frames from timestamp<PHONE_NUMBER>
2024-09-18 07:45:57,270 - root - DEBUG - Read successful
2024-09-18 07:45:57,338 - root - INFO - Detected event with detection statistic>=3.170
2024-09-18 07:45:58,263 - root - INFO - Event coalescence time found to be<PHONE_NUMBER>.787 with FAR 1.215e-04 Hz
2024-09-18 07:45:58,265 - root - INFO - Submitting trigger to file event_1410705924.json
2024-09-18 07:45:58,268 - scitokens - INFO - Validating SciToken with jti: https://cilogon.org/oauth2/f3f4a63ecccd6b73b9d7410279ffa5?type=accessToken&ts=1726669802473&version=v2.0&lifetime=10800000
2024-09-18 07:45:58,610 - urllib3.connectionpool - DEBUG - https://gracedb-playground.ligo.org:443 "POST /api/events/ HTTP/1.1" 201 585
2024-09-18 07:46:00,586 - scitokens - INFO - Validating SciToken with jti: https://cilogon.org/oauth2/f3f4a63ecccd6b73b9d7410279ffa5?type=accessToken&ts=1726669802473&version=v2.0&lifetime=10800000
2024-09-18 07:46:00,871 - urllib3.connectionpool - DEBUG - https://gracedb-playground.ligo.org:443 "POST /api/events/G2230631/log/ HTTP/1.1" 201 397
2024-09-18 07:46:01,059 - scitokens - INFO - Validating SciToken with jti: https://cilogon.org/oauth2/f3f4a63ecccd6b73b9d7410279ffa5?type=accessToken&ts=1726669802473&version=v2.0&lifetime=10800000
2024-09-18 07:46:01,283 - urllib3.connectionpool - DEBUG - https://gracedb-playground.ligo.org:443 "POST /api/events/G2230631/log/ HTTP/1.1" 201 414
2024-09-18 07:46:10,617 - scitokens - INFO - Validating SciToken with jti: https://cilogon.org/oauth2/f3f4a63ecccd6b73b9d7410279ffa5?type=accessToken&ts=1726669802473&version=v2.0&lifetime=10800000
2024-09-18 07:46:10,795 - urllib3.connectionpool - DEBUG - https://gracedb-playground.ligo.org:443 "POST /api/events/G2230631/log/ HTTP/1.1" 201 411
2024-09-18 07:46:10,796 - root - DEBUG - Reading frames from timestamp<PHONE_NUMBER>

As a result of (at least) these two bugs, we have quite a few events in gracedb that are separated by less than the 8 second window that we set:

Great catches (and only discoverable thanks to the verbose logging).
Curious about a couple of things:

Are we doing that Sci-token validation every gdb upload, or is that something done by gracedb? seems to take a significant amount of time
In that second output, it looks like we detect an event in frame<PHONE_NUMBER>, but read 3 future frames before submitting. Can you explain this?

The scitoken validation is something that's done by gracedb, unfortunately. There seems to be some caching, as the subsequent validations don't take much time, but scitokens expire after an hour (and I'm generating a new one every half-hour to be safe), so the full validation needs to happen relatively often. In a non-MDC setting, I'd imagine it would need to happen every time.
Yeah, we need to wait resampling_padding + whitening_padding + integration_length = 1 + 0.5 + 1 seconds after the event before it's potentially detectable (another one of the nice things about NGDD - the resampling padding can be just a 16th of a second).

Ahh got it, adds up. It's great how easy we are able to diagnose these bottlenecks

As I think about it some more, it's kind of strange that the scitoken validation should take so long. @mcoughlin, is this something that the low latency group is aware of? Maybe we're doing something wrong on our end? Add 10 seconds of latency to perform authorization seems not great.

This 10 seconds obviously doesnt happen every time considering we have many events with sub-10 latency.
Are there any hints under which conditions this added latency occurs?

Not that I can see right off the bat, but let me scrape the logs and see if there are any patterns to it.

Just as an example, for our lowest latency event, the validation steps seem to be the same, but happen much faster:
2024-09-13 20:32:43,121 - root - INFO - Detected event with detection statistic>=7.015
2024-09-13 20:32:43,340 - root - INFO - Event coalescence time found to be<PHONE_NUMBER>.232 with FAR 2.240e-07 Hz
2024-09-13 20:32:43,343 - root - INFO - Submitting trigger to file event_1410319978.json
2024-09-13 20:32:43,347 - scitokens - INFO - Validating SciToken with jti: https://cilogon.org/oauth2/3a4fa254fbbabd4069f11ae7d2f5d142?type=accessToken&ts=1726284601735&version=v2.0&lifetime=10800000
2024-09-13 20:32:43,709 - urllib3.connectionpool - DEBUG - https://gracedb-playground.ligo.org:443 "POST /api/events/ HTTP/1.1" 201 1267
2024-09-13 20:32:45,237 - scitokens - INFO - Validating SciToken with jti: https://cilogon.org/oauth2/3a4fa254fbbabd4069f11ae7d2f5d142?type=accessToken&ts=1726284601735&version=v2.0&lifetime=10800000
2024-09-13 20:32:45,522 - urllib3.connectionpool - DEBUG - https://gracedb-playground.ligo.org:443 "POST /api/events/G2214279/log/ HTTP/1.1" 201 397
2024-09-13 20:32:45,700 - scitokens - INFO - Validating SciToken with jti: https://cilogon.org/oauth2/3a4fa254fbbabd4069f11ae7d2f5d142?type=accessToken&ts=1726284601735&version=v2.0&lifetime=10800000
2024-09-13 20:32:45,884 - urllib3.connectionpool - DEBUG - https://gracedb-playground.ligo.org:443 "POST /api/events/G2214279/log/ HTTP/1.1" 201 414
2024-09-13 20:32:45,926 - scitokens - INFO - Validating SciToken with jti: https://cilogon.org/oauth2/3a4fa254fbbabd4069f11ae7d2f5d142?type=accessToken&ts=1726284601735&version=v2.0&lifetime=10800000
2024-09-13 20:32:46,094 - urllib3.connectionpool - DEBUG - https://gracedb-playground.ligo.org:443 "POST /api/events/G2214279/log/ HTTP/1.1" 201 411
2024-09-13 20:32:46,095 - root - DEBUG - Reading frames from timestamp<PHONE_NUMBER>

Ah, wait, at least some of the time here is coming from creating PE/p_astro. Those scitokens lines correspond, respectively, to event submission, corner plot submission, sky map submission, and p_astro submission. We need better logging in that area. To know how much is coming from GDB and how much is on our end.

Got it - probably worth doing this asynchronously now?

Definitely. Still, in the second example in the top comment, there's a 10-second gap between between "Submitting event" and the first scitoken validation line for the first event, and that's not coming from us (all that occurs is here). So I think also worth looking at how much time authorization is costing us.


Post-processing of extracted text > Sentences which have been split into separate passages

Where a sentence or paragraph spans more than one page, it will be considered to be an independent text block by the parser. This will result in the last sentence in the first text block being truncated, and the first sentence in the second text block starting in the middle of the sentence. See previous example.
Passages may be broken across multiple text blocks, depending on the layout of the document. These text blocks would need to be merged to ensure that they contain meaningful passages. For example, in the following the parser has correctly retained the correct natural reading order, but these have been parsed into separate lines.
Options:

Consider for alpha the impact of just displaying, rather than resolving, these breaks on a user viewing the full text and search.
Store metadata at passage level which could be use to control how passages that are broken across pages/columns are displayed.

Hey team! Please add your planning poker estimate with ZenHub @eurolife @kdutia @opyate

Another idea I floated in the sprint meeting: if we have the bounding box of the passage, take a PNG snapshot of the PDF at those coordinates, and render the PNG on the website (instead of the entire PDF, or cleaned-up text).

The Adobe Extract API returns a paragraphSpan element when it predicts a paragraph is split over pages or columns. We should measure the extent to which this solves this issue.


refactor!: pydantic v2

Refactors to use pydantic v2 API and updates most of the other pins to be more recent versions (removed dateutil)
Resolves issues with AnyURL by leveraging a trick that creates a field which uses it for validation, but retains it as a string
Additionally refactors the ISO-8601 datetime strings using a similar trick that is compatible with v2 and is clearer and easier to use
Lastly, adds a test showing that the model is compatible for use with FastAPI as a data field item (for forms and such)
fixes: https://github.com/spruceid/siwe-py/issues/46
fixes: https://github.com/spruceid/siwe-py/issues/58
fixes: https://github.com/spruceid/siwe-py/issues/50
fixes: https://github.com/spruceid/siwe-py/issues/54
fixes: https://github.com/spruceid/siwe-py/issues/56

A note that python 3.8 and 3.9 not supported on macos-latest right now

Sorry @sbihel @ameyarao98, CI should be fixed now

Thank you!

Published as part of v3.0.0


[C++] Enable metadata writing in the ORCWriter

Enable writing of ORC Metadata in the ORCWriter.
See also ARROW-9299 (read ORC Metadata).
Reporter: Ian Alexander Joiner / @iajoiner
Note: This issue was originally created as ARROW-12535. Please see the migration documentation for further details.

Hey, is this still an issue to potentially work on? If so, I would like to do it.

Hey, is this still an issue to potentially work on? If so, I would like to do it.

I think you're welcome to go for it.  Do you have any questions?  The guide in https://arrow.apache.org/docs/dev/developers/cpp/index.html is probably a good place to start.


Connection timeout for geonames index and idea for cache location

The URL for downloading GeoNames seems to be unreachable currently and when launching Eidos, it errors at the "Downloading the GeoNames index from http://clulab.cs.arizona.edu/models/geonames+woredas.zip" line. In relation to this immediate issue (which is probably external to Eidos), I've been thinking whether it would be better to have a fixed location for the geonames cache on a given machine (e.g. in a folder called ~/.eidos/) rather than always looking for the cache in the current working folder (which may be any path from which Eidos is being used, often changing).  Any thoughts?

You're right: the zip file is not accessible through HTTP, which makes me believe the CS dept. web server is down. Just FYI, the file is accessible through the local file system, once I log in. I will start a CS ticket for this issue.
I like the .eidos idea! @kwalcock, what do you think?

I'm not sure about the web servers.  cs.arizona.edu seems to be OK, but clulab.cs.arizona.edu is not.  Any idea where this is hosted?  The downloading of geonames at runtime is slated to be replaced with a compile-time dependency, so that problem should eventually go away.  The cache directory is set in eidos.conf and can be changed to an absolute rather than relative path.  I'm not sure whether ~ is allowed, but suspect it is not.  It doesn't seem outlandish to insist that the program be called from the same working directory in order for it to reuse a cache.  It is possible to have multiple caches that were built using different options.  Is there a use case in which eidos needs to be called from different working folders?

I see, given all these factors, I guess it's not worth changing anything now. By the way, the use case in which this comes up is when we use Eidos as a dependency of INDRA and so Eidos is invoked from application-specific folders for different reading/assembly runs.

Another option would be to make a symbolic link from ./cache in each of these directories to ~/cache.


Learning again

@WuraolaOyewusi I think we merged adding this file already. If you want to make changes, you'll have to update your local master, create the new branch, and then edit the file, instead of creating it from an outdated master before the file was added.

Ok, I will do that and add this to what I've learnt


Added all segmentation to poly metrics

Adds metrics that converts segmentation masks into polygons and then uses our existing polygon metrics.
TODO:

[ ] Allow disabling of iou matching. We don't want to always match the highest iou object (a seg mask might be split into two polys).

LGTM, code looks super clean, well-documented and structured 🌟
Are the metric available for a user to call from the SDK, if so, should we add some notes in the CHANGELOG and up the version?

Yup, changelog and bump!


refactor(x/gamm): remove CreatePool and pool id index boilerplate in gamm (merge)

Closes: #XXX
What is the purpose of the change
Currently, we update the pool id index in both x/gamm and x/swaprouter. Since the goal is to fully migrate the index to x/swaproter, we start to remove any references it the old x/gamm index.
We still need to keep the state entry in x/gamm. However, every method, function or query, utilizing it can be refactored to either swithc to querying x/swaprouter index or be removed entirely.
Brief Changelog
As a result, of sunsetting the index in x/gamm , the folllowing changes have been made:

removed CreatePool method from x/gamm. Its callers instead use CreatePool in x/swaprouter.
removed GetNextPoolIdAndIncrement function in x/gamm since the updates to the index should not happen anymore
make pool-incentives keeper take swaprouter keeper instead of gamm for retrieving pool indexes

Testing and Verifying
This change is already covered by existing tests
Documentation and Release Note

Does this pull request introduce a new feature or user-facing behavior changes? no
Is a relevant changelog entry added to the Unreleased section in CHANGELOG.md? no
How is the feature or change documented? not applicable

Going to merge with 1 approval once CI passes. This has been QAed on concentrated-liquidity-main. Additionally, we discussed this change on the call with ValarDragon


Feature: A better share button NeedMoreShare2

PR Checklist
Please check if your PR fulfills the following requirements:

[x] The commit message follows our guidelines.
[x] Tests for the changes have been added (for bug fixes / features).

[x] Muse | Mist have been tested.
[x] Pisces | Gemini have been tested.

[ ] Docs have been added / updated (for bug fixes / features).

PR Type
What kind of change does this PR introduce?

[ ] Bugfix.
[x] Feature.
[ ] Code style update (formatting, local variables).
[ ] Refactoring (no functional changes, no api changes).
[ ] Build related changes.
[ ] CI related changes.
[ ] Documentation content changes.
[ ] Other... Please describe:

What is the new behavior?
Add a new share button which is a pure javascript lib.
It will be the only sharebutton that supports HTTPS in NexT (for now) and it has a better UI.
It's useful in China.
https://github.com/revir/need-more-share2

Screens with this changes:

1 Desktop Postbottom

when hover:

when click:

2 Desktop Float

when hover:

when click:

3 Mobile Postbottom

4 Mobile Float

How to use?
In NexT _config.yml:
# NeedMoreShare2
# This plugin is a pure javascript sharing lib which is useful in China.
# See: https://github.com/revir/need-more-share2
needMoreShare2:
  postbottom:
    enable: true
    Options:
      iconStyle: box
      boxForm: horizontal
      position: middleCenter
      networks: Weibo,Wechat,Douban,QQZone,Twitter,Facebook
  float:
    enable: true
    Options:
      iconStyle: box
      boxForm: horizontal
      position: middleRight
      networks: Weibo,Wechat,Douban,QQZone,Twitter,Facebook

Does this PR introduce a breaking change?

[ ] Yes.
[x] No.

@LEAFERx and can u show live demo on it? Why on your blog i don't see this feature?

@ivan-nginx now it's on my blog

Also, need to test it on other schemes (Pisces | Gemini).

Also, need to add all supported networks from needShareButton:
default: 'Mailto,Twitter,Pinterest,Facebook,GooglePlus,Reddit,Delicious,Tapiture,StumbleUpon,Linkedin,Slashdot,Technorati,Posterous,Tumblr,GoogleBookmarks,Newsvine,Pingfm,Evernote,Friendfeed,Vkontakte,Odnoklassniki,Mailru'
options: Any of the network name's from pervios row comma separated

Also, take a look on this screen:

As u can see, there are main widget what include:

Social rating.
VK likes.
FB likes & shares.

OK. I'll try.

1 lowercase in config
2 put everything into needsharebutton.swig and needsharebutton.styl
3 rewrite the description
4 put the postbottom button into the post-widget
5 test it in Pisces&Gemini and find it works in the right way

Ok, i'll test it now too, wait a moment...

needmoreshare2:
  enable: true
  postbottom:
    enable: true
    options:
      iconStyle: box
      boxForm: horizontal
      position: bottomCenter
      networks: Weibo,Wechat,Douban,QQZone,Twitter,Linkedin,Mailto,Twitter,Reddit,Delicious,Tapiture,StumbleUpon,Pinterest,Facebook,GooglePlus,Linkedin,Slashdot,Technorati,Posterous,Tumblr,GoogleBookmarks,Newsvine,Pingfm,Evernote,Friendfeed,Vkontakte,Odnoklassniki,Mailru
  float:
    enable: true
    options:
      iconStyle: box
      boxForm: horizontal
      position: middleRight
      networks: Weibo,Wechat,Douban,QQZone,Twitter,Linkedin,Mailto,Twitter,Reddit,Delicious,Tapiture,StumbleUpon,Pinterest,Facebook,GooglePlus,Linkedin,Slashdot,Technorati,Posterous,Tumblr,GoogleBookmarks,Newsvine,Pingfm,Evernote,Friendfeed,Vkontakte,Odnoklassniki,Mailru

It seems some css styles need to improvement.

I'm trying but i meet some troubles when making web font.
It may take some time since I don't have much time now.

I'm going to make webfont on fontello like what revir does, but i find that Tapiture,Slashdot,Technorati,Googlebookmarks,Nwsvine,Pingfm,Mailru dont have icons on fontello.
Is it acceptale to remove those social networks?

@LEAFERx hi again.

Tapiture
Slashdot
Technorati
Googlebookmarks
Nwsvine
Pingfm
Mailru

Is it acceptale to remove those social networks?

I think yes, but we must try to add all of possible icons if this options already exists and created.
Well, i think need a little wait for answer from revir's issue.
Plugin is really good and i think need to modify him with greatest usability and options.

I've fixed the icon problem by a hacky way that is put two webfont file together. But it's not the best way because I'm not sure how many icons are repeated in two files.
BTW, I still think that I should put the min file because other libs are minified like pace, three and so on. But if you insist, I'll put the non-min file.

by a hacky way that is put two webfont file together. But it's not the best way because

Yea, i agree with this.

BTW, I still think that I should put the min file because other libs are minified like pace, three and so on. But if you insist, I'll put the non-min file.

All of JS and CSS files in NexT are non-minified. This was do for better maintenance, but it for NexT-libraries at least. For vendors (lib directory) libs there are min and non-min libs.
I think the better solution is to place both (min and non-min, 4 files (2 js and 2 css)) and use min versions. As will need to edit some of they, we can see code directly and after edit, for example, can to minify and use edited version.

How can i embed a font file into a css?

U mean what code (\e800) will have font?

@LEAFERx > add seperate line and make some refactors in css
No need to do this. I already refactor this styles but do not finish some button's mastering. U need just concentrate on JS and Fonts, ok?
For example, Hound report u about errors. So, need to fix they. And need to do test it on non-minify files. And only after that do minify. After all work finished and before global merge to NexT master, we just switch from non-minify to minified file. That's what i'm talking about u: no need to add minification — minification do hard to maintenance and only good for prodaction; but for prod's anyone may use minifications plugins.

I think the js is ok now

«Line is too long» — just place in new line like that:
Before:
width = window.innerWidth ? window.innerWidth : document.documentElement.clientWidth ? document.documentElement.clientWidth : screen.width,

After:
width = window.innerWidth ?
        window.innerWidth :
        document.documentElement.clientWidth ?
        document.documentElement.clientWidth :
        screen.width,

It's better for vision. Houndy is smart. :smirk:
https://github.com/iissnan/hexo-theme-next/blob/217b4d857f5fa0b9bee00bb16d021f779923b693/.jshintrc#L10

lines is cut now

See? Hound is happy now. :smiley:

Please test it if you have time

Already done? Sure, i just finishing my test cases with buttons. Take a moment..

Check it. If all alright, need to do some mastering and finally — merge it.

Bugs

In Gemini scheme with post button when click — social show under button, ok; when click again (or if click on any other place) — social dissapear, but blinked in the bottom (change default show place).

I've remade the fontello font file and i can say that there is no repeat in two font files.
The blink should be the css animation problem.

@LEAFERx alright, dude, my time is to late, need to sleep. I believe in u remastering this cool things and we create best social plugin for NexT. :+1:

blink bug is fixed

Hi! Some small bugs found:

Titles on social icons when mouse move on them — lowercases. Can we do, for example, instead of

Like this:

?

Identifier 'network_lc' is not in camel case.

network_lc » networkLC

So, need to resolve conflicts, test it and can be merged i think.
P.S. Sorry for conflicts, it's my mistake with branches and new gui soft. See last commits in master branch.

Sorry but it was the first time i merge such a complex branch
the git auto-merging deletes some lines in some files without noticing me and i really dont know if i left something
can you please resolve the conflicts?

Yeah, i just don't understand what's the problem with this merge, all was fine just yesterday.

can you please resolve the conflicts?

But i can make another: i can merge your this branch directly in master or dev branch to solve it, but need to do end of the maintenance. Merge ready? All possible bugs and another things fixed? Need to delete all commented lines and make code clean in source/css/_common/components/third-party/needsharebutton.styl, for example. So, please, finish it and i'll try to push this merge directly.

The program is «GitKraken». It's free and have nice GUI. Recommended for use.

I find a strange css bug
the button has different style and hover behavior in four theme scheme
in muse it is black background and white hover background
in mist it is always white background whether hover or not
in pisces&gemini it is white background and black hover background

It's not a bug, that styles reused from main buttons NexT styles and on different schemes — different styles. See live demo with buttons here.
Ok, do not edit for now anymore. I just try to resolve this conflicts and merge it into master, afterthat we can resume to fix all bugs, etc.

ok

@LEAFERx check please button-refactor branch. All NeedMoreShare options there? All latest commits, etc.?

yes, and I think it's ready to merge

Ok, i'll do it soon.
P.S. Conflict was. i think, because u or i don not stage pushes into your commit directly. It was my fault, i was must to create separated pull or branch with my own button refactor changes:

No need to create so must difficult commits in next time.

Alright, branch was merged. Any additions/features/fixes to this pull are welcome.
Anyway, god job @LEAFERx! Also greatings thanks for Gitment pull and halp with adaptation of Gitmint! I really appreciate it.

About buttons: well, it's inherit styles from main readmore buttons and buttons tags too. Need just redefine in _custom.styl to .btn class any style u want — all buttons (and share button too) will show in this styles.

@LEAFERx hi again! How are u? Fine? Ok, there is 2 bugs:

Then user click on any share button and share it, buttons panel after that don't close automatically.
Facebook share not work:

In mobilw style icons not flexed in main window:

Sorry that i dont have time to fix it now, my school is begin and a important exam is waiting for me.
No until Nov will i have time i guess.
Sorry again.

Ok, no problem, dude. Just take a time and try to fix it later. No need to hurry somewhere, it's opensource.

@ivan-nginx, как по мне, то нужно добавить условие, чтобы NeedMoreShare2 (боковая кнопка, которая прилипает к левой стороне браузера) не выводился на главной странице, а только на страницах заметок блога. Что скажешь?

@online так в этом и прикол, что он сделал 2 кнопки:

Боковая левая выводится везде.
Кнопка для постов выводится в постах.

Вообще, заметил, что всё же соц-иконы Бирмана выдают цифру репостов некоторых. Скорее всего интегрирую и их тоже + там также через JS, оказывается, сделано (то бишь ссылки индекс и вес страниц забирать не будут).


Feature request: ability to specify background color when tooltip=true

Would like to be able to pass in custom colors and margins when tooltip=true. :)
Thanks!

+1 this would be awesome

+1


Annotation for Default Value

during the creation of the schema, there is no notation that says "in the case of a null field or not present to assign them a default"
example:
when I define a mongo scheme
new Schema({
    count: { type: Number, default: 0 }
});

I also want that when I don't pass the field the value of default appears in the object.
Expected behavior
@DefaultValue(0)
int count;

Actual behavior
every time I send the object I initialize the field to let me add in mongo.
@Field
int count = 0;

I also accept changes or suggestions.

Is this supposed to be for Spring Session MongoDB or Spring Data MongoDB?

@gregturn is for Spring Data MongoDB

Then you may wish to close this issue and visit jira.spring.io.

https://jira.spring.io/browse/DATAMONGO-2350


YouTube playlist not working?

Hi Jules,
First off: great plugin! Easy to set up and minimal!
I do have some trouble getting a YT playlist to work though.
I made a test page on http://www.johnnygoody.com/test.html and I tried everything, but can't get it to work. Would it be possible to help me out? Thanks!
Cheers!

The video says: invalid parameters and shows the YouTube grainy screen.
In the Iframe src, the playlist parameter seems to be missing:

Could it be broken and not working anymore?

After trying longer I found out that you shouldn't put the playlist ID in there, but a list of video ID's seperated by a comma.


S3's native webserver is pretty basic (HTTP 1.1 only, no caching, limited control of headers). Not likely an issue at this stage, but with scale S3 GET costs increase quickly.
Probably warrants a separate ticket, but this could also host the entire Vue portion of the site with only any (future) API bits needing to run on actual compute hardware.

Oh yeah, it would be nice to serve the site from s3 with a CDN. I think I have a branch for that, although I could be thinking of another one of my projects. Good idea


nixos-option should show the type of an option

Issue description
We can currently display a NixOS module option type using, for example:
(types.enum ["foo" "bar"]).description.
It seems to me that, to maximize usability, nixos-option should display this string.
Also, the NixOS options browser should display this information.
Perhaps it should be toggleable, though, since options involving submodule types could be quite large.
EDIT: Apparently the description field is currently "submodule" for any submodule type. It would be nice to either pretty-print the whole type or output some kind of structured data there.
Steps to reproduce
Run nixos-option services.postgresql.enable. There is no indication of the type of this option other than the default/example values.
Technical details

System: (NixOS: nixos-version, Ubuntu/Fedora: lsb_release -a, ...)
Nix version: (run nix-env --version)
Nixpkgs version: (run nix-instantiate --eval '<nixpkgs>' -A lib.nixpkgsVersion)
Sandboxing enabled: (run grep build-use-sandbox /etc/nix/nix.conf)

cc @nbp

This was fixed in 26c45dfec2716b1508e5d7ac8efdbdf570f2eb99.


feat(upgrade): add v1.1.2 pre-hook

Problem:
In old clusters, the status.dynamic of the multi-cluster-management feature is false, so we can't change the feature's value.
Solution:
Set status.dynamic to true, so we can update the value of the multi-cluster-management feature to true in the new version.
Related Issue:
https://github.com/harvester/harvester/issues/2679
Test plan:

Install a v1.1.2 cluster.
Upgrade the cluster to the version of this branch.
Check whether the status.dynamic of the multi-cluster-management feature is true.

the pre-hook runs before the rancher upgrade, as a result the changes are lost.
this can be verified easily:

pre-upgrade:

perform upgrade:

the manifest is patched before the rancher helm chart upgrade, which seems to reset the changes

post-upgrade:

can observe a change in revision number but no change of status

This PR can be closed in favor of: https://github.com/harvester/harvester/pull/3894

Fixed in https://github.com/harvester/harvester/pull/3894, thanks @FrankYang0529 !


Default value for -j does not take into account CPU set limitation

When running a Docker container with the --cpuset-cpus=0 flags, the nproc command inside the container returns 1. However, when running ninja --help it seems that this limitation isn't taken into account by the default -j value:
-j N     run N jobs in parallel [default=10, derived from CPUs available].

Would #1224 fix this?

Unfortunately no. I tried using your branch of ninja, as well as run this sample program inside a docker container ran with --cpuset-cpus 0 (where the output of nproc is 1) and got "8" printed twice:
#include <unistd.h>
#include <iostream>

int main() {
        std::cout << sysconf(_SC_NPROCESSORS_ONLN) << std::endl;
        std::cout << sysconf(_SC_NPROCESSORS_CONF) << std::endl;
        return 0;
}

Do other apps under docker handle this flag well? Is there some other sysconf we're supposed to check? Or is this a docker-side bug?

The nproc command shows the expected output, which is 1. I'm not sure that this is a bug Docker, but instead it is the fact that Docker uses namespacing instead of virtualization, so the real computer hardware information is exposed to the container despite the fact that it is limited to be scheduled only on one CPU. The actualy number of CPUs isn't hidden from the container by any means (if you run htop inside a container limited to a specific CPU you'd still see al the CPU information). After opening this issue I asked in Stackoverflow how to achieve the same result of nproc using syscalls. However, the only answer I got was that it is a messy issue full of ifdefs and platform specific logic.

Can you still reproduce this? On my machine (16 logical CPU cores), running
> docker run --cpuset-cpus=0 -t alpine /bin/sh -c "apk add ninja && ninja --help"

prints
  -j N     run N jobs in parallel [default=2, derived from CPUs available]

Unfortunately it only works on alpine, which seems to hide the real CPU count.
Examples on my 4 CPUs machine:
docker run --rm --cpuset-cpus=0 -t alpine /bin/sh -c "apk update; apk add ninja && ninja --help; nproc --all; nproc"

ninja defaults to 2
nproc --all: 1
nproc: 1

docker run --rm --cpuset-cpus=0 -t ubuntu /bin/sh -c "apt update && apt install ninja-build && ninja --help; nproc --all; nproc"

ninja defaults to 6
nproc --all: 4
nproc: 1

docker run --rm --cpuset-cpus=0 archlinux/base /bin/sh -c "pacman --noconfirm -Sy ninja && ninja --help; nproc --all; nproc"

ninja defaults to 6
nproc --all: 4
nproc: 1

BTW this is how nproc does it: https://github.com/coreutils/gnulib/blob/ddbd29c308b46cfba8a21d52d6e7a0dbd561bdbc/lib/nproc.c#L69

Thanks! I've managed to fix it without introducing a dependency on pthread. Now we need some further testing, if this behaves the same when there's no CPU_SET limitation (it does on my system). Please check out the master branch and report results here, thanks :)

According to the manpage, sched_getaffinity() and CPU_COUNT() are only available if _GNU_SOURCE is defined before including sched.h. So this new code isn't actually being enabled at build time.
ninja says it should go to 50:
bash-4.3# ninja -h
usage: ninja [options] [targets...]

if targets are unspecified, builds the 'default' target (see manual).

options:
  --version      print ninja version ("1.9.0")
  -v, --verbose  show all command lines while building

  -C DIR   change to DIR before doing anything else
  -f FILE  specify input build file [default=build.ninja]

  -j N     run N jobs in parallel (0 means infinity) [default=50 on this system]
  -k N     keep going until N jobs fail (0 means infinity) [default=1]
  -l N     do not start new jobs if the load average is greater than N
  -n       dry run (don't run commands but act like they succeeded)

  -d MODE  enable debugging (use '-d list' to list modes)
  -t TOOL  run a subtool (use '-t list' to list subtools)
    terminates toplevel options; further flags are passed to the tool
  -w FLAG  adjust warnings (use '-w list' to list warnings)

My cpuset only has 4 CPUs:
bash-4.3# grep Cpus_allowed /proc/self/status
Cpus_allowed:   0010,01001001
Cpus_allowed_list:      0,12,24,36

Assembly dump of GetProcessorCount() shows its still using sysconf():
000000000041f780 <_Z17GetProcessorCountv>:
  41f780:       48 83 ec 08             sub    rsp,0x8
  41f784:       bf 54 00 00 00          mov    edi,0x54
  41f789:       e8 12 33 fe ff          call   402aa0 <sysconf@plt>
  41f78e:       48 83 c4 08             add    rsp,0x8
  41f792:       c3                      ret
  41f793:       0f 1f 00                nop    DWORD PTR [rax]
  41f796:       66 2e 0f 1f 84 00 00    nop    WORD PTR cs:[rax+rax*1+0x0]
  41f79d:       00 00 00


Yarn can't run the dev command

Hey.
Having issues using yarn on this
yarn run v0.15.1
$ "node build/dev-server.js"
sh: node build/dev-server.js: No such file or directory
error Command failed with exit code 127.
info Visit http://yarnpkg.com/en/docs/cli/run for documentation about this command.

But it works perfectly when using npm.
npm -v is 5.5.1
node -v is 9.0.0
yarn -v is 0.15.1
vue -V is 2.9.1

I needed to upgrade yarn, now running v1.2.1.
See here: https://github.com/yarnpkg/yarn/issues/1139
Sorry for the unnecessary issue - but hopes it helps someone in the future.

Thanks for posting the solution back 🙂


Incorporate lightray library as replacement for tuning module

I stripped out our tuning scripts and generalized things a bit into lightray which is also now on pypi, the motivation being to be able to incorporate it into AMPLFI as well.
This PR removes our tuning modules in favor of using lightray
This PR also

Generalizes how we find ssh keys for git-sync, looking in Path.home() / .ssh / id_rsa
Fixes bug in running Tune pipeline that was introduced in the inference overhaul PR
Removes charts directory and github workflow which was moved to lightray - now we use the remote chart published on that repos github for deploying the ray cluster

@wbenoit26 let me know what you think - was able to get the cluster deployed but can't testing the actual tune jobs until the train container gets rebuilt and pushed with the updated environment

Yeah there's probably a bug or two, but shouldn't affect the rest of the tasks


Clarify assembly parameter definitions

Description
The assembly parameters that relate to conditions at charge time have been clarified to indicate that they are conditions at the most recent charge, not necessarily when the assemblies first entered the core.

Checklist

[X] The code is understandable and maintainable to people beyond the author.
[ ] Tests have been added/updated to verify that the new or changed code works.
[X] There is no commented out code in this PR.
[X] The commit message follows good practices.
[X] All docstrings are still up-to-date with these changes.

If user exposed functionality was added/changed:

[X] Documentation added/updated in the doc folder.
[ ] New or updated dependencies have been added to setup.py.

Exactly. I dug into the code to see what was actually happening, and figured I could make it clearer so others don't have to do the same later.


User objects not exploded in cloud function params

[x] You've met the prerequisites.
[x] You're running the latest version of Parse Server.
[x] You've searched through existing issues. Chances are that your issue has been reported or resolved before.

Environment Setup
localhost
node 3.6
mongo 3.0
Steps to reproduce
Parse.User.extend({
  method: function() { return 'hello world'; }
});
Parse.Cloud.define('customFunction', function(request, response) {
  response.success(request.user.method());
});

When called with a valid session token,
customFunction will return hello world on Parse Cloud Code
customFunction will throw TypeError: request.user.method is not a function on parse-server

Perhaps you need to init a User before calling request.user.method()?

We noticed this too but worked around it by either creating a new User with request.user.id or by querying the request.user. Not sure what the official stance is on this.
We extend as follows:
var User = Parse.Object.extend('_User', {
...
var query = new Parse.Query(User);

I think the official stance should be to match the behavior on hosted Parse Cloud Code.
In that case handleCloudFunction can be updated to user: new Parse.Object('_User', req.auth.user)
I can make a PR if the owners can confirm the expected behavior

I recommend making the PR anyway as it will have higher visibility and get things going

This might we WAI
I couldnt get this test to fail:
https://github.com/drinklynk/parse-server/commit/136ee6def5f5fff84b1554a307943d54429072c7

Yes, we are trying to match Parse.com behaviour in as many cases as we can. If you can submit a PR that gets us closer to Parse.com behaviour that would be much appreciated :)

So I resolved this one and just wanted to make a note of it in case someone comes looking later.
The issue was that there were TWO instances of Parse that were in my node_modules one under root and one under parse-server/node_modules
might be useful to have a warning when this is happening


datagrid event

Select one ...  (check one with "x")
[x ] bug
[ ] feature request
[ ] enhancement

Expected behavior
Datagrid's "clrDgSingleSelectedChange" event should only be emitted when "clrDgSingleSelected" changed.
Actual behavior
Datagrid's "clrDgSingleSelectedChange" event come with pagination's "clrDgPageChange" event.
Reproduction of behavior
change page and then the "clrDgSingleSelectedChange" is invoked
https://plnkr.co/edit/jctXlyKsSDD1wANL3Bp2?p=preview
Environment details

Angular version: 2.0.X

Clarity version:
"clarity-angular": "^0.8.10",
"clarity-icons": "^0.8.10",
"clarity-ui": "^0.8.10",

OS and version:
Windows 7

Browser:
Chrome XX

Hi @beautifulinda,
This happens as in the current implementation, the selection is lost on page change. So, there actually IS a change in selection. From [😞, 😃, 🐱] to [].
Clarity team has planned to change this, so I suggest you check this issue https://github.com/vmware/clarity/issues/484, where you can see the new specs and follow the updates.
Best regards


Maven appcenter distribute artifact seems to be unpublished for 2.4.0

The AppCenter distribute documentation suggests declaring the native android dependency with a specific version like this:
dependencies {
   def appCenterSdkVersion = '2.3.0'
   implementation "com.microsoft.appcenter:appcenter-distribute:${appCenterSdkVersion}"
}

However, changing the appCenterSdkVersion to the latest release (2.4.0) results in an error.

ERROR: Failed to resolve: com.microsoft.appcenter:appcenter-distribute:2.4.0

It appears the module that corresponds to the latest release is not published in maven. Either that, or the versioning scheme is misleading.
Describe the solution you'd like
Locking the native dependency to the current release should be possible. Having the versioning scheme be the same on npm and maven implies that they should be locked side by side. Making this not possible implies risk that they won't work together.
Describe alternatives you've considered
This works if you use the "+" syntax. It probably pulls in the 2.3.0 maven artifact, which again, doesn't seem to align with the npm version.
.

@luisnaranjo733
What's misleading is that the github repo https://github.com/microsoft/appcenter-sdk-android shows the develop branch by default rather than the master one.
At time of writing, the version 2.4.0 is on their develop branch, not master, so it's kind of expected imo.
https://github.com/microsoft/appcenter-sdk-android/blob/master/versions.gradle
https://github.com/microsoft/appcenter-sdk-android/blob/develop/versions.gradle
And version 2.4.0 is marked as under development at https://github.com/microsoft/appcenter-sdk-android/blob/develop/CHANGELOG.md#version-240-under-development

Great point, I had not realized that. Totally agree that it's misleading that the GH repo shows the develop branch. That definitely seems like it would be a common source of confusion

Hey @luisnaranjo733,
As already mentioned above, 2.4.0 is not released yet.
The latest release on GitHub and Maven is 2.3.0
Our docs is also point to 2.3.0 as it's the latest version for Android. So, I'm curious why you thought that it isn't the latest one?

Ah, "npm". React-Native, right?
It was updated to bring the fixes from iOS native SDK, but it uses Android SDK version 2.3.0 since it's the latest one.

that they won't work together.

Android and iOS versions may not match (but they usually do). Currently React-Native SDK wraps Android 2.3.0 and iOS 2.4.0 SDK. It's definitely intended and the compatibility is verified, no worry.

Hi, we haven't heard from you in a while so I'm closing this issue. Please reopen/comment if you have more questions or run into this again.


Native Messaging with snark-RPC

As titled. This extension should automatically add torrents to snark via the RPC interface.

Checklist:

[not done] Add torrents with a dialog that is familiar to Firefox users(Similar to the HTTP/S downloads one).
[not done] Add torrents download to downloads drop-down menu

[not done] Portray progress accurately

[not done] Remove torrents by manipulating downloads drop-down menu

Done!


Icon Packer

Icon Packer
Icon Packer Repo
Changes:

Can be used as a template for more Icon Packs.
Changes Every Icon on the Main UI, Main Menu (Just the tabs), QAM Menu (Just the tabs), Controller Configuration menu, and Steam Button Overlay (when held down).
To prevent performance issues this is done using the local hosting feature. This probably shouldn't be merged until that is available.
Included submission instructions to add more icon packs to the theme itself. If preferred, the theme can just be a template for icon packs and all icon packs can be instead submitted directly to to the theme DB as their own theme.

Checklist
Failure to complete this checklist or deleting boxes from the checklist will result in the closing of your pull request unless this is a theme update. Please write any comments regarding this checklist at the bottom of your pull request.
Check every box.

[x] I am the original author of this theme or have permission from the original author to make this pull request.
[x] All copyright of this theme's contents belong to the listed author or is cited in the repository linked above.
[x] This theme's target has been marked appropriately and only styles said target.
[x] This theme works properly on the latest versions of SteamOS for Steam Deck, decky-loader and SDH-CssLoader.
[x] This theme only uses * or !important if absolutely necessary.
[x] This theme is under 10MB in size and uses the least disk space possible.
[x] This theme's preview image does not include text unless it is necessary to describe changes that can be made.
[x] This theme is safe for work and does not contain any sexual, drug-related, or profane content.
[x] This theme prefixes any CSS variables with a unique identifier.

Check one box.

[x] I am not bundling a part of another theme with this theme to encourage mixing and matching themes.
[ ] Themes included with this theme are toggleable using a patch.

Check one box.

[ ] This is a keyboard theme applied to the default keyboard.
[ ] This is a system-wide theme applied to the default keyboard. The keyboard is toggleable using a patch.
[x] This theme does not target the keyboard.

Failed check due to what ima assuming is the .png files. This should be allowed eventually for the local image host right?

Eventually, yes. When i implement it you'd need to rebase your repo

Sounds good.


[TF2.0] trainable=True on tf.keras.Embedding result in "constant folding failed: Invalid argument: Unsupported type: 21"

Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10 x64
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: No
TensorFlow installed from (source or binary): Binary
TensorFlow version (use command below): tensorflow-gpu==2.0.0-beta1
Python version: 3.7.3
Bazel version (if compiling from source): No
GCC/Compiler version (if compiling from source): No
CUDA/cuDNN version: CUDA 10.0.130_411.31 / cuDNN <IP_ADDRESS>
GPU model and memory: GTX 1060 6GB

You can collect some of this information using our environment capture
script
You can also obtain the TensorFlow version with: 1. TF 1.0: python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)" 2. TF 2.0: python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"
Describe the current behavior
When enabling masking on Embedding layer using keras subclassed model, an error is raised during model.fit():
2019-07-09 11:00:33.231719: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:502] constant folding failed: Invalid argument: Unsupported type: 21
This has a huge impact on training speed, that scales with the amount of data, in my full code, each epoch take more than 1 minute to complete(dataset of 869 tokenized strings, for train an encoder/decoder model), the code for reproduce this error dont have a noticeable impact because there is only one sample, already tokenized.
The error only is showed when an Embedding layer with mask_zeros=True, if followed by an LSTM/GRU layer
Describe the expected behavior
LSTM/GRU supporting zero masking, and having no impact of training speed
Code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate the problem.
import tensorflow as tf
import numpy as np

class lstm(tf.keras.Model):
    def __init__(self):
        super(lstm, self).__init__()

        self.embedding = tf.keras.layers.Embedding(300, 2, mask_zero=True, trainable=True)
        self.encoder = tf.keras.layers.LSTM(2, return_sequences=True, return_state=False)

    def call(self, inputs):
        output = self.embedding(inputs)
        output = self.encoder(output)
        return output[0]

input_questions = np.array([[5, 12, 13, 189, 10, 95, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]).astype(float)
output = np.array([[-0.00299482, 0.00096033]])

dataset = tf.data.Dataset.from_tensor_slices((input_questions, output)).shuffle(2).batch(1)

model = lstm()
model.compile(tf.keras.optimizers.Adadelta(1.0), tf.keras.losses.MeanSquaredError())
model.fit(dataset, epochs=10, verbose=2)
for sample, target in dataset.take(1):
    model(sample)

Other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.
What the minimal code example above show in the console for me
2019-07-09 11:10:25.468004: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library nvcuda.dll                                                                                2019-07-09 11:10:26.569016: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:    name: GeForce GTX 1060 major: 6 minor: 1 memoryClockRate(GHz): 1.733                                                    pciBusID: 0000:01:00.0                                                                                2019-07-09 11:10:26.588454: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.                                                                                2019-07-09 11:10:26.604012: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0      2019-07-09 11:10:26.619218: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2                                                                       2019-07-09 11:10:26.643472: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:    name: GeForce GTX 1060 major: 6 minor: 1 memoryClockRate(GHz): 1.733                                                    pciBusID: 0000:01:00.0                                                                                2019-07-09 11:10:26.670895: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.                                                                                2019-07-09 11:10:26.688869: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0      2019-07-09 11:10:28.438683: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:                                                                                2019-07-09 11:10:28.466078: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0                             2019-07-09 11:10:28.480348: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N                             2019-07-09 11:10:28.500656: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4712 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060, pci bus id: 0000:01:00.0, compute capability: 6.1)

Epoch 1/10

WARNING: Logging before flag parsing goes to stderr.
W0709 11:10:30.293739 15940 deprecation.py:323] From C:\ProgramData\Anaconda3\envs\tf2.0-gpu\lib\site-packages\tensorflow\python\keras\backend.py:3868: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where

2019-07-09 11:10:38.997986: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:502] constant folding failed: Invalid argument: Unsupported type: 21

2019-07-09 11:10:39.653444: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:502] constant folding failed: Invalid argument: Unsupported type: 21

1/1 - 13s - loss: 1.5416e-05
Epoch 2/10
1/1 - 1s - loss: 1.1537e-05
Epoch 3/10
1/1 - 1s - loss: 8.7570e-06
Epoch 4/10
1/1 - 1s - loss: 6.8737e-06
Epoch 5/10
1/1 - 1s - loss: 5.6433e-06
Epoch 6/10
1/1 - 1s - loss: 4.8554e-06
Epoch 7/10
1/1 - 1s - loss: 4.3538e-06
Epoch 8/10
1/1 - 1s - loss: 4.0314e-06
Epoch 9/10
1/1 - 1s - loss: 3.8191e-06
Epoch 10/10
1/1 - 1s - loss: 3.6735e-06

Press any key to continue . . .

Edit 01: i think is somewhat ralated to this on https://github.com/tensorflow/tensorflow/issues/29525

I manage to throw out some warnings, leaving exclusively:
2019-07-09 17:19:34.150417: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:502] constant folding failed: Invalid argument: Unsupported type: 21
The training speed impact of this error is still there, and the unique way i found for disable it completely is making trainable=False on the layer constructor:
self.embedding = tf.keras.layers.Embedding(300, 2, trainable=False)
here is the new code:
import tensorflow as tf
import numpy as np

class lstm(tf.keras.Model):
    def __init__(self):
        super(lstm, self).__init__()

        self.masking = tf.keras.layers.Masking()
        self.embedding = tf.keras.layers.Embedding(300, 2, trainable=True)
        self.encoder = tf.keras.layers.LSTM(2, "sigmoid", return_sequences=True, return_state=False)

    def call(self, inputs):
        output = self.masking(inputs)
        output = self.embedding(output)
        output = self.encoder(output)
        return output[0]

input_questions = np.array([[5, 12, 13, 189, 10, 95, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]).astype(float)
output = np.array([[-0.00299482, 0.00096033]])

dataset = tf.data.Dataset.from_tensor_slices((input_questions, output)).shuffle(2).batch(1)

model = lstm()
model.compile(tf.keras.optimizers.Adadelta(1.0), tf.keras.losses.MeanSquaredError())
model.fit(dataset, epochs=10, verbose=2)
for sample, target in dataset.take(1):
    model(sample)

As you can see, now instead of use mask_zeros=True in the Embedding layer constructor, i leave it false (as default), and use the tf.keras.layers.Masking, this give me the masking feature without any errors

@ElPapi42 I tried reproducing the issue and i do not see any errors while fitting the model. Please, see https://colab.sandbox.google.com/drive/11g3kQiNYk4XTzp4h550-gPGCbUdemGGR#scrollTo=ewDZ7_374EJO for your reference.Thanks!

@ravikyram I tried reproducing the issue too, the reported warnings can be found in the collab runtime logs

@ElPapi42 I tried reproducing the issue and i do not see any errors while fitting the model. Please, see https://colab.sandbox.google.com/drive/11g3kQiNYk4XTzp4h550-gPGCbUdemGGR#scrollTo=ewDZ7_374EJO for your reference.Thanks!

I dont have access to the link, request an account with access permisions

Please, see screenshot.Thanks!

@ravikyram I tried reproducing the issue too, the reported warnings can be found in the collab runtime logs

@ravikyram Please look at the runtime logs. The fact that this kind of issue will only show up in logs when ran in collab has already been noted in #30263 - and this is an actual issue, reported on multiple occasions but seemingly unsolved as of now...

Adding Eugene who is the expert of grappler.

@rmlarsen rings any bells?

Any advance on this? its really annoying

I manage to throw out some warnings, leaving exclusively:
2019-07-09 17:19:34.150417: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:502] constant folding failed: Invalid argument: Unsupported type: 21
The training speed impact of this error is still there, and the unique way i found for disable it completely is making trainable=False on the layer constructor:
self.embedding = tf.keras.layers.Embedding(300, 2, trainable=False)
here is the new code:
import tensorflow as tf
import numpy as np

class lstm(tf.keras.Model):
    def __init__(self):
        super(lstm, self).__init__()

        self.masking = tf.keras.layers.Masking()
        self.embedding = tf.keras.layers.Embedding(300, 2, trainable=True)
        self.encoder = tf.keras.layers.LSTM(2, "sigmoid", return_sequences=True, return_state=False)

    def call(self, inputs):
        output = self.masking(inputs)
        output = self.embedding(output)
        output = self.encoder(output)
        return output[0]

input_questions = np.array([[5, 12, 13, 189, 10, 95, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]).astype(float)
output = np.array([[-0.00299482, 0.00096033]])

dataset = tf.data.Dataset.from_tensor_slices((input_questions, output)).shuffle(2).batch(1)

model = lstm()
model.compile(tf.keras.optimizers.Adadelta(1.0), tf.keras.losses.MeanSquaredError())
model.fit(dataset, epochs=10, verbose=2)
for sample, target in dataset.take(1):
    model(sample)

As you can see, now instead of use mask_zeros=True in the Embedding layer constructor, i leave it false (as default), and use the tf.keras.layers.Masking, this give me the masking feature without any errors. The only one still there is the topic of the Issue.

I discover something, in the code above, i make sigmoid the activation of the lstm for get rid of:
2019-07-19 09:01:52.787220: W tensorflow/core/grappler/optimizers/implementation_selector.cc:199] Skipping optimization due to error while loading function libraries: Invalid argument: Functions '__inference___backward_cudnn_lstm_395_571' and '__inference___backward_cudnn_lstm_395_571_specialized_for_Adadelta_gradients_lstm_lstm_1_StatefulPartitionedCall_grad_StatefulPartitionedCall_at___inference_keras_scratch_graph_1552' both implement 'lstm_ca0befc9-b0cc-4e66-adc9-7da601846478' but their signatures do not match
activation="sigmoid" avoid use the cdnn implementation of the LSTM layer. Playing around with the code i notice that leaving activation="tanh" (allowing cDDN implementation), suppress the error Unsupported Type 21, but creates a new one related to the signatures, the one that push me before to make activation="sigmoid" on lstm layer
here is the new code, that not produce Unsupported Type 21 error
import tensorflow as tf
import numpy as np

class lstm(tf.keras.Model):
    def __init__(self):
        super(lstm, self).__init__()

        self.masking = tf.keras.layers.Masking()
        self.embedding = tf.keras.layers.Embedding(300, 2, trainable=True)
        self.encoder = tf.keras.layers.LSTM(2, "tanh", return_sequences=True, return_state=False)

    def call(self, inputs):
        output = self.masking(inputs)
        output = self.embedding(output)
        output = self.encoder(output)
        return output[0]

input_questions = np.array([[5, 12, 13, 189, 10, 95, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]).astype(float)
output = np.array([[-0.00299482, 0.00096033]])

dataset = tf.data.Dataset.from_tensor_slices((input_questions, output)).shuffle(2).batch(1)

model = lstm()
model.compile(tf.keras.optimizers.Adadelta(1.0), tf.keras.losses.MeanSquaredError())
model.fit(dataset, epochs=10, verbose=2)
for sample, target in dataset.take(1):
    model(sample)

This can point that the Unsupported Type 21 error is produced on the Standard LSTM implementation of Tensorflow
The problem with the new error is that tells skip the optimization completely, and that sounds bad

Obviously the Unsupported Type 21 Error is something that must be checked, there is something broke

This was fixed in https://github.com/tensorflow/tensorflow/commit/24174643a75e819b8ce01fd70d45d03616e50071, should be in nightly build

For the implementation selector problems I think there is work in progress by @qlzh727.

Can you give the pip command for indtal tf2.0 nightly build?

any advances on this issue?

Any patches or solutions ? please we need some help . thank per advance

This was fixed in 2417464, should be in nightly build

pip install tf-nightly-gpu-2.0-preview or pip install tf-nightly-2.0-preview (depending whether you want GPU support or not ; you could also want and get a nightly build of tf 1.15 instead)
That being said, the fix that was deployed three weeks ago merely avoids the warning message (which saves a little time and avoids wary print outs), but I believe the underlying issue has not been fixed yet (hopefully this is indeed work in progress).

@ElPapi42 Is this still an issue? Can you check with TF2.0 and let us know whether the issue persists with latest TF version. I ran with TF2.0 and don't see any constant folding error. Please check the gist here Thanks!

I think this was solved, but i dont have time to check right now. The project where this appeared is in an unusable state right now, must work on it for test.

I am closing the issue as it was resolved. Please feel free to open it if the issue persists again. Thanks!


[Enhancement] Support asynchronous profile processes

Fixes #issue
What type of PR is this:

[ ] BugFix
[ ] Feature
[x] Enhancement
[ ] Refactor
[ ] UT
[ ] Doc
[ ] Tool

Does this PR entail a change in behavior?

[ ] Yes, this PR will result in a change in behavior.
[x] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

[ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
[ ] Parameter changes: default values, similar parameters but with different default values
[ ] Policy changes: use new policy to replace old one, functionality automatically enabled
[ ] Feature removed
[ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

[ ] I have added test cases for my bug fix or my new feature
[ ] This pr needs user documentation (for new or modified features or behaviors)

[ ] I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

[ ] I have checked the version labels which the pr will be auto-backported to the target branch

[x] 3.2
[ ] 3.1
[ ] 3.0
[ ] 2.5

[FE Incremental Coverage Report]
:disappointed: fail : 17 / 52 (32.69%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/load/loadv2/LoadLoadingTask.java
0
1
00.00%
[219]

:large_blue_circle:
com/starrocks/task/ExportExportingTask.java
0
1
00.00%
[376]

:large_blue_circle:
com/starrocks/qe/QeProcessorImpl.java
4
24
16.67%
[111, 112, 113, 114, 115, 116, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 205, 206]

:large_blue_circle:
com/starrocks/qe/StmtExecutor.java
8
21
38.10%
[623, 624, 626, 780, 781, 782, 785, 787, 788, 789, 1668, 1684, 1687]

:large_blue_circle:
com/starrocks/qe/SessionVariable.java
3
3
100.00%
[]

:large_blue_circle:
com/starrocks/common/Config.java
2
2
100.00%
[]

[BE Incremental Coverage Report]
:heart_eyes: pass : 0 / 0 (0%)

[FE Incremental Coverage Report]
:disappointed: fail : 37 / 103 (35.92%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/load/loadv2/LoadLoadingTask.java
0
1
00.00%
[219]

:large_blue_circle:
com/starrocks/sql/ExplainAnalyzer.java
0
5
00.00%
[365, 366, 367, 368, 370]

:large_blue_circle:
com/starrocks/task/ExportExportingTask.java
0
1
00.00%
[376]

:large_blue_circle:
com/starrocks/load/streamload/StreamLoadTask.java
0
1
00.00%
[1060]

:large_blue_circle:
com/starrocks/qe/SessionVariable.java
1
4
25.00%
[2114, 2118, 2119]

:large_blue_circle:
com/starrocks/qe/StmtExecutor.java
8
32
25.00%
[333, 615, 623, 624, 781, 782, 784, 785, 786, 787, 788, 789, 791, 792, 793, 794, 795, 796, 797, 920, 1677, 1684, 1685, 1703]

:large_blue_circle:
com/starrocks/qe/DefaultCoordinator.java
5
18
27.78%
[421, 949, 950, 952, 953, 955, 956, 957, 959, 960, 961, 962, 964]

:large_blue_circle:
com/starrocks/qe/scheduler/QueryRuntimeProfile.java
2
6
33.33%
[221, 222, 225, 226]

:large_blue_circle:
com/starrocks/common/util/concurrent/MarkedCountDownLatch.java
8
15
53.33%
[65, 66, 110, 111, 112, 113, 114]

:large_blue_circle:
com/starrocks/qe/QeProcessorImpl.java
10
17
58.82%
[114, 115, 116, 118, 124, 125, 129]

:large_blue_circle:
com/starrocks/scheduler/PartitionBasedMvRefreshProcessor.java
1
1
100.00%
[]

:large_blue_circle:
com/starrocks/common/Config.java
2
2
100.00%
[]

[BE Incremental Coverage Report]
:heart_eyes: pass : 0 / 0 (0%)

[FE Incremental Coverage Report]
:disappointed: fail : 49 / 108 (45.37%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/load/loadv2/LoadLoadingTask.java
0
1
00.00%
[219]

:large_blue_circle:
com/starrocks/sql/ExplainAnalyzer.java
0
5
00.00%
[365, 366, 367, 368, 370]

:large_blue_circle:
com/starrocks/task/ExportExportingTask.java
0
1
00.00%
[376]

:large_blue_circle:
com/starrocks/load/streamload/StreamLoadTask.java
0
1
00.00%
[1060]

:large_blue_circle:
com/starrocks/qe/StmtExecutor.java
8
35
22.86%
[333, 615, 623, 624, 781, 782, 786, 788, 789, 790, 793, 794, 795, 796, 798, 799, 800, 801, 803, 804, 805, 806, 929, 1686, 1693, 1694, 1712]

:large_blue_circle:
com/starrocks/qe/scheduler/QueryRuntimeProfile.java
2
6
33.33%
[221, 222, 225, 226]

:large_blue_circle:
com/starrocks/qe/SessionVariable.java
2
4
50.00%
[2118, 2119]

:large_blue_circle:
com/starrocks/common/util/concurrent/MarkedCountDownLatch.java
8
15
53.33%
[65, 66, 110, 111, 112, 113, 114]

:large_blue_circle:
com/starrocks/qe/QeProcessorImpl.java
12
19
63.16%
[114, 115, 116, 117, 119, 125, 126]

:large_blue_circle:
com/starrocks/qe/DefaultCoordinator.java
14
18
77.78%
[421, 950, 953, 964]

:large_blue_circle:
com/starrocks/scheduler/PartitionBasedMvRefreshProcessor.java
1
1
100.00%
[]

:large_blue_circle:
com/starrocks/common/Config.java
2
2
100.00%
[]

[BE Incremental Coverage Report]
:heart_eyes: pass : 0 / 0 (0%)

[FE Incremental Coverage Report]
:disappointed: fail : 50 / 117 (42.74%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/load/loadv2/LoadLoadingTask.java
0
1
00.00%
[219]

:large_blue_circle:
com/starrocks/sql/ExplainAnalyzer.java
0
5
00.00%
[365, 366, 367, 368, 370]

:large_blue_circle:
com/starrocks/task/ExportExportingTask.java
0
1
00.00%
[376]

:large_blue_circle:
com/starrocks/load/streamload/StreamLoadTask.java
0
1
00.00%
[1060]

:large_blue_circle:
com/starrocks/qe/StmtExecutor.java
8
35
22.86%
[333, 615, 623, 624, 781, 782, 786, 788, 789, 790, 793, 794, 795, 796, 798, 799, 800, 801, 803, 804, 805, 806, 929, 1686, 1693, 1694, 1712]

:large_blue_circle:
com/starrocks/qe/scheduler/QueryRuntimeProfile.java
2
6
33.33%
[221, 222, 225, 226]

:large_blue_circle:
com/starrocks/common/util/concurrent/MarkedCountDownLatch.java
9
24
37.50%
[66, 67, 68, 112, 113, 114, 115, 116, 120, 123, 124, 125, 129, 130, 132]

:large_blue_circle:
com/starrocks/qe/SessionVariable.java
2
4
50.00%
[2118, 2119]

:large_blue_circle:
com/starrocks/qe/QeProcessorImpl.java
12
19
63.16%
[114, 115, 116, 117, 119, 125, 126]

:large_blue_circle:
com/starrocks/qe/DefaultCoordinator.java
14
18
77.78%
[421, 950, 953, 964]

:large_blue_circle:
com/starrocks/scheduler/PartitionBasedMvRefreshProcessor.java
1
1
100.00%
[]

:large_blue_circle:
com/starrocks/common/Config.java
2
2
100.00%
[]

[BE Incremental Coverage Report]
:heart_eyes: pass : 0 / 0 (0%)

[BE Incremental Coverage Report]
:heart_eyes: pass : 0 / 0 (0%)

[FE Incremental Coverage Report]
:disappointed: fail : 40 / 125 (32.00%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/load/loadv2/LoadLoadingTask.java
0
1
00.00%
[219]

:large_blue_circle:
com/starrocks/sql/ExplainAnalyzer.java
0
5
00.00%
[365, 366, 367, 368, 370]

:large_blue_circle:
com/starrocks/task/ExportExportingTask.java
0
1
00.00%
[376]

:large_blue_circle:
com/starrocks/load/streamload/StreamLoadTask.java
0
1
00.00%
[1060]

:large_blue_circle:
com/starrocks/qe/StmtExecutor.java
8
41
19.51%
[616, 624, 625, 782, 783, 787, 789, 790, 791, 792, 796, 797, 798, 799, 800, 801, 804, 805, 806, 807, 809, 810, 811, 812, 814, 815, 816, 817, 940, 1697, 1704, 1705, 1723]

:large_blue_circle:
com/starrocks/qe/scheduler/QueryRuntimeProfile.java
2
8
25.00%
[222, 223, 226, 227, 228, 229]

:large_blue_circle:
com/starrocks/qe/SessionVariable.java
1
4
25.00%
[2143, 2147, 2148]

:large_blue_circle:
com/starrocks/qe/DefaultCoordinator.java
5
18
27.78%
[422, 950, 951, 953, 954, 956, 957, 958, 960, 961, 962, 963, 965]

:large_blue_circle:
com/starrocks/common/util/concurrent/MarkedCountDownLatch.java
9
24
37.50%
[66, 67, 68, 112, 113, 114, 115, 116, 120, 123, 124, 125, 129, 130, 132]

:large_blue_circle:
com/starrocks/qe/QeProcessorImpl.java
12
19
63.16%
[114, 115, 116, 117, 119, 125, 126]

:large_blue_circle:
com/starrocks/scheduler/PartitionBasedMvRefreshProcessor.java
1
1
100.00%
[]

:large_blue_circle:
com/starrocks/common/Config.java
2
2
100.00%
[]

[BE Incremental Coverage Report]
:heart_eyes: pass : 0 / 0 (0%)

[FE Incremental Coverage Report]
:disappointed: fail : 71 / 129 (55.04%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/load/loadv2/LoadLoadingTask.java
0
1
00.00%
[219]

:large_blue_circle:
com/starrocks/sql/ExplainAnalyzer.java
0
5
00.00%
[365, 366, 367, 368, 370]

:large_blue_circle:
com/starrocks/task/ExportExportingTask.java
0
1
00.00%
[376]

:large_blue_circle:
com/starrocks/load/streamload/StreamLoadTask.java
0
1
00.00%
[1060]

:large_blue_circle:
com/starrocks/qe/StmtExecutor.java
8
41
19.51%
[616, 624, 625, 782, 783, 787, 789, 790, 791, 792, 796, 797, 798, 799, 800, 801, 804, 805, 806, 807, 809, 810, 811, 812, 814, 815, 816, 817, 940, 1697, 1704, 1705, 1723]

:large_blue_circle:
com/starrocks/qe/SessionVariable.java
2
4
50.00%
[2147, 2148]

:large_blue_circle:
com/starrocks/qe/DefaultCoordinator.java
15
22
68.18%
[422, 951, 954, 966, 967, 968, 969]

:large_blue_circle:
com/starrocks/qe/QeProcessorImpl.java
14
19
73.68%
[114, 115, 116, 117, 119]

:large_blue_circle:
com/starrocks/qe/scheduler/QueryRuntimeProfile.java
7
8
87.50%
[223]

:large_blue_circle:
com/starrocks/common/util/concurrent/MarkedCountDownLatch.java
22
24
91.67%
[113, 114]

:large_blue_circle:
com/starrocks/scheduler/PartitionBasedMvRefreshProcessor.java
1
1
100.00%
[]

:large_blue_circle:
com/starrocks/common/Config.java
2
2
100.00%
[]

[BE Incremental Coverage Report]
:heart_eyes: pass : 0 / 0 (0%)

[FE Incremental Coverage Report]
:disappointed: fail : 54 / 129 (41.86%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/load/loadv2/LoadLoadingTask.java
0
1
00.00%
[219]

:large_blue_circle:
com/starrocks/sql/ExplainAnalyzer.java
0
5
00.00%
[365, 366, 367, 368, 370]

:large_blue_circle:
com/starrocks/task/ExportExportingTask.java
0
1
00.00%
[376]

:large_blue_circle:
com/starrocks/load/streamload/StreamLoadTask.java
0
1
00.00%
[1060]

:large_blue_circle:
com/starrocks/qe/StmtExecutor.java
8
41
19.51%
[616, 624, 625, 782, 783, 787, 789, 790, 791, 792, 796, 797, 798, 799, 800, 801, 804, 805, 806, 807, 809, 810, 811, 812, 814, 815, 816, 817, 940, 1697, 1704, 1705, 1723]

:large_blue_circle:
com/starrocks/qe/scheduler/QueryRuntimeProfile.java
2
8
25.00%
[222, 223, 226, 227, 228, 229]

:large_blue_circle:
com/starrocks/common/util/concurrent/MarkedCountDownLatch.java
9
24
37.50%
[66, 67, 68, 112, 113, 114, 115, 116, 120, 123, 124, 125, 129, 130, 132]

:large_blue_circle:
com/starrocks/qe/SessionVariable.java
2
4
50.00%
[2147, 2148]

:large_blue_circle:
com/starrocks/qe/QeProcessorImpl.java
12
19
63.16%
[114, 115, 116, 117, 119, 125, 126]

:large_blue_circle:
com/starrocks/qe/DefaultCoordinator.java
18
22
81.82%
[422, 951, 954, 971]

:large_blue_circle:
com/starrocks/scheduler/PartitionBasedMvRefreshProcessor.java
1
1
100.00%
[]

:large_blue_circle:
com/starrocks/common/Config.java
2
2
100.00%
[]

[BE Incremental Coverage Report]
:heart_eyes: pass : 0 / 0 (0%)

[FE Incremental Coverage Report]
:disappointed: fail : 54 / 129 (41.86%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/load/loadv2/LoadLoadingTask.java
0
1
00.00%
[219]

:large_blue_circle:
com/starrocks/sql/ExplainAnalyzer.java
0
5
00.00%
[365, 366, 367, 368, 370]

:large_blue_circle:
com/starrocks/task/ExportExportingTask.java
0
1
00.00%
[376]

:large_blue_circle:
com/starrocks/load/streamload/StreamLoadTask.java
0
1
00.00%
[1060]

:large_blue_circle:
com/starrocks/qe/StmtExecutor.java
8
41
19.51%
[616, 624, 625, 782, 783, 787, 789, 790, 791, 792, 796, 797, 798, 799, 800, 801, 804, 805, 806, 807, 809, 810, 811, 812, 814, 815, 816, 817, 940, 1697, 1704, 1705, 1723]

:large_blue_circle:
com/starrocks/qe/scheduler/QueryRuntimeProfile.java
2
8
25.00%
[222, 223, 226, 227, 228, 229]

:large_blue_circle:
com/starrocks/common/util/concurrent/MarkedCountDownLatch.java
9
24
37.50%
[66, 67, 68, 112, 113, 114, 115, 116, 120, 123, 124, 125, 129, 130, 132]

:large_blue_circle:
com/starrocks/qe/SessionVariable.java
2
4
50.00%
[2147, 2148]

:large_blue_circle:
com/starrocks/qe/QeProcessorImpl.java
12
19
63.16%
[114, 115, 116, 117, 119, 125, 126]

:large_blue_circle:
com/starrocks/qe/DefaultCoordinator.java
18
22
81.82%
[422, 951, 954, 971]

:large_blue_circle:
com/starrocks/scheduler/PartitionBasedMvRefreshProcessor.java
1
1
100.00%
[]

:large_blue_circle:
com/starrocks/common/Config.java
2
2
100.00%
[]

@Mergifyio backport branch-3.1