[extractor/nebula] Support downloading public videos without an account

IMPORTANT: PRs without the template will be CLOSED
Description of your pull request and other information

As the linked issues describes, some Nebula videos can be watched without an account. The extractor has been updated to reflect that the app always requests a guest Bearer token, but to get this token we don't need to be authenticated.
I don't have a premium account, but maybe @hheimbuerger or someone else can test it with pay-walled videos just to make sure it works.
Fixes #4300
Downloading public video
[debug] Command-line config: ['https://nebula.tv/videos/tldrnewseu-did-the-us-really-blow-up-the-nordstream-pipelines', '--verbose', '-F']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2023.02.17 [a0a7c0154] (source)
[debug] Lazy loading extractors is disabled
[debug] Git HEAD: 41432863b
[debug] Python 3.10.7 (CPython x86_64 64bit) - Linux-5.19.0-31-generic-x86_64-with-glibc2.36 (OpenSSL 3.0.5 5 Jul 2022, glibc 2.36)
[debug] exe versions: ffmpeg N-108931-g4dda3b1653-20221104 (setts), ffprobe N-108931-g4dda3b1653-20221104, phantomjs broken, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.15.0, certifi-2022.09.24, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Loaded 1782 extractors
[Nebula] Authorizing to Nebula
[Nebula] Extracting URL: https://nebula.tv/videos/tldrnewseu-did-the-us-really-blow-up-the-nordstream-pipelines
[Nebula] tldrnewseu-did-the-us-really-blow-up-the-nordstream-pipelines: Fetching video meta data
[Nebula] tldrnewseu-did-the-us-really-blow-up-the-nordstream-pipelines: Fetching video stream info
[Nebula] tldrnewseu-did-the-us-really-blow-up-the-nordstream-pipelines: Downloading m3u8 information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), channels, acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[info] Available formats for 63f64c74366fcd00017c1513:
ID            EXT  RESOLUTION FPS │   FILESIZE   TBR PROTO │ VCODEC          VBR ACODEC     MORE INFO
─────────────────────────────────────────────────────────────────────────────────────────────────────────
audio-Unknown m3u8 audio only     │                  m3u8  │ audio only          unknown    [und] Unknown
456           mp4  640x360     30 │ ~ 29.19MiB  456k m3u8  │ avc1.42C01E    456k video only
815           mp4  960x540     30 │ ~ 52.16MiB  815k m3u8  │ avc1.42C028    815k video only
1361          mp4  1280x720    30 │ ~ 87.11MiB 1362k m3u8  │ avc1.640029   1362k video only
1273          mp4  1920x1080   30 │ ~ 81.45MiB 1273k m3u8  │ hvc1.2.4.L123 1273k video only
2008          mp4  2560x1440   30 │ ~128.48MiB 2009k m3u8  │ hvc1.2.4.L150 2009k video only
4419          mp4  3840x2160   30 │ ~282.66MiB 4419k m3u8  │ hvc1.2.4.L153 4419k video only

Template

Before submitting a pull request make sure you have:

[x] At least skimmed through contributing guidelines including yt-dlp coding conventions
[x] Searched the bugtracker for similar pull requests
[x] Checked the code with flake8 and ran relevant tests

In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under Unlicense. Check all of the following options that apply:

[x] I am the original author of this code and I am willing to release it under Unlicense
[ ] I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

[x] Fix or improvement to an extractor (Make sure to add/update tests)
[ ] New extractor (Piracy websites will not be accepted)
[ ] Core bug fix/improvement
[ ] New feature (It is strongly recommended to open an issue first)

I ran a quick test (downloading a channel) with a subscription token and everything's still A-OK. ✅
That said, I find this approach slightly problematic. By (potentially repeatedly) exploiting their "first video's on us, then you need a subscription" policy, we're basically circumventing the requirement for a subscription entirely. Or am I missing something, that allows you to do this only once? (But once per... what?)
You mention that "some Nebula videos can be watched without an account", but is it really video-specific? Could you link to a video that can not be watched without an account — or even better: create a test case, so we can see what the UI message is and how it will be handled?
Admittedly, they didn't put effective measures in place to stop this. So I guess they'll do that once they hear about this, and then this feature will be gone again.

That said, I find this approach slightly problematic. By (potentially repeatedly) exploiting their "first video's on us, then you need a subscription" policy, we're basically circumventing the requirement for a subscription entirely. Or am I missing something, that allows you to do this only once? (But once per... what?)

Once per "browser session" according to their implementation. It is fair game for us to use that imo. As it stands, anyone with basic knowledge of cookies can easily circumvent it. If they wanted better restrictions, they should implement it per-IP etc

By (potentially repeatedly) exploiting their "first video's on us, then you need a subscription" policy, we're basically circumventing the requirement for a subscription entirely.

Yep, it's video specific. For example this channel (https://nebula.tv/johnnyharris) publishes his next video (Unemployment, Explained) one week early, exclusively on Nebula.
Some other premium-only are:

https://nebula.tv/videos/extrahistory-hero-or-terrorist-john-brown-1
https://nebula.tv/videos/elizabethfilips-animal-farm-dictatorships-and-abuse-book-club

In general if the video has a blue cross or a flash icon, a subscription is required

On a side note, yt-dlp "https://nebula.tv/johnnyharris" --flat-playlist --skip-download --print "url,title" is not working, probably I'll look into it

On a side note, yt-dlp "https://nebula.tv/johnnyharris" --flat-playlist --skip-download --print "url,title" is not working, probably I'll look into it

I think that's a known issue and pukkandan has already started working on it, see https://github.com/yt-dlp/yt-dlp/pull/5979#issuecomment-1434338439.

Log of updated NebulaChannelIE
[debug] Command-line config: ['https://nebula.tv/johnnyharris', '--verbose', '-F']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2023.02.17 [a0a7c0154] (source)
[debug] Lazy loading extractors is disabled
[debug] Git HEAD: 0b85236b4
[debug] Python 3.10.7 (CPython x86_64 64bit) - Linux-5.19.0-31-generic-x86_64-with-glibc2.36 (OpenSSL 3.0.5 5 Jul 2022, glibc 2.36)
[debug] exe versions: ffmpeg N-108931-g4dda3b1653-20221104 (setts), ffprobe N-108931-g4dda3b1653-20221104, phantomjs broken, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.15.0, certifi-2022.09.24, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Loaded 1782 extractors
[nebula:channel] Authorizing to Nebula
[nebula:channel] Extracting URL: https://nebula.tv/johnnyharris
[nebula:channel] johnnyharris: Retrieving channel
[download] Downloading playlist: Johnny Harris
[nebula:channel] johnnyharris: Downloading page 1
[nebula:channel] johnnyharris: Downloading page 2
[nebula:channel] johnnyharris: Downloading page 3
[nebula:channel] johnnyharris: Downloading page 4
[nebula:channel] Playlist Johnny Harris: Downloading 90 items of 90
[download] Downloading item 1 of 90
[Nebula] Authorizing to Nebula
[Nebula] Extracting URL: https://nebula.tv/videos/johnnyharris-unemployment-explained/
[Nebula] johnnyharris-unemployment-explained: Fetching video meta data
[Nebula] johnnyharris-unemployment-explained: Fetching video stream info
[Nebula] Reauthenticating to Nebula and retrying, because last bearer call resulted in error 401
ERROR: [Nebula] johnnyharris-unemployment-explained: This video is only available for registered users. Use --username and --password, or --netrc (watchnebula) to provide account credentials
  File "/home/amish/Documents/Programming/yt-dlp/yt_dlp/extractor/common.py", line 693, in extract
    ie_result = self._real_extract(url)
  File "/home/amish/Documents/Programming/yt-dlp/yt_dlp/extractor/nebula.py", line 238, in _real_extract
    return self._build_video_info(video)
  File "/home/amish/Documents/Programming/yt-dlp/yt_dlp/extractor/nebula.py", line 111, in _build_video_info
    fmts, subs = self._fetch_video_formats(episode['slug'])
  File "/home/amish/Documents/Programming/yt-dlp/yt_dlp/extractor/nebula.py", line 77, in _fetch_video_formats
    stream_info = self._call_nebula_api(f'https://content.watchnebula.com/video/{slug}/stream/',
  File "/home/amish/Documents/Programming/yt-dlp/yt_dlp/extractor/nebula.py", line 62, in _call_nebula_api
    self._perform_login()
  File "/home/amish/Documents/Programming/yt-dlp/yt_dlp/extractor/nebula.py", line 120, in _perform_login
    self._nebula_api_token = self._perform_nebula_auth(username, password)
  File "/home/amish/Documents/Programming/yt-dlp/yt_dlp/extractor/nebula.py", line 27, in _perform_nebula_auth
    self.raise_login_required(method='password')
  File "/home/amish/Documents/Programming/yt-dlp/yt_dlp/extractor/common.py", line 1153, in raise_login_required
    raise ExtractorError(msg, expected=True)

[download] Downloading item 2 of 90
[Nebula] Extracting URL: https://nebula.tv/videos/johnnyharris-milk-is-a-lie/
[Nebula] johnnyharris-milk-is-a-lie: Fetching video meta data
[Nebula] johnnyharris-milk-is-a-lie: Fetching video stream info
[Nebula] johnnyharris-milk-is-a-lie: Downloading m3u8 information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), channels, acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[info] Available formats for 63d91620a645210001b587e5:
ID            EXT  RESOLUTION FPS │   FILESIZE    TBR PROTO │ VCODEC           VBR ACODEC     MORE INFO
──────────────────────────────────────────────────────────────────────────────────────────────────────────
audio-English m3u8 audio only     │                   m3u8  │ audio only           unknown    [en] English
606           mp4  640x360     24 │ ~124.51MiB   606k m3u8  │ avc1.42C01E     606k video only
1359          mp4  960x540     24 │ ~279.16MiB  1360k m3u8  │ avc1.42C028    1360k video only
3072          mp4  1280x720    24 │ ~630.80MiB  3072k m3u8  │ avc1.640029    3072k video only
3197          mp4  1920x1080   24 │ ~656.44MiB  3197k m3u8  │ hvc1.2.4.L123  3197k video only
5193          mp4  2560x1440   24 │ ~  1.04GiB  5194k m3u8  │ hvc1.2.4.L150  5194k video only
12033         mp4  3840x2160   24 │ ~  2.41GiB 12033k m3u8  │ hvc1.2.4.L153 12033k video only
[download] Downloading item 3 of 90
[Nebula] Extracting URL: https://nebula.tv/videos/johnnyharris-how-youtubers-are-escaping-putins-censorship-machine/
[Nebula] johnnyharris-how-youtubers-are-escaping-putins-censorship-machine: Fetching video meta data
[Nebula] johnnyharris-how-youtubers-are-escaping-putins-censorship-machine: Fetching video stream info
[Nebula] johnnyharris-how-youtubers-are-escaping-putins-censorship-machine: Downloading m3u8 information

I reverted all changes unrelated to the original PR. I will open a new PR fixing --flat-playlist


$api->getNoWatermark

$api->getNoWatermark("https://www.tiktok.com/@zachking/video/6829303572832750853");
why result returns false?

update your copy of package to tag 1.1.5
The issue has been fixed.

hi
getNoWatermark return false at the moment
i try with this uri : tiktok.com/@ohmygoal/video/6906975178568338690


app-snmpd: convert lua CBI to JS

Eliminates luci-compat, but no other changes.
Signed-off-by: Karl Palsson<EMAIL_ADDRESS>

Merged with slight changes via 693bf9523ebe5a008baa2ff017e7e267a9e72ceb


[RTC] Include /etc/map url mappings in archetype

User Story:
As an AEM admin/developer, I want url mappings from long urls to short urls to be provided so that I can easily provide SEO friendly URLS
Acceptance Criteria:

/content/{siteName} is hidden from the urls when written in links by AEM components
AEM project archetype auto-generates /etc/map rules based on archetype inputs
/etc/map rules can be customized by project team during development as needed eg. to make the mappings environment specific.

Addition: The mappings should only be applied on publish.

CQ-4266078

Hi @shsteimer, this makes sense!
This will probably require the addition of a domain name variable to the archetype, so that the mapping can be setup accordingly.

I think we should support rewrite rules as part of a Dispatcher definition. /etc is known to be deprecated and until there is a new location, IMO, the best place for managing these is in the dispatcher.

We have basic url rewrite rules in place. Closing this as the remaining /etc rules is not something we want to add anymore, because of it being deprecated.


Is Postgres-XL supported yet?

Unfortunately Postgres-XL support is not planned for the near future.

Hi @LucasManto - We built cstore_fdw to work with Postgres and test changes against recent Postgres versions.
When users are looking to scale out their Postgres installations, we recommend using Citus. Citus extends Postgres (rather than forking it) and works with all Postgres extensions, including cstore_fdw.
Could you elaborate a bit more on the use case you were planning to use cstore_fdw for?

I was really looking to scale out. Citus is just what I need.
Thank you @ozgune!


[ADDON] - Add ACK support

As an application team, if ACK is available to me, then I can define my application's AWS resources in my application definition and manage its lifecycle so that I don't have to depend on platform team to provision and manage my team's resources.

ACK support is availlable now. Closing this ticket


Create an Action for Our Sales Funnel

User Story-
I want to implement a targeted action within our sales funnel to nurture leads, address customer needs, and guide them towards making a purchase decision.
Actions-
Clearly outline the objectives of the action within the sales funnel, specifying desired outcomes such as increased lead engagement, improved lead quality, and higher conversion rates.
Results-
Successfully guide leads to progress further in the sales funnel, measured by advancements from initial interest to consideration and, ideally, conversion stages.

Action: Sales page with price tags and pictures and descriptions of the specific product. User-friendly stores that many people can use very easily.


Mumble initialising with onesync enabled

Hi, I'm using onesync up to 48 slots so not the one with the patreon argentum level, the server tells me that onesync is enabled as per photo, but mumble keeps giving me the initialising message

Did you find a fix to this?


Keypresses outside of readchar_linux() are dropped

Chars that are pressed while outside of readchar will be dropped. I presume this is because readchar enters raw mode right before calling read, but returns to cooked mode thereafter.
This means that if you press keys faster than you do your processing keys will get dropped. This manifests as characters showing on your display (as part of the line-editing mode of your tty) but not being captured by readchar.
To reproduce:
from readchar import readchar
from time import sleep
from sys import stdout

captured = ""

for i in range(10):
    c = readchar()
    captured += c
    stdout.write(c)
    stdout.flush()
    sleep(0.1)  # placeholder for "very slow processing"

print("\nactually captured", captured)

Run this, and mash the keyboard for approximately 1 second.
I originally presumed the cause of the problem is that readchar enters raw mode right before calling read, but returns to cooked mode thereafter. However, commenting out the return-to-cooked-mode part of the code does get rid of echoing of uncaptured characters, but does not actually capture the characters that are typed while being outside of readchar.
Original issue below (which reflects less understanding of the issue but has more context)
===
original below.
I'm using python-readchar for a small utility to practice my typing speed. When 2 characters are pressed almost simultaneously, sometimes the following happens: 1 of the 2 chars ends up on screen, but is not captured by python-readchar. This was hard to reproduce, but in the end I managed by using the simple utility script.
This reveals the following timings:
0.095735 1                   <= captured character
0.000025 1                   <= uncaptured character

As you can see, we're in sub-milisecond territory here.
I'm not sure where to take it from here, since I'm a bit out of my depth in the functioning of tty's etc.
Potentially relevant parts of my system:

bash --version GNU bash, version 5.1.8(1)-release (x86_64-pc-linux-gnu)
xfce4-terminal 0.8.10 (Xfce 4.16)
Python 3.9.7
readchar==3.0.4

I wasn't 100% sure this was a readchar-specific problem, but I am now.
Here's a bit of code I "copied of the internet" (with modifications) that does basically the same thing as the "to reproduce" example in the above, but does it correctly, i.e. chars that are pressed during sleep are buffered.
import fcntl
import termios
import sys
import os
import time

class NonBlockingInput(object):

    def __enter__(self):
        # canonical mode, no echo
        self.old = termios.tcgetattr(sys.stdin)
        new = termios.tcgetattr(sys.stdin)
        new[3] = new[3] & ~(termios.ICANON | termios.ECHO)
        termios.tcsetattr(sys.stdin, termios.TCSADRAIN, new)

        # set for non-blocking io
        # orig_fl = fcntl.fcntl(sys.stdin, fcntl.F_GETFL)
        # fcntl.fcntl(sys.stdin, fcntl.F_SETFL, orig_fl | os.O_NONBLOCK)

    def __exit__(self, *args):
        # restore terminal to previous state
        termios.tcsetattr(sys.stdin, termios.TCSADRAIN, self.old)

with NonBlockingInput():
    while True:
        c = sys.stdin.read(1)
        sys.stdout.write(c)
        sys.stdout.flush()
        time.sleep(1)

I found the source of the problem (at least on my system, I don't claim to be knowledgeable enough about TTY stuff to make sweeping statements here):
https://github.com/magmax/python-readchar/blob/ece56e0d6af9f8c7317f5adf72564690064ebc16/readchar/readchar_linux.py#L14
Calling tty.setraw makes it so that subsequent calls to read() don't read any of the typed-but-not-read characters. Reading the code, this could be expected since tty.setraw is called without explicit parameters, and the default for when is TCSAFLUSH, which means:

If optional_actions is TCSAFLUSH, the change shall occur after all output written to fildes is transmitted, and all input so far received but not read shall be discarded before the change is made. (emphasis mine)

Indeed, changing the call to be with TCSADRAIN seems to solve the problem of missing keystrokes.
However, that solves the problem that any keystrokes that occur outside of readchar are dropped fully, but it does not solve the problem that they will be echoed. (They will be echoed because they occur at a point in time that stdin is in "cooked" or "canonical" (non-raw) mode).
My personal conclusion is that the whole idea of switching modes for each character is simply bad architecture, and will never work reliably. Hence, in my own code, I'm going with a solution like that of the decorator.

I also briefly researched what further differences between my working example and setraw could be, but in the end I just looked up the code for setraw which is quite straight-forward. Better just to use setraw then, but not use it for each keypress.

I've created a quickfix PR that at least fixes the problem of missing keystrokes; though it does not fix the problem of echoing them if they are outside the readchar call. For the latter a bigger rewrite would be necessary.

I experimented with this a little bit and also didn't find a really good soloution either. We could use TCSANOW which would allow the keypresses to persist into the readchar() call. But they will still be echoed as you described. I also experimented with implementing a context handler as part of the v4.0.0 development, but it got to complicated very fast, so it's very hard to make libary ready.
So I belive this should be noted as a limitation of the libary, it will only reconfigure the terminal inside its own codebase and ensure that only keypresses after its call will be captured. This probably what most people expect anyway

As I had to revoke my previos statement, The only soloution seems to be to note this as a limitation of the libary on linux. It will not propegare its teminalsettings outside of its own code, so it can only capture keypresses while blocking. As this works fine when running quick enough (or on a seperate thread handeling the input) this seems acceptable, even if its sad.
Maybe developing additional functionalaty for the libary in form of a context-manager would be a interesting idea. But this will be challenging to get libary ready

Hi @Cube707, what exactly is the intent with context manager? Looking at the code, I am not entirely sure what the approach would be. Could you elaborat
Something that occurs to me is attaching some custom handling to "signals" to make sure terminal is reset after KeyInterrupt as demonstrated here.


57 add documentation

To resolve issue #57
There are a few remaining issues to resolve.

The pull request has 2 example notebooks: one with and one without storing and reloading the ordered data before performing the timing tests. To be determined which is better. This one should be called demo_order_stm.ipynb and the other removed.

It seems like stmat.copy does not have the desired result: when I try to run STM operations on the cloned data, there seems to be some parts missing (I think it was the whole .stm part).

It seems that when using small chunk sizes, storing to zarr fails. Solved by storing using 1000 size chunks and rechunking after loading. This is a bit ugly.

Possible issue with storing the ordered data to zarr when this needs to overwrite existing data. I think shutil.retree also sometimes gives issues (maybe when the is no directory).

Timing tests using the prerequisite small chunk size (500 in this case) become very slow. Don't know whether we can really fix this though.
Currently running the final timing tests on both notebooks. To be committed when done.

Thanks @vanlankveldthijs for the implementation. Following our discussion, the next steps can be:

Demonstrate the performance enhancement from re-ordering in a separate md page.
Elaborate on the factors affecting the re-ordering effect: 1) homogeneity of point distribution; 2) spatial spreadness of points. These two factors can influence the choice of chunksize and spatial closeness of points in the same chunk
Only do performance comparision on subset, which should be enough
Use %%timeit for a robust profiling
Still keep an example notebook for the illustration of re-order operation (you can decide either include this in the existing notebooks or make a new one)

Attaching here two example notebooks I made for comparison: notebooks.zip

Consolidated demo notebooks into one.
Fixed stm.copy issue (should've been stm.copy())
Changed demo notebook to only test subset operation and to test this on large and small chunks (relative to dataset) to see impact.

Fixed pytests by checking for existence of time in chunksizes.
Please check whether implementation could be nicer (better python).


feat(prow-jobs): add presubmit job for pingcap/tidb-binlog

Currently it's working in progress.
Signed-off-by: wuhuizuo<EMAIL_ADDRESS>

/approve


remove delegationtoken parameter

What changes were proposed in this pull request?
it is used for build each session's Hive object, no need now and we will have new way to proxy hive auth

Looks fine

Looks fine

add two method for future proxy.


flix: 0.45.0 -> 0.46.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/flix/flix/releases.
meta.description for flix is: The Flix Programming Language
meta.homepage for flix is: https://github.com/flix/flix
Updates performed

Version update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 0.46.0 in filename of file in /nix/store/s1nwdp6fkhjxwanc7ny529960x2qng0k-flix-0.46.0

Rebuild report (if merged into master) (click to expand)

1 total rebuild path(s)

1 package rebuild(s)

First fifty rebuilds by attrpath
flix

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/s1nwdp6fkhjxwanc7ny529960x2qng0k-flix-0.46.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A flix https://github.com/r-ryantm/nixpkgs/archive/036cfda40d303074bac166e46dad352830a62437.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/036cfda40d303074bac166e46dad352830a62437#flix

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/s1nwdp6fkhjxwanc7ny529960x2qng0k-flix-0.46.0
ls -la /nix/store/s1nwdp6fkhjxwanc7ny529960x2qng0k-flix-0.46.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review --extra-nixpkgs-config '{ allowInsecurePredicate = x: true; }' run on x86_64-linux 1

1 package built:

flix

Maintainer pings
cc @athas for testing.

Result of nixpkgs-review pr 304786 run on x86_64-linux 1

1 package built:

flix


Add keycloak section to docs

@nbkrause should probably do a run through to verify this works or is satisfactory

I've manually merged this into #1248


Does grpc-server support dynamically support adding/deleting listening addresses?

When detected a new network port is up, should add the address into the server without affecting other services on other network ports. When detected a network port is down(e.g. disconnected from the hot point due to weak wifi signal), the address should be removed from the server without affecting other services.
Currently,  when detected the above situation, I have to shut down the server, and then reinitializes it again, which will affect the services of other normally working network ports.

gRPC doesn't support this at present but I think this might be a nice-to-have feature. I have no idea when this will become a priority though.

Thanks @yashykt


[v1.7.x] Additional option to read to the end

Hi, I really miss such a function, so I decided to try to implement it myself. I am waiting for an answer to how much this option is correct and whether it will be added to the package.

Hi @StounhandJ, can you explain this feature a little bit?

As I planned, she should receive messages from the topic until the last one, as soon as the answer comes that the topic is empty, he will stop his work.

So if I understood correctly, it should stop the consumer as soon as it detects the end of the topic? If so, I think it is better to rename the method from isReadingToEnd to shouldStopAfterLastMessage and withReadingToEnd to stopAfterLastMessage. Also, it will be helpful only when reading a topic which already has some messages, because if I create a new topic, create a consumer with this option and start consuming, as soon as the first message is read, the consumer will stop.

That's right. If you think this naming is more correct, then I will rename it. Yes, if you start in a new topic where there are no messages, for example, then he will immediately break through, but I believe in some special tasks it will come in handy.

Ok. Yes, please rename it. Also, is it possible to add test?

I haven't examined the tests in this project yet, so it may take some time.

No problems, on your time.

There is also a question about the Makefile, since the assembly for testing inside it is outdated, and specifically this Dockerfile

Unfortunately, I was considering this option. If there is an error in "stopAfterLastMessage", the consume method freezes without the possibility of exiting. May put some expectation in addition to this test.

I think it is worth adding this test, at least we are covering the case when everything works as expected.

What if we try to use it?

Add annotation @medium

Yes, this is exactly the option I was considering. Unfortunately, I can't run the tests locally, so I won't bother configuring it. I described the problem with running tests above.
In docker-compose-test.yaml also crashes the test container at startup.

The docker image was missing. I've just merged a PR to fix it.


java.lang.IllegalStateException: closed when trying to access response in onResponse(Response response)

I get
java.lang.IllegalStateException: closed
            at com.squareup.okhttp.internal.http.HttpConnection$FixedLengthSource.read(HttpConnection.java:455)
            at okio.Buffer.writeAll(Buffer.java:594)
            at okio.RealBufferedSource.readByteArray(RealBufferedSource.java:87)
            at com.squareup.okhttp.ResponseBody.bytes(ResponseBody.java:56)
            at com.squareup.okhttp.ResponseBody.string(ResponseBody.java:82)

when trying to access the response body using:
response.body().string();

inside onResponse(Response response) callback.

You can only call string() once.  true

You can call string() once. But I don't know why

Because response body can be huge so OkHttp doesn’t store it in memory, it reads it as a stream from network when you need it.
When you read body as a string() OkHttp downloads response body and returns it to you without keeping reference to the string, it can’t be downloaded twice without new request.
On Tue, Jul 19, 2016 at 5:46 PM fattolium
<
mailto:fattolium<EMAIL_ADDRESS>
wrote:

You can call string() once. But I don't know why
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly,
https://github.com/square/okhttp/issues/1240#issuecomment-233655904
, or
https://github.com/notifications/unsubscribe-auth/AA7B3EORk4IpQyujb1gMCpIcFpT2uJkWks5qXOMogaJpZM4DMQVK
.
https://github.com/square/okhttp/issues/1240#issuecomment-233655904

once requst  once download

response.body.string should be defined as a variable.but in fact ,i also don't why this.

Pffft. At least give people the option to do that if they want! Exceptions don't help much.

@artem-zinnatullin thanks very much!

Thanks a lot....solved a huge problem for me 👍

if you have used response.body().string() once then you can new one same response  and return in case that you will use  response.body().string() again.

At least it would be nice to add explanation to javadocs about this.
I just broke my production because of twice call to string()

Which Javadocs did you read? That’s where we should add such a note.

It's documented on ResponseBody.
On Wed, Sep 13, 2017 at 11:04 AM Włodzimierz Rożkow <
<EMAIL_ADDRESS>wrote:

At least it would be nice to add explanation to javadocs about this.
I just broke my production because of twice call to string()
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/square/okhttp/issues/1240#issuecomment-329197139, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAEEEZm2BB9oiE_q1GHhuSJ8E3vD_JiRks5sh-72gaJpZM4DMQVK
.

I've read javadocs for string() method in ResponseBody.

Try the following:
ResponseBody responseBodyCopy = response.peekBody(Long.MAX_VALUE);
responseBodyCopy.string();

it works for me

@KaNcHeR It looks like a dirty hack, but it works :D Thanks a lot

Don't try to use response.body.string() directly. Just using like below:
ResponseBody responseBody = response.body();
String content = responseBody.string();
// Do something with "content" variable

@Override
public void onResponse(@NonNull Call call, @NonNull Response response) {
if (response.isSuccessful()) {
                try {

                    final String url = response.request().url().toString();
                    Log.i(TAG, " url: " + url);

                    Thread diffThread = new Thread(new Runnable() {
                    @Override
                     public void run() {

                            try {

                                System.setProperty("http.keepAlive", "false");

                                ResponseBody responseBody = response.body();

                                if (responseBody == null) {
                                    return;
                                }

                                String source = responseBody.string();
                                } catch(Exception e){
                                  ---> Exception! message:closed
                                }
  }

How can I do that? It gives an error if it is in a different thread ?

Response body is not stored in memory. .string() reads as stream from network. That is it.

Try storing response in in string variable separately like this,
String res = response.body().string();
Then res.

how come. it is not working

For me the problem was the wrapper function that returned Response object. Like this
callAPI(...){
.....
client.newCall(request).execute().also {
        return it
    }
...

I tried something like this
Response response =  chain.proceed(newRequest);
Response responseTmp =  chain.proceed(newRequest);
responseTmp.body().string();
and it works

That makes two HTTP requests. You don't want to do that.

Adding this bit as it may help someone else. I was puzzled to encounter the very same exception while calling ResponseBody.string() only once.
The problem, I realized later, was due to the execution context of my request: using a try with resources block with the response causes the same problem when the response body is read outside of the try block:
try (Response response = client.newCall(request.build()).execute()) {
	return response;
} //response.close() called implicitly by the JVM

If response is processed outside this method, the IllegalStateException is equally thrown. That's because the try with resources block will call response.close(). An easy workaround for such a design is to read the body in the try block itself and return it (or both it and the response object in a different data structure)

just call response.close();
@Override
public void onResponse(Call call, Response response) throws IOException {
                try {

                    final int code  = response.code();
                    final String body = response.body().string();
                    response.close();

                    Login.this.runOnUiThread(new Runnable() {
                        @Override
                        public void run() {
	                //do any thing with body variable
			Print(body);
                        }
                    });
                } catch (Exception ex) {
                    String sss = ex.getMessage().toString();
                }

            }

I encountered the same issue, receiving an Exception when reading the response, e.g. responseBody.string().
However in my code I only use it once.
BUT: When debugging, I added (by accident) a watch expression to responseBody.string() which then caused the actual execution to fail.

Try the following:
ResponseBody responseBodyCopy = response.peekBody(Long.MAX_VALUE);
responseBodyCopy.string();

it works for me

why each api gets called twice ?
I am stuck with this


C version broken as of latest pacman update (pacman v6.0.0 - libalpm v13.0.0)

https://aur.archlinux.org/packages/alpm_octopi_utils/
The C version depends on alpm_octopi_utils for searching the databases, which as of the latest pacman update has been broken. Will likely be fixed soon. Later versions of the C program may not include alpm_octopi_utils and use the alpm library directly.

Fixed as of latest alpm_octopi_utils build, update and rebuild avg-audit to fix.


Profiling made configurable to avoid thousands of windows popping up when profiling all tests at once

@CodiumAI-Agent /describe

Title
Profiling made configurable to avoid thousands of windows popping up when profiling all tests at once

PR Type
enhancement

Description

Introduced a profileAnalysis parameter to control whether profiling data is collected during diagnostic analysis.
Updated diagnostic methods to conditionally start and save profiling data based on the profileAnalysis flag.
Enabled profiling in specific test scenarios to gather performance data.

Changes walkthrough 📝
Relevant filesEnhancement

DiagnosticVerifier.Helper.csAdd configurable profiling option to diagnostic methods

MiKo.Analyzer.Tests/Helpers/DiagnosticVerifier.Helper.cs
Added a profileAnalysis parameter to control profiling data collection.  Conditional profiling data collection based on the profileAnalysis flag.

+18/-6

AnalyzerTests.csEnable profiling in performance test method

MiKo.Analyzer.Tests/Rules/AnalyzerTests.cs
Enabled profiling in the Performance_ test method by passing true.

+1/-1

DiagnosticVerifier.csAdd profiling configuration to diagnostics collection

MiKo.Analyzer.Tests/Verifiers/DiagnosticVerifier.cs
Added profileAnalysis parameter to GetDiagnostics methods.  Disabled profiling by default in Collect_issues_in_folder_.

+2/-2

💡 PR-Agent usage: Comment /help "your question" on any pull request to receive relevant information


Disable examples when alsa/libsndfile are not present

@erikd This also solves problems for users when ALSA and/or libsndfile aren't installed. Instead of erroring out, the targets are skipped gracefully.

@erikd done


feat: Adding at_getting_started very simple examples in Dart

very simple atKey and notification examples as simple starting point
- What I did
Added simple (but powerful) examples to get people started with dart
- How I did it
Migrate code from my personal repo and added basic ReadMe and asciinema examples
- How to verify it
ran and recorded the results in the Readme
- Description for the changelog
Added pre starting examples using Dart and atKeys/Notifications.

Reviewed. looks good to me

Reviewed. looks good to me

I resync with trunk so is fine to approve/merge TY

Reviewed. looks good to me

Could you approve for me?


Precompilation failing on FUGAKU

Because of (among other things?) pinthreads(:numa).

For the benefit of future readers, the work around is to disable precompilation with PrecompileTools, i.e. in your environment create the file LocalPreferences.toml with content
[ThreadPinning]
precompile_workload = false


Request - add callback to know when the content gets rendered

Hi,
With each expanded item I'm dynamically loading more data.
I'm trying to move the slider into focus with slickGoTo after the new columns get loaded. To do such a thing I'm taking a slider ref, but since there is no slider lifecycle event handler there is no way to know when the ref will be ready.
Is there a way to know when the data has been rendered by the slider?

rtfm onReInit

I'm re-opening this because I thought that the onReInit would be enough, but unfortunately, it's behavior is making my implementation difficult.
The problem is that the onReInit or componentDidUpdate event is being triggered even when the arrows are being used. I don't believe that the navigation should trigger the onReInit event - componentDidUpdate.


Cannot read property 'rawText' of undefined

Hey, there appears to be a change in the structure of the forge website which causes the forge installer to crash.
This line const url = tt.querySelector("a") || elem.querySelector("a") gets the dom element with the following html:

     Changelog

then it tries to get its first child node (the i tag) and tries to get its rawText, which it does not have because 'Changelog' is now the next element.
Heres the stackstrace:
TypeError: Cannot read property 'rawText' of undefined
    at I:/Projects/catlunch/node_modules/@xmcl/forge-installer/forgeweb.ts:32:57
    at Array.map (<anonymous>)
    at I:/Projects/catlunch/node_modules/@xmcl/forge-installer/forgeweb.ts:27:26
    at Array.map (<anonymous>)
    at parse (I:/Projects/catlunch/node_modules/@xmcl/forge-installer/forgeweb.ts:24:18)
    at Object.getIfUpdate (I:/Projects/catlunch/node_modules/@xmcl/net/index.ts:40:16)
    at runMicrotasks (<anonymous>)
    at Object.getWebPage (I:/Projects/catlunch/node_modules/@xmcl/forge-installer/forgeweb.ts:136:22)
    at Install.run (I:/Projects/catlunch/src/commands/install.ts:67:18)

the crash should be reproducable by simply getting the forge versions with the forge installer module:
    const page = await ForgeWebPage.getWebPage(
      {mcversion: installcfg.minecraft_version}
    )

Ah, thank you to report this bug. The current way to grab info from forge website is not really a reliable solution. (It's just a workaround). But I cannot think a better way to do this easily.
Hope I can fix and publish new version for it in next 12 hours....

yeah i also have no better way to do this. Ive looked at multimc but didnt find any request related to forge versions so i wonder how they do it, i probably just missed it tho ^^


Limit staking parameter changes

Staking parameters are currently adjustable at will by stakers/provers. This encourages short-term competition, with provers constantly adjusting their parameters based on others' values. To discourage this reactive behavior and promote long-term stability, I suggest limiting parameter changes to once per week or day. This will ensure provers' staking parameters better reflect their actual proving capacity and costs, avoiding potentially harmful competition.
Another desired feature is to somehow encourage provers to submit proofs ASAP. This feature remains to be discussed and potential implemented.

Staking based prover pool contracts have been deprecated.


Where can I download the SNIPS dataset？

Hi,
You can download it from https://drive.google.com/drive/folders/1ydalMtB-hpfS3SIEaR5UbRfEe2m8bFcj (datasets are in the data folder)

Thank you for your timely response!

Thank you for your timely response!


Search not working when keyboard connected

When I have the Smart Keyboard Folio attached, I cannot use the Search field in Githawk. Tapping it immediately selects and deselects it. It works if I detach the keyboard, tap, and then use the on-screen keyboard to search for something.
App Information:
App Name: GitHawk for GitHub
App Version: 1.26.0<PHONE_NUMBER>)
Installed App Version: 1.26.0<PHONE_NUMBER>)

Device Information:
Device: iPad8,5 (2018 iPad Pro 12.9 inch)
iOS Version: 12.1
Language: en-US (English (United States))
Carrier: N/A
Timezone: EST
Architecture: N/A
Connection Status: WiFi

Yuck, @BasThomas have you hit this?
Sent with GitHawk


Update debug logging

Description
Re-opening #538, because can't agree that it was just a duplicate of #537. I mentioned in #538 that it happened that we both worked on the same thing, but I believe that my version is a proper usage for developer.log(), since I use both error and stacktrace parameters provided by log() to have a better output in IDEs. Here is an example, where I added 2 lines to one of examples to show the difference in outputs:

Note that the first one is how all logs are shown currently with #537 merged with just a plain string. On the second line I passed the same batteryLevel param as error to show that it is shown in a more informative way.
Additionally, #538 added some more logs in places where they were missing.
Checklist
Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]).
This will ensure a smooth and quick review process. Updating the pubspec.yaml and changelogs is not required.

[x] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[x] My PR includes unit or integration tests for all changed/updated/fixed behaviors (See Contributor Guide).
[ ] All existing and new tests are passing.
[x] I updated/added relevant documentation (doc comments with ///).
[x] The analyzer (flutter analyze) does not report any problems on my PR.
[x] I read and followed the Flutter Style Guide.
[x] I am willing to follow-up on review comments in a timely manner.

Breaking Change
Does your PR require plugin users to manually update their apps to accommodate your change?

[ ] Yes, this is a breaking change (please indicate a breaking change in CHANGELOG.md and increment major revision).
[x] No, this is not a breaking change.

No worries, we can merge this one on top with improvements. I saw some analysis issues in the CI, once those are resolved I will merge this one. Thank you!

thank you!


wishlist: Way to express Fn trait like (FnOnce() -> !)

Spawned off of https://github.com/rust-lang/rust/issues/25325
As written by @diwic originally:
Here's a minimal example that shows the problem:
    fn never_return<F: FnOnce() -> !>(f: F) -> ! {
        f();
    }

which fails with:
    error: expected type, found `!`
         fn never_return<F: FnOnce() -> !>(f: F) -> ! {

...AFAICS, there's no good reason this syntax is not allowed.

It is possible to fake this using a "void" enum like enum Void { } as the return type and providing an fn unreachable<T>(_: Void) -> T { unsafe { ::std::intrinsics::unreachable() } }, cf. github.com/reem/rust-void for an example.

You don't need any unsafe code there, fn unreachable<T>(v: Void) -> T { match v {} } will suffice.

@sfackler interesting, does that have the same optimization results (i.e. does LLVM know that this unreachable is actually unreachable)?

Yep: http://is.gd/JVCrHf
enum Void {}

impl Void {
    fn unreachable(&self) -> ! {
        match *self {}
    }
}

O0 IR:
%Void = type {}

; Function Attrs: noreturn uwtable
define internal void @_ZN4Void11unreachable20h428ace37d3624b8ehaaE(%Void* noalias readonly) unnamed_addr #0 {
entry-block:
  %self = alloca %Void*
  store %Void* %0, %Void** %self
  %1 = load %Void** %self, !nonnull !0
  unreachable

join:                                             ; No predecessors!
  unreachable
}

Cool, thanks for bringing it up further.
I'm not familiar with the empty enum type, and I'm quite surprised it's allowed by the Rust language at all. As I understand it, the trick is that it's impossible to create a Void so therefore the closure must panic or loop infinitely - but it's entirely possible to create a Void and have the closure just return that (although it's unsafe):
struct A; let b: Void = unsafe { std::mem::transmute(A) }; b

Also I'm not sure why the unreachable function is needed at all - this works fine too:
enum Void {}

fn never_return<F: FnOnce() -> Void> (f: F) -> ! {
    f();
    unreachable!();
}

although it's unsafe

For a very good reason.

@diwic writes:

but it's entirely possible to create a Void and have the closure just return that (although it's unsafe):

The fact that such a transmute works today is an implementation artifact that, in an ideal world, would be statically rejected; see also #1076 and https://github.com/rust-lang/rust/issues/4499

Relevant: https://github.com/rust-lang/rfcs/issues/1001
Is there a reason not to make ! a plain ol' type?

@canndrew I woudn't want ! to be a "plain ol' type".
For example, it does not make sense to use it in many contexts where you should be able to put "normal types", such as argument positions for functions.
As I said on another ticket, if we were to add ! as a type, I would want use of it to require a ?-marker, something like ?Converge.   But I am not yet convinced there is sufficient value in re-adding it again.

it does not make sense to use it in many contexts where you should be able to put "normal types"

Doesn't make sense in what way? Merely not useful to do so, or nonsensical (cannot work) at some deeper level?

if we were to add ! as a type, I would want use of it to require a ?-marker, something like ?Converge.

For what reason? (Sorry, you may have already answered this somewhere - but not on the ticket you linked.)
At a technical level, let me put forth the following proposal as a straw man: (1) ! should be nothing more than special syntax for a particular empty enum type (perhaps defined as a lang item); and (2) there should be an automatic coercion from this type to every type. This at least resolves the "specialness" problem, because (as far as I know) empty enum types can appear anyplace where any other type could appear. The drawback of using one would be that it needs to be eliminated explicitly (with e.g. an unreachable()), which is resolved by adding the coercion. In what way would this not be satisfactory?

Sure it makes sense. What's wrong with writing
fn id(x: !) -> ! {
    x
}

True, you could never call this function so there's no point in writing it. But it has a meaning so there's no point in disallowing it either. There are times where it would be useful for ! to be a type. Maybe I want to return a Result<T, !> from a function that has to return a Result but which I know can never fail. Or maybe I want Vec<!> for a Vec that's that guaranteed to have no elements. Or maybe I want to write FnOnce() -> ! to refer to the trait of functions that never return.

As I said on another ticket, if we were to add ! as a type, I would want use of it to require a ?-marker, something like ?Converge.

What's the reason for this?

But I am not yet convinced there is sufficient value in re-adding it again.

What's the value in not having it? It just makes the language more complicated. I mean, imagine if we took some other type (eg. ()), gave it some special syntax (let's say #), but arbitrarily decided that it's not "really" a type so you can only use it an few places that you can use any other type, (like here: fn() -> #, but not here: Result<i32, #>). And then we go to the extra effort of specially coding this usage into the compiler. Wouldn't that be insane? That may sound like a contrived but it's exactly what C does with void.

Sorry, I hadn't seen @glaebhoerl's very similar comment when I posted mine.

That may sound contrived but it's exactly what C does with void.

In case it's not obvious what I'm saying here:
Rust fixes C's mistake of not treating empty structs like plain ol' types and instead giving them their own weird syntax and semantics that breaks generic code.
But then rust goes and makes the exact same mistake with empty enums. I we can fix this, we should.

@glaebhoerl wrote:

it does not make sense to use it in many contexts where you should be able to put "normal types"

Doesn't make sense in what way? Merely not useful to do so, or nonsensical (cannot work) at some deeper level?

I will concede that, in theory, many mental models of what "types" denote can be made to work with the !-type.  (e.g. in "types-as-propositions", ! classifies a particular set of expressions that definitely diverge; in "types-as-sets-of-values", ! denotes the empty set.  Both of these models seem to accommodate the addition fine.)
My problem is that on the implementation side of Rust, I worry about e.g. code-gen for a function that takes an argument of type !.

Would we actually generate code for such a function?  (Would we continue our existing pattern of representing them, like zero-variant enums, by a zero-sized type?  Note that I regard that as an anti-pattern...)
Am I going to end up adding special cases all over the compiler for for this odd new type variant?

From skimming over the PR where this was removed: PR #17603. I would like to infer that the impact on the compiler would be minimal (and thus I would withdraw the above misgivings).

The problem is that I don't think it is necessarily valid to draw that conclusion -- allowing ! in arbitrary type positions would represent a generalization that may expose problems that were not dealt with at the time prior to PR #17603

@glaebhoerl wrote:

if we were to add ! as a type, I would want use of it to require a ?-marker, something like ?Converge.

For what reason?

Because I believe the real-world usage of ! would be limited, and therefore a bound like this would help catch bugs.
Just my opinion; i don't have any data to support it.

I think that would be misleading because it implies that a Converge bound guarantees termination, which isn't the case.

@Diggsey yes okay, the choice of name was probably not ideal.  (That's sort of a separate topic as to whether using such default bound is a good idea at all; I am willing to concede that both the name and the idea are potentially bad.)

Would we actually generate code for such a function?

Ideally, any code that handles a value of type ! is dead code and should be eliminated. If a function needs code generated (because it's being exported from a library or something) then just implement it as a call to unreachable!().

Would we continue our existing pattern of representing them, like zero-variant enums, by a zero-sized type? Note that I regard that as an anti-pattern...

From the user's point of view, values of type ! shouldn't have a representation. Any attempt to inspect their representation (for example, using size_of) should be a static error. From the compiler's point of view, if it ever needs to know the size of ! then it's already gone down the rabbit hole into the world of undefined behavior so the "size" you use is pretty much arbitrary.
However this all seems moot because in 1.0 empty types have size 0 and I don't see how that can be changed without breaking backwards compatibility.

Am I going to end up adding special cases all over the compiler for for this odd new type variant?

No, because it's not new. We already have empty types. If you moved @reem's rust-void into libcore and made ! an alias for Void then you could remove all the special casing for the current ! type. You'd also need to make empty types unify with all other types but that's something that would be useful to add to the language anyway.

However this all seems moot because in 1.0 empty types have size 0 and I don't see how that can be changed without breaking backwards compatibility.

This is not quite true; you cannot tag empty enums with #[repr(C)], and thus their representation is considered undefined for the purposes of stability and backwards-compatibility.

This code compiles and runs though:
use std::mem;
enum Void {}
fn main() {
    assert_eq!(0, mem::size_of::<Void>());
}

Or are you saying you'd be okay with breaking this behavior?

If we guaranteed the stability of size_of return values, we'd effectively be forbearing from ever changing our current data representations except in minor ways, whereas we explicitly leave the representation of everything not #[repr(C)] undefined because we wish to reserve the right to change them. (By which I'm just predicting that @pnkfelix's answer will be "yes" - but I'll leave it to him to make it come true or false.)

Ah yeah, of course.

I just noticed this old RFC of @glaebhoerl's for anonymous enums. If we implemented that then ! could just be the syntax for the empty anonymous enum.

Triage: Fixed by #1216


T.U.R.D. recipe replacements always show as new recipes in crafting menu

Mod source
Factorio Mod Portal
Operating system

=Windows 10

What kind of issue is this?

[ ] Compatibility
[ ] Locale (names, descriptions, unknown keys)
[x] Graphical
[ ] Crash
[x] Progression
[ ] Balance
[ ] Pypostprocessing failure
[ ] Other

What is the problem?
Several T.U.R.D. selections replace existing recipes, which then show up as new recipes in the crafting menu. These notifications can be cleared, but then show up again whenever a new session is started, making it impossible to ever fully clear them.
Steps to reproduce

Select any T.U.R.D. that replaces an existing recipe.
Go into the crafting menu and see the new recipe flagged.
Mouse over the new recipe to clear the notification.
Exit and restart the session.
See that the the replaced recipes are still flagged as new.

Additional context

The Moss Farm, Biosample, Vrauks and Fish are affected in this screenshot, as these have all been touched by T.U.R.D. replacements. I've had all of these unlocked for many hours!
Log file
factorio-current.log

Related: https://forums.factorio.com/viewtopic.php?f=28&t=113737


OAuth2/SAML2 Documentation tells you to hardcode credentials into settings.dist.py instead of recommending Environmental Variables

OAuth2/SAML2 Documentation
dojo/settings/settings.dist.py is where Environmental variables and their defaults are stored. If the environmental variables is not set in the runtime environment of the application, then the default is used.
Problem
Hard-coding secret values can lead to embedded secrets if they build a docker image with them. Recommending users to edit the dojo/settings/settings.dist.py file and edit/replace the following information is poor security practice and can lead to users inadvertently exposing their secrets to others if they upload their docker image to a non-private repository.
Reproduction

Setup OAuth2/SAML2 per the documentation
Docker compose / build
Docker push to docker.io
Secrets are now accessible by all on docker.io

Fix
These values should be set within the runtime environment and using environmental variables. Instead of recommending to edit the defaults to a hard-coded secret, it should recommend to use environmental variables.
kubernetes, AWS, GCP, Azure, etc all allow you to inject secrets as environmental variables. This should be the default and recommend way of setting these secret values.
If just using docker, then recommend storing them in a safe place, like a secrets manager or password manager:
Vault:
docker run -e  DD_SOCIAL_AUTH_...="$(vault read ...)"
LastPass:
docker run -e  DD_SOCIAL_AUTH_...="$(lpass show ...)"

Thank you @ed-wp for your observation, you are absolutely right. I will update this page and https://defectdojo.github.io/django-DefectDojo/getting_started/configuration/ as well.


Can I reduce the time it takes to initiate the camera?

I'm working on an application that requires that the device wakes up and takes a picture extremely quickly. So far I've been able to get this down to 267ms which meets spec for the project. But to do this I had to initialize the file system AFTER I initialized the camera. I need to be able to apply settings to the camera that are stored on the SD card. However reading the SD card takes an additional ~250ms which pushes the overall time to picture up to nearly 600ms which is a little longer than I want. I want to see if I can shave off a few more milliseconds.
One of the things I would like to look at is the camera boot process. Right now it shows this:
VOE]ext_in = 0 sync = 0
[VOE]md init success
[VOE]algo ver ea63d5a
[VOE]pack_v 0x0000 0x0002 cus_v 0 iq_id 0 dn 0 day 0 night 1 other 2 offset 224 length 16436 iq_size 32716
[VOE]Ver 0x0001000c Fast3A Cnt AE 1 AWB 1 period AE 5 AWB 1 delay 0 0
[VOE][isp_mod_sensor_init][1581]Errhdr_mode 0 is over sensor driver num 1
[VOE]hdr_mode 0 sensor driver num 1
[VOE]fps:<PHONE_NUMBER>, pclk: 0, hts:<PHONE_NUMBER>
[VOE]fps max 30.000000 min 5.000000
[VOE]exposure_step 29.629629
[VOE]change sensor mode =><EMAIL_ADDRESS>- 'linear'
[VOE]min_fps 5.000000, max_fps 30.000000, exp_step 29.629629
[VOE]md ver 0x6d640100
[VOE]ae ver 0x61650200
[VOE]awb ver 0x77620100
[VOE]cur_hdr_mode = 0
[VOE]VOE MEM Size = 11447 Used=  3550 KB Free= 7897 KB (8086944)
[VOE]stream 0  buffer 0: 0x7054ac00 size 460800
[VOE]stream 0  buffer 1: 0x705bb500 size 460800
[VOE]first_config_osd2_block_num[0]: 1
[VOE]osd2_block_num[0]: 24
[VOE]NV12 640x480 1/30
[VOE]dynamic set fps 0 -> 30 ok
[VOE]sensor power on
[VOE]zoom crop default setting
[VOE]status == 1718
[VOE]buffer size == (0x7063bbc0)<PHONE_NUMBER>0) queue(20)

I want to get into the code that generates this output to see what I can tighten up. However I can't seem to find the .c/.cpp files generating this output. I'm seeing header files (.h) but no matching .c or .cpp files. I am also seeing a lot of .a files which makes me suspect that some of the files are precompiled binaries and not source code. Are the source files used to compile those binaries available? Is there a way I could have access to them? Or am I just looking in the wrong place?
Also do you have an environment for this camera that is outside of the Arduino library? The ESP32 had the C-level ESP-IDF. This product is advertised as having RTOS capabilities so is there a C-level library that can be used for development?
Thanks,
Jonathan L Clark

@jonathan84clark
For your situation, yes you are right, the boot process are prebuild as bin files for the SDK to use. For more sensors supported currently there are 2, GC5035 and JXF37.
To go further, I think currently it is very hard to push the whole process faster. You may want to try running your application at multi-thread.

there is a feature FCS, but have not port to Arduino SDK yet

Closed as no further request.


nginx reverse proxy + mod_security = segfault

I just made a test with nginx 1.7.9 and mod_security for nginx, following this tutorial: http://www.nginxtips.com/how-to-install-mod_security-on-nginx/
I configured nginx as reverse proxy with mod_security enabled. After few requests nginx was segfaulting, always when serving images. I checked the coredump, this is the output:
Program terminated with signal SIGSEGV, Segmentation fault.
#0  ngx_http_upstream_copy_allow_ranges (r=0x2678680, h=0x7fff406e6f40, offset=) at src/http/ngx_http_upstream.c:4571
4571        if (r->upstream->conf->force_ranges) {
so I tried to set proxy_force_ranges flag (which was unset) in configuration and the segfault went away.
Note that if I turn off modsecurity the proxy alone works fine.

This appears to be an issue in nginx. The ngx_http_upstream_copy_allow_ranges routine just gets the ngx_http_request_t *r, that modsecurity got passed as an (opaque) parameter. Modsecurity doesn't appear to touch the ngx_http_request_t. Setting proxy_force_ranges as you suggested works around the issue by causing r->upstream->conf to be defined, but I think this would be preferable:
This diff works around this issue for me, without having to force proxy ranges:
--- orig/nginx-1.7.9/src/http/ngx_http_upstream.c	2014-12-23 16:28:40.000000000 +0100
+++ nginx-1.7.9/src/http/ngx_http_upstream.c	2015-02-10 14:29:10.211240721 +0100
@@ -4569,5 +4569,5 @@
     ngx_table_elt_t  *ho;

-    if (r->upstream->conf->force_ranges) {
+    if (r->upstream->conf && r->upstream->conf->force_ranges) {
         return NGX_OK;
     }

I have tried the version before v1.7.7 does not have any segmentation issue until reaching to 1.7.7. The proxy_force_ranges directive appears from v1.7.7 and it is default to off. To solve, i suggest to turn it on if using Modsecurity modules for Nginx.
http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_force_ranges

Nice, @driehuis patch fixes the issue, thanks! :)

The official answer from the nginx maintainers is https://trac.nginx.org/nginx/ticket/690. To quote:

The ngx_http_upstream_copy_allow_ranges() function is a handler called to copy headers within upstream module. It is expected to be only called when r->upstream exists and r->upstream->conf is set. If it's called with r->upstream->conf not set - it means the bug is elsewhere.

So, the workaround patch has been rejected and the mod_security interface must be rewritten to eliminate the ngx_http_upstream_copy_allow_ranges() call, or make it at the appropriate time. I do not know enough of the innards of both nginx and mod_security to be able to help here, unfortunately.

I have just ran into this issue also, using modsecurity-2.9.1.
Do we know if this will be getting correctly handled in the mod_security interface?

No longer a concern in libModSecurity. Marking it as won't fix for 2.x. Further information about libModSecurity available here:
https://github.com/SpiderLabs/ModSecurity/tree/v3/master


Add Delta Lake Connector

Build connector to extract metadata from DeltaLake.
This requires to run spark sql to fetch the tables information or we can use the following sqlalchemy driver
https://flynn.gg/blog/databricks-sqlalchemy-dialect/

@harshach Could you please kindly provide the docs of how to use this connector?
I went through the official website below, yet I did not find the related instruction.
https://docs.open-metadata.org/connectors
Thanks.


Edit pass

Description
These are a couple of early guides that I wanted to ensure meet style conventions.

Hey Alex - as always, excellent eye for detail. I agree with you on these changes throughout. Would you mind bringing the heading 2 style question to the mob sesh on Monday morning? That would be great to get writer's to look at. And come up with some feedback, thoughts. I'll pop online a little early to talk about that with you.

https://xeroshoes.com/shop/shoes/oswego-women/
Size 8.5 / color = purple

This seems stale. I'm going to go ahead and close this for now. Feel free to open this back up if you'd like to see this work added in.

Sorry for the delay on this. I'm working through it now.


node: WritableStream.write() should use a union for Buffer|string

I'm converting a node program I originally wrote in JS to TS, and
without this, the following pattern causes tsc to error out (tested
under 1.6):
let buf = current.read(); // current is a ReadableStream
if (buf !== null)
    output.write(buf);

With:
src/bin/cat.ts(28,17): error TS2345: Argument of type 'string | Buffer' is not assignable to parameter of type 'string'.

thanks mate!


SVG import doesn't work well if viewBox contains negative numbers

Here is an example SVG that does not import correctly.
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" viewBox="-395 197 168 168">
    <circle class="st0" cx="-311" cy="281" r="84" fill="#04c07d"/>
    <g class="st1">
        <path class="st2" d="M-268.3 240.7s15.2 13.1-1.6 24.1c0 0-27.2 21-14.9 63.2 0 0 15.1 26.2 14.5 68.8 0 0-.8 14.5-8.3 33.8l-97.5-32.6s14.6-18.9 22.1-41c0 0 18.8-59.2 7.5-84.2 0 0-4.3-2-10.6 1.5 0 0-7.1-7.4 7.2-47 0 0 12.2-26.1 39.9-24.4-.1-.1 30.6 3.5 41.7 37.8z" fill="#7f4200"/>
        <path class="st3" d="M-280.3 336.3l-71.3 12s-2.9 22.2-24.5 49.6l97.8 32.7c-.1.1 21-45.6-2-94.3z" fill="#fa0"/>
        <circle class="st4" cx="-332.8" cy="255.6" r="2.9" fill="#351800"/>
        <path class="st5" d="M-268.3 240.7s-15 3.8-31.2 16c0 0-4.3-15.8-13.7-17.8 0 0-7.6-2.9-19.8.6 0 0-13.7 3.5-20.8-.9 0 0 4.7-36.7 43.7-35.7 0-.1 34.2 2.2 41.8 37.8zm-57.2 51.8s-23.6 8.6-26.7-20.2c0 0 14.5-9.7 26.7 20.2z" fill="#333"/>
        <path class="st6" d="M-284.6 327.8s-23.4 20.3-63.8 7.3l-17.1 27 86.1-23.5s-.9-6.8-5.2-10.8z" fill="#fff"/>
    </g>
</svg>

All of the paths import properly but the two circles end up getting weirdly distorted.
The SVG was exported using Adobe Illustrator.

The reason is because svgpathdata.js doesn't yet know how to transform elliptical arcs (there's a TODO in SvgPathData#transform()).
For future reference, it looks like this code might do what we want: https://gist.github.com/alexjlockwood/c037140879806fb4d9820b7e70195494#file-flatten-js-L441-L547

Fixed.


[BUG] Click target for "no symptoms" in AoE is too large

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior
click target for "no symptoms" checkbox is too large, clicking far to the right outside of text unchecks box
Steps To Reproduce
check "no symptoms" then click farther to the right of text
Expected Behavior
click target should be smaller, limited to checkbox and text
Additional context
also applies to symptom checkbox
Does this issue have a priority label?

[X] I have added a priority label

Has issue been assigned to a team?

[X] I have added a team label

Choosing not to address this because there's a planned refactor of the whole AoE form. We'll fix it then.


Remove confusing methods from Traversable that treat it as 0..1 container

There will still be changes to the type hierarchy in the next major release Vavr 1.0.0. For example #1295
We should remove all methods from Traversable that treat a collection as container that has 0..1 elements.
For example @Mousaka stated:

when/how can I use Map's getOrElseThrow(Supplier) ? What does it mean to get something from a Map without giving a key?

He is absolutely right - this is an uncommon collection behaviour. We are able to achieve the goal of removing the mentioned methods from Traversable by performing the following changes:

Traversable does not extend Value anymore
We factor out commonalities between Value and Traversable into a non-public interface that is extended by both

Uncomment the following code block in io.vavr.PartialFunction after this issue is solved:
    /* TODO: currently the name is ambiguous. uncomment after #2023 is finished!
    default PartialFunction<T, R> orElse(PartialFunction<? super T, ? extends R> fallback) {
        return of(t -> applyOrElse(t, fallback), t -> isDefinedAt(t) || fallback.isDefinedAt(t));
    }
    */

Already done in master (= vavr 1.0 branch) by removing Value interface.


Lightmap Probes not casting light on dynamic objects

Godot version
v4.0.alpha4.official
System information
Windows 10, 2070 SUPER, Vulkan Clustered
Issue description

dynamic objects are remaining unlit from the Baked Lightmap with the generated lightmap probes, it returns these errors when moving the dynamic object in editor:
servers/rendering/renderer_rd/renderer_storage_rd.cpp:7557 - Condition "!lm" is true. Returning: false
servers/rendering/renderer_rd/renderer_storage_rd.cpp:7563 - Condition "!lm" is true. Returning: AABB()
Steps to reproduce
Make a scene and light it with Lightmap Probe and Emissive materials. Make sure 'gen probes' is enabled in the lightmap. Bake the lightmap and insert a mesh, make it's 'GI mode' be either 'dynamic' or 'disabled'. The mesh either remains unlit, both in editor and in game.
Minimal reproduction project
No response

Duplicate of https://github.com/godotengine/godot/issues/57890.


Merged latest master from upstream

@bitprophet thank you  :+1:

Only a few commits, really :grin:

@bitprophet the thanks applies not only to these few commits but to all the other great stuffs you bring to the world like Fabric and more :heart:


Delete more unused code

What
Describe what the change is solving
How
Describe the solution
Breaking Changes
Are there any breaking changes in this PR?

#463  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @hjkatz and the rest of your teammates on  Graphite

Merge activity

Oct 23, 5:07 PM EDT: A user merged this pull request with Graphite.


feat: make cow swapper wallet agnostic

Description

Pull Request Type

[ ] :bug: Bug fix (Non-breaking Change: Fixes an issue)
[ ] :hammer_and_wrench: Chore (Non-breaking Change: Doc updates, pkg upgrades, typos, etc..)
[ ] :nail_care: New Feature (Breaking/Non-breaking Change)

Issue (if applicable)

closes #
Risk

Testing

Engineering

Operations

Screenshots (if applicable)

Current dependencies on/for this PR:

develop

PR #5187

PR #5194   👈

This comment was auto-generated by Graphite.


can we use one jio account on multiple devices

Your Environment

Add-on name: [e.g. Jiotv]
Add-on version: [e.g. 2.0.0]
Kodi version: [e.g. Kodi 19]
Operating system version/name: [e.g. Windows 10, LibreElec 9.0, etc...]

Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.
Debug log

The debug log can be found from this link:

Yes. But it has some limits obviously. I am not sure how many devices but for general use, it'll be fine for 2-3 devices.


Move eng/common check to end of analyze stage

Run eng-common-workflow-enforcer.yml at the end of the analyze stage.

/check-enforcer override


[Cpp][DP] Kadane's Algorithm

🚀 Feature
Implement Kadane's Algorithm with C++.
Have you read the Contributing Guidelines on Pull Requests?
Yes, I have read the guidelines!
Motivation & Pitch
Kadane's Algorithm (also called Maximal Subarray sum algorithm) is one of the most basic Dynamic Programming Algorithms. it is used to find a subarray with a maximum sum. The normal algorithm would take O(n^2) time (where n is the size of an array), but when optimized with Dynamic Programming, the time complexity reduces to O(n).
So Kadane's Algorithm is an efficient way to calculate maximal subarray sum

@Nkap23 Kadane's Algorithm has already been implemented here.


[WIP] Enable Chunked Sorting to Reduce Peak Memory Usage in Top-P-Top-K Sorting

This is the PR collaborated with @youkaichao for chunked_sorting to reduce peak memory (try to solve the OOM issue in computing logits in Issue #5907). Please don't merge for now because it is still experimental.
The issue we want to address here is that Pytorch sorting on logits would incur a significant peak memory usage, which turns out to be a major memory bottleneck during the decoding time, especially when users request to obtain logits. Our approach is that instead of sorting the large logits as a whole, we first split them into chunks and do sorting on each smaller chunk. In that way, intermediate variables that created during sorting will all become much smaller and the memory can be recycled timely.
Effectiveness Proof:
We conducted a simple verification: On a standard 4 GPU server, we ran vllm serve meta-llama/Meta-Llama-3-8B --tensor_parallel_size 4 and monitored the memory usage. We have successfully managed to reduce the peak memory usage from 5.0 GB to 4.5 GB on Rank 0 (since sampling and sorting only happened at Rank 0). Considering roughly 3.7 GB non-deductible model weights (plus some minor usage by NCCL and CUDA graphs), and the relatively small code edits, we do see this peak memory reduction as a promising direction (i.e., reducing 50% peak memory usage) to work on.
Current TODOs include:

FlashInfer experiments.
Introduce new args to enable this chunked sorting behavior.

FIX #5907 (link existing issues this PR will resolve)

@yangalan123 thanks for the investigation! I changed the chunk_size budget to always be max-num-seqs , this keeps the current behavior, and only fixes the OOM issue when people ask for prompt logprobs.
we can investigate in the future, if people ever want to directly control the chunk_size . if that is the case, we can expose a flag to users.
cc @robertgshaw2-neuralmagic I think it can solve most of issues linked in https://github.com/vllm-project/vllm/issues/5907 .

for record, the memory cost of torch.sort seems to be a known issue , see https://github.com/pytorch/pytorch/issues/77049#issuecomment-1121397826

Nice work, will review on my flight today

Thanks! Left some minor comments.

Can you run this through an lm-eval-harness test that uses prompt logprobs as a sanity check for correctness?
Is there any impact on speed?

Thanks! Left some minor comments.

Can you run this through an lm-eval-harness test that uses prompt logprobs as a sanity check for correctness?
Is there any impact on speed?

Thanks for the review and comments! (and Happy New Year!) As the initial author for this PR, I can provide some initial insights on the correctness of this chunked sorting approach:

Correctness: I think here prompt_logprobs is not a direct testing for this PR, though this PR is motivated by the OOM issues resulted from computing prompt_logprobs. A better and more straightforward correctness testing, is directly comparing naive torch.sort with our chunked sorting solution. I run the following very simple benchmarking codes to compare these two kinds of sorting (partial credit goes to Claude, as I am kind of lazy in holiday seasons :-) ):

# benchmark_runner.py
import torch
import time
import json
import argparse
from pathlib import Path
from tqdm import tqdm

def naive_sort(tensor: torch.Tensor) -> torch.Tensor:
    return torch.sort(tensor, dim=-1)[0]

def chunked_sort(tensor: torch.Tensor, chunk_size: int) -> torch.Tensor:
    chunks = torch.split(tensor, chunk_size, dim=0)
    sorted_chunks = [torch.sort(chunk, dim=-1)[0] for chunk in chunks]
    return torch.cat(sorted_chunks, dim=0)

def run_benchmark(args):
    results = {
        'time': [],
        'accuracy': [],
        'type': 'naive' if args.chunk_size is None else f'chunked_{args.chunk_size}'
    }

    for _ in tqdm(range(args.num_rounds)):
        tensor = torch.randn(args.num_tokens, args.vocab_size, device='cuda')
        start_time = time.perf_counter()
        if args.chunk_size is None:
            result = naive_sort(tensor)
        else:
            result = chunked_sort(tensor, args.chunk_size)
            if args.check_accuracy:
                naive_result = naive_sort(tensor)
                accuracy = float(torch.allclose(naive_result, result, rtol=1e-5))
                results['accuracy'].append(accuracy)

        torch.cuda.synchronize()
        results['time'].append(time.perf_counter() - start_time)

        del tensor, result
        torch.cuda.empty_cache()
        torch.cuda.synchronize()

    Path(args.output_dir).mkdir(exist_ok=True)
    output_file = Path(args.output_dir) / f"results_{results['type']}.json"
    with open(output_file, 'w') as f:
        json.dump(results, f)

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--num_tokens', type=int, default=8192)
    parser.add_argument('--vocab_size', type=int, default=200000)
    parser.add_argument('--chunk_size', type=int, default=None)
    parser.add_argument('--num_rounds', type=int, default=5)
    parser.add_argument('--check_accuracy', action='store_true')
    parser.add_argument('--output_dir', type=str, default='benchmark_results')
    args = parser.parse_args()

    if not torch.cuda.is_available():
        raise RuntimeError("CUDA not available")

    run_benchmark(args)

if __name__ == "__main__":
    main()

I run with `num_tokens=8192` and `vocab_size=128256` on an A40 GPU to simulate the running of Llama-3 models. The running results show that, no matter what `chunk_size` we choose, after 5 rounds of checking, the accuracy (in terms of sorting results match) is 100%, which verifies the correctness of our implementation. This is expected as the chunking-and-merging happens not at the dimension of vocabulary, but at the dimension of tokens.

Efficiency: We would definitely face some overhead here -- because we need to do some chunking first. Also we might lose the opportunity to expose higher concurrency, as we now are sorting smaller chunks of logits and may not fully saturate the GPU computation resources. Nevertheless, reusing the above codes for benchmarking and removing extreme observed metrics in running (potentially due to shared cluster usage), I find that setting chunk_size to 32 only incurs around 0.2 more seconds (1.21 v.s. 1.02, about 18% slower).  Note, here this is already a stress testing when we use the full input window (num_tokens=8192). It is definitely possible in real applications, this overhead is more negligible with smaller logit tensors to sort. I think for users suffering from OOM issues (e.g., when computing prompt_logprobs), running a bit slower is much better than just getting the CUDA OOM error and the whole running fails.
Please note that, though we can also do a memory profiling here, for both running time and memory profiling results reported by this simple simulation program, we should take it with a grain of salt because the special distributed environments and setup at vLLM runtime. I made a simulation report here only for quick reference.

Other issues on codes -- as @youkaichao is already working to polish my initial commits, perhaps he can provide more thoughts and details.
Happy New Year again to everyone and thanks all for reviews and polishing this PR!


[AdvancedInvite] Fix AttributeError

Type

[x] Bugfix
[ ] Enhancement
[ ] New Feature
[ ] Other

Description of the changes
This PR is to fix "AttributeError: 'extra_links' is not a valid registered Group or value." by changing "extra_link" in "_config_structure" to "extra_links"

Hey, thanks for noticing this.
Since I use extra_link for grabbing the value from the config in the invite command it would be easier to swap around the setting's command's method

I use extra_link for both now.


Default params for calculating bleu

Hi, what exactly are the default set of parameters (tokenize, smooth_method, smooth_value) to calculate BLEU? I see default smooth_method is floor and tokenize='none' (which uses 13a tokenizer) in this repo while huggingface datasets has default smooth_method=exp and tokenize=None (which uses BaseTokenizer)
would like to know the default one used in research.
And also, whats the difference b/w raw_corpus_bleu and corpus_bleu methods in compat.py?

There are different default values for different use cases (sentence-level BLEU, legacy methods). If you run e.g. sacrebleu -i output.detok.txt -t wmt17 -l en-de without any other parameters, you will see also the SacreBLEU signature, which is in this case nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.0.0. So the default tokenization is 13a and the default smoothing method exp.

See the source code of BaseTokenizer and I hope you'll understand why the respective parameter is called --tok=none. This means that no spaces are added to the lines, so tokens are separated at original whitespace only. Note that this is useful in very rare cases only (checking the effect of non-standard tokenizers) and a warning is issued if --tok=none is used.
So why is it called BaseTokenizer rather than NoneTokenizer when it does not even separate commas? It is the base class for other tokenizers.
Python None is the usual way for encoding a default value, so I think it should be kept as is and I don't see any risk of ambiguity.

I can see how the use of "none" to mean "don't use any tokenization" can be confusing at the code level. For the v2.2 release, I created a separate tokenizer and added some comments that I hope will help clarify this.


Add drop zone while dragging file over

Clear indication of drag and drop functionality

Hello!~
I've tested this on my machine
Issues:
Flickers or doesn't appear when dropping to center of the page
Sometimes doesn't appear, but scrolls the page up or down, depending on mouse position
The attached video shows the flicker issue
Please take a look at this example, it fixes the issues
https://user-images.githubusercontent.com/38132413/117087879-79144c00-ad59-11eb-9b9e-73fe63acb136.mp4

Hey @Ciremun
Thank you for testing it out, i fixed all the issues that you mentioned, although it still scrolls a bit on firefox


Support form deep-linking

Sorry for the wall of text, just want to leave this here for posterity.
The only user-facing changes we should see with this PR is the new "FormList" view and a slightly crumby animation when moving between form panels.
This PR does not do anything for our quick editing feature that I've previewed, that will follow in another PR, and should be less invasive than this one.
Introduces cms.state and cms.dispatch
See code
Most of Tina is not reactive, and changing properties on classes always requires us to also set up subscriptions, or listen for events. As complexity grows, this has become pretty brittle, and pain points were felt while building out the logic for deep-linking. The need to effectively switch the "active" form meant that looking for it in cms.forms (a plugin) the values wouldn't always be there. This issue, along with many others necessitated a move towards state that is kept in React-land.
I'm not sure exactly how far we want to lean into the top-level React state, but I think when top-level React state doesn't fit, local React state should, and we can probably start to treat the CMS object as more of a large config class that shouldn't be mutated once initialized (eg. move Events/Alerts and possible Forms into React state).
FormBuilder handles responsibilities that GroupForm, GroupListForm and Blocks were handling
When you have nested properties, we've previously been mounting a React portal over top of the previous group of properties. There were performance issues with this approach, not to mention an endless battle for the highest z-index for each subsequent portal that was mounted (since they all mount at the same insertion point). But also, this strategy prevented us from being able to "deep-link" into a form. Say you want to see the fields at blocks.0.cta.actions.2. Previously we would need to mount and traverse through each layer of fields. Now, the FormBuilder is given the form object, and the active field name (blocks.0.cta.actions.2), and it grabs the appropriate fields. Deep-linking is trivial at this point and I expect some performance improvements, but the z-index issues for things that are still using the form portal like rich-text nested forms and the blocks selector have been avoided for now, since I think we're hoping to restructure things in a more sane manner in the near future.
cc @spbyrne I tried to keep some animation intact, but it kind of sucks. I'm sure we can do something better but every time I tried it ended up being the beginnings of a large overhaul, which I want to avoid in this scope.
I've left much of the code in GroupForm, GroupListForm and Blocks untouched, just commented out things that needed to be, hope to remove large chunks of it soon.
A new UI for the "FormList"
See code

This view is what you see when you have a list of forms and click the "Return to Form List" text above a form. Previously, it was something we'd mostly avoided, and moving forward, I dont expect it to get much attention. But it now shows all forms on the page, and keeps track of where they are in the query hierarchy. It's not the prettiest, but at least it should help people navigate once we land "quick editing". This view is updated from the graphql-reducer, and managing the hierarchy is separate from managing the forms themselves, both info is stored in the top-level cms.state.

Noticed a UX issue/regression: https://www.loom.com/share/e4ebe6030dff4b639947fc105dcbda4e

Contextual editing is broken if you go to a document with an other -> edit in CMS -> go to a different collection
See: https://www.loom.com/share/0f275794279e490c96eff7e2400c86d4

@jeffsee55  Seeing an issue with drag and drop
Note: this is not happening on main
https://www.loom.com/share/465f8a29e38a42e0a8027c4b4efa8719

Thanks @logan-anderson - this was due to the <Transition> component, which DnD doesn't like https://github.com/atlassian/react-beautiful-dnd/issues/128.
cc @spbyrne the <Transition> I added sucked, and now knowing that it breaks other stuff I'm inclined to leave animations out of this PR altogether and we can tackle it in the future.


storage: allow deleting images via storage API

This commit introduces the ability to wipe storage volumes for images from a
specific storage pool.
Closes #3539.
Signed-off-by: Christian Brauner<EMAIL_ADDRESS>

@brauner needs rebase

@stgraber rebased.

jenkins: test this please