Fix migration bug

What type of PR is this：

[x] bug
[ ] feature
[ ] enhancement
[ ] others

Which issues of this PR fixes ：

Fixes #4328
Problem Summary(Required) ：

When clone tablet from path A to path B, and then back to path A, it will failed because the original tablet remains in path A. The original tablet is GCed async every one hour. Fix it by removing shutdown tablet when clone. And add ut to test this case.

run starrocks_fe_unittest


Error on adding filter

Hi
I only want to put a simple filter, but I get nft error.
Command:
root@xxx:/home/xxx/nft-geo-filter# ./nft-geo-filter --allow HU
Response:
ERROR - show_subprocess_run_error - Failed to run: (1, ['/usr/sbin/nft', 'add', 'chain', 'inet', 'geo-filter', 'filter-chain', '{', 'type', 'filter', 'hook', 'prerouting', 'priority', 'filter', '-', '200;', 'policy', 'drop;', '}'])
ERROR - show_subprocess_run_error - Command exit status: 1
ERROR - show_subprocess_run_error - Command stdout: b''
ERROR - show_subprocess_run_error - Command stderr: b'Error: syntax error, unexpected string, expecting - or number\nadd chain inet geo-filter filter-chain { type filter hook prerouting priority filter - 200; policy drop; }\n                                                                              ^^^^^^\n'
Failed to add the target nftables chain

I get only en empty geo-filter table:
table inet geo-filter {
}

Hmm, what version of nft are you using? I guess old versions of nft don't support math operations on chain priorities.

I just checked the nftables changelog and it looks like support for arithmetic expressions in chain priorities was only added in nftables 0.9.1 (https://www.spinics.net/lists/netfilter/msg58736.html). Since Debian Buster is still using nftables 0.9.0 (and other LTS distros even older versions of nftables), I have gone ahead and removed the arithmetic expression from the "add chain" command and replaced the priority with just "-200". Please try the script on the latest commit and let me know if it fixes your issue.

i'm new to git and never used it before. Can you help me a lil, how I can start that script? I've nft up and running, installed git on centos but I can't figure out how to install/run it. :)

@frankofno You don't need git. You can just download the script from here: https://raw.githubusercontent.com/rpthms/nft-geo-filter/master/nft-geo-filter, save it somewhere on your server (probably in /usr/local/bin) and then run the script. Look at the examples in the README to learn how to use nft-geo-filter.

the "... and run the script."-part is what I can't figure out. just typing in "nft-geo-filter" or nft-geo-filter -h" or "nft-geo-filter --allow MC" is not working. not a directory not a command messages showing up. "nft-geo-filter --table-family netdev --interface enp1s0 MC" is giving me command not found

oh, chmod +x was missing, one step closer. now i'm getting
[root@mail bin]# nft-geo-filter --table-family netdev --interface eth0 MC
Traceback (most recent call last):
File "/usr/local/bin/nft-geo-filter", line 406, in
if gFilter.add_table() == -1:
File "/usr/local/bin/nft-geo-filter", line 81, in add_table
subprocess.run(nft_command.split(), capture_output=True, check=True)
File "/usr/lib64/python3.6/subprocess.py", line 423, in run
with Popen(*popenargs, **kwargs) as process:
TypeError: init() got an unexpected keyword argument 'capture_output'

@frankofno  Looks like your Python version is too old and doesn't support the capture_output argument. Could you create a separate issue so that we don't derail @TeleMediaCC's issue? Also, specify which distro you're using in that issue.

I installed 0.9.6
root@xxx:/home/xxx/nft-geo-filter# nft -v
nftables v0.9.6 (Capital Idea #2)

root@xxx:/home/xxx/nft-geo-filter# ./nft-geo-filter --allow HU
ERROR - show_subprocess_run_error - Failed to run: (1, ['/usr/sbin/nft', 'add', 'chain', 'inet', 'geo-filter', 'filter-chain', '{', 'type', 'filter', 'hook', 'prerouting', 'priority', 'filter', '-', '200;', 'policy', 'drop;', '}'])
ERROR - show_subprocess_run_error - Command exit status: 1
ERROR - show_subprocess_run_error - Command stdout: b''
ERROR - show_subprocess_run_error - Command stderr: b'Error: syntax error, unexpected string, expecting - or number\nadd chain inet geo-filter filter-chain { type filter hook prerouting priority filter - 200; policy drop; }\n                                                                              ^^^^^^\n'
Failed to add the target nftables chain

@TeleMediaCC Could you tell me what distro you're running the script on? And what version of the distro? Also, what version of Python 3 have you got installed on your system?

Fresh installed Debian 10.5
#cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 10 (buster)
Release:        10
Codename:       buster

# uname -r
4.19.0-10-amd64

# python -v
Python 2.7.16 (default, Oct 10 2019, 22:02:15)
[GCC 8.3.0] on linux2

# python3 -v
Python 3.7.3 (default, Jul 25 2020, 13:03:44)
[GCC 8.3.0] on linux

Are you sure you're using nftables 0.9.6 on Debian 10.5? I just tried installing nftables on the latest Buster release and the highest version of nftables I could get was nftables 0.9.0.
root@debian:~# cat /etc/debian_version
10.5
root@debian:~#
root@debian:~#
root@debian:~#
root@debian:~#
root@debian:~#
root@debian:~#
root@debian:~# apt update && apt install nftables
Hit:1 http://deb.debian.org/debian buster InRelease
Hit:2 http://deb.debian.org/debian-security buster/updates InRelease
Hit:3 http://deb.debian.org/debian buster-updates InRelease
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  libjansson4 libnftables0
The following NEW packages will be installed:
  libjansson4 libnftables0 nftables
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 286 kB of archives.
After this operation, 896 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://deb.debian.org/debian buster/main amd64 libjansson4 amd64 2.12-1 [38.0 kB]
Get:2 http://deb.debian.org/debian buster/main amd64 libnftables0 amd64 0.9.0-2 [203 kB]
Get:3 http://deb.debian.org/debian buster/main amd64 nftables amd64 0.9.0-2 [45.2 kB]
Fetched 286 kB in 0s (11.1 MB/s)
Selecting previously unselected package libjansson4:amd64.
(Reading database ... 29491 files and directories currently installed.)
Preparing to unpack .../libjansson4_2.12-1_amd64.deb ...
Unpacking libjansson4:amd64 (2.12-1) ...
Selecting previously unselected package libnftables0:amd64.
Preparing to unpack .../libnftables0_0.9.0-2_amd64.deb ...
Unpacking libnftables0:amd64 (0.9.0-2) ...
Selecting previously unselected package nftables.
Preparing to unpack .../nftables_0.9.0-2_amd64.deb ...
Unpacking nftables (0.9.0-2) ...
Setting up libjansson4:amd64 (2.12-1) ...
Setting up libnftables0:amd64 (0.9.0-2) ...
Setting up nftables (0.9.0-2) ...
Processing triggers for man-db (2.8.5-2) ...
Processing triggers for libc-bin (2.28-10) ...
root@debian:~#
root@debian:~#
root@debian:~#
root@debian:~#
root@debian:~# nft -v
nftables v0.9.0 (Fearless Fosdick)

Sure. I built from source based on these doc: https://dev.to/isabelcmdcosta/installing-nftables-from-sources-ondebian--4ic

@GaborTorma Could you download the latest version of the script from here: https://raw.githubusercontent.com/rpthms/nft-geo-filter/master/nft-geo-filter and try running the script again? I had replaced the "filter - 200" with an explicit "-200" priority, so maybe it should work for you?
Is there any special reason why you're compiling nftables yourself? You could also try using the nftables package from the buster-backports repo which is at version 0.9.3 (which should hopefully work, but I haven't tested it yet).

@GaborTorma I've just pushed a fix to the script. The script now works in Buster 10.5. The issue was that nft was parsing the "-200" part as an option to nft instead of treating it as a part of the add chain command (https://bugzilla.redhat.com/show_bug.cgi?id=1778883). Adding a "--" argument after nft fixes the issue. Please download the script again and give it a shot.
root@debian:~# cat /etc/debian_version
10.5
root@debian:~#
root@debian:~#
root@debian:~#
root@debian:~#
root@debian:~# apt update && apt upgrade
Hit:1 http://mirrors.linode.com/debian buster InRelease
Hit:2 http://mirrors.linode.com/debian-security buster/updates InRelease
Hit:3 http://mirrors.linode.com/debian buster-updates InRelease
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
root@debian:~#
root@debian:~#
root@debian:~#
root@debian:~#
root@debian:~#
root@debian:~#
root@debian:~#
root@debian:~#
root@debian:~# apt install nftables
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  libjansson4 libnftables0
The following NEW packages will be installed:
  libjansson4 libnftables0 nftables
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 286 kB of archives.
After this operation, 896 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://mirrors.linode.com/debian buster/main amd64 libjansson4 amd64 2.12-1 [38.0 kB]
Get:2 http://mirrors.linode.com/debian buster/main amd64 libnftables0 amd64 0.9.0-2 [203 kB]
Get:3 http://mirrors.linode.com/debian buster/main amd64 nftables amd64 0.9.0-2 [45.2 kB]
Fetched 286 kB in 0s (8,992 kB/s)
Selecting previously unselected package libjansson4:amd64.
(Reading database ... 29491 files and directories currently installed.)
Preparing to unpack .../libjansson4_2.12-1_amd64.deb ...
Unpacking libjansson4:amd64 (2.12-1) ...
Selecting previously unselected package libnftables0:amd64.
Preparing to unpack .../libnftables0_0.9.0-2_amd64.deb ...
Unpacking libnftables0:amd64 (0.9.0-2) ...
Selecting previously unselected package nftables.
Preparing to unpack .../nftables_0.9.0-2_amd64.deb ...
Unpacking nftables (0.9.0-2) ...
Setting up libjansson4:amd64 (2.12-1) ...
Setting up libnftables0:amd64 (0.9.0-2) ...
Setting up nftables (0.9.0-2) ...
Processing triggers for man-db (2.8.5-2) ...
Processing triggers for libc-bin (2.28-10) ...
root@debian:~#
root@debian:~#
root@debian:~#
root@debian:~#
root@debian:~#
root@debian:~#
root@debian:~# nft -v
nftables v0.9.0 (Fearless Fosdick)
root@debian:~#
root@debian:~#
root@debian:~#
root@debian:~#
root@debian:~#
root@debian:~# curl -O https://raw.githubusercontent.com/rpthms/nft-geo-filter/master/nft-geo-filter
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 21622  100 21622    0     0  72073      0 --:--:-- --:--:-- --:--:-- 72073
root@debian:~#
root@debian:~#
root@debian:~#
root@debian:~#
root@debian:~#
root@debian:~#
root@debian:~# chmod u+x ./nft-geo-filter
root@debian:~# ./nft-geo-filter MC && nft list ruleset
table inet geo-filter {
        set filter-v4 {
                type ipv4_addr
                flags interval
                auto-merge
                elements = { <IP_ADDRESS>/22, <IP_ADDRESS>/20,
                             <IP_ADDRESS>/19, <IP_ADDRESS>/21,
                             <IP_ADDRESS>/19, <IP_ADDRESS>/18,
                             <IP_ADDRESS>/24, <IP_ADDRESS>/20,
                             <IP_ADDRESS>/22, <IP_ADDRESS>/22,
                             <IP_ADDRESS>/22, <IP_ADDRESS>/21,
                             <IP_ADDRESS>/23, <IP_ADDRESS>/23,
                             <IP_ADDRESS>/19, <IP_ADDRESS>/21,
                             <IP_ADDRESS>/19 }
        }

        set filter-v6 {
                type ipv6_addr
                flags interval
                auto-merge
                elements = { 2a01:8fe0::/32,
                             2a07:9080::/29,
                             2a0b:8000::/29 }
        }

        chain filter-chain {
                type filter hook prerouting priority -200; policy accept;
                ip saddr @filter-v4 drop
                ip6 saddr @filter-v6 drop
        }
}

Works. Thanx!


Members not showing in public mode

Even when an organization has public members, running in public mode isn't picking them up. Need to investigate what the API is sending back.

Fixed in 144557c81d0e1b3e39d0b5175033ba368fea0fef


[BEAM-5467] Use process SDKHarness to run flink PVR tests.

Use process instead of docker to run flink python validates runner test cases
Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.
Post-Commit Tests Status (on master branch)

Lang
SDK
Apex
Dataflow
Flink
Gearpump
Samza
Spark

Go

---
---
---
---
---
---

Java

Python

---

---
---
---

It looks like the SDK worker terminates unexpectedly. You would get the workers console output running the job server with debug logging or temporarily change: https://github.com/apache/beam/blob/5720c1d22771a65ad5d7be6a06ad8aa0754fa64b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/ProcessManager.java#L43

Run Seed Job

Run Python Flink ValidatesRunner

Run Python Flink ValidatesRunner

Run Python Flink ValidatesRunner

Run Python Flink ValidatesRunner

Run Python Flink ValidatesRunner

Run Python Flink ValidatesRunner

Successful run: https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink_PR/19/

R: @tweise @mxm @herohde

@angoenka the current solution of using the container boot binary prevents this from running on another architecture. Can we come up with a solution that would also work locally on macOS and other platforms. We had already solved this for Lyft.

Reusing it would be better. Can we get that code to master?

Of course. We just need to agree how we want to enable the environment provider/factory. Probably best as separate PR.

I was hopping to fit it via process environment factory. Does it need a separate environment factory to be registered?

It is a separate environment factory: https://github.com/lyft/beam/blob/release-2.8.0-lyft/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java#L47
Alternatively it would need to be part of the default process environment factory with a parameter to control the python specific execution.

Looking at the code, it seems that we can easily plug it in using standard PROCESS environment with a custom command. Shall we use the standardized approach of PROCESS environment?
However, in the meanwhile till we don't have a replacement, shall we move with the current approach of using boot.go as worker. This will make our PostCommit consistent.

Note the implementation is Python SDK worker specific, not just the command. But it can be controlled with environment parameters if needed. I would love to get rid of the custom code.. Agreed that we should do that as follow-up.
BTW the PR has a merge conflict.

Run Python Flink ValidatesRunner

Run Python Flink ValidatesRunner

Run Python Flink ValidatesRunner

Run Python Flink ValidatesRunner

Run Python Flink ValidatesRunner

@angoenka @mxm the relevant JIRA for follow-up work / custom environments is https://issues.apache.org/jira/browse/BEAM-4819

Thanks!
Addressed the comments.

Run Python Flink ValidatesRunner

Run Python Flink ValidatesRunner

Run Python Flink ValidatesRunner

Run Python Flink ValidatesRunner

@tweise
I ran the Python PVR 3 times https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink_PR/36/ to https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink_PR/38/
which all passed.
Not sure if it fixes the issue but it seems more reliable

Run Seed Job

Run Python Flink ValidatesRunner

@youngoli Was experiencing several "in process" issues when attempting to run pipelines with the ULR.


Shared String: findTile to accept an array of tile labels

Feature
A common scenario is to find the closest tile from a variety of labels. For example, in the case of extending the selection again, we would wish to find the next paragraph mark tile, or a table cell mark, or a section break mark. The idea would be for findTile to return the first marker that has any of the tile labels given.

Keeping the issue alive.


How can I run the baseline models

Hi, thank for your interest in our work.
The current codebase does not include the baselines. Adding baselines after cleaning up the code is still on our to-do list, but in the meantime, I can send you the relevant code from our development branch via email if you need it.

closing this issue due to inactivity.


Fix workflow runs on PR from other contributors

When a contributor opens a PR, the GitHub workflows do not run. I have to approve each run manually. When I run a workflow (eg. testing), the run fails on adding the test results on the PR:

The testing workflow needs to be able to run on any PR, and publish the results on the PR.
I don't want the Docker pipeline to run though, so not any image will be pushed to Docker hub.

Tried setting workflow permissions to r/w (as stated here, but still doesn't seem to work.

As stated in this thread, changed pull_request to pull_request_target and now it works on forked repos!

Did not solve the issue. Need to look into more.

Created a possible fix in #59 but will have to wait for someone else to open a PR to verify this.

Still an issue. #67 failed with Resource not accessible

Reverted back to marocchino/sticky-pull-request-comment, but added check in step that only PR created by @dickwolff will add coverage. Not ideal but don't have a better solution. The tests run, which is the most important thing. Closing.


Record no-op set() doesn't return same instance

The docs (IIRC) state that Record is implemented with a Map, yet:
> m1 = new Immutable.Map({ foo: 0 });
> m2 = m1.set('foo', 0);
> Immutable.is(m1, m2)
true
> m1 === m2
true

> MyRecord = Immutable.Record({ foo: 0 });
> r1 = new MyRecord();
> r2 = r1.set('foo', 0);
> Immutable.is(r1, r2)
true
> r1 === r2
false

What am I missing..?

This is a duplicate of #536, what was I thinking...

Thanks for the report! I'll make sure to keep track of this case in #536


Multiple uses of existential/question-mark variables in function definitions do not work as expected

Minimal example below:
$ cat foo.k
module FOO
  imports INT
  imports BOOL

  syntax Seq ::= List{Int, ":"}
  syntax Seq ::= "foo" "(" Int ")" [function]
  rule foo(N) => ?_:Int : foo(N -Int 1)
    requires N >Int 0
  rule foo(0) => .Seq

  syntax X ::= "foo"
  rule <k> foo => foo(3) </k>
endmodule

$ ./ext/k/k-distribution/target/release/k/bin/kompile foo.k --backend haskell
[Warning] Compiler: Could not find main syntax module with name FOO-SYNTAX in
definition.  Use --syntax-module to specify one. Using FOO as default.

$ cat foo
foo

$ ./ext/k/k-distribution/target/release/k/bin/krun foo
Reading the input file TimeSpec {sec = 0, nsec = 84834}
Parsing the file TimeSpec {sec = 0, nsec = 45296501}
Verifying the definition TimeSpec {sec = 0, nsec = 25025880}
Reading the input file TimeSpec {sec = 0, nsec = 31449}
Parsing the file TimeSpec {sec = 0, nsec = 121358}
Verifying the pattern TimeSpec {sec = 0, nsec = 126137}
Executing TimeSpec {sec = 0, nsec = 27401608}
<k>
  'Unds'12 : 'Unds'12 : 'Unds'12 : .Seq
</k>

Expected output is
<k>
  'Unds'12 : 'Unds'13 : 'Unds'14 : .Seq
</k>

Summary: at first sight, it seems to be both a frontend issue (universal quantification instead of existential) and a backend issue (i.e. not renaming conflicting variables). I think that we implicitly assumed existential quantification for new variables in rules (assumption that should go away at some point, I guess), so the main problem should be in the backend.
The rule foo(N) => ?_:Int : foo(N -Int 1) axiom gets translated to
axiom{R} \implies{R} (
    \equals{SortBool{},R}(
        Lbl'Unds-GT-'Int'UndsUnds'INT-COMMON'UndsUnds'Int'Unds'Int{}(
            VarN:SortInt{},
            \dv{SortInt{}}("0")
        ),
        \dv{SortBool{}}("true")),
    \and{R} (
        \equals{SortSeq{},R} (
            Lblfoo'LParUndsRParUnds'FRESH-VARS'UndsUnds'Int{}(VarN:SortInt{}),
            Lbl'UndsColnUndsUnds'FRESH-VARS'UndsUnds'Int'Unds'Seq{}(
                Var'Unds'12:SortInt{},
                Lblfoo'LParUndsRParUnds'FRESH-VARS'UndsUnds'Int{}(
                    Lbl'Unds'-Int'UndsUnds'INT-COMMON'UndsUnds'Int'Unds'Int{}(
                        VarN:SortInt{},
                        \dv{SortInt{}}("1")
                    )
                )
            )
        ),
        \top{R}()
    )
)

or, in a more readable form:
axiom{R} \implies{R} (
    ((varN <Int 0) == true)
    \equals{SortSeq{},R} (
        foo(VarN),
        Coln(Var_12:SortInt{}, foo(VarN -Int 1))
    )
)

Note that Var_12:SortInt{} is universally quantified, and foo(VarN) is supposed to be a singleton, so the above axiom should not work (isn't the axiom pattern always bottom if Coln is not constant in the first argument?). @traiansf in case he has comments, but, I guess, it should be quantified existentially.
Regardless, if I were to apply this to foo(3), the first step would go like this:
foo(3) = Coln(Var_12, foo(2))

But then, when replacing foo(2) with Coln(Var_12, foo(1)), should't Var_12 be renamed? I mean, that was one of the important thing that Target variables were supposed to do, so I'll assume that the backend has a bug here, too. @ttuegel in case he has comments about this.
Regardless of the above comments, I'll try to get a better understanding of the backend issue.

@virgil-serbanuta Ah, I realized what the problem is! One of the first things we do when we apply a rule is rename all the rule's implicitly-quantified variables so they do not conflict with any variables in the configuration. But, when we apply a function rule, we only apply the rule to a sub-term, so we cannot see all the variables in the pattern! One solution would be: any time we introduce a variable when applying a rule, that variable should be existentially quantified in the resulting term. We never did this before, so I think this bug has always been lurking. Existentially quantifying all the new variables might cause problems for us because we can't always simplify the binders away. In principle, if we could lift the quantifier all the way to the top of the configuration, we could remove it.

I agree with this in principle, but we should be a bit careful: existential quantifiers could become universal when lifting in predicates (in theory, they could become universal in terms, too), we can't change the quantifier order, and we can only remove existential ones at the top. So, in cases where lifting would produce an universal quantifier at the top, it might be better to leave the existential where it occurred.
However, the Scala backend lifted all quantifiers, so I wonder if there's something I'm missing. Anyway, I would only lift existential quantifiers if they have an even number of negation above, and the universal ones when they have an odd number of negations.
Indeed, the bug has been lurking for a long time. However, the initial implementation for variable uniqueness attempted to generate globally unique IDs, which, for all its drawbacks, didn't have this problem. On the other hand, we never used explicit quantifiers for variables introduced by rule application, so, implicitly, we also assumed that all functions (including the ones in predicates) had an even number of negations above them.

I wouldn't want to just lift the quantifiers immediately, but have the simplifier move them up level-by-level, until they can be removed (if desired) at the top level. Does that address your concern about the parity of negation?

I meant this: if we have a term sigma(exists x . phi, not (exists y . psi)), we can get any of these when lifting quantifiers:
forall y . exists x . sigma(phi, not psi)
exists x . forall y . sigma(phi, not psi)
exists x . sigma(phi, not (exists y . psi))

The first formula seems useless: We can't remove any quantifier present now, and none of the new quantifiers that may show up in the formula.
The second formula seems slightly more useful: we can remove exists x, but it will be the last quantifier that we'll ever remove. With some effort, we could make sure that our simplification generates this instead of the first formula.
The third formula seems the most useful, since we can remove exists x and many of the future quantifiers that may show up in the formula. The effort needed to implement this seems similar to the effort needed to distinguish the second formula from the first, but I might be wrong.
So then we have the following options (maybe others, too):

Stop executing and crash when we get a forall at the top.
Figure out how to deal with forall at the top (perhaps we just need to learn to ignore quantifiers at the top and hope that either we don't see forall at the top or the SMT can handle it).
Never lift quantifiers that will become forall at the top (again, hoping that the SMT can either handle or ignore it). Perhaps we can even try to narrow the scope for these quantifiers.

I think that all of the above could work. The first one will let us know if this can happen in real-life cases and we'll have a concrete example that may provide hints about what to do. The latter will let us ignore the quantifier for a long time, hoping that the SMT can deal with it (assuming that we have negations only in predicates, which seems to be reasonable for now).

Never lift quantifiers that will become forall at the top (again, hoping that the SMT can either handle or ignore it).

Right now, I don't think we lift quantifiers around \not, so this should not be a problem. I'm assuming we prefer to work with \exists whenever possible. We could add simplification cases that lift \not \forall into \exists \not and lower \forall \not into \not exists.

Well, IIRC, right now we don't lift any quantifiers at all...

I just merged #684 which fixes the "the resulting pattern is broken" issue. Should we keep this issue open until we lift quantifiers, or should we close it now?

Let's close it now. We will discover more and different problems that will require lifting quantifiers.


Release 0.18.3

Scheduled release date: TBD
This is a tracker to aggregate issues that should be part of 0.18.3.
Please add specific Issues or PRs prefixed with the "Depends on" keyword
(and remember that PRs need to be backported to the release-0.18 branch).
Make sure to follow the manual steps listed below before starting the release process.
Manual Steps
For detailed information regarding each step, please check: https://submariner.io/development/building-testing/ci-maintenance/

[ ] Custom GitHub Actions
[ ] GitHub Actions
[ ] Kubernetes Versions
[ ] Shipyard Linting Image Software

Depends on https://github.com/submariner-io/cloud-prepare/pull/1046
Depends on https://github.com/submariner-io/lighthouse/pull/1677
Depends on https://github.com/submariner-io/submariner/pull/3221
Depends on https://github.com/submariner-io/subctl/pull/1269
Depends on https://github.com/submariner-io/submariner-operator/pull/3276

Assigning to myself

Released: https://github.com/submariner-io/releases/releases/tag/v0.18.3


How to use time-machine with Ember-data

Hi,
I want to use Ember-Time-Machine on all of my models, so I can rollback user changes with '''undoAll''.  Is there any documentation about how to use TM with EmberData models.
My first attempt was to reopen the 'DS.Store.' and wrap TimeMachine around the 'findRecord' function like this:
'''
DS.Store.reopen({
findRecord() {
let record=this._super(...arguments);
return TimeMachine.Object.create({content: record});
}
'''
But this (ofcourse) doensn't work, because the TimeMachine object is not the same as what 'findRecord' returns normally :-)

@henderikus the example in the demo uses TM + ED. Unfortunately, I dont believe what you are trying to do is possible nor best practice. I suggest the following:
import Ember from 'ember';
import TimeMachine from 'ember-time-machine';

export default Ember.Route.extend({
  model() {
    return this.store.findRecord('user', 1);
  },

  setupController(controller, model) {
    controller.set('model', TimeMachine.Object.create({
      content: model
   }));
  }
});

Once you have a TM set for a single model, all child models will be tracked as well.

@henderikus Im gonna go ahead and close this one, feel free to reopen if you need some more help.


Pressing Command Key (while text selected) in Left-hand Side of Diff Editor Makes it Vanish

VSCode Version: Insiders 25dc9851ef6d41fb4939a37312590cadaee90327
OS Version: macOS

Steps to Reproduce:

From the Git view, open a diff containing some changes from an unstaged change
Select some text on the left-hand side
Press the Cmd key (as if you were going to press a hotkey to copy the selected text).
Note that the left-side of the diff view is now gone.

@bgashler1 Any messages in the console ?

Great catch!

Cannot reproduce in 40243d1e01, which doesn't have the fix. Any instructions for verification?

I can't repro either. I checked out vscode at the commit before the fix

Verifications steps:

Change a CSS file in a git repo
Open the diff using the git viewlet
Hold Ctrl and move mouse around in the readonly editor

@ramya-rao-a I verified this


Add List fine-grained personal access tokens with access to organization resources API

Fixes: #3213.

@arymoraes @gmlewis Could you please check who will be the next reviewer?

@arymoraes @gmlewis Could you please check who will be the next reviewer?

In this repo, we depend upon volunteers to review.  Since you are another contributor to this repo, you are welcome to review this PR and give your feedback. If you approve, please LGTM+Approve and we can move forward with this PR. Thank you.

Thank you, @AbhishekAg !
Merging.


Writing Inertia Documentation

In my opinion, Inertia will make a huge step for Laravel community, really thank you for this awesome work. I'll be very happy to help you out writing the docs using my package larecipe if you want of course. Thank you again 😍❤

@saleem-hadad All help is appreciated. I haven't had a chance to look at your package in-depth. But my first thoughts are that it would help tremendously.
However, from what I can gather from your package description your package helps document features inside a Laravel app and seeing as this is just an adapter and not a fully fleshed out app I'm guessing it might not work as intended.
If I'm mistaking then I'm sure @reinink is all for everything getting documented. 😄


upgrade React and Fluxible

"fluxible": "1.1.0",
"fluxible-addons-react": "0.2.8",
"fluxible-plugin-fetchr": "0.3.9",
"fluxible-router": "0.4.14",
"react": "15.0.2",
"react-dom": "15.0.2",
"react-intl": "2.1.2",

Thanks @leplay ! Awesome 👍

😀  This example really helped me to understand Fluxible. Thanks.


break point not working in ipad chrome

[x] bug
Swiper Version:  4.1.6.
Platform/Target and Browser Versions: IOS 11 chrome (no issue in safari)
Live Link or JSFiddle/Codepen or website with isssue: http://idangero.us/swiper/demos/380-responsive-breakpoints.html

Expected Behavior
break point ==>
1024: {
slidesPerView: 4,
spaceBetween: 40,
},
768: {
slidesPerView: 3,
spaceBetween: 30,
}
four slides in landscape
three slides in portrait
Actual Behavior
three slides in both mode after orientation change.

I'm Very sorry to be away for long....
yes exactly... its new chrome issue
i can not reproduce this now, i think it fixed by chrome.
THANK YOU SO MUCH for your great response 👍 .


MacOS collision between OGRE1.9 and OGRE2.1

While I was debugging the issue with OGRE + QML on MacOS I founded this issue:
objc[37280]: Class OgreConfigWindowDelegate is implemented in both /usr/local/opt/ogre1.9/lib/libOgreMain.1.9.0.dylib (0x14b388dc0) and /usr/local/opt/ogre2.1/lib/libOgreMain.2.1.0.dylib (0x1539eb838). One of the two will be used. Which one is undefined.

I talked with @iche033 about this issue in the past:
As for loading the right OgreMain version, I think on ubuntu, we worked around it by installing to different directories, e.g. /usr/lib/x86_64-linux-gnu/OGRE-2.1 and /usr/lib/x86_64-linux-gnu/OGRE-1.9.0.  I think we are using the same approach on MacOS but it's not working properly.

I think we made the ubuntu debs work with debian rules in ignition-forks/ogre-2.1-release#1 but since macOS builds and installs ogre2 from source, the two ogre libraries both just get installed to the same directory.

I'm not sure this a problem that can be solved by installing ogre and ogre2 to different directories. The issue seems to be that for macOS both Ogre libraries contain an objective C class with the same name: when the render engine attempts to load the second ogre rendering plugin the choice is not uniquely defined.
No fix at this stage, but some more detail on causes below.
Test case: ign-rendering/examples/mesh_viewer
The following traces are from the example ign-rendering/examples/mesh_viewer modified to load both engines "ogre" and "ogre2".
The behaviour can be replicated using either CMAKE_BUILD_TYPE=Debug or CMAKE_BUILD_TYPE=RelWithDebInfo.
The application works when either ignition-rendering-ogre or ignition-rendering-ogre2 is loaded,
but fails when both are loaded. Furthermore the ogre2 case will fail if there is any linkage with Ogre1.9,
for example if mesh_viewer is linked against Ogre1.9 in order to catch Ogre::Exception.
The good news is that some of the Ogre2 examples work on macOS, just not combined with Ogre1
The conflict error is the same as given above:
@"objc[54119]: Class OgreConfigWindowDelegate is implemented in both /usr/local/Cellar/ogre1.9/1.9.0/lib/libOgreMain.1.9.0.dylib (0x10fb93628) and /usr/local/opt/ogre2.1/lib/libOgreMain.2.1.0.dylib (0x11cffb290). One of the two will be used. Which one is undefined.\r\n"

On macOS Ogre libraries are installed in different directories. Both have symlinks in /usr/local/lib but I do not think that is the problem. Inspecting the symbols shows both have an entry for _OBJC_CLASS_$_OgreConfigWindowDelegate:
$ nm /usr/local/Cellar/ogre1.9/1.9.0/lib/libOgreMain.1.9.0.dylib | grep OgreConfigWindowDelegate
00000000002886c0 t -[OgreConfigWindowDelegate cancelButtonPressed:]
0000000000288280 t -[OgreConfigWindowDelegate dealloc]
0000000000288a90 t -[OgreConfigWindowDelegate getConfigWindow]
0000000000288b00 t -[OgreConfigWindowDelegate getOgreLogo]
0000000000288b40 t -[OgreConfigWindowDelegate getOptionsPopUp]
0000000000288b20 t -[OgreConfigWindowDelegate getOptionsTable]
0000000000288ab0 t -[OgreConfigWindowDelegate getOptions]
0000000000288ad0 t -[OgreConfigWindowDelegate getRenderSystemsPopUp]
0000000000287820 t -[OgreConfigWindowDelegate init]
0000000000288790 t -[OgreConfigWindowDelegate numberOfRowsInTableView:]
0000000000288710 t -[OgreConfigWindowDelegate okButtonPressed:]
0000000000288360 t -[OgreConfigWindowDelegate popUpValueChanged:]
0000000000288aa0 t -[OgreConfigWindowDelegate setConfigWindow:]
0000000000288af0 t -[OgreConfigWindowDelegate setOgreLogo:]
0000000000288ac0 t -[OgreConfigWindowDelegate setOptions:]
0000000000288b30 t -[OgreConfigWindowDelegate setOptionsPopUp:]
0000000000288b10 t -[OgreConfigWindowDelegate setOptionsTable:]
0000000000288ae0 t -[OgreConfigWindowDelegate setRenderSystemsPopUp:]
0000000000288750 t -[OgreConfigWindowDelegate tableView:objectValueForTableColumn:row:]
00000000002887b0 t -[OgreConfigWindowDelegate tableView:shouldSelectRow:]
0000000000288680 t -[OgreConfigWindowDelegate windowShouldClose:]
0000000000393628 S _OBJC_CLASS_$_OgreConfigWindowDelegate
0000000000393610 S _OBJC_IVAR_$_OgreConfigWindowDelegate.mCancelButton
00000000003935e0 S _OBJC_IVAR_$_OgreConfigWindowDelegate.mConfigWindow
00000000003935e8 S _OBJC_IVAR_$_OgreConfigWindowDelegate.mOgreLogo
0000000000393608 S _OBJC_IVAR_$_OgreConfigWindowDelegate.mOkButton
0000000000393618 S _OBJC_IVAR_$_OgreConfigWindowDelegate.mOptionLabel
0000000000393620 S _OBJC_IVAR_$_OgreConfigWindowDelegate.mOptions
00000000003935f8 S _OBJC_IVAR_$_OgreConfigWindowDelegate.mOptionsPopUp
0000000000393600 S _OBJC_IVAR_$_OgreConfigWindowDelegate.mOptionsTable
00000000003935f0 S _OBJC_IVAR_$_OgreConfigWindowDelegate.mRenderSystemsPopUp
0000000000393650 S _OBJC_METACLASS_$_OgreConfigWindowDelegate
0000000000392ea0 s __OBJC_$_INSTANCE_METHODS_OgreConfigWindowDelegate
00000000003930a0 s __OBJC_$_INSTANCE_VARIABLES_OgreConfigWindowDelegate
00000000003931c8 s __OBJC_$_PROP_LIST_OgreConfigWindowDelegate
0000000000392e30 s __OBJC_CLASS_PROTOCOLS_$_OgreConfigWindowDelegate
0000000000393210 s __OBJC_CLASS_RO_$_OgreConfigWindowDelegate
0000000000392e58 s __OBJC_METACLASS_RO_$_OgreConfigWindowDelegate

$ nm /usr/local/opt/ogre2.1/lib/libOgreMain.2.1.0.dylib | grep OgreConfigWindowDelegate
000000000023fb31 t -[OgreConfigWindowDelegate cancelButtonPressed:]
000000000023fdf9 t -[OgreConfigWindowDelegate getConfigWindow]
000000000023fe3f t -[OgreConfigWindowDelegate getOgreLogo]
000000000023fe67 t -[OgreConfigWindowDelegate getOptionsPopUp]
000000000023fe53 t -[OgreConfigWindowDelegate getOptionsTable]
000000000023fe0d t -[OgreConfigWindowDelegate getOptions]
000000000023fe21 t -[OgreConfigWindowDelegate getRenderSystemsPopUp]
000000000023eff5 t -[OgreConfigWindowDelegate init]
000000000023fbf0 t -[OgreConfigWindowDelegate numberOfRowsInTableView:]
000000000023fb78 t -[OgreConfigWindowDelegate okButtonPressed:]
000000000023f939 t -[OgreConfigWindowDelegate popUpValueChanged:]
000000000023fe03 t -[OgreConfigWindowDelegate setConfigWindow:]
000000000023fe35 t -[OgreConfigWindowDelegate setOgreLogo:]
000000000023fe17 t -[OgreConfigWindowDelegate setOptions:]
000000000023fe5d t -[OgreConfigWindowDelegate setOptionsPopUp:]
000000000023fe49 t -[OgreConfigWindowDelegate setOptionsTable:]
000000000023fe2b t -[OgreConfigWindowDelegate setRenderSystemsPopUp:]
000000000023fbb0 t -[OgreConfigWindowDelegate tableView:objectValueForTableColumn:row:]
000000000023fc06 t -[OgreConfigWindowDelegate tableView:shouldSelectRow:]
000000000023faf6 t -[OgreConfigWindowDelegate windowShouldClose:]
00000000002ce290 s _OBJC_CLASS_$_OgreConfigWindowDelegate
00000000002ce278 s _OBJC_IVAR_$_OgreConfigWindowDelegate.mCancelButton
00000000002ce248 s _OBJC_IVAR_$_OgreConfigWindowDelegate.mConfigWindow
00000000002ce250 s _OBJC_IVAR_$_OgreConfigWindowDelegate.mOgreLogo
00000000002ce270 s _OBJC_IVAR_$_OgreConfigWindowDelegate.mOkButton
00000000002ce280 s _OBJC_IVAR_$_OgreConfigWindowDelegate.mOptionLabel
00000000002ce288 s _OBJC_IVAR_$_OgreConfigWindowDelegate.mOptions
00000000002ce260 s _OBJC_IVAR_$_OgreConfigWindowDelegate.mOptionsPopUp
00000000002ce268 s _OBJC_IVAR_$_OgreConfigWindowDelegate.mOptionsTable
00000000002ce258 s _OBJC_IVAR_$_OgreConfigWindowDelegate.mRenderSystemsPopUp
00000000002ce2b8 s _OBJC_METACLASS_$_OgreConfigWindowDelegate

The error is raised as soon as the second plugin ignition-rendering-ogre2 is loaded here:
https://github.com/ignitionrobotics/ign-rendering/blob/9ca3a59c721e0a11cc835083d8efad88919fdcf7/src/RenderEngineManager.cc#L467
The application subsequently exits with an Ogre RenderingAPIException.
In the case we load the ignition-rendering-ogre plugin before the ignition-rendering-ogre2 plugin the exception is thrown in: OgreGLTexture.cpp - GLTexture::_createSurfaceList() on a GLTextureBuffer call. It's not clear at this stage why the reported name conflict is causing this to fail. It's possible this is not the only conflict, just the first reported.
Location of conflicting Ogre class
In OGRECave/ogre the conflicting class is currently defined here (it appears to have moved since 1.9):
https://github.com/OGRECave/ogre/blob/2dc0861786db9019d8e862ada666f1450edf3006/Components/Bites/src/OgreOSXConfigDialog.mm#L41
In OGRECave/ogre-next (Ogre2) it is here:
https://github.com/OGRECave/ogre-next/blob/184741a085af54464210fb6de80fd29bfcf9b401/OgreMain/include/OSX/OgreConfigDialogImp.h#L42


A blinking cursor should push only one frame (#3445)

Prior to this patch, we were pushing two frames each time the cursor blinked.
In turning the cursor on or off, the markNeedsPaint call was triggering another
frame to be scheduled because we cleared a bit in the scheduler at the
beginning of the frame instead of at the end of the frame.
To implement scheduling correctly, we actually need two bits: one for
ensureVisualUpdate, which just promises to get to the end of the pipeline soon,
and scheduleFrame, which promises to get to the beginning of the pipeline soon.
(Reland)

@Hixie

This patch is almost the same as the earlier one.  The difference is in the widgets/bindings.dart

lgtm. test?

The trouble with writing a test for this issue is that this is the part of the system we've mocked out in the testing harness in order to let the test drive frames.


How can i add SupabaseVectorStore in LangFlow

I'm currently reading LangFlow code, but I still can't figure out where the implementation of chains, agents, llms, and so on takes place.
So I want to know here how you can add your components (in this case SupabaseVectorStore). Thanks in advance.

Hey, @YoungGoofy.
You can add it to the config.yml and then test it in a flow.
If it does not work, then you should find the code for that in src/backend/langflow/interface/vector_store.
The most important part of adding new components is to test them.


Add some broadcasting syntax for ITensors

This adds some broadcasting syntax to ITensors, such as:
i = Index(2,"i")
A = randomITensor(i,i')
B = randomITensor(i',i)
c = 2
B .= c .* A

to overwrite B with c*A, without any allocations. Internally, this will just call mul!(B,c,A). Take a look at the tests for a more complete list of example operations that are supported now.
@emstoudenmire, this made me think that we could abuse the broadcasting syntax a bit to get lazy ITensor contractions. For example, it would be simple enough to make:
i = Index(2,"i")
A = randomITensor(i,i')
B = randomITensor(i',i'')
C = randomITensor(i,i'')
c = 2
C .= c .* A  .* B

do an in-place multiplication of A and B into C, scaling by c, without any allocations. This would be a bit of an abuse of the notation, since for Julia Arrays, this operation would not do a matrix multiplication but instead does an elementwise multiplication (a Hadamard product). However, ITensors act differently enough from Julia Arrays that maybe this is ok.

Very nice. It’s great to see this can be added with a fairly small bit of code, covering all storage types! On the other hand, it still seems unfortunate to me that Julia works this way, namely that one can’t get this kind of behavior already with the standard, undotted notation.
One question about this feature: I suppose the reason for the exceptions in the last unit tests is the ordering of indices, correct? And that the broadcasted operations requires the same index ordering?

I think this code is good to go -

Very nice. It’s great to see this can be added with a fairly small bit of code, covering all storage types! On the other hand, it still seems unfortunate to me that Julia works this way, namely that one can’t get this kind of behavior already with the standard, undotted notation.

That is only partially true. Even for an operation like B .= c .* A, this adds something beyond doing B = c * A in the C++ version, since the two dots are "fused" into a single operation, so no temporaries are made (it actually works by lazily handling the c .* A operation).

One question about this feature: I suppose the reason for the exceptions in the last unit tests is the ordering of indices, correct? And that the broadcasted operations requires the same index ordering?

The broadcast operations here are done by overloading other general ITensor operations like mul! and add!, so they don't require the same index ordering (the permute is handled as usual, note that all of the other tests involve tensors that are transposes of each other). It is actually a bit more subtle, mostly just that those operations correspond to ones we don't have nice ITensor functions for right now. There is no reason we can't support them, I was just keeping this PR simpler and we can add more functionality later.
What do you think about using C .= A .* B as a syntax for contracting A and B and storing the result in C, i.e. without temporaries?

Agreed that it's doing something we don't even have in C++. I was more saying I wish there was a better approach to this sort of thing that either Julia's or C++; there could be, I think, if language designers thought about this case ahead of time.  Maybe later on Julia will have a way to introspect into the garbage collecting system or do custom garbage collection to see if references are unique. What makes me nervous is that I don't want users to think they should try to dot everything just in case the undotted code is slow; but we can lead there by showing examples.
Regarding .* for ITensor contractions, I'm supportive of it, so let's go ahead. I suppose the only reason not to would be because it would mean a different thing for Julia arrays? But then like you said, we are already interpreting * differently for ITensors, so I think it's very sensible to have .* mean the broadcasted version of the same thing.
To offer an element-wise broadcasting operation, we could overload a different symbol for it. I believe there are a number of available mathematical symbols for overload, such as a circle with a dot in the center, etc. That might even be closer to the math tensor literature notation. And of course we could just have a function for that operation when it's not occurring in a broadcasting operator expression.
Thanks for explaining about the indices -

Agreed that it's doing something we don't even have in C++. I was more saying I wish there was a better approach to this sort of thing that either Julia's or C++; there could be, I think, if language designers thought about this case ahead of time. Maybe later on Julia will have a way to introspect into the garbage collecting system or do custom garbage collection to see if references are unique. What makes me nervous is that I don't want users to think they should try to dot everything just in case the undotted code is slow; but we can lead there by showing examples.

I agree it would be nice if there was better reference counting functionality in Julia, as well as the ability overloading = and *=.
Definitely giving good examples will be useful, and eventually a lot of the usage of dots can be replaced by ITensor-level lazy operations (I think basically everything except for .=, .*= and .+= could be).

Regarding .* for ITensor contractions, I'm supportive of it, so let's go ahead. I suppose the only reason not to would be because it would mean a different thing for Julia arrays? But then like you said, we are already interpreting * differently for ITensors, so I think it's very sensible to have .* mean the broadcasted version of the same thing.

Yes, that is the only reason. There is some interpretation that the dot means an element-wise operation, and contraction is not quite that, but it seems ok to stretch the definition since contraction is clearly much more useful for ITensors.

To offer an element-wise broadcasting operation, we could overload a different symbol for it. I believe there are a number of available mathematical symbols for overload, such as a circle with a dot in the center, etc. That might even be closer to the math tensor literature notation. And of course we could just have a function for that operation when it's not occurring in a broadcasting operator expression.

Yeah, it is not something that has even really come up so far, but it would be pretty easy to overload another symbol for that operation. Wikipedia uses an open circle or an open circle with a dot for a Hadamard product.

Thanks for explaining about the indices -


fix: [M3-7746] - Fix $0 region price error in "Enable All Backups" drawer

Description 📝
This fixes a couple issues in the "Enable All Backups" drawer when one or more Linodes are in a $0-priced region.

Fixes calculation of total backup cost
Removes error indicator

Changes  🔄
List any change relevant to the reviewer.

Fixes calculation of total backup price when one or more Linodes is in a $0 region
Removes error indicator from backup rows for Linodes in $0 regions
Adds and updates tests to account for these states

Preview 📷
Include a screenshot or screen recording of the change
:bulb: Use <video src="" /> tag when including recordings in table.

Before
After

How to test 🧪
We should be able to rely on the unit tests and Cypress tests to cover this, since getting the "Enable All Linode Backups" drawer to appear can be difficult depending on the state of your account.
As an Author I have considered 🤔
Check all that apply

[x] 👀 Doing a self review
[ ] ❔ Our contribution guidelines
[ ] 🤏 Splitting feature into small PRs
[ ] ➕ Adding a changeset
[x] 🧪 Providing/Improving test coverage
[ ] 🔐 Removing all sensitive information from the code and PR description
[ ] 🚩 Using a feature flag to protect the release
[ ] 👣 Providing comprehensive reproduction steps
[ ] 📑 Providing or updating our documentation
[ ] 🕛 Scheduling a pair reviewing session
[ ] 📱 Providing mobile support
[ ] ♿  Providing accessibility support

I saw a case where the total wasn't be displayed as expected if the totalBackupsPrice is 0 in BackupDrawer.tsx.
Oh awesome catch, thanks @mjac0bs! I think this even warrants its own test case, so I'll handle that all at once tomorrow.

Also totally agreed on that const name, I'll take care of that too 👍 Thank you!

@mjac0bs Just pushed a fix for that $0 total price issue, as well as some unit tests for the BackupDrawer component


Outline-map shows much less items than the built-in outline

Example project: https://github.com/QuarticCat/nbe-at-large/blob/160881a8916c5564955363c03b1505899157fb38/grammar/src/syntactic.rs#L96
With the rust-analyzer extension.
In VSCode's built-in outline view:

In the outline map:

Well, after using this extension for a while, I think this might be expected behavior. So I'll close this issue. :)

Here's one more example: https://github.com/archlinux/archinstall/blob/master/archinstall/lib/disk/filesystem.py
With the official Python and Pylance extension.
In VSCode's built-in outline view (I filtered variables):

In the outline map:

It turns out to be the problem that these items are not automatically expanded. This issue might be duplicated.


allow buildah cross compile for a darwin target

the goal here is to allow ourselves to cross compile buildah for a darwin
target.  we are doing this to eventually protect from regressions that could
creep into buildah so we don't dig ourselves a deeper hole.
the simplified and non-variable approach to the make darwin was done with
intent to keep this simple until we can exploit things a little more.
once this PR merges, i will create a CI test that will test for regressions
on a make darwin.  we should also be doing a gofmt with a darwin target so the
!linux|darwin tagged files are also checked for completeness.  initially the
test can be optional for passing with the long-term idea that it be made
a firm requirement at the buildah maintainers behest.
Signed-off-by: baude<EMAIL_ADDRESS>

LGTM

@rh-atomic-bot r+


React: Render Conditionally from Props Incorrect Line Reference

The challenge seed has since been updated, and this paragraph is out of line 😉

First, you'll need a simple expression that randomly returns a different value every time it is run. You can use Math.random(). This method returns a value between 0 (inclusive) and 1 (exclusive) each time it is called. So for 50/50 odds, use Math.random() >= .5 in your expression. Statistically speaking, this expression will return true 50% of the time, and false the other 50%.

This should be reworded to not make a line reference.
I suggest:

...Statistically speaking, this expression will return true 50% of the time, and false the other 50%. In the render method, replace null with the above expression to complete the variable declaration.

Challenge: https://www.freecodecamp.org/learn/front-end-libraries/react/render-conditionally-from-props

I agree.

Can I have a bash at this?


added some docs for HAL sources of truth; updates #426

Gonna try and merge it before it gets out of hand and maybe having something in there will inspire more people to make it better.
Most notably,

it solves the "noisy building problems" by putting all integration tests into one target that can be different based on precoimpile flags
makes it possible to deploy to several prototypes according to Nucleo's embedded serial number
A bash script that demonstrates how one would deploy multiple tests to machine in sequence. This will of course be improved to also take readings from external sensors before we add it to CI.
Some stubs of code for relay and LabJack (mostly Riley, a bit of me just trying out different pin numbers).
Notes on streaming webcam data out of the server.

Sorry it's such a mishmash. Kept sporadically working on it prior to covent but now realized it got kind of big enough to ship even in its ugly state.
Changelog as follows:
gitignore entry for CLion IDE
added Labjack I2C code
added python script for 8 channel relay
added lots of docs about integration tests
added info on where prototype is
added notes on webcam business
added motion config
added buzzer test
renamed buzzer test, stepper test on channel 1
added idle test
simplified integration test building, made script to run a few in sequence
configured platformio to deploy to multiple prototypes, made a script for running a few tests in sequence

LGTM with precommits fixed.  You should be able to run pre-commit run --from-ref $(git merge-base HEAD origin/master) --to-ref HEAD.


Make create_logger code a little more concise

The default level for a handler is logging.NOTSET (which has a numeric value of 0). We only need to change the level for the logger.

LGTM :slightly_smiling_face:


Cleanup tests

The current tests were too slow. I improved this

Moving model definition to conftests.py to reduce duplication.
Set workers to 0 to deactivate pytest parallel fit. This one is a bit a mystery why it improves performance during tests, but not during normal model fitting.

I'm still looking into this, my local tests are slower than on azure. I saw speedups locally, but only 1 min faster on azure.

For example, here is pytest on local for me.
-- Docs: https://docs.pytest.org/en/stable/warnings.html
Profiling (from /Users/benweinstein/Documents/DeepForest/prof/combined.prof):
Tue Oct  3 11:46:14 2023    /Users/benweinstein/Documents/DeepForest/prof/combined.prof

         50523311 function calls (49528540 primitive calls) in 663.158 seconds

   Ordered by: cumulative time
   List reduced from 5970 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       84    0.001    0.000  663.179    7.895 runner.py:106(pytest_runtest_protocol)
       84    0.338    0.004  663.147    7.895 runner.py:114(runtestprotocol)
 1691/924    0.008    0.000  662.809    0.717 hooks.py:272(__call__)
      252    0.002    0.000  662.808    2.630 runner.py:212(call_and_report)
 1691/924    0.001    0.000  662.802    0.717 manager.py:90(_hookexec)
 1691/924    0.004    0.000  662.802    0.717 manager.py:84(<lambda>)
 1691/924    0.019    0.000  662.799    0.717 callers.py:157(_multicall)
      252    0.001    0.000  662.765    2.630 runner.py:240(call_runtest_hook)
      252    0.002    0.000  662.761    2.630 runner.py:298(from_call)
      252    0.001    0.000  662.759    2.630 runner.py:255(<lambda>)
       84    0.000    0.000  626.554    7.459 runner.py:153(pytest_runtest_call)
       84    0.000    0.000  626.553    7.459 python.py:1639(runtest)
       84    0.950    0.011  626.549    7.459 python.py:176(pytest_pyfunc_call)
       21    0.000    0.000  430.850   20.517 call.py:30(_call_and_handle_interrupt)
       21    0.002    0.000  430.844   20.516 trainer.py:854(_run)
       21    0.000    0.000  390.264   18.584 trainer.py:966(_run_stage)
       71    0.312    0.004  347.718    4.897 retinanet.py:571(forward)
       12    0.000    0.000  332.843   27.737 trainer.py:489(fit)
       12    0.000    0.000  332.843   27.737 trainer.py:524(_fit_impl)
24394/161    1.696    0.000  325.961    2.025 module.py:1124(_call_impl)

========================= 84 passed, 4014 warnings in 672.84s (0:11:12) ==========================
(DeepForest) benweinstein@Bens-MacBook-Pro DeepForest %

I would still like to merge this PR because it has good benefits readability, but if tests are about 7min on azure, i'm not sure I'm going to improve that. @henrykironde for your review.


fix warning

$ unzip -t youtube-dl
Archive:  youtube-dl
warning [youtube-dl]:  22 extra bytes at beginning or within zipfile
  (attempting to process anyway)
    testing: youtube_dl/YoutubeDL.py   OK
    testing: youtube_dl/__init__.py   OK
    testing: youtube_dl/__main__.py   OK
...

Ermm... Better adding --quiet

It's not supposed to be unzipped by hand.

It's not supposed to be unzipped by hand, yes.
But the zip offsets are incorrect, after all.


Thinking about adding a dark mode?

Is it possible to consider adding a dark mode switch, which could be great for writers used to writing in dark mode?

I've been able to implement dark-mode in tailwind with the typography plugin with just prose dark:prose-invert.

@notjustinshaw 👀 that's awesome! Feel free to make a PR to add dark mode to Novel! 😄

I could get darkmode to work, I'm using cookies to store current mode

I could get darkmode to work, I'm using cookies to store current mode

this dark mode ui looks in consistent , that toggle button size is large , continue writing icon missing etc  also y store in cookie  ? i think it is better to store the theme state in local storage than cookies

Yeah, I fixed all the issues you mentioned in the above commits already. 😁
Why cookies: it's hard to manage global state in server components, local storage could be done but this was a very straightforward approach, so I went with it

Yeah, I fixed all the issues you mentioned in the above commits already. 😁
Why cookies: it's hard to manage global state in server components, local storage could be done but this was a very straightforward approach, so I went with it

Ne olduğunu anlamayan yine ben

Dark mode is now live!! #80


Investigating performance issues in philox

copying @gargrahul
@jszuppe could you or someone help here? One of the teams is investigating some performance regression issues in rocRAND with changes in compiler (hcc/hip in ROCm). When running (fox example): benchmark_rocrand_kernel --engine philox --dis uniform-float
we are seeing slowdowns. Could you point to the source kernels that are launched with this command? Is it somewhere here:
https://github.com/ROCmSoftwarePlatform/rocRAND/blob/master/library/src/rng/philox4x32_10.hpp
Also explain some top level details about the kernel, and the kernel launch parameters?
It would also help if you could give some tips/ways to simplify the kernel as we track down the perf issues to compiler changes. Thanks

Does this happen only to Philox? And what about benchmark_rocrand_generate --engine philox --dis uniform-float?
How significant are these slowdowns?
It would be great if you show benchmarks for all engines and all distributions on the compilers (both old and new results):
benchmark_rocrand_kernel --engine all --dis all
benchmark_rocrand_generate --engine all --dis all

Regarding this particular benchmark.
The kernel is here: https://github.com/ROCmSoftwarePlatform/rocRAND/blob/master/benchmark/benchmark_rocrand_kernel.cpp#L84
And the main code for Philox is here: https://github.com/ROCmSoftwarePlatform/rocRAND/blob/master/library/include/rocrand_philox4x32_10.h

@ex-rzr There are minor differences in other benchmark results though philox results for benchmark_rocrand_kernel, with HIP PR 535 https://github.com/ROCm-Developer-Tools/HIP/pull/535 are significantly slower than without HIP PR 535 included.
Attached are detailed results. Philox results on Fiji before and after HIP PR 535 are as follows:-
(Before HIP PR 535)
philox:
uniform-uint:
Throughput =  123.534 GB/s, Samples =   30.883 GSample/s, AvgTime (1 trial) =    4.047 ms, Time (all) =   80.949 ms, Size = 134217728
uniform-float:
Throughput =  130.166 GB/s, Samples =   32.542 GSample/s, AvgTime (1 trial) =    3.841 ms, Time (all) =   76.825 ms, Size = 134217728
uniform-double:
Throughput =  119.575 GB/s, Samples =   14.947 GSample/s, AvgTime (1 trial) =    8.363 ms, Time (all) =  167.259 ms, Size = 134217728
normal-float:
Throughput =   78.683 GB/s, Samples =   19.671 GSample/s, AvgTime (1 trial) =    6.355 ms, Time (all) =  127.092 ms, Size = 134217728
normal-double:
Throughput =   25.275 GB/s, Samples =    3.159 GSample/s, AvgTime (1 trial) =   39.566 ms, Time (all) =  791.311 ms, Size = 134217728
log-normal-float:
Throughput =   63.957 GB/s, Samples =   15.989 GSample/s, AvgTime (1 trial) =    7.818 ms, Time (all) =  156.356 ms, Size = 134217728
log-normal-double:
Throughput =   17.524 GB/s, Samples =    2.191 GSample/s, AvgTime (1 trial) =   57.064 ms, Time (all) = 1141.270 ms, Size = 134217728
poisson:
lambda 10.0
Throughput =    2.363 GB/s, Samples =    0.591 GSample/s, AvgTime (1 trial) =  211.597 ms, Time (all) = 4231.938 ms, Size = 134217728
discrete-poisson:
lambda 10.0
Throughput =   51.815 GB/s, Samples =   12.954 GSample/s, AvgTime (1 trial) =    9.650 ms, Time (all) =  192.994 ms, Size = 134217728
discrete-custom:
Throughput =   52.257 GB/s, Samples =   13.064 GSample/s, AvgTime (1 trial) =    9.568 ms, Time (all) =  191.362 ms, Size = 134217728
(After HIP PR 535)
philox:
uniform-uint:
Throughput =   33.059 GB/s, Samples =    8.265 GSample/s, AvgTime (1 trial) =   15.125 ms, Time (all) =  302.491 ms, Size = 134217728
uniform-float:
Throughput =   32.981 GB/s, Samples =    8.245 GSample/s, AvgTime (1 trial) =   15.160 ms, Time (all) =  303.205 ms, Size = 134217728
uniform-double:
Throughput =   36.494 GB/s, Samples =    4.562 GSample/s, AvgTime (1 trial) =   27.402 ms, Time (all) =  548.040 ms, Size = 134217728
normal-float:
Throughput =   33.265 GB/s, Samples =    8.316 GSample/s, AvgTime (1 trial) =   15.031 ms, Time (all) =  300.617 ms, Size = 134217728
normal-double:
Throughput =   25.809 GB/s, Samples =    3.226 GSample/s, AvgTime (1 trial) =   38.747 ms, Time (all) =  774.934 ms, Size = 134217728
log-normal-float:
Throughput =   33.157 GB/s, Samples =    8.289 GSample/s, AvgTime (1 trial) =   15.080 ms, Time (all) =  301.598 ms, Size = 134217728
log-normal-double:
Throughput =   17.797 GB/s, Samples =    2.225 GSample/s, AvgTime (1 trial) =   56.191 ms, Time (all) = 1123.813 ms, Size = 134217728
poisson:
lambda 10.0
Throughput =    0.527 GB/s, Samples =    0.132 GSample/s, AvgTime (1 trial) =  948.613 ms, Time (all) = 18972.261 ms, Size = 134217728
discrete-poisson:
lambda 10.0
Throughput =   29.536 GB/s, Samples =    7.384 GSample/s, AvgTime (1 trial) =   16.928 ms, Time (all) =  338.568 ms, Size = 134217728
discrete-custom:
Throughput =   29.901 GB/s, Samples =    7.475 GSample/s, AvgTime (1 trial) =   16.722 ms, Time (all) =  334.433 ms, Size = 134217728
Do you see any possible reason for this big difference? How can I build philox benchmark with a single dist?
Post_HIP_535_benchmark_rocrand_generate.txt
Post_HIP_535_benchmark_rocrand_kernel.txt
Pre_HIP_535_benchmark_rocrand_generate.txt
Pre_HIP_535_benchmark_rocrand_kernel.txt

I can't check HIP with and without PR 535 right now, so here are my theoretical thoughts:

Only Philox uses vector types (uint4)

Philox host-API functions (benchmark_rocrand_generate) do not suffer this 75% performance degradation.

Philox device-API functions call next() while host-API functions call next4(): https://github.com/ROCmSoftwarePlatform/rocRAND/blob/master/library/include/rocrand_philox4x32_10.h#L196

I suspect that the line 198 causes the slowdown: https://github.com/ROCmSoftwarePlatform/rocRAND/blob/master/library/include/rocrand_philox4x32_10.h#L198

There is a dynamic indexing of an array stored in registers. This probably generates buffer_load instructions in ISA or something like that.
Can you replace this line with this code and run the benchmarks again?
        unsigned int ret;
        switch(m_state.substate)
        {
            case 0:
                ret = m_state.result.x;
                break;
            case 1:
                ret = m_state.result.y;
                break;
            case 2:
                ret = m_state.result.z;
                break;
            default:
                ret = m_state.result.w;
                break;
        }

If it works I'll create a PR.

Thanks @ex-rzr . Your suggestion works well. Please go ahead with PR.

Closed by #33


Dismiss Pop Up

added functionality to tap on map to dismiss pop up and also added the correct logic to only show the remaining times the bus will come.

Make the minor change and you're good to merge :)

commit: ef6b53b


video stream size prediction: measure input video stream size

Currently we have a couple of methods to use the measured sample encode percent to predict the final encoded video stream size of the whole input.

input_file_size * encode_percent Original method, it can work fairly well when the video stream is the vast majority of the file size. This falls down otherwise, e.g. when there are multiple large audio tracks.
encode_size * input_duration / sample_duration This can be better when the previous method would over-estimate. However, sample sizes tend to be a worse approximation of average size than they are VMAF indicators so this is also not super accurate.

Since the first methods main problem is over-estimating we take the minimum of these two predictions.
A more accurate approach would be to use the input video stream size. input_vstream_size * encode_percent which directly works around the issues with the first approach.
The problem with this is to calculate the input video stream size we need to fully scan the file with ffmpeg. This could be undesirable in cases where the input is on slower storage. It's a relatively expensive operation and only helps the prediction video stream size output which is only shown on crf-search.
On the other hand we may be able to scan the input concurrently with the crf-search, since it isn't a CPU intensive operation. So may be worth it depending on how valuable the predicted size is to the user.

That's an interesting idea, though I suspect it'll end up being an approximation of similar accuracy to the current encoded-sample-size * duration calculation. The sample vstream to audio stream proportion will fluctuate in the same way as sample vstream size (which is the cause of the current inaccuracy). It will also mean writing more samples which is mildly undesirable. We could do better by just including all streams in the samples (or first sample) and doing a scan of them to get the stream sizes.
However, I think I'd prefer doing an async scan of the input which will produce the video stream size. This option will be more accurate and I think simpler.

It is more accurate than the current method because audio streams are not affected by the video streams fluctuation (current inaccuracy).
I’m not saying to use video to audio proportions, which do fluctuate the same way and would maintain the same source inaccuracy, but to use absolute numbers of the audio streams, taking advantage of the constant bitrate nature of most of the audio and subtitle codecs, as shown in the equations I wrote above.
It only requires creating 1 at most 2 more samples with all the streams. Then the absolute size of all streams that are not video can be obtained with a simple substraction of full sample minus video sample.
This way you can completely avoid having to scan and parse anything.
The scanning method is of course the most accurate way without a doubt, but it needs extensive HDD operations in big files that may take more than 10mins to complete, like it did for one of my files. And the higher the size, the longer it would take, like with higher resolution 4k files, uncompressed raw video, and so on.
I honestly think this new approximation could work and proof to be more reliable and accurate, so scanning could be completely avoided.
I thought you preferred to avoid that, so this new method could be worth a try.
This idea is a different kind of approach and should result in a noticeably more accurate approximation. It did in my tests. I can show you an example with real numbers later, but trust me, it makes a difference compared to the current method, they are not similar.

Source file 22,9GB = 18,4GB vstream + 4,5GB other streams (2 audio + 2 subtitle streams)
Real encoded video stream size is 3,86GB:

Whole input size prediction method: crf 18 VMAF 98.30 predicted full encode size 4.90 GiB (21%) taking 2 hours
Encode sample size approx. method: crf 18 VMAF 98.30 predicted video stream size 4.01 GiB (21%) taking 2 hours
Audio stream size approx. method (to reach vstream size by subtraction):
not_vstreams_lossless_sample_size = 62319KB  - 50684KB = 11635KB
not_vstreams_full_size = 11635KB * 7219s / 20s = 4199653,25KB / 1024 = 4101,22MB / 1024 = 4,01GB
approx_vstream_full_size = 22,9GB - 4,01GB = 18,895GB
final_encoded_size_prediction = 18,895GB * 21% = 3,97GB

This case should have been a bit challenging for the new method as both audio streams were variable bitrate (unless I was extremely lucky with the random sample I created), but even so the results were even better than the previous method, which was already pretty good.

Source file 11,4GB = 10,3GB vstream + 1,1GB other streams (2 audio + 6 subtitle streams)
Real encoded video stream size is 2,27GB:

Whole input size prediction method: crf 17 VMAF 97.09 predicted video stream size 2.57 GiB (22%) taking 2 hours
Encode sample size approx. method: crf 17 VMAF 97.09 predicted video stream size 2.51 GiB (22%) taking 2 hours
Audio stream size approx. method (to reach vstream size by subtraction):
not_vstreams_lossless_sample_size = 17879KB  - 15079KB = 2800KB
not_vstreams_full_size = 2800KB * 7560s / 20s = 1058400KB / 1024 = 1033,594MB / 1024 = 1,01GB
approx_vstream_full_size = 11,4GB - 1,01GB = 10,39GB
final_encoded_size_prediction = 10,39GB * 22% = 2,29GB

This would be like the best case scenario, because both audio streams are constant bitrate, and it shows in the result being so accurate.

Source file 14,2GB = 11,2GB vstream + 3GB other streams (4 audio + 8 subtitle streams)
Real encoded video stream size is 5,15GB:

Whole input size prediction method: crf 18-19 VMAF 97.07-97.41 predicted video stream size 5.84-7.28 GiB (41-51%) taking 2 hours
Encode sample size approx. method: crf 18.5 VMAF 97.24 predicted video stream size 6.11 GiB (46%) taking 2 hours
Audio stream size approx. method (to reach vstream size by subtraction):
not_vstreams_lossless_sample_size = 13890KB  - 6000KB = 7890KB
not_vstreams_full_size = 7890KB * 7687s / 20s = 3032521,5KB / 1024 = 2961,45MB / 1024 = 2,89GB
approx_vstream_full_size = 14,2GB - 2,89GB = 11,31GB
final_encoded_size_prediction = 11,31GB * 46% = 5,2GB

Another easy case for the new method, as all the audio streams are CBR, but thanks to that it offered a far better approximation than the current method, that overestimated the encoded size.

Source file 17,9GB = 16,9GB vstream + 1GB other streams (2 audio + 3 subtitle streams)
Real encoded video stream size is 2,37GB:

Whole input size prediction method: crf 17 VMAF 95.46 predicted full encode size 2.59 GiB (14%) taking 2 hours
Encode sample size approx. method: crf 17 VMAF 95.46 predicted video stream size 2.52 GiB (14%) taking 2 hours
Audio stream size approx. method (to reach vstream size by subtraction):
not_vstreams_lossless_sample_size = 37903KB  - 36333KB = 1570KB
not_vstreams_full_size = 1570KB * 9326s / 20s = 732091KB / 1024 = 714,93MB / 1024 = 0,698GB
approx_vstream_full_size = 17,9GB - 0,698GB = 17,2GB
final_encoded_size_prediction = 17,2GB * 14% = 2,41GB

It seems that the smaller the audio stream (which is the most common situation), the more accurate this method becomes, but even in extreme conditions with 12 audio streams the results were pretty good, better than current predictions.
I'm more convinced the more I test it, you may try and see it for yourself.

OK, the last one, I tried to find one of the most complex files I have.
I also simplified the process to take the non video streams sample size with a single ffmpeg command, so it is not necessary to compare against another sample with the video stream to get the size. That allows to get a sample from any random point in the file and could be of any duration, so if necessary the sample duration could be extended to get improved more accurate results.
However, I did all my tests with a single random 20s sample per test and the results were already pretty good, so unless some particular file is exceptionally difficult, like with lots of variable bitrate audio streams, that should be enough.
This is the ffmpeg command (aparently writing the seeking parameters in the beginning helps accelerating the process very significantly): ffmpeg -ss 1100 -t 20 -i ".\00000.m2ts" -map 0:a -map 0:s -c copy "non_vstreams_sample.mkv"
It results in a sample file with all audio+subtitle streams, without video, so the file size of that sample is all we need to start calculating the approximation.
Source file 29,9GB = 22,4GB vstream + 7,5GB other streams (13 audio + 28 subtitle streams)
Real encoded video stream size is 4,26GB:

Whole input size prediction method: crf 17 VMAF 95.32 predicted full encode size 5.78 GiB (19%) taking 2 hours
Encode sample size approx. method: crf 17 VMAF 95.32 predicted video stream size 3.49 GiB (19%) taking 89 minutes
Audio stream size approx. method (to reach vstream size by subtraction):
non_vstreams_sample.mkv = 16897KB
non_vstreams_full_size = 16897KB * 9225s / 20s = 7793741,25KB / 1024 = 7611,08MB / 1024 = 7,433GB
approx_vstream_full_size = 29,9GB - 7,433GB = 22,467GB
final_encoded_size_prediction = 22,467GB * 19% = 4,27GB

I hope you find this method effective, it really works and requires minimal processing.

Thanks! You've convinced me it's worth trying this out. And perhaps i can just include audio & subs in all lossless samples and parse to audio+sub stream sizes from ffmpeg to make the calculation. Videos can have attachments too like fonts, but it's probably something we can look closer at later.
I'll test using this approach when I have some time.


handle toast onTap

There is a problem with handling clicks on toast. I display a toast when receiving an event, and I want to perform some actions when clicking on this toast
I suggest the following solution:
Add VoidCallback onTap in .show() and add onTap: onTap in GestureDetector

Thank you for your feedback. I will look into it and get back to you soon.
On Wed, 26 Jun 2024 at 6:45 PM Lev @.***> wrote:

There is a problem with handling clicks on toast. I display a toast when
receiving an event, and I want to perform some actions when clicking on
this toast
I suggest the following solution:
Add VoidCallback in .show()
—
Reply to this email directly, view it on GitHub
https://github.com/gaurishankar007/flutter_sliding_toast/issues/1, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/ANXQP6Y3BM72Q3SGPR5DR3TZJK3QHAVCNFSM6AAAAABJ53FQDCVHI2DSMVQWIX3LMV43ASLTON2WKOZSGM3TKMRYGA4DOOI
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

The onTap callback is added in the new version.


Properly Handle a case when Text size is more than model handle

When this project was started initial intention was to handle only short text. But now we have added Google News and Crawlers, hence there is need to handle longer text as well.
As we know that most of BERT based model support 512 max tokens (with few exceptions like BigBird). Currently Analyzer ignore (#113) excessive text.
@akar5h Suggested nice ways to split based on tokenizer output instead of char size.

So would it not be more optimal that instead of splitting texts, we split the tokenizer output ( tokenizer.batch_encode_plus output) like input_ids , attention mask into sequences of length 512 and feed these splits to the model ?

We can see if this can be incorporate here as suggested by @akar5h -

https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/question_answering.ipynb#scrollTo=n9qywopnIrJH&uniqifier=1
Scroll down to Preprocessing the training data , . Here we can see how huggingface tokenizer provides a way to split longer length documents as an inbuilt operation .


Notes on Upgrading to Angular 8

I'm submitting a...

[ ] Regression
[x] Bug report
[ ] Feature request
[ ] Documentation issue or request
[ ] Support request => Please do not submit support request here, instead post your question on Stack Overflow.

Current behavior

Everything works with Angular 7, but if you upgrade your cli and then upgrade the angular universal app:
npm install -g @angular/cli@next
ng update @angular/cli @angular/core codelyzer --next

A couple of problems appear:

Angular complains about the lazy loading of modules unless you change the "compilerOptions"->"module":"esNext" in the tsconfig.app.json file
If you try the SSR build, you get the error already reported here - https://github.com/nestjs/ng-universal/issues/64
The SSR build fails, because the ng-universal/webpack-config.factory.js has this section:

test: /\module.ts$/,
                        loader: 'string-replace-loader',
                        options: {
                            search: '../server/main',
                            replace: '../dist/server/main',
                            flags: 'g'
                        }

and apparently in the latest version of string-replace-loader, including any flags in the options turns on regex, and because the periods are in there unescaped, it totally messes the replacement-
Expected behavior

Should work with Angular 8 :)
Minimal reproduction of the problem with instructions

See above
What is the motivation / use case for changing the behavior?

Getting ready for Angular 8!
Environment

Nest version: 6.0.1

For Tooling issues:
- Node version: 10.15.3
- Platform:  Mac

Others:

Thanks for sharing your experiences. Once Angular 8 is officially published, I'll update an example.

@kamilmysliwiec Now that Angular 8 wsa officially released, can we expect this to work? Looking forward to it. Thanks!

I just started a new project with the latest ng cli (v8), would be happy to use universal-nest

*despite the warnings, v1.1.1 seems to work fine with angular 8.x

Repo updated :)


Training Speedup with Sparse Dataloader

This is probably dependent on the particular setup. But for the machine I am working on, and with high-dimensional data, training with your dataloader is much slower than it needs to be. The basic problem is that sending high-dimensional ATAC data between RAM and VRAM takes a lot of time, and this really slows down training. My fix for this is to load the counts from the dataset without making them dense, turning them into a sparse torch tensor, sending it to the device, and only filling the zeros in the VRAM. For my use case (60k peaks, my machine) this gives me a 3x speed boost, while not hurting performance for 3k dimensional RNA data.
If anyone wants to test whether the speedup is specific to my setup, I implemented this in a self-contained file that implements SparseUnsupervisedTrainingMixin as a plug-in replacement for UnsupervisedTrainingMixin (on version 0.20.3, not sure if the new version breaks this). If trained normally nothing changes, if device="cuda:0" or equivalent is passed to model.train() it uses my approach for sparse data and directly sends all tensors to device. The actual implementation is just a few lines, most of the code is duplicated since I wanted something stand-alone (and didn't know how to get the appropriate device from within AnnDataLoader).
I'm happy to help with implementing this in the normal package (I don't know if you have good reasons to keep the dataloader fully on the CPU, for me this approach is faster even for low-dimensional data so I just use it as the default for all sparse data), although I'm not that deep into your code.

This is very interesting, thanks for sharing your code. If I'm understanding this correctly, these changes would load sparse data as follows:
scipy csr -> torch csr -> torch csr on device -> torch dense on device

As opposed to what we have currently:
scipy csr -> numpy dense -> torch dense -> torch dense on device

This makes sense given that sending a dense tensor to VRAM probably takes longer than a sparse one, although I wouldn't have anticipated it would give a 3x speed boost. Do you happen to have your benchmark code and results on hand? I'd be interested to try this on my setup.
Even if the speedup is not as dramatic, this would be a valuable addition to our codebase. Would you be interested in opening a PR with these changes?

Ok I ran a simple benchmark with these changes (~20,000 cells x 27,000 genes) and I'm seeing a ~1.3x improvement per training epoch and specifically a 4.6x improvement during data loading. We can probably add this as a non-default option, though we definitely need to run some benchmarks to ensure that model performance is not significantly affected by this.

Ok I ran a simple benchmark with these changes (~20,000 cells x 27,000 genes) and I'm seeing a ~1.3x improvement per training epoch and specifically a 4.6x improvement during data loading. We can probably add this as a non-default option, though we definitely need to run some benchmarks to ensure that model performance is not significantly affected by this.

The speedup is obviously larger for higher dimensions (60k, 120k etc. peaks), but I expected that it's going to vary with how the GPU is connected etc.

This is very interesting, thanks for sharing your code. If I'm understanding this correctly, these changes would load sparse data as follows:
scipy csr -> torch csr -> torch csr on device -> torch dense on device

Yes, that's the idea.
About the PR, I'm not sure I'm the right person to implement this in a way that fits well with the rest of the code. It doesn't require that many changes, what I did was basically dropping the .toarray() for sparse data in AnnTorchDataset, and changing the collating function used in AnnDataLoader. Instead of the default collate_fn=torch.utils.data.default_convert I use a short extension that deals with sparse data as proposed (default_convert just passes sparse scipy data through without converting it):
def sparse_convert(x, device=None):
    """ Torch default default_convert for non-sparse data.
        Sparse data is turned into torch csr, sent to device, and only then turned to dense.
    """
    if device is None: # If no device is given, use the torch default.
        return torch.utils.data.default_convert(x)
    else:
        if isinstance(x, dict): # If dictionary {"x":..., "y":..., etc.} do separately for each
            return {key: sparse_convert(x[key], device) for key in x}
        else:
            if scipy.sparse.issparse(x):
                # If sparse, go through torch csr, to device, then dense
                # Manually converting indptr etc. to torch is faster for some reason, as_tensor to not copy the data
                tensor_sparse = torch.sparse_csr_tensor(torch.as_tensor(x.indptr), torch.as_tensor(x.indices), torch.as_tensor(x.data), dtype=torch.float32, size=x.shape)
                return tensor_sparse.to(device).to_dense() # first to device, then to dense
            else: # Default is not sparse
                return torch.utils.data.default_convert(x).to(device)

Two issues with that are that I don't know how to properly get the correct device from within AnnDataLoader, and that the current dataloaders don't send data to the GPU at all. So sending it to the GPU already here might break something elsewhere for all I know. So it might be better if someone else who has a better overview over what the rest of the code does adds this in a way that doesn't break anything.

Sounds good - Lightning automatically sends data from the DataLoader to the correct device, so we don't explicitly set that on our end. I can open up a PR and tag you in it if that works for you.


Use --namespace-scope-mode for Quickstart

Motivation
Quickstart should use namespace scope mode for simplicity. In current version there is an inconsistency wrt ServingRuntimes vs ClusterServingRuntimes.
Modifications

docs/install/install-script.md (correct a typo)
docs/quickstart.md (change install command for ns scope mode and add missing torchserv serving runtime)
scripts/install.sh (fix sed -i commands)

Result
Quickstart install steps work like a charm again :-)
TODOs
Once merged into main, we will need to update the release-0.10 branch as well

/cc @njhill
/assign @njhill

/lgtm


📎 Improve useNamingConvention options

Description
It would be nice to support all cases of @typescript-eslint/naming-convention rule, such as the following usage:
    '@typescript-eslint/naming-convention': [
      'error',
      {
        selector: 'typeLike',
        format: ['PascalCase'],
        leadingUnderscore: 'allow',
      },
      {
        selector: 'enumMember',
        format: ['UPPER_CASE'],
      },
    ],

Note that we already have an option for enum member case. I am open to relax some default and adding some options. Could you be more explicit about the need?


Feature Request: Sort markets by pool size

I think this is a great suggestion, and should pretty pretty simple a la https://github.com/manifoldmarkets/manifold/commit/688ebb797a9964fb6523f0ea2a1eaa90dfb9d860
Unfortunately I can't get it to work on my build, I think there are some firestore changes that may need to be added as part of this.

Sorry what I meant was sort by the size of the betting pool, not the size of the liquidity pool. ie how much money goes to the winners. Could be called Most Invested or something. Since selling has a negative amount, I think you can just sum the amount of all the bets. Compared to Most Traded which is the sum of the absolute value of all the bets.
Sort by liquidity pool would be cool too though. Except free response markets don't have a liquidity pool?

oh pool is an object not a number, so it makes sense you can't just sort by it
https://github.com/manifoldmarkets/manifold/blob/dca957708c03430948d20d67b2de321a390d591b/common/contract.ts#L65

added and removed (because with limit orders, pool size is no longer a good measure of liquidity)


[TextField] Enhancement for prefix and suffix on input

You'll notice the Polymer Paper version of input which is Google's interoperation of text input allows for the addition of a prefix and/or suffix on inputs. This would be a nice enhancement.
https://elements.polymer-project.org/elements/paper-input

Screenshot, use case?

This is an example from an example from MD docs:
https://www.google.com/design/spec/components/text-fields.html#text-fields-multi-line-text-field
Notice the $ as part of the input "$10.00" vs it being in front.

This is from the demo examples in paper-input documentation.

we do this internally present.... e.g. we have <FieldInteger postfix="mins" value={this.state.minutes} /> and FieldCurrency ~= <FieldDecimal prefix={this.state.currencySymbol} value={this.state.amount} /> and FieldPercentage ~= <FieldDecimal postfix="%" />

Thanks I'll pass along to our devs.

@bramick : I hope this issue was resolved for you. I am closing this for now. If you still have issues with this, do let us know. Thanks!

@chrismcv Can you elaborate what you mean? I am trying to figure out how to do this and haven't found any examples of it being done

@tintin1343 Why is this closed - is there a way now to add a prefix and/or suffix? Can that be documented here? TextField doesn't support this and there's no such thing as FieldDecimal / FieldInteger as mentioned by @chrismcv

@REBELinBLUE @caesay
Hi there, I have solved this using a similar approach to the one briefly outlined by @chrismcv
Keep in mind that this is more of a proof of concept I've quickly typed out, however it should provide sufficient detail for anyone looking for a solution.

Create a new component 'TextFieldCustom'.
Within the 'TextFIeldCustom' render method, check for a prop called 'prefix'.
If a prefix is passed to 'TextFieldCustom', show it next to the field.

TextFieldCustom
import React, { Component } from 'react';
import TextField from 'material-ui/TextField';

// Component styles.
const styles = {
	// Make sure the field label is on the left.
	label: {
		left: 0
	},
	// Allow absolute positioning within the field's wrapper.
	// We need this to make the prefix show inline with the field.
	fieldWrapper: {
	    position: 'relative'
	},
	prefix: {
            position: 'absolute',
	    left: -12,
	    bottom: 9,
	    fontWeight: 600,
	}
};

export default class TextFieldCustom extends Component {

	// Constructor.
	constructor(props) {
		super(props);

		// Bind 'this' scope to TextFieldCustom, allowing us to refer to it within handleChange.
		this.handleChange = this.handleChange.bind(this);
	}

	// Handle field change event, pass name and value to the onChange callback.
	handleChange(event, index) {

		// Run callback with the name and value.
		this.props.onChange(this.props.name, event.target.value);
	}

	// Render component.
	render() {

		// Optional symbol prefix, example use: currency or percentage.
		const prefix = this.props.prefix || '';

		return (

				{prefix ?
					{prefix}
				: null}

				<TextField
	                                type="text"
	                                value={this.props.value}
	                                onChange={this.handleChange}
	                                floatingLabelText={this.props.label}
	                                floatingLabelFixed={true}
	                                name={this.props.name}
	                                fullWidth={true}
	                                multiLine={this.props.multiLine}
					floatingLabelStyle={styles.label}
					disabled={this.props.disabled || false}
	                        />

		);
	}
}

Extra point
I also like to have a consistent way of receiving change events from different field types (select field, text field, etc.). This is another reason why I have a wrapper around the Material UI field component.
Within other components (such as ContactForm), I have just one method called 'handleChange', which receives 2 arguments (name, value). My custom fields (select, text, etc.) get that 'handleChange' method as a prop. They then process the value from the Material UI components, and call the 'this.props.handleChange' exactly in my desired format (name, value).


Better check for ODX file extension

thanks

(this now works because of #57, right @kayoub5 ?)

(this now works because of #57, right @kayoub5 ?)

Yes


Adding maintainers from upstream

Adding maintainers from upstream. See the comment below.

[x] @jklaise
[x] @ascillitoe

Comment: https://github.com/SeldonIO/alibi/issues/579#issuecomment-1019889351

Checklist

[x] Used a personal fork of the feedstock to propose changes
[x] Bumped the build number (if the version is unchanged) 0 --> 1
[x] Reset the build number to 0 (if the version changed) NA
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@conda-forge-admin, please rerender