chores: editorconfig and dockerfile

I was just exploring the code but wanted to tidy up a bit while I was at it.
EditorConfig
Adds an EditorConfig file, so contributors can use consistent code styles with the rest of the project.
This makes it more convenient to work on the project when a developer's editor settings aren't the same as yours.
Optimizes the Dockerfile
This reduces the number of intermediate layers, and makes the build marginally faster.
Its good practice to minimize layers in a Docker image to make the final image smaller, especially if there's a lot of file system changes between layers. In this case, the difference in size is negligible.

A Docker image consists of read-only layers each of which represents a Dockerfile instruction. The layers are stacked and each one is a delta of the changes from the previous layer.
…
Each instruction creates one layer:
FROM creates a layer from the ubuntu:18.04 Docker image.
COPY adds files from your Docker client’s current directory.
RUN builds your application with make.
CMD specifies what command to run within the container.
— https://docs.docker.com/develop/develop-images/dockerfile_best-practices/

Optimize apt-get

Removes the cached lists from /var/lib/apt/lists. (Reduces image size by 0.02 GB.)
No longer installs recommended packages. (Reduces image size by another 0.02 GB.)

More info:

https://ubuntu.com/blog/we-reduced-our-docker-images-by-60-with-no-install-recommends
https://hackernoon.com/tips-to-reduce-docker-image-sizes-876095da3b34

Before
$ time docker build -t autosub:dev --build-arg model=0.9.3 --no-cache .
real    3m57.379s
user    0m1.650s
sys     0m1.776s

$ docker images -a
REPOSITORY                  TAG               IMAGE ID       CREATED              SIZE
autosub                     dev               2565f52660cb   About a minute ago   1.86GB
<none>                      <none>            4fd064c626ea   About a minute ago   1.86GB
<none>                      <none>            d9d3ae1ed1db   About a minute ago   1.86GB
<none>                      <none>            cf1540452ffc   2 minutes ago        714MB
<none>                      <none>            dfa2fae14669   2 minutes ago        714MB
<none>                      <none>            9ada5732266d   2 minutes ago        714MB
<none>                      <none>            d6821a416a2c   3 minutes ago        416MB
<none>                      <none>            c918464fff64   3 minutes ago        416MB
<none>                      <none>            594b7f697317   5 minutes ago        114MB
<none>                      <none>            e5f83a59a602   5 minutes ago        114MB
python                      3.8-slim-buster   4728acd2148c   2 days ago           114MB

After
$ time docker build -t autosub:dev --build-arg model=0.9.3 --no-cache .
real    3m38.463s
user    0m1.252s
sys     0m1.209s

$ docker images -a
REPOSITORY                  TAG               IMAGE ID       CREATED          SIZE
autosub                     dev               41dbaca6c36b   7 minutes ago    1.82GB
<none>                      <none>            117e3c54c5cf   7 minutes ago    1.82GB
<none>                      <none>            a911683cc89d   10 minutes ago   114MB
<none>                      <none>            7cf695bce4db   10 minutes ago   114MB
<none>                      <none>            264a99e3d9d1   10 minutes ago   114MB
python                      3.8-slim-buster   4728acd2148c   2 days ago       114MB

Hey
Thanks for this!
I didn't know about EditorConfig before. Now I'll make sure to include it in future projects :)

No. I say we should actually remove the dockerfile changes from this commit. The rest is great, but I disagree with the dockerfile. First I have already implemented the removal of apt cache in my own PR and the rest actually conflicts with my own changes. Second, I disagree with the approach on reducing layers. The goal should not be to minimize layers at the expense of readability and also development needs.
For example, in this commit, if anything in the code base changes the dockerfile will now copy the changes and then reinstall everything including things that are not dependent on code changes because we merged it into one line.
As a comparison please take a look at what i did in my own PR. The only thing I grouped into a single layer was the apt stuff because that logically only needs to occur once.  The rest, including the copying of files, is split into individual lines. So now if I change a file, only that and related files will be added because docker will now reuse previous layers that were unchanged. If we group it all into a single line then even if a part of a layer changes it will recompute the entire layer. This is a waste of resources and will actually INCREASE build time for developers which is the only build time that matters because users will only need to built it once whereas devs will build multiple times. Also, docker automatically will recompute any unchanged layers if they occur after a changed layer, thus order matters too and ideally copy commands should as late as possible. This why I copy the large .pbmm and .scorer files first and THEN the code from this project. This way for a minor kb sized change it won't copy the gigabyte length .pbmm file. Please remove the dockerfile from this commit and instead just use the dockerfile that I published.

Basically, there is no point trying to optimize that which Docker is already trying to optimize for us. There is a benefit from having more layers not less. The only time fewer layers can potentially make sense is when you publish the image for others and you want to simplify the amount they have to download and process, in which case there are other tools available that will just "squash" multiple layers into one after the fact. But for a development utility, there is no point trying to marginally minimize layers like that.

Just going to put my rational forward.
I don't have a strong enough opinion to make a dispute out of it, so if you still disagree then power to you.

The goal should not be to minimize layers at the expense of readability and also development needs.

The image isn't just for development needs, it can be used by users.
Not to mention, in my opinion it's still effortless to read. (Unless every .sh file is also considered unreadable.)

if anything in the code base changes the dockerfile will now copy the changes and then reinstall everything including things that are not dependent on code changes because we merged it into one line.

However, you should then consider spacing out your image in development, not the final image that's pushed and going to be used by users in production.

But for a development utility

I don't believe this Dockerfile is just for a "development utility".

There is a benefit from having more layers not less.

Have fewer layers can be helpful, especially for future reference. The final image retains all changes of the layers before. As the build process gets larger, the more unnecessary changes get brought along to the final image.
I'm the one who opened an issue suggesting adding a Dockerfile, with the purpose of production use and publishing.

I understand what you are saying and if one chooses to follow your style of development then your changes certainly make sense. I just disagree with that style. To me end users should just expect to run a docker pull autosub command and get what they want. Users have no business building their own images. In which case, dockerfiles should be made for the developers since we are the ones running the build command every time we want to test a change. So, in my personal opinion Dockerfiles are a development utility exclusively. Docker images are for the user. Also, I don't understand what you are saying in that last part there? Who cares what extraneous things are included in intermediate layers as long as it saves me time during the development build process? And you said yourself the size differences here were marginal, so no problem the way I see it.

Who cares what extraneous things are included in intermediate layers as long as it saves me time during the development build process?

The extra things in intermediate layers impact the size of the final image. (Including what is pushed to Docker Hub.)
In this particular image, the different was marginal, I agree. I'd still consider it a good practice in general as for many images it can save a lot more. Rather than experimenting with "Is it worth it?" before deciding, it's faster to just reduce layers in the first place.
For example, the change here reduces the image from 236 MB to 160 MB:
https://github.com/apache/nifi/pull/5037/files
This reduces the image from 2.91 GB to 1.96 GB:
https://github.com/apache/accumulo-docker/pull/14/files
This reduces the image from 2.37 GB to 1.61 GB:
https://github.com/apache/fluo-docker/pull/18
All of them do nothing but merge RUN commands together.

I like what you were trying to achieve as optimizations should be made where possible and image size is great to reduce, but this type of optimization is just plain wrong ... in my opinion. You are entitled to yours. Take it or leave it. I personally don't like having to download an entire gigabyte file and reinstall entire packages that never changed every time I add a comma to one of the python files. I guess some people do. To each their own.

OK, so as briefly discussed in this comment thread, this pull request was merged before it was ready and is causing some headaches, so it's been reverted and re-applyed.
Basically, it's 3 logically distinct commits squashed into one:

Add .editorconfig file (simple change that nobody has any problems with)
Apply the new formatting rules based on the new .editorconfig
Changes to the Dockerfile

It's the formatting/whitespace/layout changes have caused a huge number of conflicts (and wasted time for me) as I try to merge the several Pull Requests and forks from the past several months.
The Dockerfile changes here are relatively trivial to reconstruct compared to @yash-fn's branch that adds GPU Dockerfile changes.
So what I've done is I've split the commits into the .editorconfig commit (which I've applied already) and the Dockerfile changes.
There is debate on the merits of the changes above, but the key point is the actual changes being done in the Dockerfile can be pretty trivially reconstructed on top of origin/master after @yash-fn has been merged. Also applying the EditorConfig formatting changes across the whole codebase can be done as a commit on top too, if necessary.
I am fully aware that rewriting public git history is deeply frowned upon, but in this case it's the path of least resistance to save time resolving necessary git conflicts.

Okay
So I'll revert 7b294ba and then merge #30 again. That works?


Refactored code & architecture, and added new functionality relating to order information and execution.

Added code to async get account information (required for trade executions) upon session startup.
Changed the structure of the code to be more OO-friendly and cleaner.
Added new models for trading accounts and orders.
Moved the tasty session object into its own model.
Unfortunately this is still not complete but we now have working orders

@krsvojte , would appreciate your input and criticism on this PR. Most important to me is the class architecture and what patterns might be useful for this.


When transaction submitted, no link to explorer

When a user submits a transaction, there is no link to explorer and no progress notification of any kind. The user kind of just waits for it and tries to refresh the page I guess.
Will this be tied in with the notification system in the next build?

Use https://ergexplorer.com/ for all explorer links

Add the link to the pop-up (snackbar).
Keep snackbar open indefinitely until user closes, so that they can use the link.


Do not hide error while rotated tile out of bounds

026c92905e496db1c7a9797d198790f21b9795b5 and c8d771edbca368eed0f55635306b5473d307285d brings functions to rotate tile in Row and Field. But currently they hide the fact that tile (probably) is out of bounds. Instead, they needs to returns Either of Success or Error. One thought is that it will be hard to implement it without removing modifyAt in rotateFieldTileAt.

Maybe return Either Error FRow/Field?


style(ui): Tune prettier to align with the rest of the codebase

Please have a look at the individual commits for details.

It could make sense to configure separation between import groups to align with Kotlin code.

It could make sense to configure separation between import groups to align with Kotlin code.

I don't think the plugin I'm using supports that.

It could make sense to configure separation between import groups to align with Kotlin code.

I don't think the plugin I'm using supports that.

Oh, didn't notice it was different, trivago's seems to be more popular. But your choice supports is as well.

It could make sense to configure separation between import groups to align with Kotlin code.

I don't think the plugin I'm using supports that.

Oh, didn't notice it was different, trivago's seems to be more popular. But your choice supports is as well.

Yes, but then we need to invent an import order instead of using the default one, which seems to work well. I'm open to suggestions as to the definition of the import order.

Yes, but then we need to invent an import order instead of using the default one, which seems to work well. I'm open to suggestions as to the definition of the import order.

A good start would be the default but with lines between. :)


Abstract cdc crate over engines

What problem does this PR solve?
One of the last crates needed to abstract over engines before tikv-server can be started with a non-rocks kv engine.
Part of https://github.com/tikv/tikv/issues/6402
What is changed and how it works?
Just replacing RocksEngine with KvEngine typarams.
Related changes
Check List
Release note
None

Rebased to fix the sign-off of a commit.

Rebased

Rebased.

/merge


Provided definition for cloud

I have written a definition for cloud

hello, please submit an issue with the template instead of a PR https://github.com/learntocloud/cloud-dictionary/issues


Added Emojica to Text.

Project URL
https://github.com/xoudini/emojica
Description
Edited README, added Emojica to the Text section, since the framework handles strings.
Why it should be included to awesome-ios (optional)
Well, there isn't another framework/library for iOS that replaces the standard Apple emoji with custom emoji sets.
Checklist

[x] Only one project/change is in this pull request
[x] Addition in chronological order (bottom of category)
[x] Supports iOS 9 or later
[x] Supports Swift 3
[x] Has a commit from less than 2 years ago
[x] Has a clear README in English

That is a really, really nice readme. 😍

@victorgama Thank you! 😊

Thank you for contributing! 🎉

Cheers! 🎉


Error importing Checkmarx results. Ignored active and verified state

Information
DefectDojo tag v1.15
Api endpoint import-scan
When you try to import the result file from Checkmarx using the API or the Web UI and changing the active state to true and the verify state to True, the findings are always setted as inactive.
Bug description
Problem importing the Checkmarx file report using the "checkmarx scan" and "checkmarx scan detailed"  importer.
The problem is reproduced using the API endpoint (import-scan) and the WEB UI. Findigns state is always "Inactive" when the import process is done successfully setting the Active and Verified flag to True. It seems that these flags are ignored:
curl -X POST "https://defect-dojo.dmain/api/v2/import-scan/" -H  "accept: application/json" -H  "Content-Type: multipart/form-data" -H  "X-CSRFToken: token" -F "scan_date=2021-06-02" -F "minimum_severity=Info" -F "active=true" -F "verified=true" -F "scan_type=Checkmarx Scan" -F "file=@ssdlc-testing.xml;type=text/xml" -F "engagement=51" -F "close_old_findings=false" -F "push_to_jira=false"

Expected behavior
Finding state must change if you set the active or verify flag status to False or True.

the file uploaded is attached
ssdlc-testing.xml.zip

Hello @devGregA , I will try to investigate how to solve the fix because is critical for me and my organisation solve it... Do you have some recommendation to locate the part of code that import Checkmarx results?
thank you,

For some reason these values are hardcoded to be false, I don't know why: https://github.com/valentijnscholten/django-DefectDojo/blob/3957c7d7f4f32dfa2339494e406ac24675864a13/dojo/tools/checkmarx/parser.py#L122. Looks like very old code, I think it should be removed and let the fields go to default value, which will also allow the settings chosen in API request or UI to propagate.

Thank for your answer @valentijnscholten , I have removed the parameters active and verified with the value False hardcoded and It solves the problem for the Checkmarx and Checkmarx Detailed Parser.
I am going to open a Pull Request with these changes. :)

yes please :-)

I can confirm that this issue has been fixed: Scan import for checkmarx is working fine with vers 2.4.1, issues are listed and active and unverified, as specified during the import.

@jcaillon Thanks for letting us know, closing an issue always feels good.


Key Vault's GetSecretAsync live test occasionally fails with 404

Every so often we get a 404 error with the GetSecretAsync live test on different platforms but with the same basic error:
Status: 404 (Not Found)

Content:

{"error":{"code":"SecretNotFound","message":"Secret not found:<PHONE_NUMBER>"}}

This may be a test parallelization issue since there are some tests that sample/test purging, like for samples and README.

Dup of #10496


Convert existing tests to work against postgres database

Copy and Modify the sqlite tests to work with postgres, and create more if needed to increase the code coverage.
Suggestions:

test all crud functions (could discuss if this is needed)
test all routers

Remove the tests folder once finished. The tests folder holds all sqlite tests. Remember to move the server/backend/tests/tests/jwt_utils.py file to the new testspg folder, as that one is reusable
Task:

Write tests
Watch the running time for running all tests
Check if could start the docker container just with postgres and flyway, skip the api service (maybe using Docker capability for compose file with 'extend', not completely unrelated two separate compose files)

Running time for the postgres testing still remains within 2 mins, didn't increase a lot when we added more tests.
The test container doesn't support start the docker container with selected service, like just with postgres and flyway, skip the api service. I created a ticket in the test container repo testcontainers/testcontainers-python#315


Discussion for possible enhancements of the new CUB support

With the great effort by @anaruse in #2090, I've seen encouraging performance boosts. Below is a list of possible improvements I can think of, either for offering extensive support or for enabling even more boost. I am interested in knowing what I've missed or misunderstood.

[x] Support complex numbers (#2538)
[x] Allow using CUDA streams: (#2555)

if users set up a context manager like this
with stream:
    arr.sum()
    # do other stuff

The non-default stream should be honored. All of the CUB functions introduced in #2090 support an optional stream argument. We just need to pick up the current stream pointer during setup and modify the wrappers.

[x] Change the CUB wrappers from def to cdef or cpdef: (UPDATE: see https://github.com/cupy/cupy/issues/2519#issuecomment-538205781)

currently they are all Python def functions. Could be be beneficial for performance. In particular, if we don't want to expose those wrappers to end users, cdef would be a nice choice.

[x] Support batch reduction for contiguous arrays (#2562):

currently only a full reduction is supported, but if a reduction over the last axes of a contiguous array of shape, say, (X, Y, Z),  is needed, this seems possible with a naive loop over the remaining axes. In other words, in this case we can use CUB to do arr.sum(axis=2) or arr.sum(axis=(1,2)), assuming arr is C contiguous. This resembles the current treatment of PlanNd in the FFT module.

[ ] Document how to enable CUB support if built from source: Need to set CUB_PATH and CUB_DISABLED.
-> could be avoided if the CUB source code is bundled (#2584)
[x] Support argmin and argmax (#2596 enables a global (no axis) search)
[x] Support half-precision floating points (#2600)
[x] Support F-contiguous arrays (#2682)
[x] Support sparse matrix operation (#2698)
[x] Honor the keepdims argument (#2725)

Question: (from https://github.com/cupy/cupy/pull/2508#issuecomment-536368493): is Jenkins configured to test CUB functionalities? UPDATE: No, see https://github.com/cupy/cupy/pull/2538#issuecomment-543507886.

Support batch reduction for contiguous arrays: currently only a full reduction is supported, but if a reduction over the last axes of a contiguous array of shape, say, (X, Y, Z),  is needed, this seems possible with a naive loop over the remaining axes. In other words, in this case we can use CUB to do arr.sum(axis=2) or arr.sum(axis=(1,2)), assuming arr is C contiguous. This resembles the current treatment of PlanNd in the FFT module.

Could also be useful? http://nvlabs.github.io/cub/structcub_1_1_device_segmented_reduce.html

Agreed that enhancements to add segmented reductions and/or looping to support batch reduction would be great.
Functions for argmin and argmax reductions are also available in CUB and could be added.
http://nvlabs.github.io/cub/structcub_1_1_device_reduce.html#a6b35963e90120b6d2c76a6068b0340a9

See also #2517 which extends support for the existing 1d reductions to include other valid axis argument and keepdims combinations

Oh cool. You move so fast @grlee77! In #2508 I enabled support of complex numbers. I wonder if we should just combine the two PRs into one. Some conflicts need to be resolved either way.

Oh cool. You move so fast @grlee77! In #2508 I enabled support of complex numbers. I wonder if we should just combine the two PRs into one. Some conflicts need to be resolved either way.

I think it was just coincidence that I happened to be working on linalg.norm earlier this week :). I was initially defining simple ReductionKernels for various norms, but then found that relying on CUB would be much faster in cases where it can be used. I don't mind rebasing after #2508 is merged (or just rebasing on it now if you prefer to go that way).

I think it was just coincidence that I happened to be working on linalg.norm earlier this week :)

I see, I thought you were looking into #2516 already 😆

I was initially defining simple ReductionKernels for various norms, but then found that relying on CUB would be much faster in cases where it can be used.

Interesting. I suppose at some point we could even rewrite the logic in ReductionKernel based on CUB!

I don't mind rebasing after #2508 is merged (or just rebasing on it now if you prefer to go that way).

Well, now that I think about it, I guess the complex numbers can wait after your #2517, which is a more fundamental work.

Change the CUB wrappers from def to cdef or cpdef:

Just tried this, no difference in timing. I wonder why...

Just FYI. You can also speedup reduction operations using cuTENSOR.
https://pspringer.gitlab-master-pages.nvidia.com/cuTensor/sphinx/api/cutensor.html#cutensorreduction
cutensorReduction is implemented in cuTENSOR 0.2 and cuTENSOR 0.2 will be available in CuPy once theis PR is merged (https://github.com/cupy/cupy/pull/2341)

There was also related discussion about using cudnnReduceTensor in #2294. However, it seems cuDNN does not have complex-valued support either.  cuTENSOR does support complex-valued tensors and also seems to support reduction over specific axes of n-d arrays.

@grlee77 I'm preparing a PR for native support of complex numbers in CUB, and I'll abandon #2508 then. I see a very promising performance (of course). Can you help me test it later?

Do you know if cuTENSOR will be bundled with an upcoming CUDA Toolkit release (like cuSPARSE, cuFFT, etc. are now) or will it remain a separate library?

It will be a separate library like cuDNN.

edit:: Just wanted to confirm that I was able to obtain cuTENSOR via the early access link I posted in this comment

Yes, you can download cuTENSOR EA via the link once you join NVIDIA Developer Program.
https://developer.nvidia.com/developer-program/signup

@anaruse, I have a quick question: Since CUB is header-only, are we legally allowed to distribute it along with CuPy's source code (and wheels) on PyPI? Bundling it would be more convenient for users.

By the way, the last update time for CUB was Feb 2018? @anaruse do you know why? Is it because CUB is absorbed into Thrust?

are we legally allowed to distribute it along with CuPy's source code (and wheels) on PyPI?

I'm not a legal person, so I'm not so sure, but it would be OK as long as distribution way conforms to BSD-3-Clause License.
https://github.com/NVlabs/cub/blob/1.8.0/LICENSE.TXT

Is it because CUB is absorbed into Thrust?

CUB is internally used by Thrust (see /usr/local/cuda/include/thrust/system/cuda/detail/cub), but it does not mean that CUB is absorbed into Thrust.

Thanks @anaruse for both answers! Sounds like we could just cherry-pick the needed CUB header files and include them in CuPy (along with the license file of course), very much like how Thrust was included. Since v1.8.0 is quite stable for a while, I suggest to pin at this version.
The next question would be if we wanna enable CUB reductions by default.

Support batch reduction for contiguous arrays:
currently only a full reduction is supported, but if a reduction over the last axes of a contiguous array of shape, say, (X, Y, Z), is needed, this seems possible with a naive loop over the remaining axes. In other words, in this case we can use CUB to do arr.sum(axis=2) or arr.sum(axis=(1,2)), assuming arr is C contiguous. This resembles the current treatment of PlanNd in the FFT module.

I got this working in https://github.com/leofang/cupy/tree/cub_enhance with CUB's device_segmented_reduce. The idea is to do a device-device copy to make reduced axes be contiguous. But, the performance numbers are not uniformly better across all test combinations (tried different axis setup for a ndim=3 array). Looking into this now.
If anyone is interested, please feel free to comment / diagnose / improve before I create a PR. I feel some decision making (like applying this to certain cases instead of making it a generic solution?) has to be done.

I got this working in https://github.com/leofang/cupy/tree/cub_enhance with CUB's device_segmented_reduce. The idea is to do a device-device copy via transpose to make reduced axes be contiguous. But, the performance numbers are not uniformly better across all test combinations (tried different axis setup for a ndim=3 array). Looking into this now. Most likely the copy is slowing things down (to contrast, CuPy's reduction kernel doesn't do any copy).
If anyone is interested, please feel free to comment / diagnose / improve before I create a PR. I feel some decision making (like applying this to certain cases instead of making it a generic solution?) has to be done.

See #2562.

Sounds like we could just cherry-pick the needed CUB header files and include them in CuPy (along with the license file of course), very much like how Thrust was included. Since v1.8.0 is quite stable for a while, I suggest to pin at this version.

Bundling CUB is attempted in #2584.

argmin() and argmax() are supported in #2596.

I implemented cub::DeviceSpmv::CsrMV for CuPy today and it seems to be 3-4 times faster than the cuSPARSE-based implementation in my initial testing. I will make a PR shortly.

Cool, I completely overlooked that section in the CUB documentation! I wonder why doesn't CUB support other kinds of sparse matrix operations...

As documented in https://github.com/cupy/cupy/pull/2725#issuecomment-559879307, we could also use CUB to accelerate mean (which is also used internally by std and var)

I am closing this meta-issue, as most of the listed tasks are completed except for the source-tree bundling (#2584) and the tests (#2598). Thanks to everyone for making all these improvements! It's been a not-so-short, perhaps a bit bumpy journey since @anaruse added the initial support. 🙂


islandora_breadcrumbs seems to not include Home

If the current theme is configured to not always hide the "Home" in breadcrumbs, I would expect that this module would include the site Home link as well.
Is this due to the initialization of $breadcrumbs in the build() function as
  $breadcrumbs = new Breadcrumb();
instead of something like this:
    $breadcrumbs = parent::build($route_match);

Thanks for reporting this @wgilling.

I looked at this a bit more, and I don't think parent::build($route_match) will work because we are implementing an interface, not extending a base class.
All of the examples of custom breadcrumbs I've found use the $breadcrumbs = new Breadcrumb(); convention but also add $breadcrumb->addLink(Link::createFromRoute($this->t('Home'), '<front>')); right after it; which will do what you suggest. Although, I'm inclined to make it configurable so we can turn it off if we want.

I am not entirely sure that you need to have a separate configuration for suppressing the Home in the breadcrumb.  The theme should take care of removing the Home breadcrumb (as in at_core and other theme code does in the preprocess_breadcrumb function based on the $config['breadcrumb_home'] being checked.

I'll run with that. 👍

Resolved via https://github.com/Islandora/islandora/commit/f5d98ab709b90327eae18898cc512dfc5fa9a798


Update info & build a little bit more packages (Fedora, Flatpak)

Some small ideas that can be implemented pretty easily:

[ ] 1. The Fast And Easy Fix: To downloads section, you probably should add link to ArchLinux AUR for SyncPlay: it works perfect on 3 different OSes (Manjaro, Arch and EndeavourOS). This will help a lot of people who uses Arch & Manjaro which is a large audience: https://aur.archlinux.org/packages/syncplay/
[ ] 2. There's no RPM package, sadly. I'd be happy to help and package it, although I don't have too much RPM packaging experience. This is second large audience.
[ ] 3. The package in SNAP (https://snapcraft.io/syncplay) is v.1.6.5 which is outdated. I don't know whom to ping to update it.
[ ] 4. Last but not least, there's Flatpak. It would be nice to do that as well.

So I wonder if Syncplay uses some scripts when the build are updated or not?
I might grasp on one point or another from that exact list.

AUR package is not maintained by any of the core Syncplay developers. This is why it was never featured in the downloads section.
RPM package might be tad difficult because Github Actions only run Ubuntu. That said, a basic package could maybe be created from the existing deb using Alien (deb-rpm converter).
SNAP worked on previous CI pipeline which was using Travis. We tried to make a SNAP package work with Github Actions and recent Syncplay versions, but in general it proved to be way more difficult than it should be. We never managed to produce a package that would work without segmentation faults on substantial fraction of systems we tested it on. As far as we are aware there is just no easy and well supported way of packaging pySide2 applications into SNAPs. All of this has lead to decision to drop SNAP in future.
When we looked at it last time Flatpack has pretty restrictive sandboxing which makes it very difficult if not outright impossible for Syncplay to communicate with media player. Making it work would likely require packaging a media player together with it, which gets very complicated not just for technical reasons but also possibly licensing.

As more of a side note - we have an AppImage which should work almost anywhere. This in one way or another covers most of the needs that would be addressed with RPM/SNAP/Flatpak.
Lastly - the biggest problem in all of this is matter of ownership and maintenance. Most of the packaging methods currently in the CI pipeline are used on regular basis by core Syncplay developers. We would like to avoid situation where a packaging pipeline we barely understand (and don't really test ourselves) is introduced as it also carries expectations of maintaining it in good shape.

Ok, I see.
Yeah, I know alien, this is kinda one-time solution; I was thinking to have a permanent one.
Oh. I've actually just tried to build SNAP, and it seems it worked. I haven't tested it yet, but this is it: https://snapcraft.io/syncplay-latest -> the process of making the package was easy and straightforward, if we'll see it's working fine, I might share it somewhere as an article.
Oh yes. I see. This is sad.

Yes, I am using AppImage on some machines and it works. I was just curious to see the package in different distributions formats because I see it as very finished and polished, and important. That could lead to larger user- and dev- base in the future.
About last point, I understand and totally agree with this position. Maybe we could implement something of the SNAP. Thanks for your fast answers and good luck with development!
Den

@sxiii I've tried your snap, but it seems to error out:
$ snap run syncplay-latest.syncplay
WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement
Traceback (most recent call last):
  File "/snap/syncplay-latest/2/usr/bin/syncplay", line 16, in <module>
    from syncplay import ep_client
ModuleNotFoundError: No module named 'syncplay'

Maybe I'm running it in wrong way - I don't really have any experience with snaps.
Also it seems like I missed your initial question about scripts used in CI pipeline - with one exception, they are all referenced in [https://github.com/Syncplay/syncplay/blob/master/.github/workflows/build.yml](our build workflow for Github Actions). The exception is Windows build which uses AppVeyor for historical reasons.
With regards to rpm, our deb package is also rather basic (i.e. it doesn't fully conform to Debian Policy). So an alien-generated rpm package would probably be just as good. Only concern I'd have is that the dependencies might have different names - especially with regard to python packages.

@daniel-123 me neither have about zero experience with snaps. Maybe I've done something wrong :) I will probably try to fix it a bit later.
Ok I see your point on the RPM thoughts. Thanks :)
Should I close the issue meanwhile?

I guess the issue can be closed with following status:

We haven't managed to create a process which would result in reliable Snap package. In light of recent inclusion of Synpcplay into Debian repositories and working AppImage - Snap is basically redundant and on its way out as far as distribution channels of Syncplay go.
I would be open to officially including RPM only if somebody would be actually using it on daily basis and they step up to committing to maintaining it.
Unless somebody comes forward and convinces us otherwise, Flatpak is not really compatibly with how Syncplay works right now.
We'll think about reworking installation/downloads part of web page. Especially with inclusion of deb package it probably warrants some rearranging.

Thanks @daniel-123 for the clarification. I agree with your points and as so, I'll close the issue. :+1:


[Bug]:  client update extents from meta and drop extents in cache leadto ek conflict

Contact Details
No response
Is there an existing issue for this?

[X] I have searched all the existing issues

Priority
low (Default)
Environment
CubeFS version:3.4.0

Current Behavior
leadto write io return error,  in debug and error log print ek conflict error messages
Expected Behavior
No response
Steps To Reproduce
No response
CubeFS Log
No response
Anything else? (Additional Context)
No response

closed by #3563


Please add a license file

I'd like to add this to awesome-zsh-plugins, but it doesn't have a license file.
Please add a license file - if you don't have one in mind,  I recommend using choosealicense.com to help choose an appropriate one.
Thanks!

@unixorn created license. Thanks


Added autosave, updated title description, version bump

Solved #12

I would advice to either go with the yarn lock if yarn is your package manager, or package-lock.json if npm is your package manager.


size mismatch when loading checkpoint (ssf2020)

Bug
I am trying to load a checkpoint after training the ssf2020 model.
However, when I load the checkpoint, a size mismatch error occurs
To Reproduce
Run the following script (provided you already have a "checkpoint.pth.tar")
net = models_video[args.model](quality=3) #args.model -> default ssf2020
net = net.to(device)

torch.save(net.state_dict(), "checkpoint.pth.tar")
net.load_state_dict(torch.load("checkpoint.pth.tar"))

You should get the following error:
Traceback (most recent call last):
  File "/home/user/videocompression/main.py", line 480, in <module>
    main(sys.argv[1:])
  File "/home/user/videocompression/main.py", line 427, in main
    net.load_state_dict(torch.load("checkpoint.pth.tar"))
  File "/home/user/videocompression/compressai/compressai/models/video/google.py", line 484, in load_state_dict
    super().load_state_dict(state_dict)
  File "/home/user/.conda/envs/compression/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1223, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for ScaleSpaceFlow:
        size mismatch for img_hyperprior.entropy_bottleneck._offset: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([192]).
        size mismatch for img_hyperprior.entropy_bottleneck._quantized_cdf: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([192, 23]).
        size mismatch for img_hyperprior.entropy_bottleneck._cdf_length: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([192]).
        size mismatch for res_hyperprior.entropy_bottleneck._offset: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([192]).
        size mismatch for res_hyperprior.entropy_bottleneck._quantized_cdf: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([192, 23]).
        size mismatch for res_hyperprior.entropy_bottleneck._cdf_length: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([192]).
        size mismatch for motion_hyperprior.entropy_bottleneck._offset: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([192]).
        size mismatch for motion_hyperprior.entropy_bottleneck._quantized_cdf: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([192, 23]).
        size mismatch for motion_hyperprior.entropy_bottleneck._cdf_length: copying a param with shape torch.Size([0]) from checkpoint, the shape in current model is torch.Size([192]).

Expected behavior
The model state loads properly.
Environment
Please copy and paste the output from python3 -m torch.utils.collect_env
Collecting environment information...
PyTorch version: 1.8.0+cu111
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.3 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: 10.0.0-4ubuntu1
CMake version: version 3.16.3

Python version: 3.9 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: NVIDIA A100-PCIE-40GB
GPU 1: NVIDIA A100-PCIE-40GB
  MIG 2g.10gb     Device  0:
GPU 2: NVIDIA A100-PCIE-40GB

Nvidia driver version: 470.57.02
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.2.4

HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.21.2
[pip3] pytorch-msssim==0.2.1
[pip3] torch==1.8.0+cu111
[pip3] torchaudio==0.8.0
[pip3] torchvision==0.9.0+cu111
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               10.2.89              hfd86e86_1
[conda] libblas                   3.9.0            12_linux64_mkl    conda-forge
[conda] libcblas                  3.9.0            12_linux64_mkl    conda-forge
[conda] liblapack                 3.9.0            12_linux64_mkl    conda-forge
[conda] liblapacke                3.9.0            12_linux64_mkl    conda-forge
[conda] mkl                       2021.4.0           h06a4308_640
[conda] mkl-service               2.4.0            py39h7f8727e_0
[conda] mkl_fft                   1.3.1            py39hd3c417c_0
[conda] mkl_random                1.2.2            py39h51133e4_0
[conda] numpy                     1.21.2           py39h20f2e39_0
[conda] numpy-base                1.21.2           py39h79a1101_0
[conda] pytorch-msssim            0.2.1                    pypi_0    pypi
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torch                     1.8.0+cu111              pypi_0    pypi
[conda] torchaudio                0.8.0                    pypi_0    pypi
[conda] torchvision               0.9.0+cu111              pypi_0    pypi

Additional context
I have noticed that if I print the actual value of the parameters in the stack trace, the corresponding tensor in the state_dict is actually empty:
net = models_video[args.model](quality=3)
print(net.state_dict()["img_hyperprior.entropy_bottleneck._offset"]) # **output: -> tensor([], device='cuda:0', dtype=torch.int32)**

update
As expected, by commenting lines 439-482 of compressai.models.video.google.py I get the loading work. However, I guess this is not a proper solution.

There are two ways to handle this.

Run update_model to modify the checkpoint:
python -m compressai.utils.update_model checkpoint.pth.tar

This also freezes the checkpoint, removes some state (e.g. optimizer), and adds a hash to the filename. If that is not desired, the alternative is...

After loading the model, call net.update(force=True):
net = models_video[args.model](quality=3)
net.update(force=True)

Personally, I've created a couple of wrappers that replace tthe
In fact, this is what I use to inject net.update(force=True) into compressai.utils.eval_model.
# personal_code/utils/compressai/update_and_eval_model.py

import compressai.utils.eval_model.__main__ as eval_model_main
import torch
import torch.nn as nn
from compressai.zoo import load_state_dict
from compressai.zoo.image import model_architectures as architectures

def load_checkpoint(arch: str, checkpoint_path: str) -> nn.Module:
    ckpt = torch.load(checkpoint_path)
    state_dict = ckpt["state_dict"] if "state_dict" in ckpt else ckpt
    state_dict = load_state_dict(state_dict)
    model = architectures[arch].from_state_dict(state_dict).eval()
    model.update(force=True)
    return model

def main(argv):
    eval_model_main.__dict__["load_checkpoint"] = load_checkpoint
    eval_model_main.main(argv)

# personal_code/utils/compressai/__main__.py

import importlib
import sys

if __name__ == "__main__":
    _, util_name, *argv = sys.argv
    if util_name == "update_and_eval_model":
        from . import update_and_eval_model

        main = update_and_eval_model.main
    else:
        module = importlib.import_module(
            f"compressai.utils.{util_name}.__main__"
        )
        main = module.main

    main(argv)

Usage:
python -m personal_code.utils.compressai update_and_eval_model checkpoint $DATASET_PATH --verbose -a=bmshj2018-factorized -p checkpoint.pth.tar > results.json

Hi, thanks for the detailed reply!
I have tried to adapt both solutions but I still get the same error.
Method 1
By using python -m compressai.utils.update_model checkpoint.pth.tar I get an error:
Traceback (most recent call last):
  File "/home/user/.conda/envs/compression/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/user/.conda/envs/compression/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/user/videocompression/compressai/compressai/utils/update_model/__main__.py", line 165, in <module>
    main(sys.argv[1:])
  File "/home/user/videocompression/compressai/compressai/utils/update_model/__main__.py", line 136, in main
    net = model_cls.from_state_dict(state_dict)
  File "/home/user/videocompression/compressai/compressai/models/google.py", line 285, in from_state_dict
    N = state_dict["g_a.0.weight"].size(0)
KeyError: 'g_a.0.weight'

Method 2
I have written the following method:
def load_checkpoint(checkpoint_path):
    from compressai.zoo import load_state_dict
    from compressai.zoo import models_video

    ckpt = torch.load(checkpoint_path)
    state_dict = ckpt["state_dict"] if "state_dict" in ckpt else ckpt
    state_dict = load_state_dict(state_dict)

    model = models_video['ssf2020'](quality=3).from_state_dict(state_dict).eval()
    model.update(force=True)
    return model

But I still get the same error of size mismatch.
I am trying to digging into this by myself but I think I am missing something..
maybe you could spot the problem more easily.
Thanks for your help

Thanks for the report.
We've confirmed that this is a bug, so we will fix it in the next official version.
(Note that the current release v1.2.0b2 is the beta version, so new features and weights for the video model could be fragile.)
In the meantime,
you can properly change "load_state_dict" in google.py under the video folder.
Simply you can leave only entropy_bottleneck related register updates codes and "super().load_state_dict(state_dict) at the end.
Thank you.

Thank you for the answer. I temporary fixed it by putting the following lines in the training script (e.g. train_video.py, right after the model initialization:
    net = models_video[args.model](quality=3)
    net = net.to(device)

    # needed for correct checkpoint saving
    net.img_hyperprior.entropy_bottleneck.update()
    net.res_hyperprior.entropy_bottleneck.update()
    net.motion_hyperprior.entropy_bottleneck.update()

This allows to load the checkpoint without modifying the load_state_dict.
Hope it helps

Hi @emmeduz,
One of our recent commits (https://github.com/InterDigitalInc/CompressAI/commit/84bf9203a708dcd09fd1d2fb7dea7c65a48ff94c) hopefully fixes your issue.
So, please check out the master up-to-date.
This ticket is closed.
Thanks for your elaborate information about this ticket.


METRON-499 Check for Metron Jar Fails During Quick-Dev Deployment

METRON-499
When deploying with Quick-Dev, it fails when checking to see if the Metron jars have been built.  When checking the local file system, it attempts to obtain elevated privileges and fails.
 @ ~/Development/incubator-metron/metron-deployment/vagrant/quick-dev-platform
$ vagrant provision
 Running with ansible-tags: ["hdp-deploy", "metron"]
 Running with ansible-skip-tags: ["solr", "yaf"]
==> node1: Running provisioner: ansible...
    node1: Running ansible-playbook...

....

TASK [metron_common : Check for Metron jar path] *******************************
fatal: [node1 -> localhost]: FAILED! => {"changed": false, "failed": true, "module_stderr": "sudo: a password is required\n", "module_stdout": "", "msg": "MODULE FAILURE", "parsed": false}

When checking the local file system for the jar, Ansible attempts to obtain elevated privileges, since most of the tasks defined in the playbooks run with elevated privileges.  Elevated privilege is not needed in this case.
Tested by running on Quick-Dev.

+1 (non-binding)
Thanks for fixing. Tested in quick dev.

Thanks for fixing!

+1 Thanks!


Can't change preferred editor

Following the README's configuration instructions, I opened the mac file to change my preferred editor, but that line is:
export pref_editor="${pref_Editor}"

Where is the pref_Editor defined?

pref_editor is prompted when script mac-cli/tools/install is called,
and then, it's stored inside 'mac' script
Here is the relevant code from mac-cli/tools/install:

    echo "${LIGHTGREEN}Please enter your favourite text editor (by entering the command that opens your favourite editor. i.e: nano):"
    echo "${NC}Press enter without any value to keep default: vi"
    read pref_editor
    if [ ! -z "$pref_editor" -a "$pref_editor" != " " ]; then
        LC_CTYPE=C sed -i'' -e 's#pref_editor="vi"#pref_editor="${pref_Editor}"#g' "$PACKAGE_DIRECTORY/mac-cli/mac"
    fi

Understood, thank you!


OffsetGenerator needs to be run as root

On at least stock Debian 12 and Ubuntu 22/23, the kallsym table only emits zero address values when called as unprivileged user. The easiest way to circumvent this is to run the OffsetGenerator as root. Since this needs to be called only once per kernel version (right?), this is not a huge issue - further versions of the PoC will most likely have a larger list of kernels/distros.
I'm not sure if this should be documented though.

yea you should run offset generator trought root for get offsets and place it in array

So if I know the root password, what's the need for this exploit?

yea you should run offset generator trought root for get offsets and place it in array

So if I know the root password, what's the need for this exploit?

As far as I understand, the offsets aren't individual for each system but for each kernel version. So for a universal exploit, you need to collect them for each kernel before exploitation, then find the correct combination for the system you are trying to exploit.
It's a PoC after all, so it's not supposed to be a fully weaponized one-shot exploit.


2.11.5611 connection issue

Definetly it is an issue with my computer since it works with current settings for others.
I get the error above.
I think in logs this is what I am encountering:
2019-02-18 17:07:51.705 SEVERE: [252] impl.protocol.sip.ProtocolProviderServiceSipImpl.register().443 No address found for ProtocolProviderServiceSipImpl((48226602070 (SIP))
2019-02-18 17:07:51.708 INFO: [252] impl.protocol.sip.SipLogger.logInfo().196 Info from the JAIN-SIP stack: the sip stack timer gov.nist.javax.sip.stack.timers.DefaultSipTimer has been stopped
2019-02-18 17:07:52.709 INFO: [252] impl.protocol.sip.SipLogger.logInfo().196 Info from the JAIN-SIP stack: the sip stack timer gov.nist.javax.sip.stack.timers.DefaultSipTimer has been stopped
not sure about that tho. We use below settings
Registrar : ctmy3s.pl Port 6050
Proxy ctmy.3s.pl Port 6050
Preferred Transport : UDP
DTMF Method : SIP INFO
Please help. I have no clue whats wrong.

ping?:(

can someone please help me or direct me to someone that can help me?

It works for everyone on the same settings except for me.

Is your information above correct you have an error in the registrar line, missing a .

oh my god...it was the dot !!!!!
I totally missed that.....geez man you are a god. it works.

Can you ping that domain from your PC?

Glad it works :)

I soooooo out of the woods. cheers man !!!


Cache Resample kernel inside transform.PitchShift()

🚀 The feature
Cache PitchShift Resample kernel to improve speed of this transform on second usage.
Motivation, pitch
In porting some augmentation code from librosa to torchaudio, I noticed that PitchShift() and Resample() are slower than librosa in CPU.
In the case of transforms.Resample(), this changes on second run as the kernel is cached. But PitchShift does no such caching, calling instead the functional.
As to the first run. I didnt dig into librosa core.resample() to see why under similar conditions on first run resample is faster in one of the directions. Perhaps this is another opportunity, but besides this suggestio.
transforms.PitchShift() calls the resample() functional, rebuilding the kernel every time.
Alt .1 declare class PitchShift(Resample): and hardcode whatever doesn't require the full set of options.
Alt.2 partial code replication in PitchShift, cache kernel buffer, and call   _get_sinc_resample_kernel() and  _apply_sinc_resample_kernel() instead of resample() inside the PitchShift functional code.
Alternatives
No response
Additional context
No response

Hi @xvdp
Thanks for the pitch. This is a good point and make sense. I think Alt.2 is somewhat better from maintaining perspective.
I will put this as help-wanted issue, as I do not have bandwidth to look into this right now. We welcome your contribution as well.

Thanks, I fixed it on my end but I have some unrelated issue on the install from source; So I cannot test it properly in its context, even if with a simple overriding the classes on _transform and functional it works. I presume you want a proper set of tests before doing pull request.


Recurring/not recurring should be a query filter

So that I could only show tasks with or without a recurrence rule set.
Should probably be something like /(does|is) (not )? (recur|repeat)(r?ing)?/

I added "is recurring" and "is not recurring" in &243.
I'd recommend a separate issue for complex logic such as:

Is recurring daily
Is not recurring on Thursdays
etc.


Missing decoration and hint

in otp field I want add the text hint like a dot
in this u didn't provide any hint field or styling to work on
plz provide that It would be grateful
Thankyou.

Hi @A7Aima  you can use otpPinInputCustom with otpPinFieldDecoration you can archive this. Kindly ref below code
NOTE:  You can give any character inside otpPinInputCustom like *,-,$,# and many more
Dart   OtpPinField(                             onSubmit: (text) {},                             otpPinInputCustom: '*',                             otpPinFieldStyle: const OtpPinFieldStyle(),                             onChange: (text) {},                             otpPinFieldDecoration: OtpPinFieldDecoration.defaultPinBoxDecoration,                             otpPinFieldInputType: OtpPinFieldInputType.custom,                           ),


Compilation fails on CLANG + Linux

As the title suggests, compilation fails on clang + linux.
A short description can be found here.

Hi Ismail
Thank you for reporting this problem.
I have tried to reproduce it unsuccessfully using GCC and CLANG, debug and release.
Maybe did it appear in a previous version? I have done many improvements the previous days.
Please I would like to ask you to check it again.
My Linux version is Ubuntu 22.04.3 LTS Jammy.
Best regards
Iñaki

Hi Ismail
I have upgraded CLANG until version 17 and I have reproduced your errors.
They are now fixed, and the changes also work with previous version 14.
My Linux version now is Ubuntu 23.04. Lunar. Kernel version is <IP_ADDRESS>. CLANG version is 17.0.6.
Please update UppHub and check it again.
Thank you!
Best regards
Iñaki

Hi Ismail
I have upgraded CLANG until version 17 and I have reproduced your errors. They are now fixed, and the changes also work with previous version 14. My Linux version now is Ubuntu 23.04. Lunar. Kernel version is <IP_ADDRESS>. CLANG version is 17.0.6. Please update UppHub and check it again. Thank you!
Best regards Iñaki

Hello Iñaki,
I've updated the package via UppHub. CLANG and GCC is at the same version, previous errors are gone but now both compiler fails with the following undeclared identifier error (this is the only one left):
upp/.config/u++/theide/UppHub/Hdf5/Hdf5/hdf5.cpp (334): error: use of undeclared identifier 'CopyRowMajor'
upp/.config/u++/theide/UppHub/Hdf5/Hdf5/hdf5.cpp (473): error: use of undeclared identifier 'CopyRowMajor'

Best regards,
İsmail

Please  disregard the above comment. I forgot to update eigen. :)
Now everthing works just fine.

Perfect, Thank you!


Kakaotalk

Why so many kakaotalk updates in last few days??
https://github.com/lukesampson/scoop-extras/search?q=kakaotalk&type=Commits

Should be fixed now. If not, then their server might return different results (maybe A/B testing)


Fix word box boundaries in rendered PDF

This addresses the dimension of the text boxes in the generated PDFs (essentially a one-off bug) that are slightly too small to accommodate the boundary of the words. Here is an example of some text selected in a sandwiched PDF:

Sandwiched document regenerated after fix:

Please provide testing case for problem and other details (e.g, which pdf viewer you use...)

I tried the patch on test/testing/8087_054.3G.tif. In Firefox the result looks pretty good. With macOS preview it has several problems: For some words the new boxes look better, but many words are no longer separate, so it is for example no longer possible to select a word by double clicking. Some lines even change the height (they are now too high). Therefore I think this needs more changes to be really good.
PDF without patch
PDF with patch

See #2879.

@stweil thanks for submitting an example. I have tested exclusively under Linux, and only seen improved results with the following PDF viewers:

evince version 3.34.2 (poppler)
chromium version 85 (pdfium)

I just got hold of a windows machine to test with Adobe DC Reader DC 2020.012.20048 (English) with improved results as well (the words are completely covered on selection). No problems selecting words by double-clicking during any of my tests. The height of the selected text stays the same from what I can see by visual inspection. I find it strange that the macOS viewer shows a different height of the text box, as this code change only tweaks the horizontal stretch. Do you see this behavior in all readers on macOS? (I don't have a mac myself, so I can't check).

(I made a Python port of pdfrenderer.cpp that reads hOCR files rather than Tesseract results, but am seeing the same problem. Code: https://git.archive.org/merlijn/archive-pdf-tools/-/blob/master/pdfrenderer.py)
This image shows the 'original' code, with the pdf_word_len++ increase in place. The center PDF is generated with pdf_word_len++ removed. The right PDF is just provided for reference, since the bounding boxes make much of the PDF not visible.
In the both left and center PDF, I pressed Ctrl+A in evince to 'highlight' all text. The reason that you actually see glyphs is because I inserted a second font that references a system font to show glyphs, which doesn't break text copy / paste. (I plan to submit a PR for this later this month)

While the new code does seem to make some bounding boxes better, it also looks like the titles overlap more (but that might be an artifact from my port, or just from the hOCR file(s))

MerlijnWajer,
First, you pasted an incorrect link. Here is the script location:
https://git.archive.org/merlijn/archive-pdf-tools/-/blob/show-text-on-selection/internetarchivepdf/pdfrenderer.py
I don't see a license informantion in your repo.

I inserted a second font that references a system font to show glyphs, which doesn't break text copy / paste. (I plan to submit a PR for this later this month)

See this comment: https://github.com/tesseract-ocr/tesseract/issues/2879#issuecomment-583897489
You should try to test with this patch, which was provided by the person that wrote the pdf output code in Tesseract.

@MerlijnWajer,
First, you pasted an incorrect link. Here is the script location:
https://git.archive.org/merlijn/archive-pdf-tools/-/blob/show-text-on-selection/internetarchivepdf/pdfrenderer.py
I don't see a license informantion in your repo.

Don't want to derail, but just wanted to say that the license will be the same as the pdfrenderer.cpp in Tesseract, of course. There is a comment in the file that already hints at that. I'm planning to finish that this week - sorry for the noise.

I inserted a second font that references a system font to show glyphs, which doesn't break text copy / paste. (I plan to submit a PR for this later this month)

See this comment: #2879 (comment)

I will open a separate issue for this later, not to derail discussion here.

You should try to test with this patch, which was provided by the person that wrote the pdf output code in Tesseract.

Ok, thanks, I will test that this week and let you know if it improves the situation.

I think @jbreiden's patch should be used instead of this PR.
https://github.com/tesseract-ocr/tesseract/issues/2879#issuecomment-583896719

Actually, this patch has a good impact on Firefox's pdf.js, unlike the patch from https://github.com/tesseract-ocr/tesseract/issues/2879#issuecomment-583896719.
Both have a bad impact on macOS's Preview.

I ran into this the other day: https://github.com/nickjwhite/gofpdf/commit/3e3f1fbc561c4ae28e33d92959d58f2aa74428c8
mentioned in https://blog.rescribe.xyz/posts/pdfs/ - maybe it has something useful there.


Unable to copy Learn Modules between Canvas courses

If this is about an issue in the guides
Describe the issue
My college uses master course shells for our courses so that online courses use the same content for every section. I have created a shell in Canvas that uses MS Learn LTI integrated content. When creating an assignment with the required modules in the Learn LTI I expect that the content would be made available in the Canvas shell is exported/copied to another course, but this is not happening. I worked briefly with Lee Stott via Teams and tried what was suggested, but this did not work.
Expected behavior
I expect to create an assignment using the Learn LTI, add the required modules, and then be able to use the same assignment in another course. I did not "publish" the assignment at the suggestion of Lee, but this did not help. The only difference between the Canvas shells is that the master shell has several participants and the copied shell only has me as instructor/developer.
Screenshot of the assignment in the master course:

Screenshot of the assignment in the copied course:

@joeperry8080 thanks let us look into this if you could email<EMAIL_ADDRESS>with your details we will schedule a call.

@Joeperry8080 so at present this is NOT supported as the tool uses unique references to specific courses held within the LMS.
We will look at this feature as an enhancement. If you could share with us a use case to why you like to have templated courses which use the Learn LTi?
One of the reasons we thought this would NOT be  used was ensuring the most relevant module in Learn were presented to the learner  i.e. New or different available modules which would be better for the course which were NOT previously available when the initial/first course was run.

@Joeparry8080
We will investigate this further and add this as a feature request https://github.com/microsoft/Learn-LTI/issues/147


Gitter integration

I think it might be worth adding Gitter integration for this project.

If this becomes a popular enough project for collab, I'll move it to whiteboards and make a room

Still need the integration link on the readme.


{unknown} type information for url::Url::parse

With version 2.1.1 of the url crate and the following code:
fn main() {
    use url::Url;
    let issue_list_url =
        Url::parse("https://github.com/rust-lang/rust/issues?labels=E-easy&state=open");
}

when I hover over Url::parse I get {unknown}.

Does it work if you move the use statement outside the function?

@flodiebold Yes. If I move the use to the top level I get the expected type information and documentation.

Then this is a duplicate of #1165.


VS Code creating a new file with same name just makes lowercase/uppercase first letter

WARNING in ./src/views/sendmail/SendMailView.jsx
There is another module with an equal name when case is ignored.
This can lead to unexpected behavior when compiling on a filesystem with other case-semantic.
Rename module if multiple modules are expected or use equal casing if one module is expected.

WARNING in ./src/views/sendmail/sendMailView.jsx
There is another module with an equal name when case is ignored.
This can lead to unexpected behavior when compiling on a filesystem with other case-semantic.
Rename module if multiple modules are expected or use equal casing if one module is expected.

VSCode Version: 1.9.1
OS Version: MacOS Sierra 10.12.3

Steps to Reproduce:

While editing file

What is the issue? Is there an error?


raspberry compatibilty with an old version

I have one question: raspberry pi 1B is compatibility to scrcpy ? or i have to buy a youngest version to use it
p.s.: i use a raspbian SO
p.s.: my inglish is not good :p

ok now a change to raspbian bullseye and have the same problens ... ;-; ... just to install all things that scrcpy doens´t install to me and i have to manually install ...

ok now a change to raspbian bullseye and have the same problens ... ;-; ... just to install all things that scrcpy doens´t install to me and i have to manually install ... with another version thats not exist p.s: the dependencias thas doesn´t install is now meson and the same dependencies

"Solved my problem": all raspberry are compatible
"New issue" (for me ;-;) and working on issue #5475: Currently the problem being the wrong system time?


implement aggregate transformer

At present the there is only a simplest aggregate transformer in echarts-simple-transform/aggregate. We should better implement a built-in aggregate transformer in echarts.
About the API
Think of whether the API of echarts-simple-transform/aggregate appropriate.
About group by
In echarts-simple-transform/aggregate only on dimension can be set in group by. Do we need to support multiple dimension group by?
About Methods
Some methods of aggregate already roughly implemented in echarts-simple-transform/aggregate:

sum
count
first
average
Q1
Q2/median
Q3
min
max

We need to make sure they are implemented correctly.
Other methods we may implement:

distinct: The count of distinct dimension values.
variance: The sample variance of dimension values.
variancep: The population variance of dimension values.
stdev: The sample standard deviation of dimension values.
stdevp: The population standard deviation of dimension values.
stderr: The standard error of dimension values.
product: The product of dimension values.
valid: The count of dimension values that are not null, undefined, '', NaN, '-', or determined by user defined validation function.
... anything else useful?

Some references: vega aggregate.
About math calculation precision
We all known that the floating precision issue (like 0.1 + 0.2). It might bring trouble in some scenarios like:

The math result is needed to be displayed in label.
The sum should equal to 100%.
Multiple by 100, 1000, ...

Some cases:
0.1 + 0.2 === 0.30000000000000004
2.3 - 0.3 === 1.9999999999999998
33643.2 - 15918.3 === 17724.899999999998
146.39 - 62.83 === 83.55999999999999
1.09 - 0.13 === 0.9600000000000001
0.1 * 0.2 === 0.020000000000000004
16.08 * 100 === 1607.9999999999998
146.39 * 100 === 14638.999999999998
9999999999999999 ===<PHONE_NUMBER>0000000
-11.699999999999986 === -11.699999999999987

We should also take care the toFixed bug like:
1.005.toFixed(2) === '1.00' // rather than '1.01'
0.1.toFixed(20) === '0.10000000000000000555'

Solution:

A. Do not care about it until some users required.
B. Try to resolve it:

For arithmetic: use addSafe we already have and implement multipleSafe in appropriate place.
For comparison of two numbers, use:

const EPSILON = (function () { // Solve compatibility issues
    return Number.EPSILON ? Number.EPSILON : Math.pow(2, -52);
})();
function compare(a, b){
    return Math.abs(a - b) < EPSILON;
}

See: IEEE 754, http://0.30000000000000004.com/, https://github.com/camsong/blog/issues/9, https://javascript.info/number#imprecise-calculations, ... (lots of related references)
Some library:

https://github.com/nefe/number-precision
mathjs(math.add(math.bignumber(0.1),math.bignumber(0.2)))

decimal.jsx = new Decimal(0.1)
y = 0.2
console.log(x.plus(y).toString()

bignumber.js
big.js

About big number
Do not support.
Test
Could find some test cases in some other math libraries.

Would be great to have this feature 💯

Seems like a pretty useful feature

This feature seems really helpful. 👍

Yes, this feature is very helpful. Would be great to have this feature.

It is a very great feature. It would be very helpful to have this feature.

Any update on merging this?

This feature that i really want it. cannot wait


Dejavu ES URL https

I've installed Dejavu and can connect to my test elastic cluster when the elastic URL is http unsecured.  However when I enable https on the elastic cluster Dejavu cannot connect via the https elastic URL.  The response shows that is having ERR_CERT_AUTHORITY_INVALID.  Is there a way to bypass the cert check on Dejavu?

@neilc2016 This isn't a Dejavu specific issue per se. Browsers would throw an error when connecting to a self-signed https cert. Try connecting outside of dejavu to your ES cluster, do you still see the error or not?
Perhaps, this can help: https://superuser.com/questions/27268/how-do-i-disable-the-warning-chrome-gives-if-a-security-certificate-is-not-trust.

Thanks Siddharth.  I exported the elastic https URL cert and imported the cert back in Chrome.  Now when using Dejavu it can connect to the elastic https URL without the error.   All our logstash/Kibana/API services have disabled cert verification so we don't have to deal with this issue but will need to update cert for Dejavu.


hi，How does the SukiWindow communicate with DemoPageBase?

How does the SukiWindow trigger events and get properties in DemoPageBase?
How does DemoPageBase trigger events and get properties in the SukiWindow?


Sigh support for personal teams

New Issue Checklist

[x] Updated fastlane to the latest version (v1.102.0 at the time of writing)
[x] I have read the Contribution Guidelines

Issue Description
When running sigh with an account that isn't in any teams, we get presented with:
No teams available on the Developer Portal
You must accept an invitation to a team for it to be available
To learn more about teams and how to use them visit https://developer.apple.com/library/ios/documentation/IDEs/Conceptual/AppDistributionGuide/ManagingYourTeam/ManagingYourTeam.html

Looking for related GitHub issues on fastlane/fastlane...

[...]

and 45 more at: https://github.com/fastlane/fastlane/search?q=Your%20account%20is%20in%20no%20teams&type=Issues&utf8=✓
/Library/Ruby/Gems/2.0.0/gems/spaceship-0.32.1/lib/spaceship/portal/ui/select_team.rb:41:in `select_team': [!] Your account is in no teams (RuntimeError)
    from /Library/Ruby/Gems/2.0.0/gems/spaceship-0.32.1/lib/spaceship/portal/portal_client.rb:55:in `select_team'
    from /Library/Ruby/Gems/2.0.0/gems/spaceship-0.32.1/lib/spaceship/portal/spaceship.rb:37:in `select_team'
    from /Library/Ruby/Gems/2.0.0/gems/spaceship-0.32.1/lib/spaceship/portal/spaceship.rb:81:in `select_team'
    from /Library/Ruby/Gems/2.0.0/gems/sigh-1.10.1/lib/sigh/runner.rb:16:in `run'
    from /Library/Ruby/Gems/2.0.0/gems/sigh-1.10.1/lib/sigh/manager.rb:7:in `start'
    from /Library/Ruby/Gems/2.0.0/gems/sigh-1.10.1/lib/sigh/commands_generator.rb:34:in `block (2 levels) in run'
    from /Library/Ruby/Gems/2.0.0/gems/commander-4.4.0/lib/commander/command.rb:178:in `call'
    from /Library/Ruby/Gems/2.0.0/gems/commander-4.4.0/lib/commander/command.rb:178:in `call'
    from /Library/Ruby/Gems/2.0.0/gems/commander-4.4.0/lib/commander/command.rb:153:in `run'
    from /Library/Ruby/Gems/2.0.0/gems/commander-4.4.0/lib/commander/runner.rb:444:in `run_active_command'
    from /Library/Ruby/Gems/2.0.0/gems/fastlane_core-0.50.3/lib/fastlane_core/ui/fastlane_runner.rb:36:in `run!'
    from /Library/Ruby/Gems/2.0.0/gems/commander-4.4.0/lib/commander/delegates.rb:15:in `run!'
    from /Library/Ruby/Gems/2.0.0/gems/sigh-1.10.1/lib/sigh/commands_generator.rb:98:in `run'
    from /Library/Ruby/Gems/2.0.0/gems/sigh-1.10.1/lib/sigh/commands_generator.rb:11:in `start'
    from /Library/Ruby/Gems/2.0.0/gems/sigh-1.10.1/bin/sigh:6:in `<top (required)>'
    from /usr/local/bin/sigh:23:in `load'
    from /usr/local/bin/sigh:23:in `<main>'

As per issue #2003 and PR #5636, this seems to be by design.
However, a comment on issue #2003 states:

Sigh could support free provisioning aka the personal team if there are no other teams as Xcode does. Looking in DVTFoundation.framework, Xcode uses the same API protocol i.e. QH65B2. Just a head's up that this functionality exists even if not exposed by the web UI.

The [feature] tag was added thereafter, but the issue was still closed in accordance with PR #5713.
Feature Request
I would like for sigh to offer a way to use the "personal team" (aka. free provisioning) of an Apple ID.

Still the same for sigh 1.11.2.

@Siguza Thanks for letting us know! We'll leave this open to track interest!

+1

still bombs, updated to the latest gems
➡  Sigh support for personal teams
https://github.com/fastlane/fastlane/issues/6022 [open] 5 💬
a week ago
➡  Crash: When no teams exist in your AppleID account
https://github.com/fastlane/fastlane/issues/2003 [closed] 12 💬
13 Sep 2016
➡  Error in fastlane init. Your account is in no teams
https://github.com/fastlane/fastlane/issues/5115 [closed] 3 💬
12 Aug 2016

Why do you want to use sigh for personal teams? It's usually good enough to use Xcode's Fix Issue button if you just want a development profile 👍

My use case is an automated script that involves signing and installing already archived ipas on a device.

I'm trying to do provisioning from the command line and from Makefiles.
My projects are based entirely on Makefiles rather than xcodeproj's (and I'm avoiding the XCode GUI altogether), and I've been able to do every step of development this way, except for provisioning.
So far I've been using "Cydia Impactor" to install IPAs along with provisioning profile on my devices, but a) that's another GUI and b) it asks for email and password every time, overall consuming quite a lot of time for something that I feel could be fully automated.


Tracking: Introduce Sqlancer

https://github.com/sqlancer/sqlancer has already supported Postgres.
We can run the test against batch queries first.
Running it for each PR seems unnecessary, maybe set it up as a daily job? Probably also allow people to manually trigger it.
Running it for stream queries? Require some extra effort by extending the existing Postgres module of Sqlancer.
Prerequisite:

[x] #3294
[x] #3406  (Haven't fully support, but enough to run Sqlancer)
[x] #3421
[ ] #3431
[ ] Support pg_collation
[ ] Support pg_operator
[ ] Support pg_am

References:
See: https://github.com/sqlancer/sqlancer/blob/78506a83a4b77d2c9536980a4d3b4d47eb7e9317/src/sqlancer/postgres/PostgresGlobalState.java#L44-L94

+1 for running daily job against batch query.
For streaming query, does it support materialized view?

For streaming query, does it support materialized view?

Yes(at least in code), https://github.com/sqlancer/sqlancer/blob/master/src/sqlancer/postgres/gen/PostgresViewGenerator.java
When I use 8 threads for testing PG with Sqlancer, the throughput is around 6000 queries per second.
Suspect that if we wait for the mv's flush synchronously by blocking, the throughput won't be this high.

cc @sumittal

Let me know if you have any questions or need input from my side about integrating SQLancer!

Suspect that if we wait for the mv's flush synchronously by blocking, the throughput won't be this high.

You may adjust checkpoint interval to a relatively small value (or just using the mock frontend / compute node as we've done in most unit tests). This might lead to better throughput.

Suspect that if we wait for the mv's flush synchronously by blocking, the throughput won't be this high.

Absolutely, we're even not done with parallel ddl! https://github.com/singularity-data/risingwave/issues/2692

Few more system tables needed:

pg_collation
pg_operator
pg_am

See: https://github.com/sqlancer/sqlancer/blob/78506a83a4b77d2c9536980a4d3b4d47eb7e9317/src/sqlancer/postgres/PostgresGlobalState.java#L44-L94
Updated the description.

Few more system tables needed:

pg_collation
pg_operator
pg_am

See: https://github.com/sqlancer/sqlancer/blob/78506a83a4b77d2c9536980a4d3b4d47eb7e9317/src/sqlancer/postgres/PostgresGlobalState.java#L44-L94
Updated the description.

Thanks for the info!
We can also hard-code these PG system tables just enough to support SQLancer.
We try to spend as little effort as possible on the compatibility of these PG system tables.

I think none of these three system tables is essential for testing. As an alternative, you could also consider adding a flag to the SQLancer Postgres implementation that specifies whether these system tables are supported (see https://github.com/sqlancer/sqlancer/blob/master/src/sqlancer/postgres/PostgresOptions.java).

Have continued work on this, you may follow the progress here:

Compatibility work on RW: https://github.com/risingwavelabs/risingwave/tree/kwannoel/sqlancer
Compatibility work on Sqlancer:  https://github.com/kwannoel/sqlancer/tree/rw

Plan:

(WIP in first two branches above) First try to get minimal working with NoREC since it seems simplest to support. This is just investigative, may hardcode some things in sqlancer:postgres, just to get it running first.
Once we know limitations etc... we can include a variant e.g. https://github.com/sqlancer/sqlancer/blob/master/src/sqlancer/citus/CitusProvider.java since there are differences in SQL features.

Many thanks to @mrigger for providing his thoughts and suggestions on these.


[bug] tauri info does not correctly identify @tauri-apps/api version

Describe the bug
When using Yarn Berry (3.2, PnP mode), the Tauri CLI does not correctly identify the installed version of @tauri-apps/api.
Reproduction

yarn init -2
yarn add -D @tauri-apps/cli
yarn add @tauri-apps/api
yarn install
yarn tauri info

Expected behavior
yarn tauri info currently returns @tauri-apps/api [NPM]: Not installed! when listing packages.
yarn tauri info should return the correct version of the installed package (e.g. @tauri-apps/api [NPM]: 1.0.0-rc.5) when listing packages.
Platform and versions
vscode ➜ /workspaces/manager-desktop (master ✗) $ yarn tauri info

Environment
  › OS: Debian 10 X64
  › Node.js: 16.15.0
  › npm: 8.10.0
  › pnpm: Not installed!
  › yarn: 3.2.0
  › rustup: 1.24.3
  › rustc: 1.60.0
  › cargo: 1.60.0
  › Rust toolchain: 1.60.0-x86_64-unknown-linux-gnu

Packages
  › @tauri-apps/cli [NPM]: 1.0.0-rc.10
  › @tauri-apps/api [NPM]: Not installed!
  › tauri [RUST]: 1.0.0-rc.9 (no lockfile),
  › tauri-build [RUST]: no manifest (no lockfile),
  › tao [RUST]: no manifest (no lockfile),
  › wry [RUST]: no manifest (no lockfile),

App
  › build-type: bundle
  › CSP: unset
  › distDir: ../src-ts
  › devPath: localhost:9009

App directory structure
  ├─ .devcontainer
  ├─ .git
  ├─ .vscode
  ├─ .yarn
  └─ src-tauri

Stack trace
No response
Additional context
Likely not required, but for exact reproduction you can use the following VSCode devcontainer.
Dockerfile
# See here for image contents: https://github.com/microsoft/vscode-dev-containers/tree/v0.234.0/containers/rust/.devcontainer/base.Dockerfile

# [Choice] Debian OS version (use bullseye on local arm64/Apple Silicon): buster, bullseye
ARG VARIANT="buster"
FROM mcr.microsoft.com/vscode/devcontainers/rust:0-${VARIANT}

# Tauri system dependencies
# See: https://tauri.studio/v1/guides/getting-started/setup/linux
ARG TAURI_SYSTEM_DEPENDENCIES="libwebkit2gtk-4.0-dev build-essential curl wget libssl-dev libgtk-3-dev libappindicator3-dev librsvg2-dev"

# [Optional] Uncomment this section to install additional packages.
RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
    && apt-get -y install --no-install-recommends $TAURI_SYSTEM_DEPENDENCIES

devcontainer.json
// For format details, see https://aka.ms/devcontainer.json. For config options, see the README at:
// https://github.com/microsoft/vscode-dev-containers/tree/v0.234.0/containers/rust
{
	"name": "Rust",
	"build": {
		"dockerfile": "Dockerfile",
		"args": {
			// Use the VARIANT arg to pick a Debian OS version: buster, bullseye
			// Use bullseye when on local on arm64/Apple Silicon.
			"VARIANT": "buster"
		}
	},
	"runArgs": [
		"--cap-add=SYS_PTRACE",
		"--security-opt",
		"seccomp=unconfined"
	],

	// Set *default* container specific settings.json values on container create.
	"settings": {
		"lldb.executable": "/usr/bin/lldb",
		// VS Code don't watch files under ./target
		"files.watcherExclude": {
			"**/target/**": true
		},
		"rust-analyzer.checkOnSave.command": "clippy"
	},

	// Add the IDs of extensions you want installed when the container is created.
	"extensions": [
		"vadimcn.vscode-lldb",
		"mutantdino.resourcemonitor",
		"matklad.rust-analyzer",
		"tamasfe.even-better-toml",
		"serayuzgur.crates"
	],

	// Use 'forwardPorts' to make a list of ports inside the container available locally.
	// "forwardPorts": [],

	// Use 'postCreateCommand' to run commands after the container is created.
	// "postCreateCommand": "rustc --version",

	// Comment out to connect as root instead. More info: https://aka.ms/vscode-remote/containers/non-root.
	"remoteUser": "vscode",
	"features": {
		"node": "lts"
	}
}

Ayyy, another yarn berry specific issue 🥳
We're invoking yarn list to read the packages and that command doesn't exist in yarn2+. In yarn2+ it's yarn info afaik

Hey @FabianLars! Happy to contribute--if you have a central tracking ticket for this I can take a stab at this (seemingly small) bug fix.
Cheers!

I suspect the fix may involve some simple detection of the Yarn version, so hopefully not a huge deal.

We always appreciate PRs!

if you have a central tracking ticket for this

Yeah, it's this gh issue here :P (Unless i misunderstood your question? But yeah, we're not tracking it elsewhere.)
For a start, this is the function that reads the version: https://github.com/tauri-apps/tauri/blob/dev/tooling/cli/src/info.rs#L193-L230


Always use AzCopy to get total resources to upload

Fixes https://github.com/microsoft/vscode-azurestorage/issues/873

Isn't this the code that was reporting the progress? https://github.com/microsoft/vscode-azurestorage/blob/main/src/commands/azCopy/azCopyTransfer.ts#L105-L109

This has been confusing me for a bit lol. So that check was originally to make sure that no progress was shown until we were sure that totalWork existed. But you're right that it clobbers whatever totalWork may have been passed when creating the TransferProgress. So I can totally remove totalWork as a class member

Isn't this the code that was reporting the progress? https://github.com/microsoft/vscode-azurestorage/blob/main/src/commands/azCopy/azCopyTransfer.ts#L105-L109

This has been confusing me for a bit lol. So that check was originally to make sure that no progress was shown until we were sure that totalWork existed. But you're right that it clobbers whatever totalWork may have been passed when creating the TransferProgress. So I can totally remove totalWork as a class member

Let me know if you want me to break this up into multiple PRs - that latest commit introduced a lot of changes

Let me know if you want me to break this up into multiple PRs - that latest commit introduced a lot of changes


Add delightfuldot application

Project Abstract
delightfuldot is a lightweight alternative solution for @polkadot/api that aims to address its high memory consumption issue in helping dapps to connect to a large number of Substrate-based networks effectively and efficiently.
Grant level

[ ] Level 1:  Up to $10,000, 2 approvals
[x] Level 2:  Up to $30,000, 3 approvals
[ ] Level 3:  Unlimited, 5 approvals (for >$100k: Web3 Foundation Council approval)

Application Checklist

[x] The application template has been copied and aptly renamed (project_name.md).
[x] I have read the application guidelines.
[x] Payment details have been provided (bank details via email or BTC, Ethereum (USDC/DAI) or Polkadot/Kusama (USDT) address in the application).
[x] The software delivered for this grant will be released under an open-source license specified in the application.
[x] The initial PR contains only one commit (squash and force-push if needed).
[x] The grant will only be announced once the first milestone has been accepted (see the announcement guidelines).
[ ] I prefer the discussion of this application to take place in a private Element/Matrix channel. My username is: @____:matrix.org (change the homeserver if you use a different one)

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Thanks for the application @sinzii very interesting. I think it would be nice to have an alternative to the Polkadot.js API.

Can you check out this other wrapper project sub-API by KodaDot and briefly compare to it?
I'm not seeing that a lot of people have adopted the Coong Wallet so far; do you plan on continuing to maintain that as well?
Considering the above how do you plan to market the API to get devs to switch to it?

Thanks!

@keeganquigley thank you for taking the time to review the proposal and for the questions

Can you check out this other wrapper project sub-API by KodaDot and briefly compare to it?

As far as from what I've taken a look, sub-API is a helper to help manage connections to multiple Substrate-based networks, this is done via a mapping between a network endpoint/prefix and a @polkadot/api ApiPromise instance. This is helpful when you need to manage a lot of ApiPromise instances since you can easily set them up and quickly/lazily get an instance by providing the network endpoint or prefix.
Most of the dapps that need to connect to multiple networks would also need this API management, for example, SubWallet is also using a similar approach to manage their connections to networks.
But under the hood, they are still using @polkadot/api as a mean of connection so they are still being impacted by the aforementioned high memory consumption & slow startup time issues.
delightfuldot is a replacement for @polkadot/api, where we offer similar functionalities but without those issues and dapps can choose to connect to networks via @polkadot/api or delightfuldot. We could probably redefine again the mapping as below:
type ApiType = ApiPromise | DelightfulApi;
const connectionMap: Map<string, ApiType> = new Map<string, ApiType>()

I'm not seeing that a lot of people have adopted the Coong Wallet so far; do you plan on continuing to maintain that as well?

This is understandable and to what we expected since we haven't made any marketing effort for Coong Wallet. Coong Wallet is now still in its early phase of development/incubation, we plan to add more & more useful features for Coong Wallet before we introduce it to the mass, and delightfuldot is a stepping stone for those.
As a website-based wallet, users tend to open multiple tabs of Coong Wallet and we don't want to see each tab consume hundreds of MB or even GB of RAM on users' devices because of the high-memory consumption issue hence the need for a better & lightweight tool such as delightfuldot for connecting a large number of networks efficiency. We also believe a lot of other dapps in the ecosystem are also in our shoes so we decide to go ahead with this proposal so potentially others could benefit from it when the whole idea becomes a reality.

Considering the above how do you plan to market the API to get devs to switch to it?

We haven't put a lot of thought into this, but I think we can start with some of the below ideas:

Prepare good/clear documentation/website.
Reach out to dapps/wallet in the ecosystem that has the issue with @polkadot/api when connecting to a large number of networks to introduce it (potentially SubWallet, Polkadot JS Plus Extension, Kodadot ...)
Start sharing the ideas/work on Polkadot Forum, Twitter, Telegram, or Matrix channels of the ecosystem so people could be aware and start trying around.
...

Thanks @sinzii I had just found it in passing so thanks for looking into it. I appreciate the thorough answers. I will go ahead and mark the application as ready for review and ping the rest of the committee members. In the meantime, it looks like the image links have broken. Can you fix these? Thanks!

it looks like the image links have broken.

Ah, thank you @keeganquigley for the catch, I've just fixed that!

have you talked to any wallet developers or other network intensive dapp developers to see if this is a common concern and that this solution is needed?

Last year, we had a chance to collab with SubWallet team to review the wallet to improve its performance and stability. At that time, the memory consumption of the wallet was quite high (I don't remember the exact number), and the team had made some effort to limit the usage of @polkadot/api and introduce the toggles to turn on/off the connection to networks (so the wallet doesn't need to connect to all of the networks at the same time, but only to those networks that users have assets on) to somehow mitigate this issue.
We haven't had any chance to talk to other wallets or dapps to have their insights but in general, we believe this is an issue that other dapps/wallets would also running into when connecting to a large number of networks via @polkadot/api

Let me know if you have any feedback/thoughts on the application, thanks!

Sorry for the delay, @sinzii. I will review the application and provide some feedback tomorrow.

Hi @Noc2, I totally understand the concern about the coordination, and to some extent absolutely agree with that.
But we would like to start working on this ourselves at least for the first development phase of the project, as the more people involved in a project, the longer it would probably take in making decisions and moving forward as a whole. We expect to make a lot of decisions about the architecture, API, project setup, and structure, especially for an open-source project there would need a lot of considerations in designing the API so that it would be extensible and open for people to contribute later on. We believe we can move faster starting this ourselves.
At some point in the future, we believe there would definitely need to have coordination between wallet/ui providers/projects as there are people/projects starting to use it and run into problems that need to address/standardize so that it would be easier for them to adopt with their use-cases/needs. We do have plans to talk to other wallets in the ecosystem that are currently relying on @polkadot/api (e.g: SubWallet, PolkadotJS-Plus, ...) to start trying this around and gather feedback to further improve the project when it's in a good shape.
As the maintainer of Coong Wallet, we also understand some of the problems that a wallet/project will likely get into when connecting to a large number of networks to design the library so that it would be easy for others to use and adapt.
Let me know how that sounds, thanks!

Thank you everyone for the feedback and suggestions and in response to that, we're currently in discussions with SubWallet team for potential collaboration on this project. We'll update more on the plan this week.

My apologies for the delayed update here, our discussion with SubWallet team has gone a bit more involved with more plans that are out of the scope of this grant. I'll update as soon as we reach an agreement and finalize the whole plan. Thanks!

Sounds good @sinzii thanks for the update, feel free to ping us all once we should take another look.

@keeganquigley Sorry for the delayed reply here, after a prolonged discussion over the past few weeks with several potential approaches, we couldn't really find a good common ground for both teams to move forward in a collaboration unfortunately.
While working in collaboration is a good approach, but for this first iteration, I don't think it works for us for now. So please do let me know if you have any other suggestions to help move this forward, I'm open to reducing the scope of the work to a smaller level to build this up.
With a plan to support 1000 parachains in the future, this further strengthens our belief in a light and fast JS client solution for Polkadot ecosystem.
Thank you!

Thanks @sinzii I appreciate the update. That's a shame, sorry to hear the collaboration didn't work out. I would personally be willing to go forward with a smaller scope for perhaps a level 2 PoC. It's up to you whether you'd want to combine the core functionalities into one milestone or keep it split into two; feel free to refactor the application.
For now I will go ahead and remove the on hold status. Happy to schedule a call as well, if you'd like to discuss further. If so you can book a slot here.

I have read and hereby sign the Contributor License Agreement.

Hi everyone, I've just reduced the proposal scope to a level 2 grant, the work will mostly be focused on building foundational modules and core functionalities. Please take your time to review that application again and let me know if you have any feedback. Thanks!

@keeganquigley Thanks a lot for your support & approval!

@takahser Thank you for taking the time to review the proposal and for the questions:

In M1 you mention "APIs to inspect pallets' events & errors". Does this mean you can only query past events or is it also possible to listen to them in real-time?

By APIs to inspect pallets' events & errors, I meant to get/inspect events and errors for each pallet from the metadata. These are similar to the @polkadot/api APIs to inspect events & errors.
The ability to query events or listen to real-time events will be supported via storage query to system pallet which will be supported in the APIs to execute pallets' storage queries part. For example, to query/subscribe to events for current block, we'll call the API: api.query.system.events, more details about this can be found here

Another thing that is unclear to me is whether there would be any key management functionality i.e. generating new keys, adding existing keys, signing with keys, etc

In the scope of this grant, we will not provide these functionalities, dapp developers can use existing packages such as @polkadot/keyring, @polkadot/ui-keyring for those purposes. We'll mostly focus on the APIs to interact with a Substrate blockchain.
On that note, we also do notice some issues with existing packages for generating & managing keys, so we do have plans to implement our own solution for those functionalities in the future.

@keeganquigley @Noc2 I've just pushed a commit to fix a small typo, so you may need to reapprove the PR. My apologies for the inconvenience.

@sinzii thanks for the prompt reply.

The ability to query past/current events or listen to real-time events will be supported via storage query to system pallet which will be supported in the APIs to execute pallets' storage queries part.

That's great. Could you make this clear in the proposal?

dapp developers can use existing packages such as @polkadot/keyring, @polkadot/ui-keyring for those purposes.

That's fine by me, if these are downstream dependencies for your package. Can you make it clear in the proposal that this will be the case? Just to make sure the integration will be seamless with your package.
After updating on these 2 issues I'll be happy to add my approval as well.

@takahser I've just updated the proposal, let me know if that addresses your concerns, thanks!

Awesome, thank you everyone for your support! 🚀

One of my wishlist items are not downloading whole metadata for parsing simple rpc calls, see if its possible. I had discussion reagarding this with Rob. He said its possible with merkel proof.

@nikw3f I'm very curious about the merkel proof that Rob mentioned, would you mind sharing it with me if he did elaborate on that term and approach to achieve that? Metadata is one thing we need to optimize to achieve a better performance, so aside from the proposed approach, we're also exploring other approaches to further optimize it.

Sure let me connect with him

Hi everyone, please expect the M1 delivery to be delayed a few weeks because we're currently joining the Polkadot Hackathon Vietnam 2023. The project is in a good progress, we're now wrapping up the last pieces of M1, you can check it out now here the repo. Thank you for your understanding!

Hi @sinzii
That's not a problem at all. Could you amend the timeline accordingly?

@takahser Sure, I've just added a PR for the amendment. Thanks!


Why redisson lock implements use WAIT command

I am using redisson lock in my project, but the redis cluster i use in product env forbidden to use WAIT command, so the redisson lock alway ends up to and RedisException with message " wait command not found", I try every lock implements in redisson, all the same, so what should i do to fix this ? change anothe redis or write a lock by myselft ?

wait is used to sync lock state with slaves and increase lock durability. With this feature RedLock isn't needed anymore.

wait is used to sync lock state with slaves and increase lock durability. With this feature RedLock isn't needed anymore.

So, does "wait" command will slow down redis in some circumstance, thus my cloud redis service provider not allow to use "wait" in cluster model.

yes, it can. I'm afraid I can't change this behavior due to incomplete implementation of Redis protocol by one vendor.


Fix Alertmanager link

The broken link is currently at the bottom of this page

Actually this page should be deleted, it has moved to under Prometheus.

Are you sure?


[Feature]: Make password widgets eye icon/toggle non default

Is there an existing issue for this?

[X] I have searched the existing issues

Summary
Can we have a way to avoid the eye icon, it would be really helpful if we had a secret field, or a way to  disable that eye icon so that user is not able to see the field  JUST BE ABLE TO EDIT IT

Why should this be worked on?
It would be very helpful if we had some secret fields that can just be edited without being able to view it

@shashi-donon thank you for the feature request!

Thanks for looking to it,

we had some secret fields that can just be edited without being able to view it

Hey @shashi-donon why is it this way? I need more information to triage this better. Is this being done for a specific reason?

So We have situation where we are just storing the password for api calls. and we have a support team who can do simple diagnostic check, and we don't to show case those password anyone.
But instead they could be able to just create a new password for the user directly.

@shashi-donon that would still not be secure as the user can find that information from the network. I suggest either

Not returning any data from the network
If for some reason you have to return it, don't show that in the Input widget, instead store it using storeValue and retrieve it directly from there
Let us know if this still doesn't work for you. We try to avoid adding too many configurations to the widgets to prevent over complicating them


[arquillian/arquillian-extension-liferay#30] New functional test for the portlet

Just started reviewing :) :octocat: Sent from GH.

Pull request submitted to https://github.com/cgoncas/arquillian-extension-liferay/pull/24. See changes here. :octocat: Sent from GH.