Very slow on Android

Hi,
on my Nexus 5 if I click to the input, there is a delay about 1-2 seconds before datepicker is displayed.
When I click to 'set' button to choose the date - again, there is a 1-2 seconds delay.
Other elements work fast.

Have you tried using the modal instead of the popup? That was faster than the popup. But still i tried in few devices, there is no considerable delay of 1-2 seconds. May be you can give it a try using the modal.

I will test with a modal and will post results here, but I prefer a popup.
You can test delay with android 5.1.1 Nexus 5 emulator.

I confirm that Modal works faster, but what's the reason?

The real problem is the years list, when the app loads years list takes a few seconds in my case

I don't think year list is leading to this delay. When we use modal, it's not even taking fraction of seconds to load. But if we use popup then only it is taking about a second to show it.

Modal  is full screen in mobile , client user maybe treat as a view ,and will use back button to cancel date select ; and pop can redefine button style  more easy. but Modal's button style use hard code.
so, that is why somebody like me like use pop

i also encounter the slowness on one of my low-end devices. Looking at the source i would suggest to use controller as syntax and scope is a bit over used.

Thanks for the suggestion @rynangeles.

+1, I am also experiencing slowness with the popup on android. Any fix for this in the works?

the slowness is all because of digest of angular. i also tried ui-bootstrap datepicker and it has the same slowness. One of this day ill try to create a slim and simple impementation of datepicker for angular mobile.

issuing same slowness on Android (Note 3), about 2 seconds

Same slowness on Android 5 on Lenovo A6000. In my case the user taps more then once thinking that the tap was not proper.

You can disable the button when the user clicks on the button for the first time. This may avoid some confusion for the users.

@rajeshwarpatlolla it's a workaround, not a solution

Yes, you are right @borNfreee.

I did a small hack. I am referring to the mimised version of the code.

I removed the template caching code for popup and converted that into an html file /lib/ionic-datepicker/ionic-datepicker-popup.html

In my app.js, I created a loop for template caching:

var templates = [
            "lib/ionic-datepicker/ionic-datepicker-popup.html"
        ];

        for(i=0;i<templates.length;i++){
            var template = templates[i];
            if ($templateCache.get(template)){
                return; //prevent the prefetching if the template is already in the cache
            }
            $http.get(template).success(function (t) {
                console.log('-----------------------got the datepicker template');
                $templateCache.put(template, t);
            });
        }

Then at the line where the ionic-datepicker-popup.html is called, I am calling /lib/ionic-datepicker/ionic-datepicker-popup.html

This has resulted in shaving off atleast 2 seconds for me. The modal is indeed fast but I cannot go to modal way dues to client's restriction.

Same snowless on Xiaomi MI3w with popup. any solutions for this?

Please any one give the model code so that i can use help me out

All you need to do is, in the object you are passing to the datepicker directive just say

templateType : ''modal"

where i need to change templateType : ''modal" please help  me out
On Sun, Feb 28, 2016 at 12:14 PM, Rajeshwar<EMAIL_ADDRESS>wrote:

All you need to do is, in the object you are passing to the datepicker
directive just say
templateType : ''modal"
—
Reply to this email directly or view it on GitHub
https://github.com/rajeshwarpatlolla/ionic-datepicker/issues/82#issuecomment-189800255
.

--
thanks and regards,
R.Mohanraj
7760342142

Please read the documentation

thanks its working
On Sun, Feb 28, 2016 at 12:24 PM, Rajeshwar<EMAIL_ADDRESS>wrote:

Please read the documentation
https://github.com/rajeshwarpatlolla/ionic-datepicker#how-to-use
—
Reply to this email directly or view it on GitHub
https://github.com/rajeshwarpatlolla/ionic-datepicker/issues/82#issuecomment-189803763
.

--
thanks and regards,
R.Mohanraj
7760342142

    Invite

                Dear rajeshwarpatlolla/ionic-datepicker,

                    I found a simple way to save on all purchases. CashbackURL.in is just a one click process to earn cashback.
                    Before buying from any of the 500+ Indian online stores, just open www.cashbackURL.in and search for store or copy paste product URL. And, continue buying normally from the store.
                    Cashback gets tracked and you can redeem the amount for either mobile recharge or cash payment.

                    Click here to Activate your Account

                ( Since I'm referring you, clicking on the above link creates an account instantly. And, you can immediately start earning on cashbackURL )

                    Cheers,
                    sree mohan

            This email has been sent by cashbackURL on behalf of your friend, sree mohan.
                Click here to Unsubscribe if you don't want to receive any emails in future.

        © LOCUS Web Services Pvt. Ltd.,2nd Floor, Pochampally House, 1-10-72/A/2, above Nike Showroom,
            Beside Nalli Silks, SP Road, Begumpet,Telangana, India.

Please check the new release v1.0.0. This issue has been addressed.

I'm still experiencing a delay of 2 seconds when I open the datepicker (popup). How can I fix this?

I am getting unknown provider after adding ionicDatePickerProvider provider to my .config method while every another thing is correct configured .


Partial Python 3 support

This PR builds upon the ideas in #22 and introduces partial support for Python 3. I'm not ready to consider Python 3 "well-supported" quite yet, as #22 (keys are coerced to strings) is more visible on Python 3.
Notes:

I went ahead and used six as the compatibility layer, rather than writing and testing a custom compatibility layer.
I've defensively added __future__ imports for Python 2 to try to keep behaviors between the major versions as similar as possible (I don't think they're doing much work in practice)
I've made some style changes as recommended by flake8. I'm enforcing flake8-correctness in Travis yet, but probably will eventually.
Keys get coerced to unicode strings instead of byte strings now. This doesn't mean much on Python 2, but it does make it difficult to retrieve bytes objects in Python 3 (previously it was difficult to retrieve str objects on Python 3).

Big +1 to flake8!
On 30 Jul 2016 5:04 p.m., "Bo Bayles"<EMAIL_ADDRESS>wrote:

This PR builds upon the ideas in #22
https://github.com/honzajavorek/redis-collections/pull/22 and
introduces partial support for Python 3. I'm not ready to consider Python 3
"well-supported" quite yet, as #22
https://github.com/honzajavorek/redis-collections/pull/22 (keys are
coerced to strings) is more visible on Python 3.
Notes:

I went ahead and used six as the compatibility layer, rather than
writing and testing a custom compatibility layer.
I've defensively added future imports for Python 2 to try to
keep behaviors between the major versions as similar as possible (I don't
think they're doing much work in practice)
I've made some style changes as recommended by flake8
http://flake8.pycqa.org/en/latest/. I'm enforcing flake8-correctness
in Travis yet, but probably will eventually.
Keys get coerced to unicode strings instead of byte strings now.
This doesn't mean much on Python 2, but it does make it difficult to
retrieve bytes objects in Python 3 (previously it was difficult to
retrieve str objects on Python 3).

You can view, comment on, or merge this pull request online at:
https://github.com/honzajavorek/redis-collections/pull/44
Commit Summary

Placate flake8
Add future imports
Add six to the mix
Partial Python 3 support
Update docs with Known Issues
Add six to requirements.txt
Test on Python 3

File Changes

M .travis.yml
https://github.com/honzajavorek/redis-collections/pull/44/files#diff-0
(3)
M docs/index.rst
https://github.com/honzajavorek/redis-collections/pull/44/files#diff-1
(11)
M redis_collections/init.py
https://github.com/honzajavorek/redis-collections/pull/44/files#diff-2
(3)
M redis_collections/base.py
https://github.com/honzajavorek/redis-collections/pull/44/files#diff-3
(26)
M redis_collections/dicts.py
https://github.com/honzajavorek/redis-collections/pull/44/files#diff-4
(36)
M redis_collections/lists.py
https://github.com/honzajavorek/redis-collections/pull/44/files#diff-5
(8)
M redis_collections/sets.py
https://github.com/honzajavorek/redis-collections/pull/44/files#diff-6
(23)
M requirements.txt
https://github.com/honzajavorek/redis-collections/pull/44/files#diff-7
(1)
A setup.cfg
https://github.com/honzajavorek/redis-collections/pull/44/files#diff-8
(3)
M setup.py
https://github.com/honzajavorek/redis-collections/pull/44/files#diff-9
(6)
M tests/base.py
https://github.com/honzajavorek/redis-collections/pull/44/files#diff-10
(3)
M tests/test_dicts.py
https://github.com/honzajavorek/redis-collections/pull/44/files#diff-11
(120)
M tests/test_lists.py
https://github.com/honzajavorek/redis-collections/pull/44/files#diff-12
(11)
M tests/test_sets.py
https://github.com/honzajavorek/redis-collections/pull/44/files#diff-13
(12)

Patch Links:

https://github.com/honzajavorek/redis-collections/pull/44.patch
https://github.com/honzajavorek/redis-collections/pull/44.diff

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/honzajavorek/redis-collections/pull/44, or mute the
thread
https://github.com/notifications/unsubscribe-auth/AARTMX2Ro71ke0--dWrXPfQF72Qb_8fFks5qa2f3gaJpZM4JY0xL
.

This is in so more drastic changes to solve #25 (keys are strings) can go forward.

I went ahead and used six as the compatibility layer

👍
I think the most difficult part of the Python 3 support is the unicode/str/bytes confusion indeed. At the time people started to request support for 3 here I didn't even use it yet (I do regularly now), so it was very hard for me to not only spot, but to correctly solve all the corner cases. Thanks for all the great work and thorough approach!


Navigating back should retain the scroll position.

Version
2.7.0
Reproduction link
https://github.com/krestaino/nuepress
Steps to reproduce

Go to http://localhost:3000/
Scroll down and click on a post
After the new page loads, use the browser's back button
Scroll position goes to the top rather than where it was

What is expected?
Navigating back should retain the scroll position.
What is actually happening?
Navigating back does not retain the scroll position.
Related Issues
https://github.com/nuxt/nuxt.js/issues/1036
https://github.com/vuejs/vue-router/issues/1373
https://github.com/nuxt/nuxt.js/issues/1182 (this issue on nuxt/nuxt.js)

Is this a vue-router or Nuxt.js issue?
One thing I noticed, if I set the height of the HTML element to something like html { min-height: 10000px }, the issue goes away. Is this happening because the post list is being rendered dynamically using v-for?

Please, check if this hasn't been reported already. It looks related to #1263
Next time, please make sure to provide a minimal repro using the fiddle of the vue issue helper, otherwise I won't be able to check it

Yeah sorry, I figured out the issue is with Nuxt.js and it has nothing to do with vue-router. I've reported it over there.
I'll make sure to isolate the issue next time and apologizes for the inconvenience.


js: Enable no-jquery/no-constructor-attributes

https://github.com/wikimedia/eslint-plugin-no-jquery/blob/master/docs/rules/no-constructor-attributes.md
The motivation for this rule is a subtle caveat buried in the documentation:
https://api.jquery.com/jquery/#jQuery-html-attributes
“While the second argument is convenient, its flexibility can lead to unintended consequences (e.g. $( "<input>", {size: "4"} ) calling the .size() method instead of setting the size attribute).”

Yikes, that's weird. Merged, thanks @andersk!


Room throws if connection to room was refused

TypeError: undefined is not an object (evaluating 'this.serializer.teardown') thrown if connection to room rejected (e.g if onJoin promise is rejected on server side)

Thanks @serjek, I've just added a null check for this! Just released new version on haxelib (0.12.5)


[modify] use mongodb 4.2 on travis

Coverage decreased (-4.3%) to 72.86% when pulling 6e7e8b66511a2268aea63e5bcd816744dd6e94da on sunny4381:features/mongodb-4.2 into 1c71a6f339af6d4adae3fcb55f4a6098ab89317d on shirasagi:master.


nushell: use --features=dataframe by default

Description of changes
The feature "extra" doesn't do anything anymore:
https://github.com/nushell/nushell/blob/ba5258d7165423b360edf1747d0ffec00052db13/Cargo.toml#L111
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Note that there's an open issue to discuss this: https://github.com/NixOS/nixpkgs/issues/210646

Closed in favor of https://github.com/NixOS/nixpkgs/pull/218030


chore(Algebra/CharP): ensure CharP.Defs doesn't depend on ring

I opened #10765 quite a few months ago and it didn't get any response :( In the meantime, Yaël correctly identified that the first step of #10765, which is to split off a CharP.Defs file with little imports, is a good idea in general, and got #12710 merged. Unfortunately, #12710 is not compatible with #10765, since CharP.Defs still transitively depends on the ring tactic. This PR fixes that by moving the one lemma that is incompatible to CharP.Basic, and changing the proof of another.
I literally put back the declaration of this lemma as they were before #12710. As far as I can tell, this doesn't revert anything.

That also seems to work apart from an apparently unrelated timeout!


Metadata not working : All Drops

previews for #12 #13 not working
https://kodadot.xyz/ahp/drops/enough

This is not a preview, but the metadata has not been updated. So, it will fall back into the collection image.
Does anyone know about waifu.deno.dev service? @kodadot/internal-dev. As far as I know, only @vikiival knows about that service. The endpoint returned 200, but extrinsic never triggered https://assethub-polkadot.subscan.io/account/15CoYMEnJhhWHvdEPXDuTBnZKXwrJzMQdcMwcHGsVx5kXYvW. Both ahk and ahp don't work; it will affect all drops minting.
The last extrinsic is this one. Does anyone have a clue about this one?
collection: 139
item: 1
data: https://media.exiledracers.com/dot/metadata/koda_test_poap3.json

Currently, these are the affected items on ahp:
Items: [
  'https://kodadot.xyz/ahp/gallery/130-2405264300',
  'https://kodadot.xyz/ahp/gallery/130-2405249231',
  'https://kodadot.xyz/ahp/gallery/127-2405420603'
]

Yes, It has been a victim of ddos attack. Looks like cron is glitched now

i see, anything that we can help?

running again
https://assethub-polkadot.subscan.io/extrinsic/0x84c00b2e1f5853362cd6bef28668e9d5ecb01951c7b979c06291c15bd28cb105

running again

thank you~
safe to close


Version 2 UI

Hello,
The road map mentions a new front end. Will it still be pure ASP.NET MVC or will you use a JavaScript framework?
Thanks, and can't wait for Version 2!

I'll provide a SPA with Angular 4, I'll keep the MVC UI too doing the same thing.
Will be 2 flavors of UI.

?

React is suggested


Repo_Manage_Section_UI

Changes Proposed

Created the UI for repository manage section

Check List

[X] The title of my pull request is a short description of the changes
[X] This PR relates to some issue:
[X] I have documented the changes made (if applicable)
[ ] I have covered the changes with unit tests

@chad11111 please have a look.

@aman1376 Please don't snake case (Repo_Manage_Section) only camel case (RepoManageSection)

@aman1376 also please fix conflicts

@aman1376 why did u close it ?

i have created a new pull request RepoManageSection


Enter hex number in HTML form

Hi guys !
I have setup the colorpicker in an HTML form, but I'm facing an issue. When I try to enter manually the hex number and press enter, it submit the form.
Any idea for a solution ?

@tchinou1 You might need to add event.preventDefault somewhere in the input logic. This might mean we need to tweak colorjoe itself as this is the sane way to do it.
See also http://stackoverflow.com/a/895231/228885 .

I've tried something like bellow, but it don't work. Have I made a mistake ?
$(document).ready(function() {
$(window).keydown(function(event){
if($('.colorPicker').css('display') == 'inline-block'){
event.preventDefault();
return false;
}
});
});


wine: tweak update script; winePackages.stable: 8.0 -> 8.0.1, winePackages.{unstable,staging}: 8.5 -> 8.7

Description of changes
https://www.winehq.org/announce/8.0.1
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Ok, it seems staging 8.6 is broken but fixed upstream.
The nix expression asserts that assert lib.getVersion wineUnstable == patch.version so I couldn't just add a different version for staging. Disabling the patchset entirely when there is already a fix seems like a waste, so I opted for this temporary workaround.

Result of nixpkgs-review pr 227785 run on x86_64-linux 1

35 packages built:

exactaudiocopy
grapejuice
grapejuice.dist
pipelight
playonlinux
q4wine
winbox
wine (winePackages.full ,winePackages.stableFull)
wine-staging (winePackages.stagingFull)
wine-wayland (winePackages.waylandFull)
wine64 (wine64Packages.full ,wine64Packages.stableFull)
wine64Packages.base (wine64Packages.stable)
winePackages.fonts (wine64Packages.fonts ,wineWowPackages.fonts)
wine64Packages.minimal
wine64Packages.staging
wine64Packages.stagingFull
wine64Packages.unstable
wine64Packages.unstableFull
wine64Packages.waylandFull
winePackages.base (winePackages.stable)
winePackages.minimal
winePackages.staging
winePackages.unstable
winePackages.unstableFull
wineWowPackages.base (wineWowPackages.stable)
wineWowPackages.full (wineWowPackages.stableFull)
wineWowPackages.minimal
wineWowPackages.staging
wineWowPackages.stagingFull
wineWowPackages.unstable
wineWowPackages.unstableFull
wineWowPackages.waylandFull
wineasio
yabridge
yabridgectl

nixpkgs-review reveals no new failures on macos 10.15

I bungled the nixpkgs-review on my x86_64-darwin machine, so it’s rerunning. The 8.0.1 changes LGTM.
If you drop 8.6 to wait for 8.7, it seems worth including 2fa8e8d16e82ddbce41698ba3d76b6467460ec15 to facilitate future updates with divergent Gecko versions.

Stripped the unstable/staging updates for now; chances are the problem resolves itself (in 8.7) at the end of the week and a separate PR can be made then. The update script should correctly update gecko then as well.

Wine 8.7 is out.

Added new attempt at bumping unstable/staging.

Result of nixpkgs-review pr 227785 run on x86_64-linux 1

1 package failed to build:

airwave

37 packages built:

exactaudiocopy
grapejuice
grapejuice.dist
pipelight
playonlinux
q4wine
vkd3d
vkd3d-proton
winbox
wine (winePackages.stableFull)
wine-staging (winePackages.stagingFull)
wine-wayland (winePackages.waylandFull)
wine64 (wine64Packages.stableFull)
wine64Packages.base (wine64Packages.stable)
winePackages.fonts (wine64Packages.fonts ,wineWowPackages.fonts)
wine64Packages.minimal
wine64Packages.staging
wine64Packages.stagingFull
wine64Packages.unstable
wine64Packages.unstableFull
wine64Packages.waylandFull
winePackages.base (winePackages.stable)
winePackages.minimal
winePackages.staging
winePackages.unstable
winePackages.unstableFull
wineWowPackages.base (wineWowPackages.stable)
wineWowPackages.full (wineWowPackages.stableFull)
wineWowPackages.minimal
wineWowPackages.staging
wineWowPackages.stagingFull
wineWowPackages.unstable
wineWowPackages.unstableFull
wineWowPackages.waylandFull
wineasio
yabridge
yabridgectl

Accidentally gc'ed the non-free source of airwave, so that broke in the review...

Wine 8.8 released.
https://www.winehq.org/announce/8.8
(No staging yet.)

This is why I stopped doing staging updates tbh, it's rather frustrating to not get approval/a merge over weeks while multiple new versions are released (and having to run nixpkgs-review for hours for each release)...
I will be a taking a break from PRs for a while in general, so I'll leave this to someone else for now.

Thanks for your work on this. I’m interested in the 8.0.1 release since it affects Darwin on macOS 13.3+ on Rosetta, so I’d like to pick this up.

Wine 8.9 released (with a Mono update).

I still want to pick this up, but my time currently is being spent on the Darwin stdenv bump to a newer LLVM. Once that gets to a place where I can flip the PR to ready, I want to look at this next (assuming no one else has picked it up).

By the way, I attempted to update to 8.9, and staging wasn't building. I checked, and... staging's latest version is 8.9.1.
Waiting for 8.10, then.


Explore supporting external compaction processes in accumulo-cluster script

The accumulo-cluster script could be modified to support starting coordinator and compactor processes for external compactions if the corresponding files exists.  Not sure if this is a good idea in general, but thought it was worth considering. Trying to determine if this is something worth pursuing, came up with the following list of pros and cons.  What other things are there to consider?
Pros

Makes it easier to start these processes on a cluster for testing.

Cons

More to maintain in the accumulo-cluster script, this may only be an issue if there is specialization for compactors.
Compactor processes run against a queue.  This may not fit well w/ the accumulo cluster script.  Would need to figure out an elegant way to do this.

Rather than create new hosts files, I would like to see the hosts files consolidated into a single accumulo-cluster.conf file which includes the hosts for all the services the accumulo-cluster script manages (there may be an existing ticket for this... I didn't check before writing this). If host configuration were streamlined into a single config file, it would be easier to specify the startup of additional services, perhaps even with separate config for each host.

I would like to see the hosts files consolidated into a single accumulo-cluster.conf

That sounds interesting.  Any thoughts on the structure and how compactor+queue info might fit into that? For compactors I was thinking of the following possibilities, had not considered a single file.

A dir name compactors with files under it for each queue.  The name of the file would be queue name and the contents of the file would be host to start compactors.
A file named compactors where the contents are lines like : <host>,<queue>.  This would repeat information alot and seems error prone.
A file named compactors where contents has sections.  The section is the queue name and the contents are host.  Like [Q1]\nhost1\nhost2\n[Q2]\nhost3\nhost4.

Of the three, option 3 is closest to what I was thinking. However, if it was done in a single file, it would strongly depend on the format for that file. If it were YAML, it might look something like:
managers:
 - host1

gc:
 - host1

tservers:
 - host2
 - host3

compaction:
  coordinator: host4
  queues:
   -
     name: queue1
     workers: [ host5, host6 ]
   -
     name: queue2
     workers:
      - host7
      - host8
      - host9

Would you expect that yamllint and yq be prerequisites for installing Accumulo so that we could use them to validate and parse the yaml file?

I have not done anything w/ yaml using command line tools.  I wonder if there are tools that would make it possible to convert the existing files to yaml in a few lines of bash.  If so the script could detect the old files and just print an error message w/ the suggested commands to run to convert them.

Install yamllint and yq:
sudo yum install yamllint
wget https://github.com/mikefarah/yq/releases/download/v4.9.6/yq_linux_amd64 -O yq

So, for this file:
---
managers:
  - localhost1
  - localhost2

monitors:
  - localhost1
  - localhost2

trace:
  - localhost

gc:
  - localhost

tservers:
  - localhost1
  - localhost2
  - localhost3
  - localhost4

# compaction:
#   coordinators:
#     - localhost1
#     - localhost2
#   compactors:
#     - queue: q1
#       hosts:
#         - localhost1
#         - localhost2
#     - queue: q2
#       hosts:
#         - localhost1
#         - localhost2

These commands  work:
yamllint -s hosts.yaml      # 0 == success, anything else is error and malformed
managers=$(yq eval '.managers | join(",")' hosts.yaml)
monitors=$(yq eval '.monitors | join(",")' hosts.yaml)
gc=$(yq eval '.gc | join(",")' hosts.yaml)
tservers=$(yq eval '.tservers | join(",")' hosts.yaml)
yq eval '.compaction' hosts.yaml     #  null --> means its commented out
yq eval '.compaction.coordinators | join(",")' hosts.yaml
queues=$(yq eval '[.compaction.compactors.[].queue] | join(",")' hosts.yaml)
queue1_hosts=$(yq eval '.compaction.compactors.[] | select (.queue == "q1") | .hosts | join(",")' hosts.yaml)

Would you expect that yamllint and yq be prerequisites for installing Accumulo so that we could use them to validate and parse the yaml file?

I've never used these. I'm not sure what we'd use to parse. However, I don't think it'd be unreasonable to add a yaml parser dependency for the accumulo-cluster script. (Note: I specified that it'd be fore the accumulo-cluster script, because I really think of that as a separate cluster-management utility that ships alongside of Accumulo, but is not part of Accumulo. It's optional, and users may wish to use a completely different cluster management mechanism. These are only included in the tarball because we've trained users to expect some kind of cluster management scripts ... aka 'start-all'... from early days. My preference is that they live in a separate repo, or live under a contrib folder. I mention this here, only to clarify the distinction between what is "Accumulo" and what is "accumulo-cluster", so as to keep the interface between the two separate and avoid tight coupling between the two).

I am concerned about adding yq as a dependency, since it's not conveniently packaged for RPM-based distros. I wonder if shyaml might be better, which wraps pyyaml.

It looks like to install shyaml and pyyaml you need to first install python and pip. In some environments you may need to specially configure pip. yq is a single binary.  It looks like rpm is the only format in which it's not packaged, although I found easy instructions for doing it. It could be installed on CentOS / RHEL via Snap: https://snapcraft.io/install/yq/rhel

If you do sudo dnf install shyaml, it automatically pulls in dependencies. You don't need to install pip at all, and since dnf/yum already depends on python, it will already be installed. That said, I already use snaps for a few things, so I didn't have a problem installing the snap for yq. So, whatever you think would be easiest for users. If you prefer, we could also just do JSON instead of YAML, only because there's a ton of parsers already available.
We could also do something in Java. We already have commons-configuration2 on the classpath, and it supports YAML. We already call Java in the accumulo-cluster script for ZooZap. It wouldn't be hard to write something small that parses our config file for us and extracts the relevant bits.

If you prefer, we could also just do JSON instead of YAML, only because there's a ton of parsers already available.

Json would work too. Unfortunately json does not allow for comments in files. So you would need two different types of files depending on whether or not you are using external compactions. The examples below are with and without external compactions.
{
  "managers": [
    "localhost1",
    "localhost2"
    ],
  "monitors": [
    "localhost1",
    "localhost2"
    ],
  "tracers": [
    "localhost1"
    ],
  "gc": [
    "localhost1"
    ],
  "tservers": [
    "localhost1",
    "localhost2",
    "localhost3",
    "localhost4"
    ],
  "compaction": {
    "coordinators": [
      "localhost1",
      "localhost2"
      ],
    "compactors": [
      {
        "queue": "q1",
        "hosts": [
          "localhost1",
          "localhost2"
          ]
      },
      {
        "queue": "q2",
        "hosts": [
          "localhost1",
          "localhost2"
          ]
      }
    ]
  }
}

{
  "managers": [
    "localhost1",
    "localhost2"
    ],
  "monitors": [
    "localhost1",
    "localhost2"
    ],
  "tracers": [
    "localhost1"
    ],
  "gc": [
    "localhost1"
    ],
  "tservers": [
    "localhost1",
    "localhost2",
    "localhost3",
    "localhost4"
    ]
}

yq is an extension of jq link and I believe that jq packages are more widely available ( see https://stedolan.github.io/jq/download/).  The following commands using jq work (very similar to the yq examples above):
managers=$(jq '.managers | join(",")' hosts.json)
monitors=$(jq '.monitors | join(",")' hosts.json)
gc=$(jq '.gc | join(",")' hosts.json)
tracers=$(jq '.tracers | join(",")' hosts.json)
tserver=$(jq '.tservers | join(",")' hosts.json)
jq '.compaction' hosts.json  # null means it does not exist
coordinators=$(jq '.compaction.coordinators | join(",")' hosts-compaction.json)
queues=$(jq '.compaction.compactors | [.[].queue] | join(",")' hosts-compaction.json)
queue1_hosts=$(jq '.compaction.compactors | .[] | select(.queue == "q1") | .hosts | join(",")' hosts-compaction.json)

JSON is uglier, but more ubiquitous. Ultimately, I think I'd prefer YAML, but if there are things we can do to make it depend on less (such as having a bit of java code using one of our existing dependencies) to make the bar lower, that would be nice. If not, I think I'd still rather users get yq or shyaml themselves than force the ugliness of JSON on them.
Could also do simple properties files as well, but the schema would be a little bit frustrating. Commons-configuration2 could easily manage it, though... but if we're going that route, it'd be better to just use YAML, since commons-configuration2 can handle that too.
manager.hosts=managerhost1,managerhost2
...
compaction.coordinator.1=server1
compaction.coordinator.2=server2
compaction.queues.q1.compactors=server3,server4
compaction.queues.q2.compactors=server5,server6

One approach to supporting yaml is that we write a Java program to parse the yaml and emit text that is easy to deal with in bash.  This avoid the dependency on yq or shyaml commands, while possibly introducing a yaml dep in java.  Then could do something like the following in the accumulo-cluster bash script.
accumulo org.apache.accumulo.core.cluster.YamlExtract <cluster conf file>

The java program could extract specific information based on arguments OR convert the yaml to CSV like the following that is easy to process in bash.
tserver,host1
tserver,host2
tserver,host3
manager,host4
manager,host4
gc,host5
monitor,host6
compactor,host1,Q1
compactor,host2,Q2
compactor,host3,Q2
compactor,host3,Q2

Not sure if its better to make the java program smart w/ lots of options or make it dumb and spit out something like CSV moving more logic to bash.

@keith-turner that's basically what I was thinking, since we probably have everything we need on the class path already in lib/. I wasn't sure which way to go either, but I think I prefer keeping the bash simpler by having arguments. Emitting lists of things as newline- or space-delimited makes it easier to loop over, if necessary, in bash.

@keith-turner that's basically what I was thinking, since we probably have everything we need on the class path already in lib/.

I didn't see your earlier message I see it now.  It seems like doing the heavy lifting in Java is the best way to go, especially if we already had the needed deps on the classpath.

I think I prefer keeping the bash simpler by having arguments.

I think I do too.  I also agree that we want to avoid creating something like yq or shyaml in Java.  The Java code and options should be tailored to the accumulo cluster file format.

@keith-turner @ctubbsii  - take a look at the draft PR, let me know if this is what you had in mind. I have not tested it yet.


Backup Azure Analysis Services to a restricted storage account?

Is it possible to backup an Azure Analysis Services instance to a storage account that is VNET restricted?
I tried to do this from the portal, with the 'Allow trusted Microsoft services to access this account.' flag enabled, but I am not able to select my storage account and the request to use it fails.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: fb6c1de7-a04c-f08e-adab-4733030668a1
Version Independent ID: d5e17386-38ff-d3de-ef43-f734bc2d4748
Content: Azure Analysis Services database backup and restore
Content Source: articles/analysis-services/analysis-services-backup.md
Service: azure-analysis-services
GitHub Login: @Minewiskan
Microsoft Alias: owend

@AdamSharif-MSFT I am assigning the issue to the author to comment on this.
@Minewiskan Can you please have a look at this one?


Exemption (free in preview feature)

The page says "Exemption (free in preview feature) " - however there are two questions with this statement

Policy exemptions no longer seems to be in preview (this commit by @timwarner-msft removed references to preview from exemption-structure page.
Policy exemptions don't seem to cost anything (going by the Azure Pricing Calculator at https://azure.microsoft.com/en-us/pricing/details/azure-policy/ where its only ARC Policy that costs money.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: d2c99b8a-78d1-eff9-ed69-2879f768dfd9
Version Independent ID: 41e21478-b834-30a6-cac1-b50aaaea0ac0
Content: Understand scope in Azure Policy - Azure Policy
Content Source: articles/governance/policy/concepts/scope.md
Service: azure-policy
GitHub Login: @timwarner-msft
Microsoft Alias: timwarner

@timwarner-msft - Thanks for raising the PR.
@bubbletroubles  - Thanks again for bringing this to our attention. We have updated the Doc and it will be live once its merged.  We will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.


cross-compiling for alpine-aarch64

Just leaving notes here as I don't think it's worth fixing per se, but this was rather a mess, so figured I might as well write it down for myself if nothing else. Feel free to close immediately unless you see something worth discussing.

normal stuff, exporting a bunch of env vars: LD=aarch64-alpine-linux-musl-ld CC=aarch64-alpine-linux-musl-gcc ARCH=arm64 CROSS_COMPILE=aarch64-alpine-linux-musl-
Normally LD and CC should come from CROSS_COMPILE, but $(CROSS_COMPILE)cc does not exist so CC has to be set to make it use gcc instead (could also be worked around with a symlink), and LD is defined without CROSS_COMPILE in the makefile
alpine-specific: musl doesn't include argparse and one needs to add -largp to the link commands in the makefile.
cargo stuff:

adding --target=aarch64-unknown-linux-musl to the cargo build command goes some way if the target has been added with rustup target add aarch64-unknown-linux-musl previously (couldn't figure how to do that with the system rust and has to use rustup)
final link requires setting CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_LINKER=aarch64-alpine-linux-musl-gcc as well on top, because why use standard environment variables...
the binary addr2line is at a slightly different location, ../sidecar/target/aarch64-unknown-linux-musl/$(if $(DEBUG),debug,release)/addr2line (extra dir after target)

the final ld relies on path so _binary___sidecar_target_release_addr2line needs to be updated to __binary_sidecar_start=_binary____sidecar_target_aarch64_unknown_linux_musl_release_addr2line

Here's a diff of the makefile for recap, with the env vars above that should contain everything for aarch64:
diff --git a/src/Makefile b/src/Makefile
index 8090f42..89e115c 100644
--- a/src/Makefile
+++ b/src/Makefile
@@ -12,7 +12,7 @@ BPFTOOL ?= $(BPFTOOL_OUTPUT)/bootstrap/bpftool

 LIBBPF_SRC := $(abspath ../libbpf/src)
 LIBBPF_OBJ := $(abspath $(OUTPUT)/libbpf.a)
-SIDECAR := ../sidecar/target/$(if $(DEBUG),debug,release)/addr2line
+SIDECAR := ../sidecar/target/aarch64-unknown-linux-musl/$(if $(DEBUG),debug,release)/addr2line
 # Use our own libbpf API headers and Linux UAPI headers distributed with
 # libbpf to avoid dependency on system-wide headers, which could be missing or
 # outdated
@@ -67,7 +67,7 @@ all: retsnoop simfail
 clean:
 	$(call msg,CLEAN)
 	$(Q)rm -rf $(OUTPUT) retsnoop simfail bpftool
-	$(Q)$(CARGO) clean --manifest-path=../sidecar/Cargo.toml
+	$(Q)$(CARGO) clean --manifest-path=../sidecar/Cargo.toml --target=aarch64-unknown-linux-musl

 .PHONY: cscope
 cscope:
@@ -128,16 +128,16 @@ $(OUTPUT)/mass_attacher.o: $(OUTPUT)/retsnoop.skel.h $(OUTPUT)/calib_feat.skel.h

 $(SIDECAR)::
 	$(call msg,CARGO,addr2line)
-	$(Q)$(CARGO) build --manifest-path=../sidecar/Cargo.toml $(if $(DEBUG),,--release)
+	CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_LINKER=aarch64-alpine-linux-musl-gcc $(Q)$(CARGO) build --manifest-path=../sidecar/Cargo.toml $(if $(DEBUG),,--release) --target=aarch64-unknown-linux-musl

 $(OUTPUT)/addr2line.embed.o: $(SIDECAR)
 	$(call msg,LLVM_STRIP,$<)
 	$(Q)$(LLVM_STRIP) -g $<
 	$(call msg,LD,$@)
 	$(Q)$(LD) -r -b binary \
-		--defsym __binary_sidecar_start=_binary____sidecar_target_$(if $(DEBUG),debug,release)_addr2line_start \
-		--defsym __binary_sidecar_end=_binary____sidecar_target_$(if $(DEBUG),debug,release)_addr2line_end \
-		--defsym __binary_sidecar_size=_binary____sidecar_target_$(if $(DEBUG),debug,release)_addr2line_size \
+		--defsym __binary_sidecar_start=_binary____sidecar_target_aarch64_unknown_linux_musl_$(if $(DEBUG),debug,release)_addr2line_start \
+		--defsym __binary_sidecar_end=_binary____sidecar_target_aarch64_unknown_linux_musl_$(if $(DEBUG),debug,release)_addr2line_end \
+		--defsym __binary_sidecar_size=_binary____sidecar_target_aarch64_unknown_linux_musl_$(if $(DEBUG),debug,release)_addr2line_size \
 		-o $@ $<

 $(OUTPUT)/addr2line.o: $(OUTPUT)/addr2line.embed.o
@@ -157,7 +157,7 @@ retsnoop: $(addprefix $(OUTPUT)/,					\
 		      mass_attacher.o)					\
 	  $(LIBBPF_OBJ)
 	$(call msg,BINARY,$@)
-	$(Q)$(CC) $(CFLAGS) $^ -lelf -lz -o $@
+	$(Q)$(CC) $(CFLAGS) $^ -lelf -lz -largp -o $@

 $(OUTPUT)/tests/simfail.o: $(OUTPUT)/tests/kprobe_bad_kfunc.skel.h	\
 			   $(OUTPUT)/tests/fentry_unsupp_func.skel.h	\
@@ -168,7 +168,7 @@ simfail: $(addprefix $(OUTPUT)/tests/,					\
 		     simfail.o)						\
 	  $(LIBBPF_OBJ)
 	$(call msg,BINARY,$@)
-	$(Q)$(CC) $(CFLAGS) $^ -lelf -lz -o $@
+	$(Q)$(CC) $(CFLAGS) $^ -lelf -lz -largp -o $@

 # delete failed targets
 .DELETE_ON_ERROR:

As a side note, if instead of cross compiling one goes the qemu-user route, cargo has a bug that makes it consume insane amount of memory when fetching its crate index while running under qemu-user.
That can be worked around by exporting CARGO_NET_GIT_FETCH_WITH_CLI=true before running make, and this worked for me -- I'll be using the binary generated that way for now.

@martinetd I'm open to Makefile changes to make this easier, retsnoop's Makefile is not as crazy as, say, selftests/bpf/Makefile in Linux repo, so adding support for CROSS_COMPILE would be fine with me. If the musl stuff can be more or less isolated to one or two checks, that should be fine as well.
As for _binary____sidecar_target_release_addr2line_start not being there. This symbol name is automatically derived from the path (you changed it to ../sidecar/target/aarch64-unknown-linux-musl/release/addr2line, so it will have extra _aarch64_unknown_linux_musl_ in the middle, I suspect. This is objcopy's convention, I don't have much control over this. But again, all that probably can be accommodated.
So if you feel like this has to be addressed, please send PR and let's talk about it.

Ah, I suspected something with the path for that ld symbol, but I kept the dashes in the target triplet instead of underscores, and ld didn't like dashes. Well, that sure works better and I've updated the first post with full instructions and an ugly diff for reference/recap, but nothing that could be merged as is.
As said initially I don't think it's necessarily worth fixing at this point -- unless someone comes in who want to e.g. package it for yocto, I'm not aware of any distribution doing cross build, I mostly looked because cargo + qemu-user was being stupid.
In particular I'm not sure how we could possibly do the mappings for cargo target and its env var; guessing the libc from CROSS_COMPILE doesn't sound like a good idea, so it's probably best to just leave some instructions instead? I don't know. If someone looks they might be able to find this issue ;)
Anyway, if it turns out I use retsnoop a bit I might look into packaging it for alpine, in which case I might send a patch just for argp, but it's a bit early for me at this point as it will be the first time I use it on an actual problem so I don't want to commit to anything right now.

@martinetd can this be closed?

I'd say cross compiling still isn't quite easy, but building the sidecar separately and pasing -largp through LDFLAGS ought to work so it should have gotten quite a bit easier than last time I tried -- and I can't think of anything that'd be easy to improve anyway.
Thanks for the reminder, let's close this for now!


⚠ Fortnite has degraded performance

In 002f945, Fortnite (https://fortnite.com) experienced degraded performance:

HTTP code: 200
Response time: 1505 ms

Resolved: Fortnite performance has improved in fa2fd9c.


listener: remove the peek from the listener filters

Commit Message: listener: remove the peek from the listener filters
Additional Description:
To avoid the listener filters to peek the data from the socket directly, allow the ActiveTcpSocket to peek the data
into the buffer. The ActiveTcpSocket will get the max data expected from the filters through the
filter interface size_t maxReadBytes(). Then ActiveTcpSocket will listen on the file event on the socket.
And peek the data into the buffer. And calling the listener filter callback onData(Buffer::Instance&).
If the filter doesn't need any data from the connection, it should return 0 on the interface 'maxReadBytes()'.
The ActiveTcpSocket won't listen on the file event if there is no filter expecting any data.
The buffer will be shared across the listener filter. no duplicated peek need anymore.
The buffer is implemented by ListenerFilterBufferImpl. It will store the data peeked from the socket, and provide
the const pointer to the listener filter to access the data. It also provides the interface for listener filter to drain the data,
and the data will be drained on the socket in the end.
Risk Level: high
Testing: unittest
Docs Changes: TODO
Release Notes: TODO
Fixes: #17229
Signed-off-by: He Jie Xu<EMAIL_ADDRESS>

cc @mattklein123 @ggreenway @antoniovicente @lambdai
I created this PoC for showing the interface design, so I didn't work out the unittest/functional test since I thought there will be some discussion on the interface design. But if you guys think the test should be work out, I will work on it.

/wait

I like the direction this is going. I'd like to take it even further, and do a read, not a peek. Then the listener filter can drain data from the buffer if it wants to consume the data (proxy proto listener filter needs to do this), or just look at the contents of the buffer if it only wants to inspect. At the end of the listener filters, pass the buffer (probably non-empty) along with already-read data.
If we went that route, then callback name just becomes onData() like in a network filter.

I like the direction this is going. I'd like to take it even further, and do a read, not a peek. Then the listener filter can drain data from the buffer if it wants to consume the data (proxy proto listener filter needs to do this), or just look at the contents of the buffer if it only wants to inspect. At the end of the listener filters, pass the buffer (probably non-empty) along with already-read data.
If we went that route, then callback name just becomes onData() like in a network filter.

thanks for your review! l will deep into the way of read instead of peek

I like the direction this is going. I'd like to take it even further, and do a read, not a peek. Then the listener filter can drain data from the buffer if it wants to consume the data (proxy proto listener filter needs to do this), or just look at the contents of the buffer if it only wants to inspect. At the end of the listener filters, pass the buffer (probably non-empty) along with already-read data.
If we went that route, then callback name just becomes onData() like in a network filter.

+1
The current duplicate peek includes independent peek among listener filters which you are addressing.
Another potential huge part is the duplicated peek attempt within each listener filter.
The latter one even introduce the risk of 1 extra byte per peek. Not an vulnerability yet but it's can be mitigated with the buffer shared by listener filter attempt.

@ggreenway @lambdai I update the PoC, using read instead of peek. But it is the same as previous the buffer kept in the ActiveTcpSocket. In the end, the buffer will transfer to the ConnectionImpl.

I like this approach a lot! This will make the listener filter interface so much nicer.
The biggest thing left to figure out is getting the data that is read for listener filters to go as input to the Transport Socket, not directly into the ConnectionImpl read buffer.
/wait

Thanks for the feedback! I will dig into the transport socket.

I like this approach a lot! This will make the listener filter interface so much nicer.
The biggest thing left to figure out is getting the data that is read for listener filters to go as input to the Transport Socket, not directly into the ConnectionImpl read buffer.

I update the PoC. I found we have a custom BIO for the SSL socket. And I think all the transport sockets are using ioHandle to read the data.
https://github.com/envoyproxy/envoy/blob/95038feabf260c3937465951d5da603d31ea3bd4/source/extensions/transport_sockets/tls/ssl_socket.cc#L75-L76
So I inject the data into the ioHandle. When read the data from ioHandle, it will return the injected data first, then ready the data from the real socket.

@mattklein123 @ggreenway I find out a way to move the inspect data into transport, looking for your feedback when you have time, thanks in advance!

Thanks for working on this. I think this will be a great improvement. Here are my high level thoughts, riffing a bit on some of @ggreenway ideas. I think the way I would think about this at a high level would look something like:

I thought I would modify the virtual ConnectionSocket& socket() PURE; ListenerFilterCallbacks function to return something other than ConnectionSocket, which is really the root of the problem (we are giving listener filters way too much power). I think I would create a new class called something like ListenerFilterSocket with a very limited set of functions to access the underlying socket/connection.
I like your idea around having each listener filter say how much data it needs to look at, but I would go a step further and have each listener filter say whether it only needs to peek the data, or if it needs to modify it. Most listener filters only need to peek the data. In these cases, you don't actually have to read the data out of the socket and do the injection dance. You can just do the peeking outside the listener filter iteration and provide a const buffer of the peeked data up to the limit, much like each filter is doing today.
In the cases where the listener filter asks for data to modify up to a limit, we would read out into a buffer, but I think we can do this entirely outside of the IoHandle() interface, and then provide this data to listener filters in their read functions (potentially both a const and non-const version depending on what they ask for).

It's possible that we can greatly simplify this if we really only have 2 cases which are:

Peeking only.
Removal only (proxy proto)

Do we have any cases of straight up modification? If not I think we can punt this and at that point we don't need to inject at all. We either peek only read and remove only.
If we do have to modify I think we could then have the logic to send the modified data + potential peeked data to further filters, and then finally inject any extra data directly into the connection when it is created.
^ is not fully formed but this is how I would think about it personally. @ggreenway wdyt?

I'm not sure we can make removing data work without passing some read data to the next filters, because we need to read the data to know exactly how many bytes we need to consume.
But I suppose a hybrid of these approaches is to always pass data to the listener filter via peek, but give the listener filter a callback to drain some number of bytes from the connection. The implementation of that would be a read() call, but discarding the data (since it has already been inspected).
I've never really liked using PEEK because if the data arrives in multiple segments, we have to re-read data multiple times. For something like a large TLS clienthello, sent pathologically (1 byte at a time), this can add up to a substantial amount of wasted effort. But it does avoid having to muck with IoHandle at all, which is nice. I think the most important thing at this point is to get the interface to the ListenerFilter correct; the interface shouldn't depend on our underlying implementation of how we read from sockets.

I think putting the data into the IoHandle makes sense, because then none of the transport sockets need to change. But I'm not thrilled with any IoHandle being able to have data injected to it at any time. Maybe just documenting the usage and adding ASSERTs is good enough. Another idea (not sure if this would work or not) is to have an IoHandle that wraps the real IoHandle, stores the listener-filter data, returns that data from read operations, and passes all other interface calls to the real/wrapped IoHandle (including reads when the buffer is empty). I'm not sure if that will actually be an improvement or not, but it would at least mean that all of these changes aren't present for outgoing/upstream connections.

Yes, totally agree with you. It is bad we have inject but actually we don't use it. It is great idea to have wrapped IoHandle for listener. If we are going to the read to use read, I will try the way you suggested.

I think I would modify the virtual ConnectionSocket& socket() PURE; ListenerFilterCallbacks function to return something other than ConnectionSocket, which is really the root of the problem (we are giving listener filters way too much power). I think I would create a new class called something like ListenerFilterSocket with a very limited set of functions to access the underlying socket/connection.

+1, totally agree that we shouldn't expose the underlayer to the listner filter

I like your idea around having each listener filter say how much data it needs to look at, but I would go a step further and have each listener filter say whether it only needs to peek the data, or if it needs to modify it. Most listener filters only need to peek the data. In these cases, you don't actually have to read the data out of the socket and do the injection dance. You can just do the peeking outside the listener filter iteration and provide a const buffer of the peeked data up to the limit, much like each filter is doing today.
In the cases where the listener filter asks for data to modify up to a limit, we would read out into a buffer, but I think we can do this entirely outside of the IoHandle() interface, and then provide this data to listener filters in their read functions (potentially both a const and non-const version depending on what they ask for).

If the listener filter ask for data modifying, then we still need the data injection dance, right?

It's possible that we can greatly simplify this if we really only have 2 cases which are:

Peeking only.
Removal only (proxy proto)

Do we have any cases of straight up modification? If not I think we can punt this and at that point we don't need to inject at all. We either peek only read and remove only.

@lambdai mentioned the data modification in the issue, maybe he has any usecase.
If no data modification usecase, I guess you mean it back to the initial version of this PoC, peek the data, and an interface for listener to tell how much data they want to remove from the socket. Then using the read to drain the data from the socket.
But I like the current interface onData(Buffer::Instance), it is simple and clear for the listener filter writer. We needn't another interface for how much the data we want to remove.

If we do have to modify I think we could then have the logic to send the modified data + potential peeked data to further filters, and then finally inject any extra data directly into the connection when it is created.

So we still need the inject data for modifying. I'm not sure about the benefit of hybrid the peek and read here, it feels like a more complex interface and implementation.

I agree that an interface of onData(Buffer::InstancePtr) is the cleanest interface for the listener filters to use. It's the same as network filters, and that symmetry is desirable.
Another half-thought-through idea: we could create a new implementation of the Buffer::Instance interface for listener filters. The peek'd data can be accessed via any of the normal const operations, drain() would do a read() on the IoHandle to remove some data, and other non-const functions could PANIC. Although I don't know if the current Buffer API really allows this, because data is commonly accessed via linearize(), and that gives a non-const pointer to the data. @mattklein123 wdyt?

Also, we need to make a decision on whether we support mutating the data from a listener filter. I would be fine with disallowing it. I haven't heard of any use case where that would be needed. @lambdai did you have anything in mind?

I'm OK with keeping onData(...). I think the main question is whether it needs to be a mutable buffer or it could be a const buffer. Obviously if it can be const that is better and unless we have a known modification use case I would recommend not handling that right now. If someone needs modification later we can add it.
Regarding a different implementation of the buffer, that's another option, though I would probably just define a different interface entirely such as ListenerFilterBuffer or something with a more limited set of interfaces (such as only read and drain, no modification or addition).
If we think that it's cleaner and performs better to do a real read and then inject that's fine with me, I'm mostly concerned with limiting the surface area of what a listener filter can do as much as possible.

Also, we need to make a decision on whether we support mutating the data from a listener filter. I would be fine with disallowing it. I haven't heard of any use case where that would be needed. @lambdai did you have anything in mind?

The current peek() + recv() cannot mutate (drain is not mutation in this context)

Regarding the unique TLS socket that relies on io handle while other replies on buffer:
A specialized io handle works. Alternatively there is BIO_s_mem in openssl api, underlying is the buffer interface. I am wondering if tls transport socket can switch to that instead of Envoy::IoHandle+adapter.

There are a couple of points:
1. Support data modification for listener filter or not.
Looks like we don't have use cases for modification now.
2. A const interface for listener filter
All the current listener filter implement is based on the raw buffer. If we use the Buffer::Instance, the tls inspect filter needs the `linearize'.
https://github.com/envoyproxy/envoy/blob/21df5e8676a0f705709f0b3ed90fc2dbbd63cfc5/source/extensions/filters/listener/tls_inspector/tls_inspector.cc#L198
And for removing the data, we need the drain. Both linearize and drain aren't const methods. So we can't use the Buffer::Instance here.
We can have a new interface ListenerFilterBuffer, which only provides the raw buffer and remove data methods.
`class ListenerFilterBuffer {
public:
RawSlice* data() const;
void remove() const;
private:
// a single slice since the Peek and Listener filter need a raw buffer
Slice slice_;
} ;`
The question is the raw buffer actually is mutable. And the remove data method is strange also, it sounds like a mutable method. And if we are using the Peek, actually it isn't a remove the buffer, it is a read on the socket.
How about another option is ListenerFilterSocket, which provides two methods, read and remove.
`class ListenerFilterSocket {
public:
// Peek from the socket through the IoHandle
IoCallUint64Result read(void* buffer, size_t length);
// read from the socket
void drain(size_t length);
}`
although it has to copy the data into its own buffer, but it should be ok since the current listeners have to copy the data into their own buffer also. It will be a clean interface not allow the listener filter to change the data.
3. Peek or Read
If using the ListenerFilterBuffer  way, feel like the read is better. If using the  ListenerFilterSocket,  feel like the peek is better.
ListenerFilterBuffer + read has to deal with injection. It may save some of the data copy in the listener filter.
ListenerFilterSocket + peek has better const interface. It still needs the current data copy the listener filter already has.  And this should be able to use read to support the modification in the future.
@ggreenway @mattklein123 WDYT?

Regarding the unique TLS socket that relies on io handle while other replies on buffer:

Maybe I misunderstand your point. It looks like other filter using the ioHandle also https://github.com/envoyproxy/envoy/blob/21df5e8676a0f705709f0b3ed90fc2dbbd63cfc5/source/extensions/filters/listener/proxy_protocol/proxy_protocol.cc#L307

At first glance ListenerFilterSocket + peek seems cleaner to me, but I'm not sure if there are any large performance implications of that compared to what we are currently doing?

At first glance ListenerFilterSocket + peek seems cleaner to me, but I'm not sure if there are any large performance implications of that compared to what we are currently doing?

I think it is same with current. ListenerFilterSocket is just a wrap of Iohandle. Basically we replace this
https://github.com/envoyproxy/envoy/blob/b9eada56a2ce8203c47796f80b64a7822616fa09/source/extensions/filters/listener/tls_inspector/tls_inspector.cc#L168
to ListenerFilterSocket::read. There is no extra data copy added here.

Do you think the other option would provide significantly better performance? If we are redoing things I suppose we have the opportunity to make the performance better?

Do you think the other option would provide significantly better performance? If we are redoing things I suppose we have the opportunity to make the performance better?

I didn't see a significant performance improvement. The current PoC is using read + Buffer:Instance, it only replaces the duplicated peek to one read. But we only have duplicated peek when we have multiple listener filters. I guess there won't be too much listener filters for each listener. so it won't be performance improvement.  ListenerFilterBuffer + read should be same.

The performance may differ in the pathological case of a client sending 1 byte at a time, so the peek has to cover more data each time. But in the common case, I think it will be an unmeasurable performance difference.
If performance of the edge case becomes an issue, we could put other guards in place (maximum number of peeks per connection, for instance), since the bad cases can only happen intentionally, eg from an attack.
So I think performance of either implementation should be about the same for the cases we care about.

@mattklein123 @ggreenway, in the end, I chose a mixed solution. Since I didn't want the ListenerFilterSocket to be a simple wrap of iohandle::peek, then the filter still needs to deal with the return of iohandle::peek. That isn't part I don't want to expose to the filter. So I turned to ListenerFilterBuffer, it will peek the data from socket, and store the data into its own buffer. But the filter won't to allow to access the raw buffer directly, instead of copy the data out of the buffer. Then it matches the goal of a const buffer.

/wait
Not finish yet. The unit tests for ListenerFilterBuffer and ActiveTcpSocket are added. The tests to be done are: inspect filter, HTTP inspect filter, proxy filter and integration test.

/wait
Almost done, only left http inspect filter.

@ggreenway thanks for the review!
The windows CI failed, I can't find out the reason from the log, I probably need to setup a windows env.

@davinci26 There is still a problem with Windows.
The test //test/extensions/filters/listener/proxy_protocol:proxy_protocol_test is timeout.
I found we unregister the read event after peek data by IoSocketHandleImpl::recv()
https://github.com/envoyproxy/envoy/blob/f041df430c04d7bfd8d1954e506fe150a2443a05/source/common/event/file_event_impl.cc#L170-L171
we won't register the read event again except IoSocketHandleImpl::recv() returns EAGAIN
https://github.com/envoyproxy/envoy/blob/f041df430c04d7bfd8d1954e506fe150a2443a05/source/common/network/io_socket_handle_impl.cc#L511-L513
But the test is still waiting for a read event. In the end, the test is timeout.
I try to change the check to if (io_result.wouldBlock() && file_event_) to if (file_event_), it still doesn't work, since the data peek won't consume the data, so the event keeping triggered. Then the test timeout due to too busy I guess.

Waiting for @davinci26 verify the issue https://github.com/envoyproxy/envoy/pull/17395#issuecomment-927788120
If the issue verify, it may leads some design changed.
one of solution is https://github.com/envoyproxy/envoy/pull/18305
But it would be great there is more easy solution.

The current implementation is problematic on Windows but I think we can improve it in such a way that it would be better on both linux and Windows.
Instead of doing MSG_PEEK we could have a single listener_filter_buffer_ that actually reads from the socket. Then we store all the bytes we read to a buffer.
When the listener_filter_buffer_ gets new data it invokes multiple ListenerFilterBufferOnDataCb from different filters. And passes the buffer to them.
Finally the listener_filter_buffer_ is passed along to the connection which will concatenate it with the buffer it gets from subsequent reads.
On Windows that will not break the polling mechanism and on linux it will lead to fewer read calls.
WDYT @soulxu @ggreenway ?

@davinci26 thanks for your feedback!
Sounds like we have to back the way using read instead of peek for ListenerFilterBuffer, then inject the buffer data back to iohandle. Or another solution is we just using read instead of peek for Windows https://github.com/envoyproxy/envoy/pull/18305
Looking for @ggreenway @mattklein123 's opinion.

Sorry @davinci26 @soulxu is there a summary of the issue? I think at one point we stopped using PEEK and did the read and move? I can't remember the sequence of changes in this patch at this point.

@soulxu correctly pointed out that when we use MSG_PEEK the windows polling model breaks. This happens because we don't clear the buffer. This can lead to connections hanging if tls_inspector/proxy_protocol need to call read more than once.
I did not follow the progress of the PR from the beginning so I am not sure if there was any point that MSG_PEEK eliminated. I think that would be an overall win both for Linux and windows. Since on Windows we won't hang. Also, on Linux we would have reduced system calls and hence better performance.
I am not sure how it was originally implemented but I think we could just read once and pass the buffer to all the inspectors (some inspectors, i.e. proxy_protocol might also drain the buffer so I dont think it can be read only) and finally pass the remaining buffer to the connection implementation.

Let me summary the history of this PR.
In the beginning, we have tried to use read instead of peek. But it has to has some magic to inject the data back for network filters. Then I implement a way to inject the ListenerFilterBuffer's data into IoSocketHandleImpl. When the network filter reads the data, the IoSocketHandleImpl will return previous injected data.
But after our discussion, we don't have any use-case for listener filter to modify the data. So there isn't any benefit bring to us, and it introduces those complex of injecting data into IoSocketHandleImpl. Then we decide to continue using the peek. (the summary https://github.com/envoyproxy/envoy/pull/17395#issuecomment-903417020)
Unfortunately, we found the Windows doesn't support the edge-triggered event, so it implements an emulated-edge-triggered event by level-triggered event. But it doesn't work well with MSG_PEEK, the root reason is MSG_PEEK won't consume the data in the socket, so the Windows will keep triggering the Read event. (here is detail of the issue https://github.com/envoyproxy/envoy/pull/17395#issuecomment-927788120)
Currently, in my mind, we have two options here:

Back to the previous design, using read inside the ListenerFilterBufferImpl. After passing all the listener filters, inject the ListenerFilterBufferImpl's data into IoSocketHandleImpl. Then network filters still can read those data which read by listener filter. Then we should reject anyone uses MSG_PEEK through IoSocketHandle interface.
Keep using the MSG_PEEK inside ListenerFilterBufferImpl, but we mimic the MSG_PEEK inside IoSocketHandleImpl by read for Windows. (I have a poc https://github.com/envoyproxy/envoy/pull/18305).

Two options are pretty similar, both have similar magic for reading injected data inside IoSocketHandleImpl. Just one is done at Envoy level, another one is done only at IoSocketHandle level for Windows. The second option may have a little more complex on mimic peek behavior.

Two options are pretty similar, both have similar magic for reading injected data inside IoSocketHandleImpl. Just one is done at Envoy level, another one is done only at IoSocketHandle level for Windows. The second option may have a little more complex on mimic peek behavior.

I have a mild preference for emulating this on Windows since it keeps the main Envoy code cleaner, especially since I think ultimately Windows will get real edge triggered behavior and we can remove it?

Why is this is a new problem on windows? The old code also did MSG_PEEK, it was just spread around in all of the listener filters; now it's consolidated.
@davinci26 we investigated not doing the MSG_PEEK and inserting the data into the connection, but decided that it was overall much more complex, so I'd prefer we don't go that route.

Why is this is a new problem on windows? The old code also did MSG_PEEK, it was just spread around in all of the listener filters; now it's consolidated.

It is due to most of time the client can send the initial data one time. If the client is slow enough, then the problem is exposed.
The current implementation, the first MSG_PEEK is done before initiate the file event. So the initial data was received by two MSG_PEEK recv, then there is no problem. This PR will initiate the file event first, so if there are more than one recv call to get enough data, then the problem is exposed.

Why is this is a new problem on windows? The old code also did MSG_PEEK, it was just spread around in all of the listener filters; now it's consolidated.

For the case like tls inspect, I think it is due to the most of the time the client is fast enough to send tls handshake by one shot. If the client is slow enough, the problem will be exposed.
For another case like proxy filter, it drains the socket data after peeking data, then won't make the Windows keep trigger the read event since the data isn't consumed.

This PR is really helpful in implementing https://github.com/envoyproxy/envoy/issues/18035 since it's much easier to wrap TransportSocket into the buffer than the network socket.

This PR is really helpful in implementing #18035 since it's much easier to wrap TransportSocket into the buffer than the network socket.

That is great! I will try my best to push forward this PR.

/retest

@davinci26 @ggreenway I got the windows CI works. appreciate it if you can take a look at it again.

If we need to support keeping the PEEK data in the io_handle to support windows, I don't think it makes any sense to write that code and then not use it on other platforms; we'd be better off just maintaining a single code path. @mattklein123 how does that sound to you?

Sorry I've lost the thread on this. I thought we were going to try to hide this complexity inside the Windows impl to keep the rest of the code simpler, and also with the expectation that later Windows would fix the OS to handle this case better? If we feel that it's a better direction to just handle this universally for all platforms, that's fine with me.

On Windows we have the plan to light up this backend https://github.com/libevent/libevent/issues/1216 but it will be on newer versions so having this for older version should be worth it imo.
I would be +1 to split this PR into two, one to implement the peek mechanism and one to implement the listener peek filter

@davinci26 and @soulxu is there a way to put the emulation code into windows-only code? If we can do that, I think I'd prefer that route. If not, I'd say break it into two PRs like @davinci26 suggested.

and thanks all your guys feedback again :)

is there a way to put the emulation code into windows-only code?

Fwiw I thought this was the plan. This is what I would prefer.

yea, I started from IoSocketHandleImpl, since there are some windows platforms specific code. but in the middle of this version code, I also feel something is not right since peek emulation code is growing and I have to add a lot of platform specific code to make the test works. But anyway, I still finished this version since already spend a lot of time on it, may be worth seeing what it look likes in the end.

/wait
Try to merge with @davinci26

/wait

/wait
just merge with main.

@soulxu what is the status of this PR? Is it ready for review or are we still waiting on the Windows Peek PR? Thank you.
/wait-any

@soulxu what is the status of this PR? Is it ready for review or are we still waiting on the Windows Peek PR? Thank you.
/wait-any

It is still waiting for the Windows Peek PR. I just keep merging with that PR, to test the CI passed.

still waiting for https://github.com/envoyproxy/envoy/pull/19291 merged

/wait
Just merge with the Windows PR to test

/retest

@mattklein123 @ggreenway I finally got the Windows PR merged (https://github.com/envoyproxy/envoy/pull/19291)
It is time to revisit this again :) Appreciate it if you can take look at this again.
This is going to unlock few things like https://github.com/envoyproxy/envoy/pull/18951 and nested filter chain

Awesome. Can you take a look at Windows CI and then we can look again?
/wait

/retest

Awesome. Can you take a look at Windows CI and then we can look again?
/wait

It works now :)

/wait
a fuzzed issue detected, working on it https://github.com/envoyproxy/envoy/pull/19748

/wait

Sorry, it is still waiting for https://github.com/envoyproxy/envoy/pull/19777 merge.

Sorry, it is still waiting for #19777 merge.

Go ahead and merge main, let's see where this stands now.

Sorry, it is still waiting for #19777 merge.

Go ahead and merge main, let's see where this stands now.

thanks! let me do a merge.

What's the status of this PR? Is it ready for review?
/wait-any

What's the status of this PR? Is it ready for review?
/wait-any

Yea, it is ready for review now, thanks!

just keeping merge with main

@mattklein123 @ggreenway let me know how can I help on moving this forward, also understand this pr isn't easy to review and merge. Thanks in advance!

Apologies I will make time to review it today.

/wait
working on fuzzing test

The fuzz test is ready

Thanks this LGTM. I would definitely like another quick pass from either @ggreenway or @antoniovicente given the high risk nature of this change. Very excited to see this land I think this is a great improvement.
/wait

Thanks! Also agree this is high risk.

only merge with main

/retest

/retest

This is looking great! The approach looks good.
/wait

thanks for the review!

/wait still left fuzz test.

/wait

/retest

/wait

/wait
Working on rewrite listener filter fuzz test which doesn't work with current implement anymore

just merge and resolve conflict.

@ggreenway @yanavlasov gently ping :)

/retest

@soulxu, @ggreenway is out this week. We will get this merged next week. Thank you for your patience.

@soulxu, @ggreenway is out this week. We will get this merged next week. Thank you for your patience.

got it, thanks!

@ggreenway @yanavlasov gentle ping :)
@soulxu DCO seems to have some issue, can you PTAL?

I can just force merge as all the commits have sign off. Thank you!

This looks great! Thanks for the all the work you've put into this!

@ggreenway @mattklein123 thanks for the guidance again!

I thought it might be due to my upgrade to Bazel 5.1.0 but I'm now getting a MacOS M1 aarch compilation error on a file changed by this PR
ERROR: /Users/moderation/Applications/envoyproxy/envoy/source/extensions/filters/listener/tls_inspector/BUILD:15:17: Compiling source/extensions/filters/listener/tls_inspector/tls_inspector.cc failed: (Aborted): wrapped_clang_pp failed: error executing command external/local_config_cc/wrapped_clang_pp '-D_FORTIFY_SOURCE=1' -fstack-protector -fcolor-diagnostics -Wall -Wthread-safety -Wself-assign -fno-omit-frame-pointer -g0 -O2 -DNDEBUG ... (remaining 141 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox
In file included from source/extensions/filters/listener/tls_inspector/tls_inspector.cc:1:
./source/extensions/filters/listener/tls_inspector/tls_inspector.h:83:12: error: virtual function 'maxReadBytes' has a different return type ('uint64_t' (aka 'unsigned long long')) than the function it overrides (which has return type 'size_t' (aka 'unsigned long'))
  uint64_t maxReadBytes() const override { return config_->maxClientHelloSize(); }
  ~~~~~~~~ ^
./envoy/network/filter.h:347:18: note: overridden virtual function is here
  virtual size_t maxReadBytes() const PURE;
          ~~~~~~ ^
1 error generated.
Error in child process '/usr/bin/xcrun'. 1
Target //source/exe:envoy-static.stripped failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 407.055s, Critical Path: 50.81s
INFO: 4027 processes: 1165 internal, 2862 darwin-sandbox.
 Thanks for pointing this out!

@kyessenov not surprised there would be a regression here. Feel free to revert until we sort it out. Thank you. cc @soulxu

I looked at the PR, and it didn't change cb.socket().close() statement after unsuccessful parse so I'm puzzled how this worked before. Will try to compare more.

Given this is something that istio depends on, can you create an integration test that captures the failure? Both to help diagnose this issue, and to prevent future regressions. It may be a failure when there are two listener filters; i don't think we have much coverage for that situation at all.

I can confirm it's this PR for the test case above. It's just one filter I think - before the socket close wasn't doing anything somehow.

I think I understand what happened - prior version was signaling ParseError on read errors. The refactor conflated the actual parse error and continued closing the socket.

Although this looks like a one liner fix to remove socket.close() and return Continue, let's revert, and an integration test for HTTP (which is missing) and then validate again.

@kyessenov Thanks for diagnosing this, let me fix it.


[ISSUE #3450]Add brpc example

#3450

Make sure that:

[x] You have read the contribution guidelines.
[x] You submit test cases (unit or integration tests) that back your changes.
[x] Your local test passed ./mvnw clean install -Dmaven.javadoc.skip=true.

Codecov Report

Merging #4309 (f103740) into master (c2dd940) will decrease coverage by 0.22%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##             master    #4309      +/-   ##
============================================
- Coverage     68.30%   68.08%   -0.23%
+ Complexity     7467     7448      -19
============================================
  Files          1015     1015
  Lines         28729    28725       -4
  Branches       2575     2570       -5
============================================
- Hits          19624    19556      -68
- Misses         7572     7639      +67
+ Partials       1533     1530       -3

Impacted Files
Coverage Δ

...client/brpc/BrpcContextRefreshedEventListener.java
68.57% <0.00%> (-0.35%)
:arrow_down:

...che/shenyu/plugin/brpc/proxy/BrpcProxyService.java
2.04% <0.00%> (ø)

.../client/server/consul/ShenyuConsulConfigWatch.java
58.33% <0.00%> (-27.09%)
:arrow_down:

...henyu/plugin/grpc/resolver/ShenyuNameResolver.java
44.68% <0.00%> (-19.15%)
:arrow_down:

...controller/ShenyuClientHttpRegistryController.java
63.63% <0.00%> (-18.19%)
:arrow_down:

...ache/shenyu/plugin/grpc/cache/GrpcClientCache.java
68.75% <0.00%> (-12.50%)
:arrow_down:

...ruptor/RegisterClientServerDisruptorPublisher.java
55.55% <0.00%> (-11.12%)
:arrow_down:

...he/shenyu/loadbalancer/spi/RandomLoadBalancer.java
81.81% <0.00%> (-9.10%)
:arrow_down:

...va/org/apache/shenyu/plugin/cache/CachePlugin.java
81.57% <0.00%> (-7.90%)
:arrow_down:

...henyu/admin/service/impl/UpstreamCheckService.java
55.33% <0.00%> (-7.34%)
:arrow_down:

... and 7 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Python 3 support

Should fix
https://github.com/python-constraint/python-constraint/issues/3
https://github.com/python-constraint/python-constraint/issues/8


[CUTLASS] Support conv2d activation fusion

@comaniac @Laurawly  @hwu36 @manishucsd @junrushao1994 @vinx13
This adds conv2d + activation fusion support for bias_add, relu, and fp32 sigmoid. I have pending PRs at the cutlass repo https://github.com/NVIDIA/cutlass/pull/378, https://github.com/NVIDIA/cutlass/pull/379 which will enable fp16 sigmoid, silu, and hardswish fusion as well.
End to end results
All code, models, and nvprof dump etc are available at https://github.com/masahi/tvm-cutlass-eval
All numbers in milli sec, using  fp16 models with fp16 accumulation running on tensorcore,  measured on RTX 3070

Model name
Input size
CUTLASS (no fusion)
CUTLASS with fusion
cuDNN
AutoTVM TensorCore
TensorRT fp16

resnet50
(8, 3, 224, 224)
3.47
3.16
4.08
4.14
2.53

efficientnet_v2
(8, 3, 224, 224)
8.18
8.58 (7.80 if use fast_math)
14.0
13.2
5.25

DETR-R50
(8, 3, 800, 750)
61.06
57.19
68.4
80.5
NA

deeplabv3_mobilenet_v3_large
(8, 3, 512, 512)
10.3
9.16
17.5
15.9
19.2 (?)

YOLOv5l
(8, 3, 512, 512)
33.6
25.4 (24.2 if use fast_math)
34.8
N/A
N/A

Observations

First of all, the cutlass results above are still suboptimal because residual blocks are not completely fused due to a limitation in cutlass discussed in https://github.com/NVIDIA/cutlass/discussions/347. So if we look at the partitioned subgraph of resnet50, for example, the main function looks like this (see the whole model at https://github.com/masahi/tvm-cutlass-eval/blob/master/resnet50/resnet50_partition.txt):

  ...
  %4 = @tvmgen_default_cutlass_main_3(%3, meta[relay.Constant][2] /* ty=Tensor[(64, 1, 1, 64), float16] */, meta[relay.Constant][3] /* ty=Tensor[(64), float16] */) /* ty=Tensor[(8, 56, 56, 64), float16] */;
  %5 = @tvmgen_default_cutlass_main_6(%4, meta[relay.Constant][4] /* ty=Tensor[(64, 3, 3, 64), float16] */, meta[relay.Constant][5] /* ty=Tensor[(64), float16] */) /* ty=Tensor[(8, 56, 56, 64), float16] */;
  %6 = @tvmgen_default_cutlass_main_9(%5, meta[relay.Constant][6] /* ty=Tensor[(256, 1, 1, 64), float16] */, meta[relay.Constant][7] /* ty=Tensor[(256), float16] */) /* ty=Tensor[(8, 56, 56, 256), float16] */;
  %7 = @tvmgen_default_cutlass_main_12(%3, meta[relay.Constant][8] /* ty=Tensor[(256, 1, 1, 64), float16] */, meta[relay.Constant][9] /* ty=Tensor[(256), float16] */) /* ty=Tensor[(8, 56, 56, 256), float16] */;
  %8 = add(%6, %7) /* ty=Tensor[(8, 56, 56, 256), float16] */;
  %9 = nn.relu(%8) /* ty=Tensor[(8, 56, 56, 256), float16] */;
  %10 = @tvmgen_default_cutlass_main_15(%9, meta[relay.Constant][10] /* ty=Tensor[(64, 1, 1, 256), float16] */, meta[relay.Constant][11] /* ty=Tensor[(64), float16] */) /* ty=Tensor[(8, 56, 56, 64), float16] */;
  %11 = @tvmgen_default_cutlass_main_18(%10, meta[relay.Constant][12] /* ty=Tensor[(64, 3, 3, 64), float16] */, meta[relay.Constant][13] /* ty=Tensor[(64), float16] */) /* ty=Tensor[(8, 56, 56, 64), float16] */;
  %12 = @tvmgen_default_cutlass_main_21(%11, meta[relay.Constant][14] /* ty=Tensor[(256, 1, 1, 64), float16] */, meta[relay.Constant][15] /* ty=Tensor[(256), float16] */) /* ty=Tensor[(8, 56, 56, 256), float16] */;
  %13 = add(%12, %9) /* ty=Tensor[(8, 56, 56, 256), float16] */;
  %14 = nn.relu(%13) /* ty=Tensor[(8, 56, 56, 256), float16] */;
  ...

The intermediate adds are the element-wise addition in the residual block, which can in principle be fused with the preceding conv2d. nvprof output shows that these unfused ops are taking more than 20% of e2e time. If we fuse them, I believe we could approach TRT-level performance.

Despite its suboptimality, enabling activation fusion still results in a good speedup.
Cutlass results are always better than cuDNN ones, even if fusion is disabled. The difference is especially large for deeplabv3 and efficientnetv2, both of which use a lot of depthwise conv2d. The nvprof output shows that cuDNN is spending a lot of time doing layout transform (see for example). Maybe our use of cuDNN is not optimal in terms of API usage and kernel selections. nvprof dumps for cuDNN-runs are available in the repository, if someone wants to compare against cutlass nvprof dumps.
Fusing sigmoid activation into cutlass conv2d results in worse runtime (see efficientnet_v2 row), which I found odd since TVM should be generating essentially the same sigmoid kernel in the unfused case. The fast-math option, discussed in https://github.com/NVIDIA/cutlass/pull/378#issuecomment-992143861, makes it a lot better at the cost of slight accuracy loss.
efficientnetv2 and deeplabv3 use a lot of depthwise conv2d, which is not currently offloaded to cutlass. They are actually the bottleneck in the results above, which can be observed by looking at nvprof dumps. Using AutoTVM for them should help bring down e2e time further.

Known issues

Kernel profiling time is extremely slow. For DETR-R50, I had to wait like 20 min. More discussion in https://github.com/apache/tvm/pull/9737#issuecomment-993909962
Trying to fuse silu activation in YOLOv5l results in a strange type inference error during MergeComposite. So the YOLOv5l result above doesn't use silu fusion. (Fixed a bug in type relation)
I also tried MaskRCNN, but currently it results in a strange output (no detection) when cutlass BYOC is used. The same issue also applies to cuDNN offload, but TVM-native kernels (target = "cuda") work correctly.

@comaniac @Laurawly Ready to merge?

Thanks @masahi


Blank line between class methods

Is there a rule that will enforce a blank line between class methods?
<?php declare(strict_types=1);

class Test
{
    public function test()
    {
    }
    protected function another()
    {
    }
    private function again()
    {
    }
}

changes to
<?php declare(strict_types=1);

class Test
{
    public function test()
    {
    }

    protected function another()
    {
    }

    private function again()
    {
    }
}

Hi and thanks for your question,
The rule you are looking for is called class_attributes_separation, please check the docs for details about it here


Updated Readme.md [Typo]

Please check if the PR fulfills these requirements:

[ ] The PR title and message describe the solution briefly.
[.] Documentation has been updated/added if relevant

What is the current behavior?

Grammar Mistake
What is the new behavior?
Corrected spelling
Other information?

@vamsireddy22 thanks for the change but unfortunately a stray white space or spellings will not be considered as a valid contribution. This PR will be marked invalid. :)


pr-template added to new docs root directory

Zero To Mastery - Webblocks 2022 🎉🎉
👉 Please remove the below and replace with your own values, leaving the headers where they are. 👈
PR contains:

Considerations for a Pull Request template to assist in information gathering from the participants?

This does not have to be implemented, and can be ignored. Discussion needed with maintainer team at large maybe?
Breaking changes

docs directory added to root
pull_request_template.md file with markdown content detailed added to docs folder

Screenshots

Template preview:

@sophiabrandt can I ask you to review and consider merging this PR? Same as the Canvas PR template, just with an updated heading.


Split and index docs at download time

Split the docs in main entries and index the docs at download time by changing the path value in the index.json to the markdown text.
It'd solve #1,  #8 and #14.

closed with #21


Improve BaseIF

This will help when troubleshooting and detect when an interface is down. It should be included by default.
Type of change
Please delete options that are not relevant.

[ x ] Refactor/improvement

Checklist

[ ] My commit message is conventional
[ ] I have run pre-commit on all files before creating the PR
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have added tests that prove my fix is effective or that my feature works
[ x ] New and existing unit tests pass locally with my changes
[ ] I have checked my code and corrected any misspellings

Hello @bran1501, I will open a new pr with the changes you propose to our develop branch.

#1123


[appsync] support for serverless aurora rds data source missing

It seems that the support for aurora serverless rds data source is missing.
If so please take this as a feature request.
If not please point me to the documentation.
Thank you

Any updates on this?
It looks like Terraform has already support for it here
Urgently needed

It looks like Terraform has already support for it here

That link is to the AWS Go SDK (vendored into terraform-provider-aws).  Are you sure this feature is exposed through the Terraform provider currently? I so not see it in https://github.com/terraform-providers/terraform-provider-aws/blob/master/aws/resource_aws_appsync_datasource.go#L15.

@lukehoban I thought pulumi is using the terraform go sdk providers, doesn't it?
I am fighting an uphill battle in our enterprise against the proponents of the cloud formation based solutions, but its hard to argue when features are taking month and month to arrive, if at all.
I hope there will be some progress. Is there anyway to contribute to speed this up?

@lukehoban I thought pulumi is using the terraform go sdk providers, doesn't it?

The @pulumi/aws package builds on the terraform-provider-aws Terraform provider, which in turn builds on top of the AWS Go SDK.
The code you linked is code within the AWS Go SDK, which does have support for Aurora appsync databases.  That code is not yet called from anything in the Terraform AWS provider - which does not yet support these.  Once support is added to the Terraform AWS provider, it will be available in @pulumi/aws within a couple days.
To contribute here - I'd suggest opening an issue and/or PR in https://github.com/terraform-providers/terraform-provider-aws.  This PR (which added initial support for AppSync Datasources) gives a sense of what is needed: https://github.com/terraform-providers/terraform-provider-aws/pull/2758.

Hi @BerndWessels,
As this is not available in the upstream terraform provider, I just opened an issue in that repository that we will use to track the progression of this work.
Once that has been implemented upstream, then Pulumi will be able to take advantage of this. I am going to close this issue out as a duplicate of the upstream issue. Please feel free to +1 that issue :)
https://github.com/terraform-providers/terraform-provider-aws/issues/8870
Thanks again
Paul


Update de_de.json

Fixing a few minor grammatical errors (capitalization, wording)
Fixed wrong translation for a plate of chicken

thanks


Compile for Qt using MinGW

I tried to compile and follow build instructions of resvg but I get the error when using cargo:
error: linker link.exe not found | = note: The system cannot find the file specified. (os error 2) note: the msvc targets depend on the msvc linker but link.exe was not found note: please ensure that VS 2013, VS 2015 or VS 2017 was installed with the Visual C++ option error: aborting due to previous error
using MinGW 73 64 bit with Qt5.12.1 on Windows 10

Can you list the commands that you used?

yes of course
set PATH=C:\Qt\Qt5.12.1\5.12.1\mingw73_64\bin;C:\Qt\Qt5.12.1\Tools\mingw730_64\bin;%PATH% set QT_DIR=C:\Qt\Qt5.12.1\5.12.1\mingw73_64 cargo.exe build --release --features "qt-backend"
by the way my cargo is in the PATH no need to set it twice

Maybe you should enable the GNU toolchain first rustup default stable-x86_64-pc-windows-gnu?

by the way my cargo is in the PATH no need to set it twice

The PATH modification is optional, so this is mentioned just in case.

ok it worked, thanks :)


(fixes #29) IE: Invalid calling object

Thanks to @brianleipprandt for figuring out the problem. This pull request is a slight improvement that fixes issues with browsers that do not have atob (such as IE9).

Sorry for the delay man :(

and thank you very much for the patch

published as v2.0.3

No worries. Thanks! =)


Pressing DOWN arrow should not jump over the first selectbox entry

The title says it all, maybe add an empty entry to the top of the filter and search selectboxes to be jumped over.
But don't add this empty entry to the history.

But don't add this empty entry to the history.
this is not solved yet

Should have been fixed in this commit: 4cbbdc3


[Fix] Avoid setting Switch.connection as None, and handled socket errors

This PR is needed for this this issue #26 on flow_manager:
Description of the change

Avoided setting switch.connection as None, since the protocol data associated with the connection is needed to derive the correct class to serialize some FlowMods. This was impacting when the connection was lost. Now, instead of completely nuking the connection (and losing protocol data info associated with it), it only nukes the socket.
Bubbled up (OSError, SocketError) from Connection.send to handle socket transmission errors that can happen to propagate back an event via this event kytos/core.openflow.connection.error, flow_manager (or any other NApp putting messages into msg_out queue) is supposed to listen for it (I'll open a new issue to cover this, it's also related to propagating and handling event errors), just so this can also be bubbled up to clients via events, something like:

   @listen_to("kytos/core.openflow.connection.error")
   def on_openflow_connection_error(self, event):
       log.info(f"got error content {event.content}")
       log.info(f"got error xid {event.message.header.xid}")

Documented on docs/developer/listened_events.rst some of the producers and consumers of certain connection events to facilitate understanding, who publishes and who subscribes.

The consistency checks also partly solves the error handling part if flow mods are asked to be forced, but since we also need NApps to react as soon whatever packet that was supposed to be sent to switch fails, we need to be able to be able to propagate that. So, this coupled with handling and propagating OFPT_ERRORs should help out us to achieve full error propagation.

@viniarck thank you for the PR. Pretty neat solution and very well documented. Thanks for updating the developer guide on listen events.

Appreciated your review :+1: . I'll merge this shortly


Calculator should evaluate multiple basic math operations

Use case:

user starts calculator app
user enters a math expression (For instance 5 * 4 + 12 - 5 / 1)
calculator evaluates the expression
user closes app

Implemented in #22


Use Pre-Trained Tensorflow Models with a GPU in SAP AI Core

Tutorials: https://developers.sap.com/tutorials/ai-core-tensorflow-byod.html
I'm using the Free Tier. I'm fairly certain that this tutorial cannot be completed with GPU nodes
I haven't been able to complete this tutorial because there are multiple issues.
The first issue appears when trying to register the artifact. It is my understanding (after having tried it multiple times) that the application must be created before registering the model.
Here are my proposed changes

Order
Current
Proposed

1
Step 1
Step 1

2
Step 2
Step 2

3
Step 3
Step 3

4
Step 4
Step 6

5
Step 5
Step 7

6
Step 6
Step 4

7
Step 7
Step 5

8
Step 8
Step 8

9
Step 9
Step 9

Additional comments on current steps

Step 6 says to set resourcePlan to infer.s which will enable GPU. Free tier only allows for starter resourcePlan. Maybe add this to the prerequisites?
Step 7 has a typo:  executables.ai.sap.com/name: "serve-executuable" -> executables.ai.sap.com/name: "serve-executable".

Step 5: command docker build -t ends in error: ERROR: failed to solve: process "/bin/sh -c pip3 install -r requirements.txt" did not complete successfully: exit code: 1

I suggest leaving requirements.txt without versions (haven't tested it yet)
# requirements.txt
scikit-learn
Flask
gunicorn

And using flag '--ignore-installed' in dockerfile to avoid conflicts with base image packages
RUN pip3 install --ignore-installed -r requirements.txt

Step 3: Error: Could not connect to the endpoint URL

Attribute endpoint is the problem. Should be s3. instead of s3-. Source

Original: endpoint = "s3-eu-central-1.amazonaws.com", # Change this
Correct: endpoint = "s3.eu-central-1.amazonaws.com", # Change this

Step 5: using latest tensor flow Docker image results in conflict while loading model.h5.

I suggest changing the Dockerfile to this:
FROM tensorflow/tensorflow:2.10.0-gpu-jupyter

Hi @jmmargo, @gianfranco-s,
The latest version of the tutorial seems to be working fine.
Regards.
(cc: @dhrubpaul)


Minor cleanup work

I got a bit sidetracked with refactoring apollo-link. I was having a hard time groking 1000 line stateful components so I started doing some cleanup work wherever I saw low hanging fruit.
I'm working my way into the editor codebase a bit to make it a bit easier to swap out the fetcher.
Here are the granular commits that I have working so far.

@timsuchanek implemented changes.

give me a sec. i'm rebasing

Thanks for the work @DavidJFelix !

No problem. Thanks for the collab. You'll probably see some more cleanup from me in the near future.


Regex not compatible on lower versions of IOS / Safari

Initial checklist

[X] I read the support docs
[X] I read the contributing guide
[X] I agree to follow the code of conduct
[X] I searched issues and discussions and couldn’t find anything (or linked relevant results below)

Affected package
remark-gfm4.0.0
Steps to reproduce
[/(?<=^|\s|\p{P}|\p{S})([-.\w+]+)@([-\w]+(?:.[-\w]+)+)/gu, findEmail]
Create JS crash on Safari / IOS(15.3)
Actual behavior
got error "Invalid regular expression: invalid group specifier name"
Expected behavior
not got error
Runtime
No response
Package manager
No response
Operating system
No response
Build and bundle tools
No response

Duplicate of https://github.com/syntax-tree/mdast-util-gfm-autolink-literal/issues/10