"Please refer to Azure Active Directory Data Locations" Power BI Report Broken

The link [Please refer to Azure Active Directory Data Locations.](https://aka.ms/aaddatamap) loads a Power BI report, but when you click the drop down menu to 'Select your company geo' an error comes up saying 'can't display this visual'.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 4b59be58-a0a1-08a0-bfed-d8958ba58e56
Version Independent ID: 4b59be58-a0a1-08a0-bfed-d8958ba58e56
Content: Data Residency for Other Microsoft 365 Services - Microsoft 365 Enterprise
Content Source: microsoft-365/enterprise/m365-dr-workload-other.md
Service: microsoft-365-enterprise
GitHub Login: @kelleyvice-msft
Microsoft Alias: kvice

Someone's updated the page :) It did go to a Power Bi report, now it just loops to the same page - why would https://aka.ms/aaddatamap mentioned on this page with an external page link, now link back to this page?

@msbemba Thank you very much for the contribution and sharing this explanation. @adamfowlerit Hope this comment is helpful for you. If you see a documentation update is required, please feel free to open an issue for the same. We proceed here to close it. Thanks for taking out some time to open the issue. Appreciate and encourage you to do the same in future also.


Certificate is not valid, Common Name (CN) TRAEFIK DEFAULT CERT

I'm trying to get SSL to work on https://code.dorianmarie.com
I'm getting Common Name (CN)	TRAEFIK DEFAULT CERT
I'm using this config/deploy.yml:
service: code
image: dorianmariefr/code
servers:
  web:
    hosts:
      - <IP_ADDRESS>
    labels:
      traefik.http.routers.blog.entrypoints: websecure
      traefik.http.routers.blog.rule: Host(`code.dorianmarie.com`)
      traefik.http.routers.blog.tls.certresolver: letsencrypt

registry:
  username: dorianmariefr
  password:
    - KAMAL_REGISTRY_PASSWORD
env:
  secret:
    - RAILS_MASTER_KEY
traefik:
  options:
    publish:
      - "443:443"
    volume:
      - "/letsencrypt/acme.json:/letsencrypt/acme.json"
  args:
    entryPoints.web.address: ":80"
    entryPoints.websecure.address: ":443"
    entryPoints.web.http.redirections.entryPoint.to: websecure
    entryPoints.web.http.redirections.entryPoint.scheme: https
    entryPoints.web.http.redirections.entrypoint.permanent: true
    certificatesResolvers.letsencrypt.acme.email<EMAIL_ADDRESS>    certificatesResolvers.letsencrypt.acme.storage: "/letsencrypt/acme.json"
    certificatesResolvers.letsencrypt.acme.httpchallenge: true
    certificatesResolvers.letsencrypt.acme.httpchallenge.entrypoint: web

I fetched the logs from the container and I got:
msg="The ACME resolver \"letsencrypt\" is skipped from the resolvers list because: unable to get ACME account: permissions 755 for /letsencrypt/acme.json are too open, please use 600

I think the fix is chmod -R 600 /letsencrypt/

I think this should be handled by kamal

A useful command:
docker logs traefik | grep level=error

show your /letsencrypt/acme.json i think is that your problem

/letsencrypt/acme.json was using the wrong permissions and it was a directory

The way I solved this was by using a Terraform script to automate server setup and write this file during bootstrap and set the permissions
I agree that this should be handled by Kamal

Refers to https://github.com/basecamp/kamal-site/pull/41

Because I come to this issue often:
rm -rf /letsencrypt && mkdir /letencrypt && touch /letsencrypt/acme.json && chmod 600 /letsencrypt/acme.json


Avoid to fill the holes

Hi all,
let me say, this repo is awesome!!
Is there a way to maintain unfilled the holes? Let me show an example:
This is the source:

This is the result:

Thank you in advance
Ps. I can pay a pizza, for a fast work
Greetings from Italy 🇮🇹🍕

ah this is actually not that trivial, I'll have a look when I have some time : ]

Any updates on this issue? It seems that any form of compound paths gets filled with black.

I wanted to give an update on this issue, depending on your use-case this might (or might not) work for you.
If you look in svgparser.js on line 59 you see the method this.recurse(this.svgRoot, this.splitPath). This is a function designed to split up compound paths. If you uncomment this it should work.
I'm not sure if it has any implications for your implementation but for me this fixed it.


Start building a GeoJSON data set

Leaflet is the JavaScript library that we plan to use for producing an interactive map on a web page. It has pretty good support for taking GeoJSON and rendering it on a map.
See if you can start building a data set (it can be fake data if you don't have it all yet) that has all of the content you want to be part of the map, but ensure that the resulting data format is GeoJSON. Below are some tools that can help make that process easier:

http://geojson.io/ (Interactive editor with live map)
http://geojsonlint.com/ (GeoJSON validator)

To see some real examples of GeoJSON, take a look at all of the GeoJSON data sets on OpenDataPhilly.
Once you have some data in GeoJSON format, try to open a pull request with the result as a .json file.
/cc @cahdeemer

This issue is resolved now, right @cahdeemer?


Azure file copy: Option to modify "Access type" of container not available

While using Azure file copy task, I was not able to find any option to modify the container's access type, which is Private by default.

@rhlgupta1383 This is the design of the task, right now.
You can add an Azure PowerShell task and modify the access type using cmdlets.
Thanks for bringing up the scenario.
@RoopeshNair Is there is a specific reason we didn't expose this option? If not, can we consider this as enhancement?

@rhlgupta1383  - can you please post your request on uservoice?


How to get mesh image with original image size?

All res is with img size 224. If I use original image size mesh.
How can I inverse the transform mat

@sicxu @YuDeng , I also want to ask this question. I noticed that the input image would be cropped by the 5 facial landmarks location into a 224x224 size, if I want to paste the reconstructed image to the original image, how should I do?
@Feywell , have you solved this problem?

I save the mat in preprocessing,. So I warp the render image use the inverted mat when pasteing the image to the original image

@Feywell , sorry to reply you so lately. Could you share your codes to invert the reconstructed image to the original one?  I also save the trans_params information in align_img function and try to recover the reconstructed image to the original size, but I got the results which matched not very well.

Not sure if this is still a question for anyone, here is my take on it:
In the function resize_n_crop_img in preprocess.py find the following two lines:
img = img.resize((w, h), resample=Image.BICUBIC)
img = img.crop((left, up, right, below))

You just need to save w,h,left,up and reverse these two operations (with usage of width and height of the original photo) to find which part of the original photo was cropped. The parameters right and below dont need to be saved as they are left, up plus 224.

Just did the math for the answer I posted before.
orig_left,orig_up,orig_crop_size = (left,up,target_size)/s
will  give you the upper right corner of the crop in the original image (orig_left,orig_up) and the size of the squared crop, orig_crop_size

I find this idea also gets slightly jitting effects when processing video.
Finally, I save s, left and up firstly.
Then, I resize the original image by s, and paste back the left, up to the resized original image, which works better and seamlessly.

Here is my take on it. In the function resize_n_crop_img in preprocess.py
orig_left,orig_up,orig_crop_size = (left,up,target_size)/s
will give you the upper left corner of the crop in the original image (orig_left,orig_up) and the size of the squared crop, orig_crop_size


Prepare release 3.0.1

oops, I didn't see this PR and already committed this change directly to master


contractcourt: create new mempool witness script template notification system

There're certain classes of edge-cases w.r.t multi-hop HTLC resolution, that can mostly be mitigated by simply watching the mempool for pre-image reveal. Watching the mempool (when available), means we'll be able to properly redeem HTLCs more quickly in certain cases when a party attempts to go to chain with a transaction that might ultimately confirm after the time lock delta has well expired.
In the daemon as it exists today, we have a websockets stream for btcd, and zmq for bitcoind. We'll need to create new subscription streams for this (or a filtration layer in front of the existing streams) that allow us to detect certain witness script templates so we can extract the pre-image early. Other related systems also need such a sub-system, so we should design it in a way that's extendible to any other witness script and witness element extraction.
Steps to Completion

[ ] Create a new abstract sub-system that's able to take a "mempool source" apply a pattern matching based on a witness script template or witness structure and disperse the witness script + witness to subscribers.

[ ] Pipeline this source into the pre-image beacon, along the way, modifying the pre-image beacon to optionally take a filter predicate (created above).

One question about the mempool source that we want to hook this template matching up to. The chain clients that we use for btcd/bitcoind both currently notify us of "relevant" transactions (those that we're registered spend notifications for, or are addresses that we want to watch). Since we'd want to use this for contractcourt, I think that all the transactions that we would want to match will fall in this set? I might be wrong here, not (yet) a btcwallet-wizard.
If the above is correct, would we want to have a mempool source that provides us with all mempool txns, not just the ones that are relevant to us?

Other related systems also need such a sub-system, so we should design it in a way that's extendible to any other witness script and witness element extraction.

^ Specifically referring to this, because I'm not sure what other use cases we have in mind.
Asking this because I think the latter would require a change to btcwallet for bitcoind:

btcd has our back with NotifyNewTransactions + OnTxAcceptedVerbose
bitcoind (I think) would need changes because filterTx removes all non-relevant txns we get from our tx stream early

If all of this is on the right track, I think that the best approach here would be to start with just providing our relevant mempool txns as the mempool source, then make the btcwallet changes and switch it out later (if required).

Hi @positiveblue feel free to unassign yourself if your plate is full

@yyforyongyu might have some spare capacity to dig into this alternatively

Cool take over.

Fixed by https://github.com/lightningnetwork/lnd/pull/7564


bsalloc segfaults on Windows

bsalloc currently constructs its mmap_alloc::MapAlloc instance as MapAlloc::default(), which doesn't commit allocated memory. This is fine on POSIX, but on Windows, it breaks (uncommitted memory must be committed manually). We have a couple of options:

Configure MapAlloc to allocate already-committed memory on all platforms
Configure MapAlloc to allocate already-committed memory on Windows only (leave the other platforms as they are, allocating uncommitted memory)
Leave it up to bsalloc's user to commit memory

I lean towards the first for simplicity's sake, but I'd be OK with the second as well. The third seems like a bad idea to me, but I'm willing to be told I'm wrong.

Hey @getynge, I know you were interested in fixing this.
@ezrosent and I finally got a chance to discuss this, and we decided that bsalloc should commit memory before returning it on Windows, but retain the current behavior on other platforms. Do you want to do a PR for that?


Bugfix 315: The export progress dialog is never dismissed

Fixes #315

This will cause the app to crash at other times when export is not triggered from within the app or by the user. Maybe better would be to check if the context is from an activity or fragment before showing the dialog, or dismissing it

Sorry, I misunderstood the code. I've updated the PR with the proper fix.


Add Carthage support

I'll close this one - I rather larger PR https://github.com/RxSwiftCommunity/RxMKMapView/pull/16 already included a Carthage file, thanks!


Update VERSION

bump VERSION to 1.0.0, after which let's tag a v1.0.0 release.
Also adds a smoothing functionality to PowerBeam b/c non-smooth beams throws foreground power to high delays.

@aelanman let me know if you have any comments on this, otherwise let's merge it in and tag a release!

@aelanman lmk if you are good w/ this PR as it stands, or if you'd like to see those edits made before we merge, thanks!

@nkern I'm fine with merging the beam smoothing changes, but I'd like to wait to raise the version number until after #18 . That PR adds licensing info as well, which we should have before tagging a first release.

Oh sure thing, can you approve it so when I make that commit I can merge it? Thanks


Bar over header

There is now an empty bar over the page header, it needs to be removed. It might have appeared after adding new stylesheets to base.html.

Fixed by removing unnecessary CSS stylesheet.


Panic when randomly going to next and previous library

Describe the bug
When checking random binaries I got this crash
^[]11;rgb:1818/1818/1818^[\/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2: missing program name
Try '/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 --help' for more information.

Backtrace omitted. Run with RUST_BACKTRACE=1 to display it.
Run with RUST_BACKTRACE=full to include source snippets.

The application panicked (crashed).
  failed to send tick event: SendError { .. }
in src/tui/event.rs:80
thread: <unnamed>

To reproduce
Open random binary
Click randomly enter + backspace
after a while click esc key - probably when "Loading screen is visible"
Expected behavior
No crash
Software information

Operating system:
Rust version: 1.81.0
Project version: 3f4f742cfc759425ef767605417ff19d257023f9

Hey, thanks for reporting! This is fixed in #62
The issue was, some ANSI escape codes were being run for querying the terminal background on each time you select a dependency. Now we are querying the background color once at the start of the application which should fix this :)


better performance of mesh/lods

I used the clipmesh of Terrain3D as a replacement for the quadtree based mesh thast there currently. It works way better and I got it running in the editor even.

Terrain3D referring to https://github.com/TokisanGames/Terrain3D ? Does clipmesh refer to the GeoClipMap defined here? https://github.com/TokisanGames/Terrain3D/blob/main/src/geoclipmap.cpp?

yes.

basically I modified the ocean shader to work as a custom terrain shader and put it into the terrain3d node. then modified ocean.gd to change the shader parameters of the terrain3d node. Note that you need to add one dummy terrain material, so that not automatically the squares debug view gets enabled.

I haven't had a chance to try this out myself, but I am very much interested in this idea. Do you have an example project you are willing to share? From what I understand, this would allow a combination of ocean and land within the same Terrain3D, correct?


Naming Style Validation

The Naming Conventions page in the Flipper documentation includes some good advice:

Pick a naming style and stick with it. The most important thing is consistency.

Is there any way to enforce this with Flipper? It'd be cool to be able to configure a naming_style parameter and have Flipper automatically validate that the name matches the style.

Good suggestion. I opened https://github.com/flippercloud/flipper/pull/880 with an impelementation.

Awesome! Thank you. 🙏


test(example): add snapshot test for demo

See #1381

Neat - likely fully unnecessary for the demo, but it's a good demo of insta. cargo-insta might make a good ratatui-ified app at some point in the future...


WIP: Allow a pod to be actively killed by the kubelet if it runs too long

Note: Pending agreement on approach, I will add the requisite unit test and validation code.
Change:
Add a new optional field to PodSpec, RunningTtl, that sets a time in seconds that a pod is allowed to run before the Kubelet will actively try to fail that pod and subsequently kill its containers.  The "Running" concept aligned with the behavior in Kubelet today by enforcing the TTL value relative to the earliest started container in the pod.  The target use case is to let a user basically time-box how long a pod can run without the need to build any additional active monitoring.  It's expected that the graceful termination period for the pod will be respected once its actually properly plumbed through the Kubelet... right now, it looks to default to 10s in all cases.
This change affects all pods independent of their RestartPolicy, so a RestartPolicy Always pod will be failed, and if its not managed by a replication controller that will create a new instance of that pod, the pod will basically be done.
@bgrant0607  - This is the follow-up to discussions we had at the last community meet-up.  Loosely related to #7856 since chosen terminology aligned with the Running pod phase.  I thought about a future use case for WaitingTtl to basically set a limit to how long we are willing to wait for a docker image pull before killing the pod to hopefully schedule to a new host, but that is for another time...
/cc @abhgupta @smarterclayton

On Ttl (guidelines should be added to api-conventions.md.):

Ttl is an acronym, so it should be all caps, similar to DNS or TCP.
That said, the field would either need to accept a time.Duration, with units specified as part of the value, or must have the units in the name of the field -- Seconds in this case, such as FooDeadlineSeconds.

Re. Running:

You want to exclude pulling the images?
What do you want to happen if containers crash and restart during execution? If the Kubelet crashes? If the node reboots?
If we implemented a Job controller (#1624), would you want the timeout to span pods, or reset every time a pod were replaced?
My impression is that users find the term "Running" confusing, since the containers may not actually be running. Something with less misleading connotations is likely to replace PodPhase at some point. Perhaps Active, which, like Ready, could apply fairly generically to multiple kinds of objects.

So, ActiveDeadlineSeconds?

I have a primary goal of limiting the amount of time a Pod can be active to a Node once the Kubelet acknowledges the Pod has been scheduled.

You want to exclude pulling the images?

We actually have had complaints that a Pod can appear in Pending status indefinitely if the Node does not have access to an external registry/hub (see https://github.com/openshift/origin/issues/2048).  As an insider to the system, I feel like the amount of time it takes to pull image(s) should not limit the amount of time I get to run the container(s), but as an end-user, I am not sure it matters for my use case.  In the near term, I am okay if we actually blend these two concepts, but in the future, I could see a need to distinguish how long we are willing to wait to pull an image versus how long we let a container actually run.  For example, I want to limit how long to wait for image to pull, but not limit how long the container can run seems valid.

What do you want to happen if containers crash and restart during execution?  If the Kubelet crashes?  If the node reboots?

If containers crash and restart, I do not want that to reset the time.  If Kubelet crashes, I am OK with rogue containers that are not stopped until Kubelet is made Healthy.  If the node reboots, I am OK if the node down time is included in the duration that is accounted against the pod.

If we implemented a Job controller (#1624), would you want the timeout to span pods, or reset every time a pod were replaced?

If I run a Job, I have different expectations for what it means for how long the Job can take to complete.  I want the time limit of a Job to be tied to the Job, and not the Pod(s) it produces.  In a scenario where I kick off a Job to run my payroll, I want that Job to fail if it runs for 30 minutes and was never able to actually run any pods (because the pods were never scheduled, because not enough pods ran, because I lacked quota, etc.).  So I think in a Job scenario, the end-user expectation for how long it can run is different than how long a single Pod can run.  That being said, the concepts may blend.  I may have a Job that I kick off that can be active for 12 hrs, but no single Pod kicked off by that Job should run for more than 30 minutes.  I measure the time of a Job to be related to when a controller was made active, not when a pod was made active.

My impression is that users find the term "Running" confusing...

I agree, and I would be lying to you if I said I did not get confused when implementing this.  Looking at getPodPhase is confusing ;-)
Upon further reflection, how do you feel about the following:

Add a timestamp to PodStatus that denotes when a Kubelet acknowledges that a Pod has been scheduled to the host.
Add a duration in seconds that controls how long since the Kubelet acknowledges a Pod before the Kubelet will actively attempt to kill the Pod.

So how about NodeACKTimestamp and NodeActiveDeadlineSeconds?

If we added an Active condition, we wouldn't need NodeACKTimestamp. That said, I wouldn't be opposed to something like PodStatus.StartTime, following whatever consistent naming convention we settle on for times (https://github.com/GoogleCloudPlatform/kubernetes/issues/7018#issuecomment-99749177).
Prefixing ActiveDeadlineSeconds with "Node" is slightly less ambiguous, but at the cost of being more verbose. I'm not sure it's worth it, so long as the description string explains it adequately, especially if there's a matching Active condition paired with it.

cc @dchen1107 @vmarmol

/subscibe

@bgrant0607 - thanks for feedback as always.
I will update this PR to the following:

PodStatus.StartTime - when a Kubelet acknowledges the pod is scheduled to its host, and begins to start pulling requisite images and run the container.
PodSpec.ActiveDeadlineSeconds - number of seconds relative to StartTime until Kubelet will actively work to kill the pod.

Once we settle on *Time fields, and representation of Duration fields I would be happy to update in a follow-on.  ActiveDeadlineSeconds is at least consistent with the graceful termination field on PodSpec.  And when we look to eliminate Phase, I agree we should be able to map PodStatus.StartTime to a relevant condition.
Will ping when ready for another pass.

Having a little bit of trouble getting a PodStatus.StartTime that does not get reset every sync loop given the current Kubelet code structure.   Hope to get a work-around soon, but currently banging head.  All status now is derived from what Docker returns, and we are aggressive about deleting status if there is a batch failure.

@vmarmol, @dchen1107, or someone else from the node team may have some suggestions.

@dchen1107 I will update this PR over weekend with latest changes and re ping with feedback based on any problems I am having on Monday

Ok, got something working, will finish clean up and update PR.  Need to run some tests with what happens if Kubelet dies and comes back.

Ok, this should be ready for a review pass now.  Tests added, swagger doc generated, etc.

LGTM but is there any way to add a test to verify that a pod that exceeds its deadline actually gets killed?

Please document and add a validation check that the deadline cannot be set when restart policy is always.

Added unit tests to ensure that SyncPods will terminate the actual container if the deadline is exceeded, and that it will not terminate the pod if the deadline has not been reached.
Updated swagger documentation to provide more detail on the meaning of StartTime.
@bgrant0607 - I do not follow your last comment.  The restart policy only affects if the containers are restarted in light of a failure, but if the pod itself is failed, nothing should restart.  Right now, if a restart always pod exceeds it active deadline, the pod is killed and all containers are killed.  If prior to that deadline being reached, any of the individual containers of the pod were to fail, they are restarted as many times as needed up until the specified deadline.  This seems like the right behavior to me, let me know if I misunderstood something.

LGTM overall except one nit.
@bgrant0607 My understanding about this feature is that it is independent from restart policy completely. Or we don't agree on that?

@dchen1107 - addressed your request to localize computation of start time to status_manager only.

Looks like signature for TestKubelet changed, fixing up.

Pushed an update to respond to change, Travis should hopefully show green now.

I would like to rebase this PR, but it appears the code generator for conversions is not working for v1 api version
GOPATH=$(godep path):$GOPATH go run cmd/kube-conversion/conversion.go -v v1 -f file1.txt -n file2.txt

Both output files are empty.

Ok, fixed the error in kube-conversion.go as part of PR so I could generate v1 conversions.
Poking this PR so it can hopefully avoid further rebases.

This carries a commit from https://github.com/GoogleCloudPlatform/kubernetes/pull/8201

LGTM pending the rebase.

rebased, should be good to go.  integrated with your admitPods function.

Shippable is green. Merging.


rbd: remove unneeded updateSnapshotDetails() function

updateSnapshotDetails() calls getImageInfo() on the rbdVolume that is
related to the rbdSnapshot. This is not needed at all, as both types
have the same getImageInfo() function, so it can be called directly on
the rbdSnapshot.

@mergifyio rebase

mini-e2e-helm_k8s-1.29 failed with:
[FAILED] backend images not matching kubernetes resource count,image count 6 kubernetes resource count 3
  backend image Info:
   [csi-snap-0cb3a66f-12fb-4de3-abc8-358c9652b925 csi-snap-253ee5c4-5629-4aee-ad41-5b73d46473b9 csi-snap-c8339722-e817-4a42-8fdb-b7d7e484c596 csi-vol-13284667-a769-4ece-9702-da14bccad413 csi-vol-29544488-b299-4b66-bda7-59cfa5c339ea csi-vol-cd3f6a6b-7d6f-4339-8abf-aaa653a1fdc4]

logs
And mini-e2e-helm_k8s-1.27 as well:
[FAILED] backend images not matching kubernetes resource count,image count 6 kubernetes resource count 3
  backend image Info:
   [csi-snap-00c89343-7d46-47c4-92a9-a126bcd81781 csi-snap-c4cba576-fc9b-4603-846b-f2523b6bc7dd csi-snap-c4fbb8cd-37b9-459e-98ef-50d02c2eae90 csi-vol-aebc4d25-5b16-4cd7-9f44-c4b5c117c316 csi-vol-d2399942-4d6e-4585-af59-7136d18acc9e csi-vol-d8a3ce9f-f614-467c-815f-2d59d9d3dcbc]

logs

/test ci/centos/mini-e2e/k8s-1.27

/test ci/centos/mini-e2e/k8s-1.27

This one is still running, but there are an operation with the given Volume ID 0001-0024-72956532-8149-4e58-8636-8f8d656a3c8d-0000000000000002-294bdff5-caa2-468c-86d0-82974fa340e4 already exists in the logs. Unclear if that is something that is introduced with the new rbdSnapshot.open() function.

/test ci/centos/mini-e2e-helm/k8s-1.28

Not continuing with this. rbdSnapshot.RbdSnapName is too much (ab)used to with relation to the existing RbdImageName, there does not seem to be a rule when RbdSnapName or when RbdImageName is supposed to be used. Only using RbdSnapName for the rbd-snapshot and RbdImageName for the rbd-image seems too tricky to do.


Make SQL joins a bit more robust

Fix regression with replacing "." with "_"
Prompt it to not add extraneous prefixes
Left join / based on best matching columns (because one time, it created two SELECT queries)
Use slugs instead of names with spaces for expression labels

There seems to still be issues referencing to previous tables


Cannot load on V3PlusClosures byte-code set

Some methods have too many literals. It would be great if we could load it in a V3 image. Dave is maintaining one with recent Trunk. Is SIT really using those DB methods from Squeak History project? Hmm...

Good question, but this is not related to SIT. The Sista requirement is posed by Squeak History, I don't know anything about this. :-)

Do you load the entire Squeak History Projekt or just a smaller part of it? We might want to improve the modularity in the Squeak History Project so that SIT does not have to load those "author-normalization" methods. Or do you need them?

As of today, author normalization is not explicitly used. But if the overhead is not too large, this might be a nice improvement for the future.
Why should we still support an older bytecode set than SistaV1?

Why should we still support an older bytecode set than SistaV1?

The number of literals in an image is hardly anything one would associate with SistaV1. It's just a neat convenience. Since it is possible to re-write methods so that they use fewer literals, it makes sense to do so.
Unless it is your personal project. Then you can decide for yourself. SIT, however, has an entry in the Squeak base image. So it makes sense to maximize compatibility.

Would you accept a PR for SqueakHistory that wraps the methods in question in a larger Compiler evaluate: send to make it V3PlusClosures-compatible?

Nevermind, I have proposed a better solution in https://github.com/hpi-swa/squeak-history/pull/15. If you approve it, I will merge it into the branch of my fork that is used for SIT. :-)


lunarvim crashes when going between brackets in printf function

lunarvim crashes when i go between brackets in a function

lunarvim crashes when i go between brackets in a function


Update and polish policy tests.

This PR

Splits the tests in to separate tests instead of the current set of sub-tests.
Adds a build tag githubtests to the tests involving remote write ops. This tag needs to be provided to run this.
Change the config and the underlying structure for branch policies.
Change the behaviour of dir flag in policy cmd to act as a prefix in case multiple repos are specified in repos flag.

There appears to be a build failure. Please fix in this PR if you can.

Merging as it's decided to take care of the failing tests later depending on the decision regarding dynamo db for keeping new builds information.


[Web App|Extension] Implement Marketing/Promotion Banner

Is your feature request related to a problem? Please describe.
We need to show the promotion banner in our Wallet when running the marketing campaign.
Example:

Describe the solution you'd like
The client will fetch the promotion banner information via API /configurations with these properties:

key
type
description

campaign
string
Campaign name, eg: Everstake

content
html
Campaign headline, eg: Join, compete & win! Prize pool of $100,000

image
string
Campaign image link

cta
string
Call to Action's link

Describe alternatives you've considered
N/A
Additional context
N/A

Implemented with news section, closing this for now.


Approvals for LIFI trades needs gas amount displayed

Overview
When completing a LIFI trade on Native that has a gas fee other than eth, it should be displayed in the amount and chain that it is charged in.
References and additional details

Log into develop.shapeshift and connect to native wallet
Navigate to trade page and enter LIFI trade that requires an approval (i.e usdc on optimism to avax)
Confirm and trade and note the approval step comes up
Note that the approval modal does not display a fee, appears as tho it is $0.00 and it costs 0 eth

Acceptance Criteria
Approvals should have gas fee displayed
Need By Date
No response
Screenshots/Mockups
No response
Ownership

[X] If my bounty needs engineering or needs product I have added the respective labels on the right
[X] As the sponsor of this bounty I will review the changes in a preview environment (ops/product) or review the PR (engineering)

Estimated effort
No response
Sponsor / Stakeholder
No response
Bounty Hunters

Join our discord
Include an expected timeline for you to complete work in the work plan when you apply for this bounty!
Please refer to this link for some basic info
Please do not start work on this issue until you are approved in Gitcoin.

closing as this appears to have been resolved in #4241


Crashes are not sending to Firebase Crashlytics

Hi, I'm able to send crashes via iOS simulator when run through Xcode.
When I stop running on Xcode and try to send crashes in Simulators, I'm getting following log and I'm not able to find successfully sent message.
Please help!!!
I'm not using Fabric/Crashlytics.
Step 1: Describe your environment

Xcode version: Version 11.3 (11C29)
Firebase SDK version: [Firebase/Crashlytics] Version 4.0.0-beta.7
Firebase Component: Crashlytics
Component version:[Firebase/Crashlytics] Version 4.0.0-beta.7
Installation method: CocoaPods | Carthage | Zip file (select one)

[REQUIRED] Step 2: Describe the problem
Steps to reproduce:
default	17:07:54.230079 +0530	AFStoreModelKiosk_universal	[Firebase/Crashlytics] Packaged report with id 'ca49f9e24e1f431581059f28954643ee' for submission

default	17:07:54.233329 +0530	AFStoreModelKiosk_universal	background session setup will wait for reply: session <__NSURLBackgroundSession: 0x6000022a0d20> with identifier com.google.firebase.crashlytics.ios.crash.background-session
default	17:07:54.235508 +0530	nsurlsessiond	Creating session with identifier: com.google.firebase.crashlytics.ios.crash.background-session for bundle id: com.aquafadas.aquafadasalpha
default	17:07:54.240515 +0530	nsurlsessiond	Session <com.aquafadas.aquafadasalpha>.<com.google.firebase.crashlytics.ios.crash.background-session> using resource timeout:<PHONE_NUMBER>00, request timeout: 60.000000 allowsCellularAccess: 1, allowsExpensiveAccess: 1 _sourceApplicationBundleIdentifier: (null), _sourceApplicationSecondaryIdentifier: (null)
default	17:07:54.241056 +0530	AFStoreModelKiosk_universal	background session setup reply received: session <__NSURLBackgroundSession: 0x6000022a0d20> with identifier com.google.firebase.crashlytics.ios.crash.background-session
default	17:07:54.243338 +0530	nsurlsessiond	Task <4D32D3B0-CF30-42E3-8459-27D22B17D7BE>.<1> uploadTaskWithRequest: <NSMutableURLRequest: 0x7fe5d4d24ee0> { URL: https://reports.crashlytics.com/sdk-api/v1/platforms/ios/apps/com.aquafadas.aquafadasalpha/reports } [allowsCellularAccess: 1] fromFile: file:///Users/sudha.chandran/Library/Developer/CoreSimulator/Devices/6F079265-2A88-49C3-BB6C-48769B173BDE/data/Containers/Data/Application/9CFF8C8A-26F2-415A-9184-924C4C98A199/Library/Caches/com.crashlytics.data/com.aquafadas.aquafadasalpha/v5/reports/prepared-legacy/855EDED2-5FE6-4B65-A8B6-7C31E2B66757.multipartmime
default	17:07:54.245053 +0530	nsurlsessiond	Task <4D32D3B0-CF30-42E3-8459-27D22B17D7BE>.<1> is for <com.aquafadas.aquafadasalpha>.<com.google.firebase.crashlytics.ios.crash.background-session>.<1>
default	17:07:54.249437 +0530	nsurlsessiond	TIC TCP [0x0] working for task {
    kConditionalConnectionActivityName = "Task <4D32D3B0-CF30-42E3-8459-27D22B17D7BE>.<com.aquafadas.aquafadasalpha>.<com.google.firebase.crashlytics.ios.crash.background-session>.<1>";
    kConditionalConnectionIsUpload = 1;
    kConditionalConnectionRequirementWorkload = 1000;
}
default	17:07:54.250707 +0530	nsurlsessiond	[C42 reports.crashlytics.com:443 tcp, url hash: 186f2200, traffic class: 200, tls, indefinite] start
default	17:07:55.030377 +0530	nsurlsessiond	Task <4D32D3B0-CF30-42E3-8459-27D22B17D7BE>.<1> didReceiveChallenge: <NSURLAuthenticationChallenge: 0x7fe5d4e13ad0> protection space: <NSURLProtectionSpace: 0x7fe5d4e07e40>: Host:reports.crashlytics.com, Server:https, Auth-Scheme:NSURLAuthenticationMethodServerTrust, Realm:(null), Port:443, Proxy:NO, Proxy-Type:(null) for client com.aquafadas.aquafadasalpha received auth challenge with type NSURLAuthenticationMethodServerTrust
default	17:07:55.338614 +0530	nsurlsessiond	TIC TCP [0x7fe5d4e09450] working for task {
    kConditionalConnectionActivityName = "Task <4D32D3B0-CF30-42E3-8459-27D22B17D7BE>.<com.aquafadas.aquafadasalpha>.<com.google.firebase.crashlytics.ios.crash.background-session>.<1>";
    kConditionalConnectionIsUpload = 1;
    kConditionalConnectionRequirementWorkload = 1000;
}
default	17:07:55.338714 +0530	nsurlsessiond	TIC TCP [0x7fe5d4e09450] working for task {
    kConditionalConnectionActivityName = "Task <4D32D3B0-CF30-42E3-8459-27D22B17D7BE>.<com.aquafadas.aquafadasalpha>.<com.google.firebase.crashlytics.ios.crash.background-session>.<1>";
    kConditionalConnectionIsUpload = 1;
    kConditionalConnectionRequirementWorkload = 1000;
}
default	17:07:55.340811 +0530	nsurlsessiond	TIC TCP [0x7fe5d4e09450] working for task {
    kConditionalConnectionActivityName = "Task <4D32D3B0-CF30-42E3-8459-27D22B17D7BE>.<com.aquafadas.aquafadasalpha>.<com.google.firebase.crashlytics.ios.crash.background-session>.<1>";
    kConditionalConnectionIsUpload = 1;
    kConditionalConnectionRequirementWorkload = 1000;
}
default	17:07:55.340917 +0530	nsurlsessiond	TIC TCP [0x7fe5d4e09450] working for task {
    kConditionalConnectionActivityName = "Task <4D32D3B0-CF30-42E3-8459-27D22B17D7BE>.<com.aquafadas.aquafadasalpha>.<com.google.firebase.crashlytics.ios.crash.background-session>.<1>";
    kConditionalConnectionIsUpload = 1;
    kConditionalConnectionRequirementWorkload = 1000;
}
default	17:07:56.148442 +0530	AFStoreModelKiosk_universal	6.22.0 - [Firebase/Crashlytics][I-CLS000000] Warning: NSUncaughtExceptionHandler is 'GAIUncaughtExceptionHandler' in '/Users/sudha.chandran/Library/Developer/CoreSimulator/Devices/6F079265-2A88-49C3-BB6C-48769B173BDE/data/Containers/Bundle/Application/7823D50E-8729-4499-8F2C-EE8205A8AC8A/AFStoreModelKiosk_universal.app/AFStoreModelKiosk_universal'
default	17:07:56.515343 +0530	nsurlsessiond	Task <4D32D3B0-CF30-42E3-8459-27D22B17D7BE>.<1> didReceiveResponse: <NSHTTPURLResponse: 0x7fe5d4e01800> { URL: https://reports.crashlytics.com/sdk-api/v1/platforms/ios/apps/com.aquafadas.aquafadasalpha/reports } { Status Code: 202, Headers {
    Connection =     (
        "keep-alive"
    );
    "Content-Length" =     (
        0
    );
    "Content-Type" =     (
        "application/json"
    );
    Date =     (
        "Thu, 16 Apr 2020 11:37:56 GMT"
    );
} }, status code: 202
default	17:07:58.440106 +0530	nsurlsessiond	[C42 reports.crashlytics.com:443 tcp, url hash: 186f2200, traffic class: 200, tls, indefinite] cancel
default	17:07:58.440221 +0530	nsurlsessiond	[C42 reports.crashlytics.com:443 tcp, url hash: 186f2200, traffic class: 200, tls, indefinite] cancelled
	[C42.1 90B05B1D-9EAA-4AA0-A68A-38FEF2960341 <IP_ADDRESS>:62033<-><IP_ADDRESS>:443]
	Connected Path: satisfied (Path is satisfied), interface: utun1
	Duration: 4.189s, DNS @0.003s took 0.185s, TCP @0.191s took 0.281s, TLS took 0.611s
	bytes in/out: 4192/37546, packets in/out: 10/34, rtt: 0.277s, retransmitted packets: 0, out-of-order packets: 0
default	17:07:58.440271 +0530	nsurlsessiond	0.000s [C42 DE1804C8-20A8-4590-B860-BB2AA7CAA61A reports.crashlytics.com:443 resolver] path:start
default	17:07:58.440326 +0530	nsurlsessiond	0.001s [C42 DE1804C8-20A8-4590-B860-BB2AA7CAA61A reports.crashlytics.com:443 resolver] path:satisfied
default	17:07:58.440384 +0530	nsurlsessiond	0.003s [C42 DE1804C8-20A8-4590-B860-BB2AA7CAA61A reports.crashlytics.com:443 resolver] resolver:start_dns
default	17:07:58.440529 +0530	nsurlsessiond	0.188s [C42 DE1804C8-20A8-4590-B860-BB2AA7CAA61A reports.crashlytics.com:443 resolver] resolver:receive_dns
default	17:07:58.443580 +0530	nsurlsessiond	0.472s [C42 DE1804C8-20A8-4590-B860-BB2AA7CAA61A reports.crashlytics.com:443 resolver] flow:finish_transport
default	17:07:58.443948 +0530	nsurlsessiond	1.084s [C42 DE1804C8-20A8-4590-B860-BB2AA7CAA61A reports.crashlytics.com:443 resolver] flow:finish_connect
default	17:07:58.444338 +0530	nsurlsessiond	1.085s [C42 DE1804C8-20A8-4590-B860-BB2AA7CAA61A reports.crashlytics.com:443 resolver] flow:changed_viability

Relevant Code:
My run script in build settings.
"${PODS_ROOT}/FirebaseCrashlytics/run"
"find ${DWARF_DSYM_FOLDER_PATH}" -name "*.dSYM" | xargs -I \{\} "$PODS_ROOT/FirebaseCrashlytics/upload-symbols" -gsp "${PROJECT_DIR}/GoogleService-Info.plist" -p ios \{\}

Can you share a project that reproduces this issue? Also, is the issue reproducible on device?

More questions for you: Do you have any missing dSYMs on the Crashlytics dashboard? If so, could you try downloading the dSYM manually either from Apple or the archive, and then uploading to the Firebase dashboard using the upload-symbols tool? More info here.
If not, could you set debug mode, and attempt to send up a test crash using the steps 1-5 from the linked doc?

Hi,
Now I'm able to send dSYM files and it is getting reflected in the Firebase console.
Thanks for your help


How to simulate typing for chatbot

Is there anyway to create this.... To simulate a Typing... state for the chatbot @bassaer

like this? @othreecodes

@bassaer  Yes,  Something like that

@othreecodes Currently not supported, but this function will be implemented.
Thanks👍


Makefile: Generate compile_commands.json

Introduce a make target for generating compile_commands.json to feed clangd and the Language Server Protocol, to gain better editor integration.
Add the relevant files to .gitignore.

sneaked in #56

That is scary. I dropped the patch from #56 and posted this separately, did not expect Github to merge it there...


fix: Rework logic for grid ordering in hero component

Jira
http://vjira2:8080/browse/BDS-2079
Summary
Fixes a bug where hero reverseOrder prop does not work when imageValign is bottom
Details
I set out only to fix the original bug.  However, I found the existing twig logic for hero to be pretty difficult to read, in part because there were a number of logic errors.  For example:
{% set _isImageFirstOnMobile = (imageValign != 'bottom') or (imageValign == 'top') %}
{% set _isImageLastOnMobile = (imageValign != 'top') or (imageValign == 'bottom') %}

First, both of these statements are logically identical if you remove the second conditions (if something is "not bottom", you don't need to also check for whether it could be "top").
{% set _isImageFirstOnMobile = (imageValign != 'bottom') %}
{% set _isImageLastOnMobile = (imageValign != 'top') %}

Digging in further, what happens if the value of imageValign is middle?  Well, both of the above statements evaluate to true, so you end up with image on mobile being both first and last. 🙃
So I ended up re-working a lot of the logic here.  Hopefully it's equivalent to what was there before without the additional bugs.
How to test
Reproduce the bug

Check out any other branch
Edit docs-site/src/pages/pattern-lab/_patterns/02-components/hero/20-hero--reverse-order.twig
Change imageValign to bottom
View the pattern at /pattern-lab/?p=components-hero--reverse-order
Note that the image is on the right (expected on the left).

Confirm the fix

Check out the branch in this PR
Confirm that the steps above give the expected results
Confirm that all other scenarios give expected results

@remydenton yeah, I think these couple of quirks are side-effects from the original implementation allowing for desktop and mobile order to both be independently configured before getting consolidated at the 11th hour... 🤦♂

@remydenton reminder: make sure you add a label based on the SEMVER release type so things get automatically published to correct version (plus add the milestone release version so we can track what work is expected to go out in which release!)

PR was released with v2.21.0


[Design] Login Page

Description:
As part of reservations - we will need to provide users with the ability to log into our services.
Development Checklist:

[ ] lo-fi mock-up designs
[ ] includes different login options (e.g., BCeID and BCSC)
[ ] review with partners and osprey for feedback
[ ] make updates as needed
[ ] showcase at a demo day and sprint review

Dependencies

Blocked by
Blocking

Wireframe(s):
Workflows:
Definition of Ready (Note: If any of these points are not applicable, mark N/A)

[ ] User story is included
[ ] User role and type are identified
[ ] Acceptance criteria are included
[ ] Wireframes are included
[ ] Design / Solution is accepted by Product Owner
[ ] Dependencies are identified (technical, business, regulatory/policy)
[ ] Story has been estimated (under 13 pts)

Definition of Done

In progress:

[ ] Acceptance criteria are tested (Functionality meets the acceptance criteria defined in the ticket)
[ ] UI meets accessibility requirements
[ ] Unit tests are written
[ ] Work is traceable in GitHub
[ ] PR linked to ticket number
[ ] If needed/required - Dev adds flag/label to highlight any migration steps necessary prior to PROD deployment

Code review:

[ ] Code is peer reviewed and has passed CI/CD tests

QA:

[ ] Acceptance criteria are tested (Functionality meets the acceptance criteria defined in the ticket)
[ ] Code is potentially shippable to the production environment
[ ] Functional features have been tested and passed by QA
[ ] UI components tested by designer
[ ] Code is deployed to PROD when moved to 'done' column (unless requested otherwise by PO)

PO Review:

[ ] Acceptance criteria are tested (Functionality meets the acceptance criteria defined in the ticket)
[ ] Reviewed and approved by Product Owner

Notes:

Update to? :
Login flow designs
Other notes:

all accounts-related things would be one flow - create account, login, update info
separate = filling out the forms, and somewhere in that is the create account stuff and he's designign around it.
So in project board, epics: if work crsoses between Epics we can just link as such.

todo: add these deets into #67


Implementations for off-heap FixedIntArray<->Integer map, and single-value multi-column reader/writer.

Created an interface IdMap that maps inserted keys to contiguous
integers starting from zero. Added specialized implementation for IdMap
with FixedIntArray key.

Added interfaces for single-value multi-columns readers, writers
and ReaderWriter. Also added fixed-byte implementation for the
ReaderWriter interface.

Moved FixedIntArray from inner class from within GroupBy code into a
more generic module for re-use.

Added unit tests and benchmarks for new features.

Codecov Report

Merging #2494 into master will decrease coverage by 0.02%.
The diff coverage is 79.08%.

@@            Coverage Diff             @@
##           master    #2494      +/-   ##
==========================================
- Coverage   69.48%   69.46%   -0.03%
==========================================
  Files         848      852       +4
  Lines       39555    39687     +132
  Branches     5330     5334       +4
==========================================
+ Hits        27484    27567      +83
- Misses      10279    10329      +50
+ Partials     1792     1791       -1

Impacted Files
Coverage Δ

...upby/NoDictionaryMultiColumnGroupKeyGenerator.java
92.92% <ø> (+2.83%)
:arrow_up:

...l/FixedByteSingleValueMultiColumnReaderWriter.java
100% <100%> (ø)

...writer/BaseSingleValueMultiColumnReaderWriter.java
4.76% <4.76%> (ø)

...gregation/function/customobject/FixedIntArray.java
66.66% <66.66%> (ø)

...din/pinot/core/util/FixedIntArrayOffHeapIdMap.java
81.57% <81.57%> (ø)

.../impl/dictionary/BaseOffHeapMutableDictionary.java
81.14% <83.33%> (+0.21%)
:arrow_up:

...ore/query/scheduler/resources/ResourceManager.java
81.81% <0%> (-13.64%)
:arrow_down:

...e/impl/dictionary/LongOnHeapMutableDictionary.java
82.22% <0%> (-13.34%)
:arrow_down:

...mpl/dictionary/DoubleOffHeapMutableDictionary.java
65.45% <0%> (-12.73%)
:arrow_down:

.../impl/dictionary/FloatOnHeapMutableDictionary.java
80% <0%> (-6.67%)
:arrow_down:

... and 24 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 96ca68d...a67e4bd. Read the comment docs.


is there any way to run it as headless?

is there any way to run it as headless?

Nice suggestion! I'll include this in the future plan

Could be a simple solution to this
https://stackoverflow.com/questions/7568899/does-selenium-support-headless-browser-testing

Nice hint, @JakeCowton !
It worked and I added with another some feats to Pull Request #11

merged


KAFKA-4514: Add Codec for ZStandard Compression

Hello. This PR resolves KAFKA-4514: Add Codec for ZStandard Compression. Please have a look when you are free. Since I am a total newbie of Apache Kafka, feel free to point out the deficiencies.
Add to the feature itself, I have a question: Should we support an option for ZStandard compression level?
According to ZStandard official documentation, it supports compression level of 1 ~ 22. Because of that, Hadoop added a new configuration option named "io.compression.codec.zstd.level", whose default value is 3. In this PR, I configured the compression level to 1 as a temporary one but wondering following problems:

Should we provide a configurable option?
Would it better to change the default value, from 1 to another one?

I am looking forward to your advice. Thanks.

2 big questions here.

whether this needs to go through the KIP process.  @ijuma  mentioned that in the jira
Whether we want to pull from luben/std-jni, which is someones private repo? It is bsd licensed but its not in apache so any bug fixes and such they would have to pull in and question about maintenance and such.  Since BSD licensed probably could just copy or could also copy the Hadoop implementation ones.  Anyway this is more question for the Kafka committers.

Also, thanks for working on this.  Did you have a chance to run any performance comparisons?

Thanks for the PR.
To answer @tgravescs's questions:

Generally we require KIPs for changes that affect the message format because we want such changes to be vetted thoroughly. In this case, we are using an additional bit in the attributes byte and we don't have many of those left, so good to get feedback from a wider group.

That is a potential concern. We would be more likely to include it if we could use an established library (I understand that this is a new compression algorithm so it may be hard to find).

Performance numbers would definitely help the discussion.

@tgravescs @ijuma
Thanks for your advice. Let me summarize:
I will conduct a test benchmark a couple of days, and submit a proposal on KIP with the results. From there, let's discuss the following topics there:

Whether we extend current message format to support ZStandard.
Whether we use the (already written) ZStandard library or write some JNI code directly on Kafka.
Whether we provide a Default compression level and what value it should be.

I just started configuring benchmark environment on AWS. Please tune in!

Is this issue dead?

@eliaslevy Sorry for late reply. For my recent works, I could not spare enough time for this PR. If anyone requests, I will continue to work - the only bottleneck of this PR is benchmark test to justify the packet modification.

I nudged about this on dev mailing list and rebased this PR against trunk:

https://github.com/bobrik/kafka/tree/pr-2267

Disclaimer: I only remember enough Scala to make it compile.
My plan is to get Zstandard support into sarama (go kafka library) and play with this a bit on real data. Early tests show 4x compression on 1MB chunks of production data, which is better than everything else available and more than enough to care about this:
2018/02/09 05:08:40   *     gzip/1 : 344890 (2.85x / 14ms)
2018/02/09 05:08:40   *     gzip/3 : 295210 (3.33x / 21ms)
2018/02/09 05:08:40   *     gzip/6 : 270250 (3.64x / 41ms)
2018/02/09 05:08:41   *     gzip/9 : 266410 (3.69x / 334ms)
2018/02/09 05:08:41   *     snappy : 423776 (2.32x / 2ms)
2018/02/09 05:08:41   *     zstd/1 : 258860 (3.80x / 4ms)
2018/02/09 05:08:41   *     zstd/2 : 244806 (4.02x / 5ms)
2018/02/09 05:08:41   *     zstd/3 : 232193 (4.24x / 6ms)
2018/02/09 05:08:41   *        lz4 : 547913 (1.80x / 6ms)

Tested with sarama and it looks good, I can produce and consume just fine.
# /usr/local/kafka/bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files /state/kafka/dev/sarama.test-0/00000000000000000000.log
Dumping /state/kafka/dev/sarama.test-0/00000000000000000000.log
Starting offset: 0
baseOffset: 0 lastOffset: 1 baseSequence: 0 lastSequence: 1 producerId: -1 producerEpoch: 0 partitionLeaderEpoch: 0 isTransactional: false position: 0 CreateTime:<PHONE_NUMBER>188 isvalid: true size: 107 magic: 2 compresscodec: NONE crc:<PHONE_NUMBER>
baseOffset: 2 lastOffset: 9 baseSequence: 0 lastSequence: 7 producerId: -1 producerEpoch: 0 partitionLeaderEpoch: 0 isTransactional: false position: 107 CreateTime:<PHONE_NUMBER>189 isvalid: true size: 245 magic: 2 compresscodec: NONE crc: 24510639
baseOffset: 10 lastOffset: 10 baseSequence: 0 lastSequence: 0 producerId: -1 producerEpoch: 0 partitionLeaderEpoch: 1 isTransactional: false position: 352 CreateTime:<PHONE_NUMBER>124 isvalid: true size: 96 magic: 2 compresscodec: ZSTD crc:<PHONE_NUMBER>
baseOffset: 11 lastOffset: 19 baseSequence: 0 lastSequence: 8 producerId: -1 producerEpoch: 0 partitionLeaderEpoch: 1 isTransactional: false position: 448 CreateTime:<PHONE_NUMBER>124 isvalid: true size: 149 magic: 2 compresscodec: ZSTD crc: 336937019

@bobrik Thanks for giving interest to this PR.
In short, I did this work one year ago but failed to prove it is much faster than the others - Kafka protocol uses 3 bits to specify the codec (that is, only eight codecs are available) but 2 bits are occupied already. So the maintainers concluded that it is hard to merge this PR unless the zstd codec is much faster than the others, especially, lzo.
For details, see<EMAIL_ADDRESS>But if anyone in the community agrees that it is promising, I can continue the work.
@ijuma How do you think?

It's not faster than Snappy or LZ4, but it's a lot more efficient. Here's my benchmark with production data (1MB batch of records) from lzbench:
lzbench 1.7.3 (64-bit MacOS)   Assembled by P.Skibinski
Compressor name         Compress. Decompress.  It's amzing even to me.

To clarify, the idea was to verify that zstd provides a significant benefit over existing codecs. My hope was that we'd be able to show that it was faster than gzip with better compression ratio than lz4. In order to show that, we want to benchmark the library we intend to use from Java with a reasonable data set.

@ijuma, where can I find a reasonable dataset? I'm happy to provide benchmarks with it.
Zstd website: http://facebook.github.io/zstd/

Library used in this PR: https://github.com/luben/zstd-jni
It's using native buildings, which means we should expect comparable results.

We deployed patched Kafka in production and I'm very happy with the results. At cost of slightly higher CPU usage on producer side (we have a lot of headroom there), we were able to get 4.28x compression ratio, with Snappy it was just 2.5x. Gzip is not even close in terms of both ratio and speed.

Drop around 22:00 is zstd level 1, then at 23:30 zstd level 6. Decompression speed is fairly constant, so the only knob we have is how much CPU we are willing to burn on producer side, which is great.

Sounds good. I suggest mentioning some of this in the KIP and receiving the KIP discussion.

@dongeforever can you do that? I can't edit or comment the KIP.

@bobrik LZ4 was plagued by not having a standard framing format out of the gates, which caused early adopters like Kafka to incompatible framing (see KIP-57). It would be good clarify in the KIP what frame format we plan to use.

Adding for posterity: https://blog.cloudflare.com/squeezing-the-firehose/
@Cyan4973, can you comment on framing format concerns?

can you comment on framing format concerns?

You mean for lz4 ?
I believe @xvrl comment is fairly correct : lz4 started its life as a pure block compressor, producing only compressed data, leaving metadata management (such as compressed size) to user application. This is consistent with practice of this domain, as it was adopted by other compressors of same category (lzo, snappy, lzx, quicklz, etc.). I suspect that's because in most cases, data produced by these compressors tend to be locally consumed. It's also because each application can specify its own better adapted metadata format. For example, if application knows that all blocks are necessarily < 64KB, no need to use 8-bytes for the size field.
Later on, emerged the need to create a fully interoperable format that multiple different applications can consume, even on different platforms (including different endianness). So the LZ4 frame format was created.
But even today, that's still 2 different worlds.
Any application can still select to use only compressed blocks, and add its own metadata system.
In general, applications which started with this design just kept it.
The newer frame format comes with the "interoperability" label, but adds costs in the form of additional header / footer. It also introduces new capabilities (such as cheksum, or block dependency) and limitations (no individual block > 4 MB). Therefore, only applications which are concerned by "interoperability" switched to the official frame format.
My understanding of KIP-57 is that Kafka made the switch, and is now using the official interoperable frame format. Therefore, data created before the switch used some kind of "Kafka specific" frame encapsulation, which is likely incompatible. Then follow all sort of expectable issues related to producer / consumer versions...
A standard practice transition scenario would require all consumers to be upgraded first, to support both old and new formats, and only then have producers switched to the new format. That requires those 2 formats to be effectively distinguishable as 2 separate formats.

Thankfully zstd already has a well defined frame format, so we are unlikely to find ourselves in the same situation as with LZ4 and KIP-57.

yep, lz4 experience has been extremely useful,
zstd directly benefits from it.
as a consequence, the logic has been reversed :
zstd has a fully specified frame format by default,
block mode is still accessible, but it's now labelled an "advanced mode", for expert users only.

thank you for the great summary of LZ4 history @Cyan4973. Glad to hear zstd learned from that experience. It should be a no-brainer to use the default zstd framing for Kafka.
As far as LZ4 backwards compatibility goes, I would just want to clarify that the java clients still support decompressing older (pre KIP-57) non-standard framing, since the Kafka brokers maintain support for older message formats. Non-java clients may not support those however, as @bobrik recently found out with the sarama client.

I added a few minor comments on the code. Is there anything else I can do to help move this forward?

@bobrik I greatly appreciate your kind review. Here is the update.

@bobrik @ijuma Here is the update, with lastest zstd binding version: 1.1.1 -> 1.3.3-4. Please have a look when you are free.

@bobrik Thanks for the approval. @ijuma @onurkaraman Please give me a ping when you need anything else.

The latest zstd release is better than ever: https://github.com/facebook/zstd/releases/tag/v1.3.4
With new fast modes zstd is on par with snappy on speed, but beats it by high margin on ratio:

@bobrik It seems like the author of the binding library is already working on it: https://github.com/luben/zstd-jni/commit/b985f9fb4823c52e933975aef6d962bc98d1a453 I will apply it as soon as the new version is released.

My point is more "look, zstd can cover whole spectrum of compression needs, we should definitely merge this PR" than "we should update this PR to new zstd before we merge it".

@bobrik I know. What I mean was, "I am keeping an eye to the release of zstd 1.3.4 binding and will apply it as soon as it is released". I thought 1.3.3.x is the lastest release before you pointed out. Thanks again for the notice. :smiley:
@scottcarey Thanks for your opinion. I am already working on that issue: luben/zstd-jni#33 As soon as this update is merged, I will open this issue in Jira and start working.

The best way to make progress on this is to update the KIP so that the community can vote on it:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-110%3A+Add+Codec+for+ZStandard+Compression
A few things that the KIP should cover that don't seem to be there:

What's the behaviour when clients that don't support zstd receive zstd compressed data.
What's the behaviour when brokers that don't support zstd receive zstd compressed data.
What message format versions will support zstd, is it just the latest or will older ones support it too.
Additional performance data gathered in this PR and the Cloudflare blog posts.

Until that's done, the PR cannot be merged (no matter how many people comment here). I would personally like to see this in Kafka because I think it's a great improvement for some use cases. I think it's a little hyperbolic to claim it's "single most important performance feature", but I'd love for people to help get this over the line!

One more thing, depending on the answers to some of the questions above, it is possible that protocol version bumps may be needed. If that is the case, we should include such details in the KIP.

it's a little hyperbolic to claim it's "single most important performance feature"

Sorry, I meant to say "single most important outstanding performance feature"
There, now its not hyperbolic at all. ;-)
There are plenty of already implemented features of great importance to performance.   I am unaware of any other great ones outstanding.

W.R.T. dealing with clients or brokers that don't support zstd, I see it falling into two main options:

Its a new protocol version.  Deal with it.  Only a new broker with newer clients can use it. This is somewhat similar to some of the major version upgrades in the past, like upgrading from 0.8 to 0.10.
The broker does its best to deal with those who don't understand zstd codecs. It repackages to lz4 if a consumer does not support it, and re-compresses with level 1 (lowest CPU use) if producers supply blocks in other codecs.

In all cases "What's the behavior when brokers that don't support zstd receive zstd compressed data." must be " some sort of error is sent to the client".  There is no way a broker that does not understand zstd can do anything with it.  Hopefully the existing protocol has a way to say "I don't know that codec, can you try with one of my supported ones?".   I do not know the current protocol well enough -- if it supported something like HTTP's Content-Encoding/Accept-Encoding then there is a softer way to deal with it than an unrecoverable error.

As far as eventually supporting compression dictionaries -- I think this is a completely separate KIP to deal with later.
I believe that would require a new protocol version so that producers and consumers could obtain appropriate dictionaries per topic/partition.  Each topic/partition would have the 'current' dictionary to use when publishing, and an index of all 'active' dictionaries for consumers to use as needed. Ideally Kafka itself could have a task that occasionally samples a partition's raw blocks and produces a dictionary, updating the dictionary that producers use.  Blocks would have to be framed with the id of the dictionary.  If Kafka retained the max offset that each dictionary has been written with, then once that offset is lost the dictionary can be dropped as well.
In general, if semantic partitioning is used, then partitions will likely benefit from independent dictionaries per partition rather than one per topic.
I'm imagining topic configrations such as:
zstd.dictionary.enabled=true zstd.dictionary.generate.interval=10d zstd.dictionary.samples=5% zstd.dictionary.size=128k zstd.dictionary.per.topic=true
On some data sets, getting the dictionary right is another factor of 1.5 to 3 in space savings, and it can lower the CPU cost of both compression and decompression.  It also works well with small block sizes, significantly reducing the compression-ratio vs latency tradeoff one usually has to make with block sizes.  http://fastcompression.blogspot.com/2018/02/when-to-use-dictionary-compression.html
Dictionaries add enough complexity that I believe having the broker manage it rather than clients is the way to go.  It is too error prone to expect users to manage dictionary use manually and find a good place to store them, not to mention know when the dictionary is no longer needed.   Like consumer offsets, some power users will want full control, but 95% of the time the built-in facility is the way to go.

I agree that we should leave dictionaries for a separate KIP.
Yes, it may not be unreasonable to only support zstd on new clients and brokers. So, to ensure we throw the appropriate error when an existing client doesn't support it, we would need to bump the produce and fetch protocol versions.
I'd prefer to avoid the option where the broker recompresses for the consumer as the performance impact is non negligible. For the producer case, it already works like that. If the topic is configured with a particular compression algorithm and it differs from what the producer sends, the broker automatically recompresses it.
Another option for the consumer would be to rely on it to simply throw an error if it sees a compression algorithm that doesn't work. For this to be viable, it would have to work like this in the current code. Looking at the Java consumer, it seems like it does.
As you say, the broker should return an appropriate error if it receives a compression algorithm it doesn't understand. Someone would have to verify this, but it seems like it currently returns UNKNOWN_SERVER_ERROR for this case. It would be good to fix this for the future although it doesn't help with existing brokers.
Given all of the above, I am tempted to think that we may be able to get away without bumping the protocol versions. But we need to verify these claims and update the wiki to include the information.

There is no need for a new protocol version to just add a new codec.  Clients should already handle messages compressed using unknown codecs.  E.g. the Zendesk Ruby Kafka client did not support LZ4 until recently, even though it supported the 0.10 protocol.  Similarly, the broker will already return an error if an unknown codec is used.

@eliaslevy, have you tested that the broker returns an appropriate error in that case? It didn't look like it did when I briefly looked at the code, but I could have missed something. As I said, I think it might be fine to rely on the clients to throw an error. But we would want a system test verifying that an older Java client does the right thing.
This is important because the compression algorithm is whatever the producer chooses by default. So, it's possible for a producer to start using zstd while older consumers can't read the produced messages. Thinking about it some more, this seems a bit unsafe, so it's not clear if it's the best solution. I suggest taking the discussion to the KIP thread though, the while point of the KIP process is for more people to be involved in the design discussions.

@ijuma produce with a codec id of 4:
[2018-04-02 12:00:56,704] TRACE [KafkaApi-0] Handling request:RequestHeader(apiKey=PRODUCE, apiVersion=2, clientId=ruby-kafka, correlationId=1) -- {acks=-1,timeout=5000,partitionSizes=[test-0=91]} from connection <IP_ADDRESS>:9092-<IP_ADDRESS>:50269-6;securityProtocol:PLAINTEXT,principal:User:ANONYMOUS (kafka.server.KafkaApis)
[2018-04-02 12:00:56,714] ERROR [KafkaApi-0] Error when handling request {acks=-1,timeout=5000,partitionSizes=[test-0=91]} (kafka.server.KafkaApis)
java.lang.IllegalArgumentException: Unknown compression type id: 4
	at org.apache.kafka.common.record.CompressionType.forId(CompressionType.java:152)
	at org.apache.kafka.common.record.LegacyRecord.compressionType(LegacyRecord.java:250)
	at org.apache.kafka.common.record.AbstractLegacyRecordBatch.compressionType(AbstractLegacyRecordBatch.java:150)
	at org.apache.kafka.common.record.AbstractRecordBatch.isCompressed(AbstractRecordBatch.java:33)
	at org.apache.kafka.common.record.AbstractLegacyRecordBatch.isCompressed(AbstractLegacyRecordBatch.java:49)
	at org.apache.kafka.common.record.AbstractLegacyRecordBatch.iterator(AbstractLegacyRecordBatch.java:231)
	at org.apache.kafka.common.record.AbstractLegacyRecordBatch.streamingIterator(AbstractLegacyRecordBatch.java:263)
	at org.apache.kafka.common.record.MemoryRecords.toString(MemoryRecords.java:290)
	at java.lang.String.valueOf(String.java:2981)
	at scala.collection.mutable.StringBuilder.append(StringBuilder.scala:200)
	at scala.collection.MapLike$$anonfun$addString$1.apply(MapLike.scala:344)
	at scala.collection.MapLike$$anonfun$addString$1.apply(MapLike.scala:344)
	at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
	at scala.collection.Iterator$class.foreach(Iterator.scala:891)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
	at scala.collection.TraversableOnce$class.addString(TraversableOnce.scala:357)
	at scala.collection.AbstractIterator.addString(Iterator.scala:1334)
	at scala.collection.MapLike$class.addString(MapLike.scala:344)
	at scala.collection.AbstractMap.addString(Map.scala:59)
	at scala.collection.TraversableOnce$class.mkString(TraversableOnce.scala:323)
	at scala.collection.AbstractTraversable.mkString(Traversable.scala:104)
	at scala.collection.TraversableLike$class.toString(TraversableLike.scala:600)
	at scala.collection.MapLike$class.toString(MapLike.scala:353)
	at scala.collection.AbstractMap.toString(Map.scala:59)
	at java.lang.String.valueOf(String.java:2981)
	at java.lang.StringBuilder.append(StringBuilder.java:131)
	at scala.StringContext.standardInterpolator(StringContext.scala:125)
	at scala.StringContext.s(StringContext.scala:95)
	at kafka.server.ReplicaManager$$anonfun$appendToLocalLog$1.apply(ReplicaManager.scala:707)
	at kafka.server.ReplicaManager$$anonfun$appendToLocalLog$1.apply(ReplicaManager.scala:707)
	at kafka.utils.Logging$class.trace(Logging.scala:36)
	at kafka.server.ReplicaManager.trace(ReplicaManager.scala:139)
	at kafka.server.ReplicaManager.appendToLocalLog(ReplicaManager.scala:707)
	at kafka.server.ReplicaManager.appendRecords(ReplicaManager.scala:458)
	at kafka.server.KafkaApis.handleProduceRequest(KafkaApis.scala:458)
	at kafka.server.KafkaApis.handle(KafkaApis.scala:98)
	at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:65)
	at java.lang.Thread.run(Thread.java:745)
[2018-04-02 12:00:56,729] TRACE Sending PRODUCE response to client ruby-kafka of 48 bytes. (kafka.network.RequestChannel)
[2018-04-02 12:00:56,729] TRACE Socket server received response to send to <IP_ADDRESS>:9092-<IP_ADDRESS>:50269-6, registering for write and sending data: Response(request=Request(processor=2, connectionId=<IP_ADDRESS>:9092-<IP_ADDRESS>:50269-6, session=Session(User:ANONYMOUS,/<IP_ADDRESS>), listenerName=ListenerName(PLAINTEXT), securityProtocol=PLAINTEXT, buffer=null), responseSend=Some(org.apache.kafka.common.network.NetworkSend@190bb315), responseAction=SendAction), responseAsString=Some({responses=[{topic=test,partition_responses=[{partition=0,error_code=-1,base_offset=-1,log_append_time=-1}]}],throttle_time_ms=0}) (kafka.network.Processor)

client:
~/.rvm/gems/ruby-2.4.2/gems/ruby-kafka-0.5.4/lib/kafka/protocol.rb:125:in `handle_error': Kafka::UnknownError (Kafka::UnknownError)

@eliaslevy Yes, that's exactly what I said, the wrong error is sent back to the client ("UnknownError" in your example).

@ijuma sorry, I misread your message. Re: "it's possible for a producer to start using zstd while older consumers can't read the produced messages", how is that different if the protocol is rev'ed?  Older consumers that have not been updated will also not be able to read the messages in that case unless the broker performs codec transcoding when it also transcodes among message format/protocol versions, no?

Yeah, it seems that the protocol version bump would only help if we decide to do decompression in the broker for that case. Another option would be to introduce a topic config for allowable compression types. That would make it possible for the user to only allow the set of compression algorithms that were supported by their clients.

@ijuma @scottcarey @eliaslevy @hachikuji Sorry for the late reply. This week, I have been busy with working the other issues and update the KIPs. It will address all the topics @ijuma pointed out. It seems like I can update the wiki document and open discussion this week.
I also have a sketch on applying zstd's dictionary feature, but since zstd-jni is not supporting the dictionary feature, it will be a future's work. (In fact, I am implementing it.)
But I have another question: the non-existing compression policy problem should be separated into another KIP? Reading the comments, it seems like this problem is not restricted to zstd only, but rather a more general problem. Any ideas?

@bobrik The question I was raising is how a producer knows whether the broker supports the new codec. If we bump the Produce protocol version, then the client can detect older brokers and either throw a meaningful exception to the user or choose a supported codec. Without a protocol bump, older brokers will probably respond with an UnknownServerError, which doesn't give the user any clue about what happened.
cc @edenhill

I think we agreed that any change to the protocol, be it semantics or added bits to bitfields, must bump the protocol version for the affected request types, and any such changes must go through the KIP process.
A producer must be able to tell if it can successfully produce messages with a new compression type, and if not provide a meaningful error message to the user.
What about the consumer side, should the broker down-convert zstd-compressed messages if an older consumer is fetching?

@edenhill The KIP is here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-110%3A+Add+Codec+for+ZStandard+Compression.
I think we were hoping to avoid a message format bump for this change. Basically the idea is to just let the consumer fail if it sees an unsupported compression codec. Would that work for old versions of librdkafka?
On the producer side, though, I thought we could do a little better by bumping the Produce version. Of course the broker may not have the codec installed locally even if it is on a newer version, but the common case (at least initially) is that of an older broker which isn't aware of the attribute change.

consumer fail if it sees an unsupported compression codec. Would that work for old versions of librdkafka?

Yes, it will bubble up an error to the application saying the message could not be decompressed, so it should work as expected.
Broker v1.1.0 seems to fail the ProduceRequest and set the error code to INVALID_MSG, if older brokers do the same thing we could perhaps get away with not bumping the ProduceRequest version.. but bumping the version allows the client to provide a more meaningful error message to the application, change compression type, etc.

Maybe bump the ProduceRequest version and add a InvalidMessageCodec error code and exception to the broker, so at least in the future the introduction of a new codec would not require increasing the version.

@eliaslevy Yeah, good suggestion.

@dongjinleekr
Is there anything I can do to help get this merged in?  We have been using the patch for a while and are getting really good results.  We are now in the process of upgrading to newer versions of Kafka where the patch no longer applies cleanly and I am going to spend some time to port it anyways, plus I really would like to avoid having to do these changes regularly.  So I am happy to put up a pull request to your repo so it is updated for master.

@revans2 Sorry for late reply. (Defending myself, I has been in crunch mode for whole this May.) Could you show me your work? I will check it before submitting revised KIP. Thanks in advance.

@dongjinleekr
Sorry if I came across badly.  Your work and the patch is great.  The conflict is really small, a dependency changed near the newly added zstd dependency causing issue.  I am happy to put up a pull request for it to your branch if that would be simpler for you.  I was mostly interested if there was anything you knew of that I could do to help push this in.  I know in the past the Facebook licenses for zstd was an issue, but that should be resolved now.
zstd-jni is a pure BSD license now and the zstd library from facebook is a BSD 3 clause, which is also OK.  in fact Spark merged in zstd support a few days ago.
https://issues.apache.org/jira/browse/SPARK-19112
So perhaps my real question is to the kafka committers is there something in particular that is blocking this from being merged in?
@onurkaraman
Or really anyone else here what is blocking this from going in?

@ijuma @bobrik @eliaslevy @scottcarey I updated the KIP page and re-sent the discussion mail.
@revans2 The only thing which is blocking this issue is my schedule. Anyway, it is ended! 😄I greatly appreciate your interest in my work!

I have added dictionary support for the Input/Output Stream implementations in zstd-jni-1.3.5-2. Dicationaries are also supported with the direct memory streams (non-standard).
If you have any suggestions how zstd-jni can be made better fit for Kafka, don't hesitate to open an issue in https://github.com/luben/zstd-jni/issues

@ijuma @hachikuji @mjsax Please have a look when you are free. Thanks.

@dongjinleekr Do you have time to address the comments above? We were hoping to get this in by for the 2.1 feature freeze next Monday.

Here is the update for your first two requests - For the final one (i.e., integration test), I can complete it by this Friday, with applying your feedback on these commits (if any). Is this okay?

@hachikuji Here is the update for your first two requests - For the final one (i.e., integration test), I can complete it by this Friday, with applying your feedback on these commits (if any). Is this okay?

@hachikuji Here is the update. Commit cb84f73 includes the integration tests for commit 0b401f9, ccd3119, and d18b557. Please have a look when you are free.

@dongjinleekr You may want to update zstd-jni dependency to 1.3.5-4 : it aligns some edge cases, e.g. decompression of empty frames, with the InputStream requirements.

@hachikuji Thanks for all the review. Do you think this PR will be ready in a day or two for 2.1.0 release?

@lindong28 I'm awaiting an update to fix the down-conversion logic. The patch is a little trickier than expected, so I think it might slip out of the release. I'll review as soon as updates are available though.

@dongjinleekr Sorry for the confusion. Let me know if you need any help.

@dongjinleekr thanks, i will take a look

@dongjinleekr This is what I had in mind: https://github.com/hachikuji/kafka/commit/37e33084ee5281c8764135f0eab59a2e626cd852. Does that look reasonable?

@hachikuji Modification on LazyDownConversionRecords. LazyDownConversionRecordsSend#buildOverflowBatch. Yes, it looks perfect! :+1:

@dongjinleekr Shall we add that commit here? i can push directly to your branch if it is ok with you.

@hachikuji Of course. Or, may I take your commit?

@dongjinleekr Yes, feel free to take it. Can you also rebase?

@hachikuji Please wait 20 minutes.

@hachikuji I just succeeded to rebase. As soon as it passes the test, I will push it here...

@hachikuji Here it is!

@hachikuji Here is the update.

@lindong28 I think we're just about ready here. I know we are a bit late, but how do you feel about this for 2.1?

@hachikuji Usually I would try to find a sustainable long term solution for a problem, which in this case is what is the criteria for decision the feature to be included in a given release. Currently for each feature release we tell developer that major feature needs to be merged before feature freeze date. I would take intelligent risk if the feature is related to availability of Kafka.
For other things like this non-so-critical feature which has waited almost two years since its first proposal, I am inclined to enforce the rule and not include the feature after feature freeze date. The concern is that if we break the rule because we subjectively think the PR is low-risk, it will be harder to enforce the rule in a fair and objective manner. Developers of other major KIPs (e.g. KIP-368) will similarly ask to include their PRs after deadline. Then future release manager has to either reject these which would seem unfair, or always include these which would increase the risk and potentially delay the time of release. It is probably more sustainable to reduce the subjective judgement involved in deciding whether a PR is safe.
So I think we can include this feature if we can agree on and modify the release plan template in such a way that is clear and easy to enforce in an objective manner. Maybe @ijuma can also comment on this. What do you think?

On a personal level, I'd like to see this in 2.1 because it's a great compression algorithm to replace gzip and complement lz4. However, I agree with @lindong28 that it's important to have a fair process for all features. As it happens, this is a bit of a grey area. The time-based releases wiki page (https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan) says:

A month before the release date, the release manager will cut branches and also publish (preferably on the wiki) a list of features that will be included in the release (these will typically be KIPs, but not always). We will leave another week for "minor" features to get in (see below for definitions)
...
Minor Feature - Feature that takes less than 2 weeks to develop and stabilize. Requires mostly one PR to complete the feature.

This PR seems to be low risk unless zstd is used, it's a single PR, it's medium sized (~+600 lines) so it could be classed as a minor feature. The bit where it should take less than 2 weeks to develop and stabilize is less clear though since development was not all done in one go.
Either way can be justified. @lindong28, I think you should make the call as the RM.

By the Apache rules, this is 100% up to the PMC member managing this release.
Also by the Apache rules, any PMC member can choose to create another release that contains this, at any time.  So, hypothetically a 2.1-zstd can follow this if deemed important enough by any individual PMC member.
As an aside, I'll probably fork and build such a release for private use if it does not make it.  This feature is just too good -- its going to save us a LOT of money.

@lindong28 Thanks for the thoughtful response. I agree it's tough for these features that are right on the edge and comparing the relative risk of different features is difficult business.  If you're not comfortable merging this late in the cycle, we can push to the next release. From my own perspective, I think it is a low-risk change, but I'll leave it up to you.

@hachikuji Here is the update, rebased onto the latest trunk!
@ijuma @hachikuji @scottcarey I'm much obliged to you! :)

@ijuma Is this okay?

@dongjinleekr I did a trivial rebase. I will merge once the build finishes.

Merged to trunk and 2.1. Thanks again for your persistence! Great contribution!

Thanks for your contribution @dongjinleekr! Is https://twitter.com/dongjinleekr, your Twitter handle?

Was a param introduced to adjust the compression level?

I hope so.  Half the power of Zstd is its huge range of compression levels
on the pareto frontier.
On Tue, Nov 13, 2018, 09:45 davewat<EMAIL_ADDRESS>wrote:

Was a param introduced to adjust the compression level?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/kafka/pull/2267#issuecomment-438368034, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABJJb7nskilsQmwWkXo_NZOYIl5Qw057ks5uuwVHgaJpZM4LPBAI
.

@dongjinleekr where is the compression level specified?  I can't seem to locate it, even in source.  Right now I am getting better compression with gzip still, and want to adjust the level.  Thanks.

It would only need to exist and be configured on the producer.  Its
irrelevant to the consumer or broker or wire protocol.  Therefore it isnt
in the KIP or specification, and is not related to compatability.
On Wed, Nov 14, 2018, 06:45 davewat<EMAIL_ADDRESS>wrote:

@dongjinleekr https://github.com/dongjinleekr where is the compression
level specified? I can't seem to locate it, even in source. Right now I am
getting better compression with gzip still, and want to adjust the level.
Thanks.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/kafka/pull/2267#issuecomment-438685807, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABJJb_54duoFa9MfwEJZFbaascNVvrWGks5uvCyAgaJpZM4LPBAI
.

@scottcarey while it is an anti-pattern, the broker can compress/recompress messages depending on the the topic's compression.type configuration, which could be set to zstd .

Producer configs have to be in the KIP (any config is considered public API). I don't think there's a way to configure the compression level at this point. If someone wants to contribute that, it would make to allow it for other compression types too.

IIRC this was disallowed for zstd.
On Wed, Nov 14, 2018, 08:21 Elias Levy<EMAIL_ADDRESS>wrote:

@scottcarey https://github.com/scottcarey while it is an anti-pattern,
the broker can compress/recompress messages depending on the the topic's
compression.type configuration, which could be set to zstd .
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/kafka/pull/2267#issuecomment-438722803, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABJJb6Qh7uWifh-g4dmuGvve64EulvFIks5uvEMXgaJpZM4LPBAI
.

I may be mistaken.  At least some of this was disallowed for zstd when
coverting to clients that dont have zstd support.  That is distinct from
broker recompression.
A broker could recompress and the level it does so is a major factor in
broker cpu use if so.
On Wed, Nov 14, 2018, 08:21 Elias Levy<EMAIL_ADDRESS>wrote:

@scottcarey https://github.com/scottcarey while it is an anti-pattern,
the broker can compress/recompress messages depending on the the topic's
compression.type configuration, which could be set to zstd .
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/kafka/pull/2267#issuecomment-438722803, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABJJb6Qh7uWifh-g4dmuGvve64EulvFIks5uvEMXgaJpZM4LPBAI
.

Setting the topic config to zstd while not compressing in the producer is allowed for all compression algorithms.

Yes i got automatic broker downconversion confused with topic compression.
I was under the impression both were disabled for zstd but only one is.
On Wed, Nov 14, 2018, 08:32 Ismael Juma<EMAIL_ADDRESS>wrote:

Setting the topic config to zstd while not compressing in the producer is
allowed for all compression algorithms.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/kafka/pull/2267#issuecomment-438727023, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABJJb-HdQwblXEQw_7r0gr9dbICoe7nWks5uvEW1gaJpZM4LPBAI
.

@scottcarey I wouldn't say it's irrelevant on the broker.  We have older producers that don't/ won't speak zstd, but our (testing) Brokers force zstd on the topic for storage/ space requirements.  The broker's ability to set the compression level is key in our use case.
I just can't figure out where in this PR the level is set - though now I am thinking we are just utilizing the default (3).

@davewat @scottcarey @ijuma Sorry for the late reply. In fact, I already investigated this issue (support compression level for zstd) but I concluded that it would be much better to put this issue into separated one and focus on implementing the ZSTD feature only.
Here is why: All compression codecs (i.e., Gzip, Snappy, LZ4 and ZSTD) support some parameters to change the degree of compression; However, only LZ4 and ZSTD supports the concept of 'level' - in the case of GZIP and Snappy, they require the block size parameter, not 'level'.
To make the compression level feature available, we must modify the API signatures of MemoryRecordsBuilder to support compression level, and add some validation logic[^1]. It requires additional modifications to read ProducerConfig value and pass it into MemoryRecordsBuilder. Of course, this work requires a bunch of modifications and some policies on various codecs. It is why I decided to put off this issue and use the default level for ZSTD, that is, 3.
How about your opinion? Do you really need it? Does it sound reasonable?
If the answer is Yes, please file it to Jira with 'needs-kip' tag; Then, I will take the issue - I will make the proposal.
[^1]: Check whether given CompressionCodec supports the concept of 'level,' and whether given compression level is valid for the CompressionCodec. (e.g., ZSTD supports 22 levels but LZ4 supports 4 levels only.)

@dongjinleekr , zstd also supports negative levels for faster compression, it's equivalent of the --fast X in the CLI

@dongjinleekr our use case needs it, not sure if we are just a corner case.  I have created the issue in Jira.  Thanks!
https://issues.apache.org/jira/browse/KAFKA-7632

Gzip supports levels 1 to 9.  The api in java hides it a bit, its has a
very significant impact on cpu use when compressing.  I suspect much of the
reason that recompressing broker side has a bad rap is because it was done
with the default level of 6. Level 1 is about 10x as fast.  Zstd lv 3 is
faster than that at higher compression rates, however.
Being able to tune the compression level is very important.  Compression is
all about CPU to I/O tradeoffs, and what tradeoff is the best is use case
dependent.  Zstd ranges from snappy-like speeds and compression levels to
lzma-like compression levels.  Setting compression levels will get even
more important if dictionary support is added.
On Thu, Nov 15, 2018, 07:51 Lee Dongjin<EMAIL_ADDRESS>wrote:

@davewat https://github.com/davewat @scottcarey
https://github.com/scottcarey @ijuma https://github.com/ijuma Sorry
for the late reply. In fact, I already investigated this issue (support
compression level for zstd) but I concluded that it would be much better to
put this issue into separated one and focus on implementing the ZSTD
feature only.
Here is why: All compression codecs (i.e., Gzip, Snappy, LZ4 and ZSTD)
support some parameters to change the degree of compression; However, only
LZ4 and ZSTD supports the concept of 'level' - in the case of GZIP and
Snappy, they require the block size parameter, not 'level'.
To make the compression level feature available, we must modify the API
signatures of MemoryRecordsBuilder to support compression level, and add
some validation logic[^1]. It requires additional modifications to read
ProducerConfig value and pass it into MemoryRecordsBuilder. Of course,
this work requires a bunch of modifications and some policies on various
codecs. It is why I decided to put off this issue and use the default level
for ZSTD, that is, 3.
How about your opinion? Do you really need it? Does it sound reasonable?
If the answer is Yes, please file it to Jira with 'needs-kip' tag; Then,
I will take the issue - I will make the proposal.
[^1]: Check whether given CompressionCodec supports the concept of
'level,' and whether given compression level is valid for the
CompressionCodec. (e.g., ZSTD supports 22 levels but LZ4 supports 4
levels only.)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/kafka/pull/2267#issuecomment-439087933, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABJJb5-43B16StZuZxPXMwzcGAYTUhHrks5uvY2JgaJpZM4LPBAI
.

When I added zstd support for Go library, I also added compression level support:

https://github.com/Shopify/sarama/pull/1044

It applies to zstd and gzip.

@scottcarey @luben Thank you for the correction. Right, ZSTD now supports negative compression level (#1 #2) and GZIP is also able to use compression levels, although it is blocked in the official API - but we can make use of it with some workaround. These features should be supported in the implementation.
@davewat Thank you for filing the issue. I just updated the issue applying the comments here and now working on the KIP. I will give you slack when I complete the document and open the discussion thread.
@bobrik Great. Sarama always guides us the direction! I will include sarama's case in the KIP.

@davewat @scottcarey @eliaslevy @ijuma I just opened the discussion thread in dev mailing list. Let's continue the discussion there.
cc/ @bobrik @luben

@dongeforever , just released zstd-jni-1.3.7-2 with support to query min/max compression levels


Usage with srcdoc & sandbox

I'm displaying some untrusted content in an iframe using srcdoc and sandbox (not allowing any scripts to run).
<iframe sandbox="allow-same-origin allow-popups allow-popups-to-escape-sandbox" srcdoc="{{iframe_srcdoc}}"></iframe>

Using this method the parent is able to autosize the iframe by listening for its load event and then determining the height of the iframe content and setting it in a manner such as:
$iframe.height($iframe[0].contentWindow.document.documentElement.height());

However, I was interested in trying out some of this repo's different height calculations and other features, so my question is: do you think it makes sense for this project to support cases like this where a JavaScript file cannot be loaded in the iframe itself?

Nope, you need to add JS to iFrame page.
On Tuesday, 4 October 2016, Phil Freo<EMAIL_ADDRESS>wrote:

I'm displaying some untrusted content in an iframe using srcdoc and
sandbox (not allowing any scripts to run).

Using this method the parent is able to autosize the iframe by listening
for it's load event and then determining the height of the iframe content
and setting it in a manner such as:
$iframe.height($iframe[0].contentWindow.document.documentElement.height());
However, I was interested in trying out some of this repo's different
height calculations and other features, so my question is: do you think it
makes sense for this project to support cases like this where a JavaScript
file cannot be loaded in the iframe itself?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/davidjbradshaw/iframe-resizer/issues/413, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAkHEqKCYtdc6cN4Fvfs_hbpCYzcurGIks5qwrFNgaJpZM4KOGRn
.

--
Sent from my iPhone.

Sorry I wasn't clear... I know that this library is currently designed that way. I was proposing a feature request where the same iframe autosizing intelligence could be applied from the parent/host page to the iframe (via contentWindow), which is important when using srcdoc+sandbox.
However if you think that's out of scope, feel free to close.

It's beyond scope to create a generic solution, but it might be a lot easier to do a one off hack to make it work.

@davidjbradshaw Has anyone tried to inject your library into the parent/host iframe srcdoc attribute after loading the external site in the parent/host page?  Do you think this will work?


Relationship to Hayagriva?

What is the difference between the two projects and their goals? Does one depend on the other? etc.

Hayagriva might depend on this in the future.

I will keep my eye on it! Thanks!


Page DID

Connect the endpoints:

[x] didMeta: DIDMetaViewModel;
[ ] fromDidList?: string; // if coming from DIDList, this will be the DIDList's query
[x] #317
[ ] #295
[ ] #319
[ ] didFileReplicasDComDOM: UseComDOM<FilereplicaStateDViewModel>
[ ] #290
[ ] didContentsComDOM: UseComDOM<DIDViewModel>
[ ] didDatasetReplicasComDOM: UseComDOM<DIDDatasetReplicasViewModel>

didParentsComDOM -> https://rucio.github.io/documentation/html/rest_api_doc.html#tag/Data-Identifiers/paths/~1dids~1{scope_name}~1parents/get
didFileReplicasComDOM -> https://rucio.github.io/documentation/html/rest_api_doc.html#tag/Replicas/paths/~1replicas~1list/post
not sure about what state should be returned
didFileReplicaStateDViewModel: will need to group data in the usecase
didContentsComDOM: https://rucio.github.io/documentation/html/rest_api_doc.html#tag/Data-Identifiers/paths/~1dids~1{scope}~1/get (with recursive flag)
didDatasetReplicasComDOM: https://github.com/rucio/rucio/blob/7f0d229ac0b3bc7dec12c6e158bea2b82d414a3b/lib/rucio/core/replica.py#L2737

@ThePhisch  I am a bit lost in figuring out what view models you need. Could use some help!
To put it simply, there are 2 view models you request: FileReplicaStateViewModel and FileReplicaStateDViewModel
In PageDIDFilereplicasD
https://github.com/rucio/webui/blob/3932395f8b344fd6eef9c5ae3361e17033a0c6d1/src/component-library/Pages/DID/PageDIDFilereplicasD.tsx#L19
you use a ComDOM to stream FileReplicaStateDViewModel
and then you print the scope:name in the table column.
https://github.com/rucio/webui/blob/3932395f8b344fd6eef9c5ae3361e17033a0c6d1/src/component-library/Pages/DID/PageDIDFilereplicasD.tsx#L27
So, looking at the entity FileReplicasStateD
https://github.com/rucio/webui/blob/3932395f8b344fd6eef9c5ae3361e17033a0c6d1/src/lib/infrastructure/data/view-model/page-did.d.ts#L14
it seems the entity contains the scope and name of a file in a dataset.
In the same file, you have a replica ComDOM that streams FileReplicaStateViewModel
https://github.com/rucio/webui/blob/3932395f8b344fd6eef9c5ae3361e17033a0c6d1/src/component-library/Pages/DID/PageDIDFilereplicasD.tsx#L63
This subpage/view renders the rse:state. However, how does this subpage know for which DID it is rendering the information for?
For example in the PageDIDFilereplicasD, when a user selects a file, the PageDIDFilereplicas  should render the replicas of the selected file. How is this information being passed?
I somehow feel we might need to re-define these 2 view models. FileReplicaStateDViewModel should contain a list of files and each file should contain a FileReplicaState.
For cases when the DID type is File, the view model will just contain FileReplicaState
Let's discuss this in person if you prefer.
Minor request: Can we rename Filereplicas to FileReplicas

working on it

All nextjs endpoints are done @ThePhisch :)

closed with #341