OpenDAL Java Bindings

Could I contribute to this issue?

@kidylee I'd suggest you briefly outline your ideas.

Do we need to build the C-binding before starting on the Java binding?

@Xuanwo Using jni-rs, we can build Java bindings without a full functional C binding. But JNI is not always the best solution.

@kidylee I'd suggest you briefly outline your ideas first.

Thanks for your advice. As @tisonkun said, we don't need a C-binding for Java, a cdylib is good to go.
A few candidates I'm looking into:

raw jni with jni-rs
j4rs - looks like this is a promising approach.
flapigen - the usage is more like PyO3, but it generate C-API wrapper first.

FYI - if we have a C binding, using JNA can avoid JNI glue code. But it still uses the JNI technology underneath.

FYI - if we have a C binding, using JNA can avoid JNI glue code. But it still uses the JNI technology underneath.

Thanks @tisonkun, it is a solid approach.
Here are snippets of my approach:
public void testFileSystem(){
        FileSystemService service  = new FileSystemBuilder()
                .root("/tmp")
                .build();

        var op = new OperatorBuilder(service)
                .build();

        op.write("hello.txt","hello world");
        var rs = op.read("hello.txt");

        assert "hello world".equals(rs);
    }

For the rust part, I'm taking assumption where operator instance keeps resource which we don't want to re- create it per calling. So Operator.java will hold a pointer to the rust operator instance, and below are the method to create and free it:
#[no_mangle]
pub extern "C" fn java_get_operater() -> *const i32 {
    let op = Box::new(Operator{});
    let op_ptr = Box::into_raw(op);
    op_ptr as *const i32
}

#[no_mangle]
pub extern "C" fn java_free_operater(ptr: *mut Operator)  {
    unsafe {
        let _ = Box::from_raw(ptr);
    }
}

@kidylee I'm glad to review a MVP, go ahead!

https://rust-lang.github.io/wg-async/vision/submitted_stories/status_quo/aws_engineer/using_jni.html
Findings for JNI x async.

Findings for JNI x async -

https://rust-lang.github.io/wg-async/vision/submitted_stories/status_quo/aws_engineer/using_jni.html
Why JNIEnv / JObject is not Send and how and I do an asynchronous callback? jni-rs/jni-rs#310

I found a possible solution: we can store JavaVM as thread local variable, and when we need to execute Java code, attach the rust-managed thread to obtain Jenv. Additionally, we cloud hold Jenv as thread local variable.
#[no_mangle]
pub extern "system" fn Java_org_apache_opendal_Operator_getOperator0(
    env: JNIEnv,
    _class: JClass,
    input: JString,
    params: JObject,
) -> jlong {
    // build operator
    let operator = 1;

    let java_vm = Arc::new(env.get_java_vm().unwrap());

    let runtime = Builder::new_multi_thread()
        .worker_threads(4)
        .on_thread_start(move || {
            JAVA_VM.with(|cell| {
                *cell.borrow_mut() = Some(java_vm.clone());
            });
            // println!("thread started {:?}", box_ptr);
        })
        .build()
        .unwrap();

    Box::into_raw(Box::new((operator, runtime))) as jlong
}

And then we can use async/await without suspending the caller thread.
#[no_mangle]
pub unsafe extern "system" fn Java_org_apache_opendal_Operator_async_write(
    mut env: JNIEnv,
    _class: JClass,
    ptr: *mut (Operator, Runtime),
    file: JString,
    content: JString,
    future: JObject,
) {
    let (op, runtime) = &mut *ptr;

    let file: String = env.get_string(&file).unwrap().into();
    let content: String = env.get_string(&content).unwrap().into();
    let future = env.new_global_ref(future).unwrap();

    let x = async move {
        op.write(&file, content).await.unwrap();
        JAVA_VM.with(|cell| {
            let java_vm = cell.borrow();
            let java_vm = java_vm.as_ref().unwrap();
            let mut env = java_vm.attach_current_thread().unwrap();
            env.call_method(
                future,
                "complete",
                "(Ljava/lang/Object;)Z",
                &[JValue::Bool(1)],
            )
            .unwrap();
        });
    };
    runtime.spawn(x);
}

If the community approves this solution, I'd love to add asynchronous API to Java binding.

Thanks @ShadowySpirits , looking forward for this feature.
Regard to the approach, I think we need other design to manage thread lifecycle in Rust side, instead of creating it each time when calling native method.
One approach is we can use the register method when Java loading dylib. You can check this doc here.

If a library L is statically linked, then upon the first invocation of System.loadLibrary("L") or equivalent API, a JNI_OnLoad_L function will be invoked with the same arguments and expected return value as specified for the JNI_OnLoad function.

We can create tokio runtime in this OnLoad method.

When the class loader containing a statically linked native library L is garbage collected, the VM will invoke the JNI_OnUnload_L function of the library if such a function is exported.

Same as register, tokio runtime could be destroyed in OnUnload method.
JVM guaranteed dynamic lib will only be loaded once:

A library L that is statically linked will prohibit a library of the same name from being loaded dynamically.

We can create tokio runtime in this OnLoad method.

Thanks @kidylee ! Great idea, I will look into this approach.

JNI ERROR (app bug): global reference table overflow (max=65535)

This limitation is enough for most scenarios. But we do need to eliminate this risk. One solution could be to provide users with the option to either throw an exception or initiate a blocking operation.

This limitation is enough for most scenarios

If each call obtains one global ref, I'm afraid that the iops is limited by 65535.

If each call obtains one global ref, I'm afraid that the iops is limited by 65535.

IMO, this limitation means we can not hold over 65535 global refs at the same time. However, this limit is not a significant concern since the lifetime of each global reference ends with the completion of its respective operation. Therefore, our theoretical IOPS can exceed 65535.
And most blob service limit IOPS, 65535 is already a big number: https://www.alibabacloud.com/help/en/object-storage-service/latest/usage-notes

We don't need to address all possible problems right now. We can document this limitation and fix it in the future.

I agree with @ShadowySpirits, the scenarios of OpenDAL are about file access, bandwidth and processor open file limitation are less than this global references threshold, as long as we managed it well(no leaks and release immediately).


SQL formatting configuration

We need to add SQL formatting configuration: tabs, commas, keywords, etc.

Could this include a way of telling Dbeaver to NOT insert tabs for indentation? I have configured it with 'Insert spaces for tabs', but auto-indentation still inserts tabs on new lines.

@quiescer Yes, and this too.

One Product to look at when it comes to SQL formating can be RedGate SQLPrompt that has a really nice formater.
Red Gate Software Ltd
Youtube

Compact SQL formatter was added in 4.3.
Also for both default and compact formatters the only configuration is:

Keywords case
Indent size
Tabs or spaces

All these parameters are consistent with SQL auto-formatting and can be configured with the same preference parameters.


Update CZML validation document and test.

Adds new properties from #4547.

Amazing work, @shunter!


Add MessagePublished Event

Also renamed event MessageReceived to MessageQueuedToSubscriber and added unit tests for all events.

Looks like this would also fit our need. Thanks for working this out.

@BoeseB, I'm glad it works for you.  I know it's not exactly what you wanted.  Thanks for helping make PubSub better!


Update add-custodians-to-case.md

tested and could see all types of groups showing up here. So changed it from Distribution groups to groups

Issue# 6575 is closed by Author so closing this PR. Thanks


Add pylint as dependency?

Pylint is not set as a dependency. You have to install it manually. Should it be a dependency at all?

It shouldn't be a.dependency of PQAnalysis, but it is included as an optional dependency in test. For development including the test dependencies just write
pip install .[test]


Upgrade envoy to v1.7.0

Resolves #443
To test this, I've upgraded the Envoy in an existing Contour setup and everything continued to work fine. If there is something more I can do to verify this change, please do tell.

@alvaroaleman thanks for your contribution. I'm marking this 0.7 and will review it when we start the 0.7 cycle.
In the interim, if you are able, please deploy a contour against this envoy and have a look at the logs coming out of envoy. Before we land this change we need to resolve the constant series of warnings envoy emits.

re: no image, thanks, i'll nag upstream to get it built.
On 29 August 2018 at 12:25, Alvaro Aleman<EMAIL_ADDRESS>wrote:

@davecheney https://github.com/davecheney Thanks for heads up, I'll try
looking into the warnings.
With regards to envoy 1.7 .1: While that version got released, there is no
image published for it, if we wanted it we would have to build it
ourselves: envoyproxy/envoy#4121
https://github.com/envoyproxy/envoy/issues/4121
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/heptio/contour/pull/650#issuecomment-417055875, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAcA40_jENMXrlEbO8gjOZe5tRT9G0rks5uVtyBgaJpZM4WMdbA
.


About when to do SendSuccessNotification Method in IntuneScepServiceClient.java

IntuneScepServiceClient.java

From the documentation, I've understood that the certificate will be processed by the certificate authority after it is issued as follows:

Client certificate is issued,
The required information on the issued client certificate is gathered and Intune is notified with the SendSuccessNotification method
(If the method finishes successfully) The certificate is distributed to mobile devices

Is this understanding correct?
*We understand that swapping 2) and 3) is a deprecated order.

It is not clear what you are asking. But that is the order things should happen yes.


Support Vertx 3.8.x and 4.x.x, Issue with SockJSHandler

A dependency upgrade to vertx 3.8.x and 4.x.x is not possible due to the change of the method signature of (SockJSHandler -> Router) SockJSHandler.socketHandler(Handler<SockJSSocket> handler).
https://github.com/vert-x3/vertx-web/blob/4711942d7801d4dab54a46b256cb72edc908cd58/vertx-web/src/main/java/io/vertx/ext/web/handler/sockjs/SockJSHandler.java#L68
First occurred on:
https://github.com/mcollovati/vertx-vaadin/blob/a03b8b0d33174b2e4af5337ed6b65a5e29bfe8e7/vertx-vaadin-flow-parent/vertx-vaadin-flow/src/main/java/com/github/mcollovati/vertx/vaadin/sockjs/communication/SockJSPushHandler.java#L167
It looks like the library only needs to be recompiled, but I am not sure if there are any side effects. The router which is returned is meant to be registered at a parent router.
I think the upgrade should be applied to the Vaadin 14 version too because it's an LTS release.

The mentioned change has been introduced with vertx 3.8.3, so I think i will ignore versions from 3.8.0 to 3.8.2.
There will be a vertx-vaadin-flow:14.0.X and a vertx-vaadin-flow:14.1 based on vertx 3.8.3+

This sounds great 👍

Published artifacts:
vertx-vaadin-flow:14.0.10 for Vaadin 14.0.x and Vert.x 3.8.3+
vertx-vaadin-flow:14.1.0 for Vaadin 14.1.x and Vert.x 3.8.3+

Again thanks for fast fixing. When will it be available on Maven?


Changed the directory structure

What do you think of this approach to lessons? I thought it might be nice to have them grouped by major topic (e.g. Git, R, Python) and have multiple "chapters" in each repository, with the README.md serving as the Table of Contents.

Great idea!


fixed sample account lookup bug and got scripts working in sample directory

I fixed the following bugs:

In the "sample" directory the config lookup failed as it was not finding "trading_ig_config.py" in the current directory when it is located one level up.

The "trading_ig" library inclusion in the "sample" directory was failing because it was looking in the current directory instead of one level up.

The lightstreamer unsubscribe was missing an argument which I added.

Fixed an account lookup issue. It was checking if the first account returned was matching with the config and if not then failing with an error. In my case there are multiple accounts so it needed to finish interating through the accounts to see if there was a match.

The streaming samples still don't return any results for me. I'm looking into that but at least the scripts run now without having to set the environment variables which is insecure.

@whittlem thanks for your contribution, sorry the documentation was so poor, I hope it is better now.
I am closing this. There's a new FAQ clarifying how to run the sample code


Add string compression functions

Adds compress(string) and decompress(string) functions that use Gmod's built-in compression system. decompress throws on an input string that isn't LZMA compressed.
The compression is very fast, with inputs of 64k+ characters barely taking any time, so I think the op counter limit is fine where it is.
I swear I'm going crazy thinking these existed. Maybe it was something else I was testing.

You definitely need to limit input string length because compress is not a linear o(n) algorithm.

Was gonna merge these after workshop deploy :P


Remove RemoveResponse and merge functionality into DeleteResponse

[#4377] refactor(responses): remove duplicate code of RemoveResponse and DeleteResponse
What changes were proposed in this pull request?
Removed the RemoveResponse class and merged its functionality into DeleteResponse to eliminate duplicate code.
Why are the changes needed?
This change reduces redundancy by combining similar classes into one.
Fix: #4377
Does this PR introduce any user-facing change?
No user-facing changes.
How was this patch tested?
Tested by ensuring all existing functionality works with the updated DeleteResponse class.

I don't think simply removing RemoveResponse class is enough, you need to do some code changes in other places.

@jerryshao can you please explain what else i should change ?

I search for RemoveResponse [search], there are some use cases which need to be addressed too.
In addition, I see the CI of your PR is failing. You can run ./gradlew spotlessApply to fix the code formatting and ./gradlew clean build to test your code changes locally. [Refer to How to build guide]


Projects details Page

https://github.com/user-attachments/assets/8b1f5e68-89e7-4045-a8cb-c3fc749d68bc
@rishicds
** Please merge after #240  Or conflicts might arise.**

@rishicds If you want any changes let me know I will do them .

fix conflicts

@rishicds Resolved

Merged


Icon web component

Web-components icons are not visible in Percy snapshots
It might have to do with the elements package not being built first

This didn't produce the icons in Percy as well
They appear in storybook so the problem is something else


fix(mongodb): use net.peer namespace for mongo host and port

Which problem is this PR solving?

#1243

Short description of the changes

Changed the attributes "net.host.name" and "net.host.port" to the correct ones according to spec "net.peer.name" and "net.peer.port".

From the spec:

Sometimes host name is only available to instrumentation as a string which may contain DNS name or IP address. net.peer.name SHOULD be set to the available known hostname (e.g., "<IP_ADDRESS>" if connecting to an URL https://<IP_ADDRESS>.com/foo).

Thank you @lmolkova for the help

approved by the component owner. merging

@osherv is the component owner, but I need another approve to merge:

Merging is blocked
Merging can be performed automatically with 1 approving review.


file not overriding child default values when alias

I have a chart with a sub-chart mk-backend (aliased as backend).
dependencies:
  - name: "mk-backend"
    repository: "@stable"
    version: "x.y.z"
    alias: backend

Since I upgraded from Helm v2.x to v3.0.0, the overrides for backend (defined in a file (-f) or with --set) are ignored: it is the default values that are used.
It seems to happen only when alias are used.
Output of helm version:
version.BuildInfo{Version:"v3.0.0", GitCommit:"e29ce2a54e96cd02ccfce88bee4f58bb6e2a28b6", GitTreeState:"clean", GoVersion:"go1.13.4"}
Output of kubectl version:
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:23:11Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T19:09:08Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}

Cloud Provider/Platform (AKS, GKE, Minikube etc.):
Minikube v1.5.2

Can you provide an example to reproduce this issue on our end? nginx-ingress is a good example to demonstrate this issue. Thanks!

This is not a question, this is a bug report. Here is an archive containing a minimal chart reproducing this error.
It defines a main chart and it's two sub-charts child and child2. Both sub-charts are identical. The difference is that child2 is aliased as childalias.
Unpack the archive, then in build.sh, set a repo name, url, user and password (for instance chartmuseum). In addition you might want to install helm-push plugin: helm plugin install https://github.com/chartmuseum/helm-push.
Have a k8s cluster running (for instance minikube).
Finally run ./build.sh.
This is the printout you'll get:
---
# Source: main/charts/child/templates/child-conf.tpl
apiVersion: v1
kind: ConfigMap
metadata:
  name: RELEASE-NAME-child
data:
    param3: runtime
    param4: default
---
# Source: main/charts/childalias/templates/child-conf.tpl
apiVersion: v1
kind: ConfigMap
metadata:
  name: RELEASE-NAME-childalias
data:
    param3: runtime
    param4: default
---
# Source: main/templates/main-conf.tpl
apiVersion: v1
kind: ConfigMap
metadata:
  name: RELEASE-NAME-main
data:
    param1: runtime
    param2: default

then later:
apiVersion: v1
items:
- apiVersion: v1
  data:
    param3: runtime
    param4: default
  kind: ConfigMap
  metadata:
    creationTimestamp: "2019-11-28T13:36:46Z"
    name: test-child
    namespace: test
    resourceVersion: "117013"
    selfLink: /api/v1/namespaces/test/configmaps/test-child
    uid: 6ba49810-98d2-44a1-9189-f277a542155c
- apiVersion: v1
  data:
    param3: default
    param4: default
  kind: ConfigMap
  metadata:
    creationTimestamp: "2019-11-28T13:36:46Z"
    name: test-child2
    namespace: test
    resourceVersion: "117015"
    selfLink: /api/v1/namespaces/test/configmaps/test-child2
    uid: 5ccae5a1-1a25-4406-bbf3-b9e75bc43de7
- apiVersion: v1
  data:
    param1: runtime
    param2: default
  kind: ConfigMap
  metadata:
    creationTimestamp: "2019-11-28T13:36:46Z"
    name: test-main
    namespace: test
    resourceVersion: "117014"
    selfLink: /api/v1/namespaces/test/configmaps/test-main
    uid: b4b581fe-591a-40f2-81fc-6fa9e8d5e030
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

The first printout is the result of helm template. It shows three config maps (one from the chart main, one from chart child and one from chart child2 aliased childalias). For both child and child2 the cm value is:
data:
    param3: runtime
    param4: default

This is because we run helm template with --set childalias.param3=runtime,child.param3=runtime, so param3 takes the value runtime. Note that there is a first issue here since param4 should take the value override since it is overridden in main/values.yaml.
Now let's move to the second printout section which corresponds to helm install (still with --set childalias.param3=runtime,child.param3=runtime), then kubectl get cm -o yaml -n test. Here we see that for childalias, the value of param3 is default (which is wrong) while it is runtime for child (which is right). In addition, both values for param4 are wrong since it was expected override.

Anyone having a look at this issue?

There was a mistake in the .tar.gz which is fixed now and solve some of the issues raised in the post above (which is updated).

You can check --dry-run --debug. In COMPUTED VALUES section I have found too many dependencies :o
For instance (let's say that it is myapp chart):
- name: rabbitmq
  alias: myrabbitmq
  version: 6.10.1
  repository: https://kubernetes-charts.storage.googleapis.com
  condition: myapp.myrabbitmq.enabled,myrabbitmq.enabled

myapp.myrabbitmq.enabled because I include myapp chart in stack chart for example. Then I want to to disable myrabbitmq. values.yaml in stack chart:
myapp:
  myrabbitmq:
    enabled: false

When I run helm upgrade --dry-run --debug etc. I can find in COMPUTED VALUES:
...
myapp:
  myrabbitmq:
    enabled: false <- that is fine
  rabbit: <- wait, what!?
    ...

It works fine in Helm2 btw. It does not work in Helm 3.0.1 and 3.0.0 as @marcadella has pointed out before.

Hi Folks, Is there any news on this?

same issue here, can't refer to dependencies by alias in parent values.yaml

contra-datadog.zip
Made a super simple test case.
helm template . --set foo.apiKeyExistingSecret=bar --debug --dry-run

Expected output:
install.go:159: [debug] Original chart version: ""
install.go:176: [debug] CHART PATH: /Users/gajus/Documents/dev/foo/contra-datadog

---
# Source: contra-datadog/charts/datadog/templates/secrets.yaml
apiKeyExistingSecret: bar

Actual output:
install.go:159: [debug] Original chart version: ""
install.go:176: [debug] CHART PATH: /Users/gajus/Documents/dev/foo/contra-datadog

---
# Source: contra-datadog/charts/datadog/templates/secrets.yaml
apiKeyExistingSecret: foo

Turns out this issue is pretty well known:
#2993
#3457
#3909
#4300
#4390
#7061
#7093
#7413

Any updates on this one?

I am unaware of any community members looking into fixing this. The core maintainer who was primarily responsible for this piece of Helm stepped down as a maintainer a few years back.
Please feel free to contribute bug fixes.

Tested all afternoon 😂

This is not a fix, but @whirm and whomever else might be blocked by this: it seems that if you use the helm 2 way of specifying requirements (same dependencies block but in a separate file next to Chart.yaml called requirements.yaml) then alias,condition, etc work as expected even in Helm 3!
Hope this at least helps some people while a fix is worked on!

This is not a fix, but @whirm and whomever else might be blocked by this: it seems that if you use the helm 2 way of specifying requirements (same dependencies block but in a separate file next to Chart.yaml called requirements.yaml) then alias,condition, etc work as expected even in Helm 3!
Hope this at least helps some people while a fix is worked on!

Nice! I'll try it right away,
Thanks a lot!

This is not a fix, but @whirm and whomever else might be blocked by this: it seems that if you use the helm 2 way of specifying requirements (same dependencies block but in a separate file next to Chart.yaml called requirements.yaml) then alias,condition, etc work as expected even in Helm 3!
Hope this at least helps some people while a fix is worked on!

@almahmoud: I've just tried it but it's behaving exactly the same way than having it on Chart.yaml for me. I'm using file:// URIs for the repository spec. I don't know if that should make any difference (I'm using the exact same helm version than you). Any suggestion?

@almahmoud: I've just tried it but it's behaving exactly the same way than having it on Chart.yaml for me. I'm using file:// URIs for the repository spec. I don't know if that should make any difference (I'm using the exact same helm version than you). Any suggestion?

Ok, that's what I've just discovered: For me aliases will work if you specify the aliased chart version and it works both helm 2 and 3 style.
So this works:
  - name: foo
    version: 1.0.0
    repository: "file://../foo"
    alias: bar

And this does not:
  - name: foo
    repository: "file://../foo"
    alias: bar

Weird. I have version specified but the subchart values using aliased name are ignored if specifying Helm 3 way (real values are using original name) but works as expected when I specify things separately eg https://github.com/galaxyproject/galaxy-helm/pull/276/files .

For background context: the dependency resolver has not been worked on since it was first introduced in 2016. The maintainer who contributed this piece of code stepped down back in 2017-2018ish, and since then pkg/downloader has been in "bugfix" mode ever since.
Issues like this one have riddled the dependency resolver for quite some time. It is probably worth considering deprecating and re-writing the entire dependency resolver system at this point. As in, we should rev the Chart.yaml's apiVersion to v3 and write a new dependency resolver for new charts going forward.
It's just not worth the time and effort to write and review code to fix this problem in the current system. There are WAY too many edge cases and uncaught if blocks making it very difficult to review.
If that sounds like something someone is interested in tackling, feel free to reach out and I'll be happy to provide next steps.

@bacongobbler I would be willing to take a shot at rewriting the dependency resolver. If you can point me in the right direction I will take a look.

bump!

+1

I don't know who, besides @bacongobbler (who seems to have gotten busy with $dayjob), has enough context to get someone started on this. If someone has the hours to spend (and I'm thinking it'll take a week or two to get up to speed and work through a solution), coming to the community helm meeting might be a good place to get started.

Bump

this issue seems to hold a lot of similar but not exactly equivalent bug report
I see it's marked as closed, but i still got some issues that i assume are related enough to this topic to reopen it.
my problem exactly:
one chart parent depends on DEP and child, child depends on DEP, DEP files appear only once in the result config, depsite all possible combination of aliases

this issue seems to hold a lot of similar but not exactly equivalent bug report I see it's marked as closed, but i still got some issues that i assume are related enough to this topic to reopen it.
my problem exactly: one chart parent depends on DEP and child, child depends on DEP, DEP files appear only once in the result config, depsite all possible combination of aliases

Same issue here, with Helm v3.10.3

Why this has issue been closed? Still an issue for helm v3.10.3

Yes, this is still an issue. Explanation given by @malmiteria is concise and correct.

Why this has issue been closed?

Because it's been open since November of 2019 and it hasn't been a big enough problem for anybody to contribute a fix. It was, apparently, insignificant enough that nobody even mentioned it for 4 months, so it was automatically closed.
If someone wants to offer to do something about it, it can be reopened but without a contribution it'll just have to be considered the expected behavior.

@joejulian could this behavior be at least mentioned in the docs as suggested in 872? In my use case, I did not do anything fancy - two local subcharts and one has to be deployed several times. It seems like straight from the documentation, but it was not working until I added version: ">=0.0.0" to the aliased subchart (helm v3.11.3).


Need a wheel-event like handler for touch scrolling

Swiping the content up or down shows the flickering of content being drawn after it enters the viewport, just like the issue we used to have with wheel scrolling. A similar solution (predict scrolling, make sure content is drawn) will probably work here, but I am unsure whether there is an easy way to see such scrolling coming, and whether analyzing touch events ourselves is a sane way to go here.
Any touch event expert want to chime in?

This is a very tricky situation on mobile devices since the onscroll event is normally updated until the scroll is finished, or if a momentum scroll is used, til it stops.
http://cubiq.org/iscroll-5
That would be an interesting place to start, maybe the underlying engine will help.  The idea is to manage the scrolling without default behavior.
I think you could incorporate this into codemirror.

@marijnh, any progress with a solution?

I've looked into this, but didn't find a good solution. The touch events' correspondence with the scrolling is platform-specific and hard to predict. Mobile browsers fire scroll events only in ~350ms intervals, since they do their scrolling on hardware, and don't actually show the up-to-date DOM during scrolling (but rather, a texture that they rendered at the start of such a ~350ms slice).
I tried to predict scrolling from the scroll events, but typically it ramps up very fast, you'll get an event for a handful of pixels scrolled as the touch gesture is done, and then it accelerates to >1000px already for the next event, and even more (>2000) for the one after that. Rendering thousands of pixels ahead on ever scroll event isn't realistic -- it'd cause awful lag.
You can try setting your viewportMargin to something bigger than the default (setting it to 500 mostly, but not entirely, solved the problem for me), but that'll cause a bigger DOM to be rendered and startup, as well as redraws, to be slower.

Since browsers have decided to prioritize scroll performance over the ability of scripts to interact with scrolling, my impression is that this is a losing battle, and the CodeMirror 6 design just gives up on this—it'll try to quickly render the content of the viewport when it's notified of the scrolling, but does away with all the awful hacks that try to predict scrolling in advance. So you'll see a flash of empty content during quick scrolling, but the library can be simpler and avoids eating battery by rendering stuff that may not bee needed.


Minor edits

Reference Issues/PRs
What does this implement/fix? Explain your changes.

Anaconda badge added to README.md
Some refactoring in NavaThread
CHANGELOG.md updated
README.md updated

Any other comments?

Codecov Report
Attention: Patch coverage is 94.11765% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 95.80%. Comparing base (ec72563) to head (da3e2ca).

:exclamation: Current head da3e2ca differs from pull request most recent head 79d1fc3. Consider uploading reports for the commit 79d1fc3 to get more accurate results

Files
Patch %
Lines

nava/thread.py
94.11%
0 Missing and 1 partial :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@           Coverage Diff           @@
##              dev      #41   +/-   ##
=======================================
  Coverage   95.80%   95.80%
=======================================
  Files           4        4
  Lines         143      143
  Branches       18       18
=======================================
  Hits          137      137
  Misses          2        2
  Partials        4        4

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Make move_sprite both more general and more illustrative.

Objective
Discussion: https://discord.com/channels/691052431525675048/743559241461399582/1063153669920526376
This example is important because it is far simpler than the breakout and ECS-guide examples and thus likely to be encountered by newcomers to Bevy. Furthermore, unlike the breakout example which uses a 1/60-second time-step, this example demonstrates movement with truly continuous time.
The previous version of this code has some shortcomings. Primarily, the Direction component is an enum and is mutated by the sprite_movement(…) system, suggesting that its reason for existence is as the mutable state required to implement the up-and-down movement rules.
Its purpose as a marker component is subtle and that represents a missed opportunity as well as producing an unintended pitfall for newcomers who might copy this examples code and remove that component – perhaps implementing movement with periodic functions like sin().
Solution
The revised example implements identical behaviour but places the emphasis on two key lessons for the reader:

How one might use a system to smoothly update the translation of sprites, moving them over time.
That components are widely re-used – particularly, Transform – and so queries must be well defined and marker components have their place.

Additionally, the style has been made more appropriate for the domain. For example, the movement update is now just translation += v × dt.

I think introducing two systems and requiring ordering between them is too complex for this example and not worth it.


fix: use correct field from response in node management

compatibility for corelightning 23

give some context please and some testing steps

cln removed the value field from the listfunds rpc in v23, which wasnt caught because regtest ran on v22

cln removed the value field from the listfunds rpc somewhere in v23, which wasnt caught because regtest ran on v22

when was "amount_msat" field first introduced? Pre "amount_msat" clightning versions will fail.

cln removed the value field from the listfunds rpc somewhere in v23, which wasnt caught because regtest ran on v22

* when was `"amount_msat"` field first introduced? Pre `"amount_msat"` clightning versions will fail.

I know that it was already present and working in v22

I know that it was already present and working in v22

ok, but if "amount_msat" was introduced, lets say, in v15 then it means clightning v14 will not work with this PR
note: v15 is just an example, it can be anything between 1 and 22
as an alternative the PR can check for "value" and if not present use "amount_msat"

it was msatoshi before
https://github.com/ElementsProject/lightning/commit/cd7e784d6f503553548654ca056c8faaa091348d

looking at cln 0.12.1 i also the amount_msat here https://github.com/ElementsProject/lightning/blob/fda3904d8c4d657985568f34444b3f59b2527aae/cln-rpc/src/model.rs#L1168


Use tram crossing tags when connecting trams and roads/paths

Broken out from https://github.com/openstreetmap/iD/issues/5990#issuecomment-674419342.
Tram tracks are rather different from other train tracks. There are separate crossing tags to indicate this: railway=tram_level_crossing and railway=tram_crossing. iD should have presets for these and account for them in the validator.

So status=proposed doesn't actually mean anything?

@maraf24 I'm not sure what you're referring to. The status=proposed tag has only two uses and isn't supported by iD.


Bug: velero pods pending

Contact Details
No response
Describe bug
Add veloro instance with  "noDefaultBackupLocation: true" without cloud-credentials secret in namespace results in velero pods in pending state.
Create cloud-credentials secret with empty values
kind: Secret
apiVersion: v1
metadata:
name: cloud-credentials
namespace:
cloud: >-
base64 encoding of AWS credential file
type: Opaque
What happened?
Expected Velero pods to be in running state.
OADP Version
0.4.x (Beta)
OpenShift Version
4.8
Velero pod logs
No response
Restic pod logs
No response
Operator pod logs
credentials file not found

New issue

[X] This issue is new

Hi @tkagn, I followed the steps you have mentioned in the description and unable to reproduce the issue. Once I created an empty secret called cloud-credentails, velero pod started and the CR reconciled successfully. Are you still seeing this issue?

Hi Savitha, The issue was that the cloud-credential had to exist even
though there was not backupservicelocation set.
v/r
On Mon, Nov 22, 2021 at 3:06 PM Savitha Raghunathan <
@.***> wrote:

Hi @tkagn https://github.com/tkagn, I followed the steps you have
mentioned in the description and unable to reproduce the issue. Once I
created an empty secret called cloud-credentails, velero pod started and
the CR reconciled successfully. Are you still seeing this issue?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/openshift/oadp-operator/issues/420#issuecomment-975875522,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/APVDR2MSRR74EVP5KARHIFTUNKPD5ANCNFSM5G5H6ISA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

--
Stephen Wilson
Senior Storage Consultant
Red Hat https://www.redhat.com/
@.***
M:<PHONE_NUMBER>
https://www.redhat.com/

Got it. Sorry I interpreted it the other way. Will take a look into it. Thanks for reporting the issue

@tkagn cloud-credentials are still used for other components beyond backup location.. such as for volume snapshotting if you are using the aws plugin. If you did not add aws plugin then we'll have to check where else this check is occuring.

Understood. This was for a velero instance that would eventually  backup to
an external S3 MCG/NooBaa bucket
On Tue, Nov 23, 2021 at 10:14 AM Tiger Kaovilai @.***>
wrote:

@tkagn https://github.com/tkagn cloud-credentials are still used for
other components beyond backup location.. such as for volume snapshotting
if you are using the aws plugin. If you did not add aws plugin then we'll
have to check where else this check is occuring.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/openshift/oadp-operator/issues/420#issuecomment-976694191,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/APVDR2MAWYDNCIWWZ2EKOP3UNOVWZANCNFSM5G5H6ISA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

--
Stephen Wilson
Senior Storage Consultant
Red Hat https://www.redhat.com/
@.***
M:<PHONE_NUMBER>
https://www.redhat.com/

MCG/NooBaa bucket usage currently relies on aws-plugin I believe... at least for our expected support case. How are you using mcg/noobaa without cloud-credentials?

We are trying to have a BSL per project. We can generate an empty cloud
credential to get past this issue.
On Tue, Nov 23, 2021 at 11:04 AM Tiger Kaovilai @.***>
wrote:

MCG/NooBaa bucket usage currently relies on aws-plugin I believe... at
least for our expected support case. How are you using mcg/noobaa without
cloud-credentials?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/openshift/oadp-operator/issues/420#issuecomment-976770508,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/APVDR2JUTWCGWXYJ2UM4FGDUNO3PVANCNFSM5G5H6ISA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

--
Stephen Wilson
Senior Storage Consultant
Red Hat https://www.redhat.com/
@.***
M:<PHONE_NUMBER>
https://www.redhat.com/


Pick a different shortcut for toggling the OSM layer

Ok. I'm having a bad day already, but this just made my day worse. 😠
I accidentally did Cmd+W instead of Option+W and it just closed the tab.
Option+W is a terrible choice for a keyboard shortcut in a web application.
I feel actually bad that this has slipped my notice until now, anyway we have to change it to something else.

Now:
Shift+O  Toggle OSM Data
Shift+N  Toggle OSM Notes
Shift+R  Toggle Rapid Data


Check required fields for getBlock... and getTransactionByHash

Delete author in EthereumBlockView - the rest mentioned in 612 are needed and the behavior is the same to what ethereum does.

Edit: Might be - due to some compiling issues earlier - that I tested with a version before the latest rpc changes - I'll redo the tests and re-assign the Review then!


Update macos install instructions

gfortran link is out-dated
R v4.2.3 looks for gfortran in a different place than the R v4.3 gfortran installer puts it (at least on arm64 machines), so need to edit .R/Makevars to include FLIBS, e.g.

FLIBS=-L/opt/R/arm64/gfortran/lib

on my machine.

i use gfortran from here: https://cran.r-project.org/bin/macosx/tools/

@msuchard : could you help me understand how you got this working? I installed thegfortran-12.2-universal.pkg mentioned at the top of https://cran.r-project.org/bin/macosx/tools/ and used sudo ln -s /opt/gfortran /usr/local/gfortran to redirect R to the right location, but it comlains:
ld: warning: search path '/usr/local/gfortran/lib/gcc/x86_64-apple-darwin18/8.2.0' not found

which indeed does not exist, but /usr/local/gfortran/lib/gcc/x86_64-apple-darwin20/12.2.0  does.
Any idea which gfortran install package I should install for Sonoma on an M3?

(I tried the gfortran-4.2.3.pkg on the same page but got some invalid architecture error)

arm64 compilers and whatnot get installed into /opt/.   please do not link to /usr/local/ (a good OS will ignore these links as user-errors).    just update your ~/.R/Makevars file.
mine reads:
F77 = /opt/gfortran/bin/gfortran
FC = /opt/gfortran/bin/gfortran

FLIBS=-L /opt/gfortran/lib

Thanks @msuchard !


Traivs CI fails because Python3 not available on OSX

Latest travis builds are showing:
-- CHECKING FOR PYTHON
960CMake Error at /usr/local/Cellar/cmake/3.16.5/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
961  Could NOT find PythonInterp (missing: PYTHON_EXECUTABLE) (Required is at
962  least version "3.6")
963Call Stack (most recent call first):
964  /usr/local/Cellar/cmake/3.16.5/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE)
965  /usr/local/Cellar/cmake/3.16.5/share/cmake/Modules/FindPythonInterp.cmake:169 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
966  CMakeLists.txt:94 (find_package)
967
968
969-- Configuring incomplete, errors occurred!
970See also "/Users/travis/build/BlueBrain/nmodl/build/CMakeFiles/CMakeOutput.log".
971The command "cmake .. -DPYTHON_EXECUTABLE=`which python3` -DCMAKE_INSTALL_PREFIX=$HOME/nmodl" exited with 1.
9720.02s$ make -j 2
973make: *** No targets specified and no makefile found.  Stop.
974The command "make -j 2" exited with 2.
9750.02s$ make test
976make: *** No rule to make target `test'.  Stop.
977The command "make test" exited with 2.
9780.02s$ make install
979make: *** No rule to make target `install'.  Stop.
980The command "make install" exited with 2.

This is because brew installation is failing with:
==> Downloading https://homebrew.bintray.com/bottles/python-3.7.6_1.mojave.bottle.tar.gz
432==> Downloading from https://akamai.bintray.com/64/643d627c2b4fc03a3286c397d299284ef8ce2d4a832737e41175f297d4f0862e?__gda__=exp=1583574775~hmac=ac16a797c358636f2a7805906b77186c8b4e672a8eeb50e8336764fcb20249b6&response-content-disposition=attachment%3Bfilename%3D%22python-3.7.6_1.mojave.bottle.tar.gz%22&response-content-type=application%2Fgzip&requestInfo=U2FsdGVkX19MBlCq2giNxxIK-dB3JN-DoXSLWhwc9raZvi3w_tL14xRXTScv-f568-U8rzH2MmBjiQChZxNkbfPqmH8YHgFIjJlq9rHzyS9MUSYKi7gl0LgACdf66QqKIHQa6FuS-_DJQJ8TNTUGUg&response-X-Checksum-Sha1=783156116f019791e81b2eb0542b8c0cde15b203&response-X-Checksum-Sha2=643d627c2b4fc03a3286c397d299284ef8ce2d4a832737e41175f297d4f0862e
433==> Pouring python-3.7.6_1.mojave.bottle.tar.gz
434Error: The `brew link` step did not complete successfully
435The formula built, but is not symlinked into /usr/local
436Could not symlink Frameworks/Python.framework/Headers
437Target /usr/local/Frameworks/Python.framework/Headers
438is a symlink belonging to python@2. You can unlink it:
439  brew unlink python@2
440
441To force the link and overwrite all conflicting files:
442  brew link --overwrite python
443
444To list all files that would be deleted:
445  brew link --overwrite --dry-run python

See https://github.com/bitcoin/bitcoin/issues/17848

fixed in #265


module.aft-account-request-framework.aws_backup_selection.aft_controltower_backup_selection causes drift

It looks like the aws_backup_selection not having a blank condition and empty not_resources properties defined causes drift. I believe that changing the order would fix this.
Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # module.aft.module.aft_account_request_framework.aws_backup_selection.aft_controltower_backup_selection has changed
  ~ resource "aws_backup_selection" "aft_controltower_backup_selection" {
        id            = "abcd1234-ab12-ab12-ab12-abcdef123456"
        name          = "aft-controltower-backup-selection"
      + not_resources = []
        # (3 unchanged attributes hidden)

      + condition {
        }
    }

Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include actions to undo or respond to these changes.

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place
-/+ destroy and then create replacement

Terraform will perform the following actions:
# module.aft.module.aft_account_request_framework.aws_backup_selection.aft_controltower_backup_selection must be replaced
-/+ resource "aws_backup_selection" "aft_controltower_backup_selection" {
      ~ id            = "abcd1234-ab12-ab12-ab12-abcdef123456" -> (known after apply)
        name          = "aft-controltower-backup-selection"
      - not_resources = [] -> null
        # (3 unchanged attributes hidden)

      - condition { # forces replacement
        }
    }

@jeremyciak possible duplicate of https://github.com/aws-ia/terraform-aws-control_tower_account_factory/issues/35

@mathewmoon good call. I created PR #38 which I will link to #35 as well.

This looks to be related to https://github.com/hashicorp/terraform-provider-aws/issues/22595

I do see the workaround you've listed in the issue I linked above. We'll see if we can't reproduce and test the workaround.

Hi, release 1.1.2 should resolve this issue. Please feel free to re-open if you're still experiencing any issues.


[webview_flutter] Add listener for scrolling and set the scrolled position programmatically in Android

Description
Adding scroll listener and ability to scroll programmatically in Android.
Related Issues
https://github.com/flutter/flutter/issues/42442
https://github.com/flutter/flutter/issues/31027
Checklist
Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]). This will ensure a smooth and quick review process.

[x] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[x] My PR includes unit or integration tests for all changed/updated/fixed behaviors (See Contributor Guide).
[x] All existing and new tests are passing.
[x] I updated/added relevant documentation (doc comments with ///).
[x] The analyzer (flutter analyze) does not report any problems on my PR.
[x] I read and followed the Flutter Style Guide.
[x] The title of the PR starts with the name of the plugin surrounded by square brackets, e.g. [shared_preferences]
[x] I updated pubspec.yaml with an appropriate new version according to the pub versioning philosophy.
[x] I updated CHANGELOG.md to add a description of the change.
[x] I signed the CLA.
[x] I am willing to follow-up on review comments in a timely manner.

Breaking Change
Does your PR require plugin users to manually update their apps to accommodate your change?

[ ] Yes, this is a breaking change (please indicate a breaking change in CHANGELOG.md and increment major revision).
[x] No, this is not a breaking change.

Thanks for the contribution!
I'm following the initial PR review policy, this PR isn't trivial to review so I'm labeling it with "backlog" and we will prioritize according to the issue's priority.
Relevant issue: https://github.com/flutter/flutter/issues/42442

@amirh any progress on this?

@amirh any progress?

@andzejsw In the meantime, you can try out this plugin, it has an implemented scroll listener

I took an initial look at this; at a high level it looks good, but per this is a case where we should land it in parallel with an iOS implementation per the policy described at https://github.com/flutter/flutter/wiki/Contributing-to-Plugins-and-Packages#platform-support
If anyone is interested in moving this forward, I encourage you to open a PR based on this one that adds an iOS implementation; once we have both we can review this one and the iOS portion of that one.
This will also need to be updated for the federation changes (which is likely currently blocked on https://github.com/flutter/flutter/issues/94051 for the moment; we hope to resolve that at the beginning of 2022).

@TheVinhLuong Are you planning on updating this for the structural changes in the plugin (federation, Android conversion to a more Dart-based implementation), so that it would be ready as the foundation for a cross-platform PR?

@stuartmorgan Sorry for my late reply. I will update the PR for the Android side, also for iOS if I can make it.

@TheVinhLuong Just a friendly check-in to see if this is still something you have on your radar?

@TheVinhLuong Just a friendly check-in to see if this is still something you have on your radar?

Yes, I have implemented it both for android and ios. I am writing test now.

@Hixie Could you please review my PR?

@TheVinhLuong @Hixie We are currently working on a new interface for webview_flutter and aren't accepting new features until it is released.
See https://github.com/flutter/flutter/issues/94051
https://github.com/flutter/flutter/issues/93732

https://github.com/flutter/flutter/issues/93732 is resolved, but this is still blocked on us finishing https://github.com/flutter/flutter/issues/94051

Status update from triage: this is still blocked, but we hope to have it unblocked soon.

webview_flutter v4 has finally been completed, which included fixing the platform interface to that we can make additions to it without breaking changes. That means that this is finally unblocked! Thanks for your patience while we worked through the restructuring.
webview_flutter v4 uses a Dart-first implementation, where the platform channel boundary is as close as possible to the underlying host APIs, and plugin logic is written primarily in Dart rather than the host language. That means that this PR will need to be updated to follow that model. Please let us know if you have questions about updating to v4 (and https://github.com/flutter/plugins/pull/6881 may be useful as an example of adding a feature in the v4 structure).

Thanks for the update. I will keep you updated on the progress of this PR.

Hi @stuartmorgan, can you review my PR? I have updated the PR according to the new plugin federation format.
Since this PR is just an implementation for the Android side the automated test for changelogs is failing for other platforms. Please add an 'override: no versioning needed' label for this PR if there is nothing wrong. Thank you!

Since this PR is just an implementation for the Android side the automated test for changelogs is failing for other platforms. Please add an 'override: no versioning needed' label for this PR if there is nothing wrong.

You just need to revert the tool-generated pubspec changes in those other packages.
I'll review when I get a chance, but in the meantime you have Dart unit test and analysis failures in CI that will need to be addressed.

Sadly, I don't think this feature is currently implementable in the web, since there's no communication mechanism between the iframe that is being injected and the flutter web app.

Left some early feedback. Are you able to do the iOS portion as well or would you like help with that?

I think I can do it. Can I open a new PR for the iOS implementation?

This PR should follow https://github.com/flutter/flutter/wiki/Contributing-to-Plugins-and-Packages#changing-federated-plugins and include the iOS implementation in this PR. Then once this has been approval, you can split the code from each package into their own PRs.

@bparrishMines Thanks for your feedback. Could you please take a look again at Android implementations for this? If no problems arise I will apply the same logic for the iOS side.

We've just completed the migration of the plugin code to the flutter/packages repository, as described in https://flutter.dev/go/flutter-plugins-repo-migration, and this repository is now being archived. Unfortunately that means that all in-progress PRs here must be moved to flutter/packages.
Please see our instructions for an explanation of how to move your PR, and if you have any issues moving your PR please don't hesitate to reach out in the #hackers-ecosystem channel in Discord.
Our apologies that your PR was caught in this one-time transition. We're aware that it's disruptive in the short term, and appreciate your help in getting us to a better long-term state!


Add weight-at-age figure to the assessment

make.wt.at.age.plot(wt.at.age, ages = 2:10, lwd = 1)
From data presentation

Sounds like we can put this figure in the SRG response appendix this year (even though we maybe don't explicitly have it as a response...

Added, but nothing written about it. Placeholder code chunk

Add this plot to the 2022 assessment document, in main figures section

Done. Haven't added text to main text yet (flagged the caption to mention this).

Fixed caption, added bold line for age 5 for clarity, and mentioned in text. Sometime earlier today.


Stem Activations

Currently this package does not support to return the track activations. Is this of interest for you? I can implement this and send a PR.

We have it implemented as Matlab code, and the author hasn't had a chance to clean it up to a releasable form. If you want to implement it in python that would be fantastic.

no worries. I can implement it, but I will wait until #15 is fixed, so that it will work on the 1.0 release as well

actually I need the activation annotation soon. So I would be happy If I can apply them on a stable master branch. What about putting your changes since 82e103 on the 1.0 or 1.1 branch?

See my comment on #16


v2.0

I'm working on a new version that uses Guzzle instead of this multi_curl abomination I wrote last year. It should be a lot cleaner. I'll think I'll have something ready in a few days. If anyone has any feature suggestions, leave em here while I'm rewriting most of the class anyway.

Sounds good, I'll be testing your project out on mine. Got any other example code you could share? I'm wanting to add a high scores page where a front end user could query and return the result using ajax or jquery. Guzzle is a lot better than all the CURL calls but you need to be careful about SSL_PEER_VALIDATION, most people have to keep it off for development or other reasons because it does not support self-signed certs. Even on CA certs it can throw the same error if it can't find a .pem file, almost everyone has it set to curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
Example:
https://github.com/guzzle/guzzle/issues/1391
So for a production script, it should have an obvious option to turn it off for an end user who may not know any coding if you ever intend to go that far with it.


Custom Tag syntax error

When a custom  tag have written like this, it throws a syntax error:
{{#usrName createdBy}}{{/usrName}}
To correct this problem, I'm adding a space between opening and closing tags than it renders correctly.
{{#usrName createdBy}} {{/usrName}}

Yes this is a bug for regular tags as well as custom tags. For now you need at least a space between open and close tags. Will be fixed in an upcoming commit.


Vulnerable to zip slip

It appears as decompress is vulnerable to archives containing files that hold path-traversal names such as ../../outside.txt.
As a PoC I have attached a .tar.gz-archive(slip.tar.gz) that will, when extracted, create a file in /tmp/slipped.txt.
Use the example code to extract:
const decompress = require('decompress');

decompress('slip.tar.gz', 'dist').then(files => {
	console.log('done!');
});

Note that this will not work out-of-the-box with .zip-archives since yauzl will throw an exception if the entry's filename contains "..".
However, since this package also supports symlinks we can use that instead to bypass this limitation.
But just adding a symlink to our desired target location won't do the trick - since the archive's files are being extracted asynchronous (index.js#44-73) we will end up in a race condition between extracting the symlink and the file referencing it.
By creating a directory and within that create a symlink to its' parent directory. Continue with creating a symlink to / and a file with a name using a looped structure of all symlinks to the target. We end up with something like;
mkdir generic_dir
ln -s ../ generic_dir/symlink_to_parent_dir
ln -s / symlink_to_root
touch generic_dir/symlink_to_parent_dir/generic_dir/symlink_to_parent_dir/[...]/symlink_to_root/tmp/slipped_zip.txt

Adding this to an archive allows us to always win the race!
Here's a PoC
(slip.zip), that when extracted will create a file in /tmp/slipped_zip.txt. Demonstrating how this also works for zip-archives using symlinks.

Now 'npm audit' warns this vulnerability as a high risk one. I am using filePond in my React app and because FilePond is dependent to this package I got 9 High Risk vulnerabilities and none of them can be solved because there is no patch available yet. Please somebody how knows this package, submit a fix.

what is the alternative solution for this decompress? any other library?

NPM security needs to get a grip. This isn't a security problem. Does this bypass filesystem security? Nope.

NPM security needs to get a grip. This isn't a security problem. Does this bypass filesystem security? Nope. i.e. every file in an archive is an "arbitrary file".

Maybe you could right somewhere or to some application that could execute that file or cause errors to applications. In my opinion this isn't minor either.

NPM security needs to get a grip. This isn't a security problem. Does this bypass filesystem security? Nope. i.e. every file in an archive is an "arbitrary file".

The problem isn't that the files are arbitrary, it's that unpacking an archive can result in files writing to arbitrary paths. So if you unpack to /tmp/foo a malicious .tar.gz can trick this library into outputing a file to /usr/bin instead, etc.
Definitely a security risk.

@kevva @sindresorhus are you guys aware of this?

The issue alone is not a security problem. But it opens the door for security issues as many packages depend on this one. I had a look at this issue and found out that it originates in the package decompress-tar.
I fixed it there and created a pull request. The build has broken due to the old project setup. @kevva should decide how to proceed. I will support if necessary.

NPM security needs to get a grip. This isn't a security problem. Does this bypass filesystem security? Nope. i.e. every file in an archive is an "arbitrary file".

The problem isn't that the files are arbitrary, it's that unpacking an archive can result in files writing to arbitrary paths. So if you unpack to /tmp/foo a malicious .tar.gz can trick this library into outputing a file to /usr/bin instead, etc.
Definitely a security risk.

Nah.  Again, you can only right to the filesystem if you have permission. /tmp/foo isn't a problem. Only if you springboard off another vulnerability can you make anything of this so called "high" security risk.
tar -xzvf my.tar.gz isn't considered insecure if if does what you ask, i.e. unpack the files (with all the arbitrary paths inside) into your filesystem. If I execute this with sudo, now we have a problem. If I execute as an ordinary user, it's gonna fail to write to /usr/bin, just like decompress will.

NPM security needs to get a grip. This isn't a security problem. Does this bypass filesystem security? Nope. i.e. every file in an archive is an "arbitrary file".

After some investigation of the behavior of the command line utilities (and notwithstanding NPM security group's habitual and lazy scarlet-lettering of packages), I recant my objection on the grounds that this is a bug for other reasons.
Looks like BSD tar (such as in Darwin) does allow .. prefixes during creation, but requires the -P (preserves paths) flag to unpack them. Linux (GNU tar) strips ../ during tar creation, always. Therefore, in practice parent relative paths should be rather rare, and the ordinary behavior would be to not expect them exist in the archive, or would otherwise expect them to be ignored by default.
I think the Linux behavior of treated ../foo/ as foo/ would at least be expected.
Workaround with current behavior to correct this behavior might be:
const decompress = require('decompress');
const path = require('path');
const assert = require('assert');

const fixPath = (file, targetDir) => {
  // resolve relative path to target. e.g. file.path of a/../../b could actually be ../b
  const resolvedRelative = path.relative(targetDir, path.resolve(targetDir, file.path));

  // get rid of any .. prefix
  const fixed = resolvedRelative.replace(/^(\.\.[/\\\\]?)+/, '');
  file.path = fixed;
  return file;
};

assert.deepEqual(fixPath({ path: '../bar'}, './foo'), { path: 'bar' });
assert.deepEqual(fixPath({ path: '../../bar'}, './foo'), { path: 'bar' });
assert.deepEqual(fixPath({ path: 'bar/../../baz'}, './foo'), { path: 'baz' })

const targetDir = path.resolve('./test-dir');

// foo.tgz has:
// ../
// ../baz/
// ../bar/
// ../bar/text.txt
// ../baz/text.txt

decompress('./foo.tgz', targetDir, {
  map: file => {
    file = fixPath(file, targetDir);
    return file;
  },
  filter: file => {
    // strip empty directory creation
    file = fixPath(file, targetDir);
    return file.path.length;
  },
})
.then(files => {
  files.forEach(file => console.log(file.path))
});
// Outputs:
// baz/
// bar/
// bar/text.txt
// baz/text.txt

Nah. Only if you are doing something insane like running node as root on a web service, and using it to unpack user provided tarballs. Again, you can only write to the filesystem if you have permission. /tmp/foo isn't a problem. Only if you springboard off another vulnerability can you make anything of this so called "high" security risk.

Yes, this is very much a security risk.  Running code as root is not a vulnerability in itself, its just not preferred practice if you can help it.  It could be something as simple as automated script.  There are still a lot of things only root can do.  Linux has been doing a better jobs about giving privileges to non root user but there are still several things that require root access. Not to mention there are still several system that require root access for something as simple as running on any port under 1024.
It might even be something more insidious like having ~/.node_modules in the path.  Where it installs a local module that could overwrite modules a lot people install globally, like npm or webpack.
Code by default should be secure, WITH the possible option of bypassing and warning the developer of possible issues.  Most *nix archivers (e.g. tar/zip) do not allow extraction outside of the specified directory (think root for that dir) unless you specify a specific option to allow files outside of it.

Nah. Only if you are doing something insane like running node as root on a web service, and using it to unpack user provided tarballs. Again, you can only write to the filesystem if you have permission. /tmp/foo isn't a problem. Only if you springboard off another vulnerability can you make anything of this so called "high" security risk.

Yes, this is very much a security risk. Running code as root is not a vulnerability in itself, its just not preferred practice if you can help it. It could be something as simple as automated script. There are still a lot of things only root can do. Linux has been doing a better jobs about giving privileges to non root user but there are still several things that require root access. Not to mention there are still several system that require root access for something as simple as running on any port under 1024.
It might even be something more insidious like having ~/.node_modules in the path. Where it installs a local module that could overwrite modules a lot people install globally, like npm or webpack.
Code by default should be secure, WITH the possible option of bypassing and warning the developer of possible issues. Most *nix archivers (e.g. tar/zip) do not allow extraction outside of the specified directory (think root for that dir) unless you specify a specific option to allow files outside of it.
unzip requires -: -> https://linux.die.net/man/1/unzip "For security reasons, unzip normally removesparent dir'' path components (../'')"
tar requires -P -> https://linux.die.net/man/1/tar

If a sys admin or coder requires continuously running your node app as root to get ports below 1024 (or any other privileged system activity), frankly they are not competent to be writing/running web services. Sure, in the most naive setups you may need to use root in some capacity to "initiate" a service, but you would never maintain that webservice acting under the powers of root under normal operating circumstances, you would perform what needed to be done with privilege and then you would de-escalate your privileges, with process.setuid(id), or daemonize your server, use pm2/spawn new low priv processes, use docker, use a proxy, etc. Sorry, if some can't see how running your webservice process as root user is far more dangerous than having a utility that walks a variable list of relative paths and performing writes, then I have no words.
tar -xzvPf  ~/foo.tgz doesn't require -P for "security reasons", it's a matter of practicality because you can execute the extraction anywhere, parent relative paths can easily become meaningless / nonsensical.
Also, you kinda missed the point. We don't nerf functionality in a language or "utility function" simply because it can be used badly. fs.writeFile() can be used badly, indeed it can write to arbitrary paths! Frankly, the original authors of this package should be commended for adding the map and filter options, making this an excellent choice for people wanting to extract archives securely, and with expected results. That being said, I've already withdrawn my objection on the grounds that I think the default behavior is indeed a bug, but ya'll need to remember that code like this is used in more circumstances than web services (such as command line utilities).
Again, "may cause a security consequences" and "no way to avoid security consequences" are different levels of risk, and the current behavior falls squarely in the former classification. Not high.

If a sys admin or coder requires continuously running your node app as root to get ports below 1024 (or any other privileged system activity), frankly they are not competent to be writing/running web services. Sure, in the most naive setups you may need to use root in some capacity to "initiate" a service, but you would never maintain that webservice acting under the powers of root under normal operating circumstances, you would perform what needed to be done with privilege and then you would de-escalate your privileges, with process.setuid(id), or daemonize your server, use pm2/spawn new low priv processes, use docker, use a proxy, etc. Sorry, if some can't see how running your webservice process as root user is far more dangerous than having a utility that walks a variable list of relative paths and performing writes, then I have no words.

That was a for instance, but it could be as simple is to access files that only root has access to, and only root SHOULD have access to.  Saying you can't write code in NodeJS for root because it has unintended side-affected security issues is a terrible excuse.

tar -xzvPf ~/foo.tgz doesn't require -P for "security reasons", it's a matter of practicality because you can execute the extraction anywhere, parent relative paths can easily become meaningless / nonsensical.

The REST of the world basically thought it was a security issue, which is why it got patched (https://nvd.nist.gov/vuln/detail/CVE-2001-1267).  Even if you don't think it was.  -P was added to tar when they stopped allowing absolute paths to change files (hackers were putting in /etc/passwd and /etc/shadow in the tar).  The problem was that they still allowed traversing outside of the extraction path which IS/WAS a security bug, and they patched it in 2001-08-27 (version 1.13.20)
        (extract_archive): Fix test for absolute pathnames and/or "..".

https://access.redhat.com/security/cve/cve-2001-1267
https://www.suse.com/security/cve/CVE-2001-1267/
https://www.tenable.com/cve/CVE-2001-1267
Now, this vulnerability, was marked as low, but the nodejs one is marked as high.  Not sure why they did that though. I personally don't think it warrants a high, maybe a medium at most.
I'm not sure if this code does it, but another issue in tar was the ability to create a symbolic link to an absolute path and then extract to that link, which they fixed as well.

Also, you kinda missed the point. We don't nerf functionality in a language or "utility function" simply because it can be used badly. fs.writeFile() can be used badly, indeed it can write to arbitrary paths! Frankly, the original authors of this package should be commended for adding the map and filter options, making this an excellent choice for people wanting to extract archives securely, and with expected results. That being said, I've already withdrawn my objection on the grounds that I think the default behavior is indeed a bug, but ya'll need to remember that code like this is used in more circumstances than web services (such as command line utilities).

You do patch these issues even if it breaks(i think thats what you meant by "nerf") functionality because developers DON'T expect underlying code to have security problems. The default act of ANY "utitlity" should be SAFE/SECURE even if it breaks functionality.  If that functionality is still needed, then the developer should provide a way to EXPLICITLY bypass said security.  It should take effort to make things insecure because you are intentionally doing so, but it should not take effort to secure it.  Almost every utility in the past has done so, and the ones that didn't, people are not using anymore.

If a sys admin or coder requires continuously running your node app as root to get ports below 1024 (or any other privileged system activity), frankly they are not competent to be writing/running web services. Sure, in the most naive setups you may need to use root in some capacity to "initiate" a service, but you would never maintain that webservice acting under the powers of root under normal operating circumstances, you would perform what needed to be done with privilege and then you would de-escalate your privileges, with process.setuid(id), or daemonize your server, use pm2/spawn new low priv processes, use docker, use a proxy, etc. Sorry, if some can't see how running your webservice process as root user is far more dangerous than having a utility that walks a variable list of relative paths and performing writes, then I have no words.

That was a for instance, but it could be as simple is to access files that only root has access to, and only root SHOULD have access to. Saying you can't write code in NodeJS for root because it has unintended side-affected security issues is a terrible excuse.

tar -xzvPf ~/foo.tgz doesn't require -P for "security reasons", it's a matter of practicality because you can execute the extraction anywhere, parent relative paths can easily become meaningless / nonsensical.

The REST of the world basically thought it was a security issue, which is why it got patched (https://nvd.nist.gov/vuln/detail/CVE-2001-1267). Even if you don't think it was. -P was added to tar when they stopped allowing absolute paths to change files (hackers were putting in /etc/passwd and /etc/shadow in the tar). The problem was that they still allowed traversing outside of the extraction path which IS/WAS a security bug, and they patched it in 2001-08-27 (version 1.13.20)
        (extract_archive): Fix test for absolute pathnames and/or "..".

https://access.redhat.com/security/cve/cve-2001-1267
https://www.suse.com/security/cve/CVE-2001-1267/
https://www.tenable.com/cve/CVE-2001-1267
Now, this vulnerability, was marked as low, but the nodejs one is marked as high. Not sure why they did that though. I personally don't think it warrants a high, maybe a medium at most.
I'm not sure if this code does it, but another issue in tar was the ability to create a symbolic link to an absolute path and then extract to that link, which they fixed as well.

Also, you kinda missed the point. We don't nerf functionality in a language or "utility function" simply because it can be used badly. fs.writeFile() can be used badly, indeed it can write to arbitrary paths! Frankly, the original authors of this package should be commended for adding the map and filter options, making this an excellent choice for people wanting to extract archives securely, and with expected results. That being said, I've already withdrawn my objection on the grounds that I think the default behavior is indeed a bug, but ya'll need to remember that code like this is used in more circumstances than web services (such as command line utilities).

You do patch these issues even if it breaks(i think thats what you meant by "nerf") functionality because developers DON'T expect underlying code to have security problems. The default act of ANY "utitlity" should be SAFE/SECURE even if it breaks functionality. If that functionality is still needed, then the developer should provide a way to EXPLICITLY bypass said security. It should take effort to make things insecure because you are intentionally doing so, but it should not take effort to secure it. Almost every utility in the past has done so, and the ones that didn't, people are not using anymore.

So you are saying we should patch, fs.writeFile() to prevent it from writing to arbitrary files? Got it.

Umm no.  Wow, you went from completely right to completely left.  The people who patched tar, didn't tell the OS people to fix their code.  The utility allowed it to happen, so they fixed it.  When i tell tar/zip to extract an archive to a directoy, it shouldn't be expected that it'll change files outside of that directory.  Thats completely insane.  They require an option to say, "Yeah I want you to possibly do bad things here, but i'm giving you permission, because I know what I'm doing".  It shouldn't be any different here.

Umm no. Wow, you went from completely right to completely left. The people who patched tar, didn't tell the OS people to fix their code. The utility allowed it to happen, so they fixed it. When i tell tar/zip to extract an archive to a directoy, it shouldn't be expected that it'll change files outside of that directory. Thats completely insane. They require an option to say, "Yeah I want you to possibly do bad things here, but i'm giving you permission, because I know what I'm doing". It shouldn't be any different here.

Obviously, my example is absurd. It's hyperbole to demonstrate that logically the cause of the vulnerability: "ability to write arbitrary files" is true of the node core utilities, of which this library is an extension. fs.writeFile() isn't the operating system of decompress, it's in the same exact user space. It could be written to only write to the process cwd, with flags to allow an override.
From a technical perspective (what your program should be able to do to the filesystem, given sufficient privilege), this is not a security problem, at all.
But let me agree with you now. It might be, from a "social" perspective. In that a security exploit strategy can be to subvert the coder's social expectations of outcome. If we write the acceptance criteria of "an archive" such as (nevermind the different skillsets/expectations of the respective user bases of tar vs zip):
An Archive (the archive rules):

will only contain children resolving relative paths
no parent relative paths, and no absolute paths are allowed

If this set of assumptions is true for all "users" of decompress (historically this is more true for zip users than tar users), than this is both a bug, and also has the potential of being exploited (if and only if another security practice is subverted, such as needlessly giving write access to the filesystem in areas outside the domain of your application)
My problem with this notice is that:

the above assumptions are not (or have not been) universally true of tar users (of which I have been one for over 20 years)
given an alternative understanding of an archive (to be one that could very well have absolute and parent relative paths contained), there could have been legitimate and safe uses of such capability (just as there is legitimate and safe uses of fs.writeFile())
the gravity is overstated, and the package authors don't deserve the negative social impact of the notice (I think we agree)
it is possible (both with command line tar and decompress package) to know what you are writing and to where, right now, with no patch, and to prevent writes to unexpected places.
the filesystem security model plays the same role in protecting fs.writeFile(), as it does tar, as it does decompress, and it's also subversive to a coder's understanding to assume this isn't true (or to ignore the fact).

Obviously, my example is absurd. It's hyperbole to demonstrate that logically the cause of the vulnerability: "ability to write arbitrary files" is true of the node core utilities, of which this library is an extension. fs.writeFile() isn't the operating system of decompress, it's in the same exact user space. It could be written to only write to the process cwd, with flags to allow an override (that would be "secure" by default, sure it would break a lot of code, but...).

No, then it would impossible to write any utilities that DO add/update any system files, explicitly.  The utility is the only place to know whats permissible or not.  Also, it would prevent extraction into a temporary directory which is outside the cwd.  Requiring a process to change directories in order to modify files and directories would cause too many side effects.

From a technical perspective (what your program should be able to do to the filesystem, given sufficient privilege), this is not a security problem, at all.

I agree, but what is expected from a security aspect is completely different than what its capable of.  For example, if you write a utility to add users to /etc/passwd and /etc/shadow, then if it has access, it should be able to.  But generalize code to extract from an archive, shouldn't arbitrarily delete all users from the system and change the root password because a third party archive sets it as a location.

But let me agree with you now. It might be, from a "social" perspective. In that a security exploit strategy can be to subvert the coder's social expectations of outcome. If we write the acceptance criteria of "an archive" such as (nevermind the different skillsets/expectations of the respective user bases of tar vs zip):
An Archive (the archive rules):
1. will only contain children resolving relative paths

2. no parent relative paths, and no absolute paths are allowed

Again, no, parent relative paths ARE allowed as long as they are contained within the extraction directory (think chroot) e.g. "test/../test.txt" is a perfectly valid file and should be extracted)

3. the gravity is overstated, and the package authors don't deserve the negative social impact of the notice (I think we agree)

Agreed, but authors should be responsive to security concerns and when they don't fix (or just close the project down).  It could also be positive too. I'm more likely to use third party, if they fix security issues in a timely manor (base on the complexity of the problem)

4. it is possible (both with command line tar and decompress package) to know what you are writing and to where, right now, with no patch, and to prevent writes to unexpected places.

Completely true, but that might not be the case for packages that automate things.

5. the filesystem security model plays the same role in protecting `fs.writeFile()`, as it does `tar`, as it does decompress, and it's also subversive to a coder's understanding to assume this isn't true (or to ignore the fact).

Of course, but to treat fs.writeFile() and tar as being similar is disingenuous.  writeFile modifies a single file and you have to specifically tell it the file to update.
Wherever the code is that writes the file, its THEIR responsibility for sanitizing the filename.  For example, JSZip doesn't have to worry about it because they give database back in a callback for the user of the utility to determine how they want to handle it.  In that case, its the responsibility of the person using JSZip to sanitize the files written to the disk.
if decompress were more like
decompress('my/archive', (fn, data) => {
  fs.writeFile(path.resolve(topath, fn), data)
})

Then I would say the problem does not lie with the decompress package but with whoever is using it.

Programmers operating under this understand will already be wary of blind trust of the contents of an archive, and when input sanitization is necessary, they will certainly do so (using the filter option).

Its always going to be blind trust, unless the programmers that are using code to decompress are creating the archives themselves (which most likely is never the case).

 with flags to allow an override (that would be "secure" by default, sure it would break a lot of code, but...).

No, then it would impossible to write any utilities that DO add/update any system files, explicitly. The utility is the only place to know whats permissible or not. Also, it would prevent extraction into a temporary directory which is outside the cwd. Requiring a process to change directories in order to modify files and directories would cause too many side effects.

I did say "allow an override". ;)

Again, no, parent relative paths ARE allowed as long as they are contained within the extraction directory (think chroot) e.g. "test/../test.txt" is a perfectly valid file and should be extracted)

Sure if that's the behavior every user expects (I think that's at least the convention the command line tools now uniformly default to), then it is at least a bug.

Agreed, but authors should be responsive to security concerns and when they don't fix (or just close the project down). It could also be positive too. I'm more likely to use third party, if they fix security issues in a timely manor (base on the complexity of the problem)

This is easy to say, but we have a multitude of small packages, and often their utility makes them popular to use, but not necessary so popular for project participation. Authors pick things up and put them down, go on vacation, focus on other things for a time.  Unfortunately, this is the source of many a fork, stale project, death of an otherwise good utility.
Also, big part of responding to these security notices is looking at them critically, not just accepting the claims themselves, assuming the actual risk or impact on real applications, not just hypotheticals, and at the potential misplace of responsibility. How mitigated is it naturally? Is this the right place to fix it? Should we look at how it is being applied? We aren't slaves to these opinions, and if ya'll are enchanted at the infallibility of these security groups, and their very fancy CVEs, you shouldn't be (respect the badge!) They are a blunt tool, run by people with a particular bent toward (often essential/useful) paranoia, but also often they lack all ability to see nuance, or to correctly interpret usage context, casting a wide net for tuna and sometimes killing dolphins in the process.
Please consider for a moment: Overzealous declarations of insecurity on small popular packages (in particular when the true risk is mitigated naturally, and the risk level is overblown LOW to HIGH) may have a higher toll than reward. I've been working with a devop engineer to deploy an application when one of these npm audit notices pops, and because it's a dependency of 20 other things, there is this long scary console pouring past my sys admins nose. Now I'm having an awkward conversation about why are we using node.js if it's so insecure, and he's telling his boss we should move to something more "rock solid" like Java. "Devs shouldn't be allowed to pick this unprofessional stack."
Don't get me wrong, I'm glad to know. But then I go back to my desk, and I'm seething because ordinary filesystem permissions mitigate the problem, and my files are on a completely different filesystem from the app and os, and the these are only build-time dev-dependencies to begin with.

Of course, but to treat fs.writeFile() and tar as being similar is disingenuous. writeFile modifies a single file and you have to specifically tell it the file to update. I wouldn't expect webpack to write to system configuration files.

Yeah, I don't think disingenuous is the word, but I know what you are saying.

Wherever the code is that writes the file, its THEIR responsibility for sanitizing the filename.

Agree and disagree. The user of the software (application of the utility) knows better the desirable outcome (in the application) than lower in the stack, and has domain context the general purpose utility does not have. In this case we have a social concept (tar), which has an emergent convention of unpacking acyclically. In this case, I agree, the default behavior should reflect that.

Its always going to be blind trust, unless the programmers that are using code to decompress are creating the archives themselves (which most likely is never the case).

Sure, I think it will always be human nature to trust that your own (the programmer's) understanding of the outcome will not be subverted. Taking advantage of the ability to create your own constraints over trusting the default behavior is still wise.

What about stop talking about if its a security issue or not and just follow other compress libraries and fix this issue? It's an issue and it should be fix.
Reference: https://snyk.io/research/zip-slip-vulnerability
See also:

https://github.com/ZJONSSON/node-unzipper/pull/59
https://pivotal.io/security/cve-2018-1261

AFAIK the maintainer hasn’t yet weighed in, but to be clear some folks have opened PRs:

https://github.com/kevva/decompress/pull/73
https://github.com/kevva/decompress/pull/75

Thanks to PR authors! Unfortunately, the last activity of @kevva was quite some long time ago. This might not even be on his radar.
Projects may have to abandon for another tool, mitigate (if they don't mind the scarlet letter on their project in the interim), or someone can fork the repo (your project can do this themselves and use a git dependency in package.json for awhile), and or somebody publish a patched utility under a new package name. 🤷♂
# How you installed from patched git repo
npm install --save https://github.com/trptcolin/decompress.git#directory-traversal-patch

Maybe<EMAIL_ADDRESS>should let authors know how easy it is to mitigate...

If you look on the profile from @kevva you might be notice that the autor is active on github since the commit today to a private repository.
Basically he is active, maybe someone should write him a mail. Maybe he is muting the whole project.

FWIW for any dependend projects - I wouldn't recommend depending on my git repo. I wouldn't trust myself to keep that branch around, particularly if the PR gets merged. No objections if somebody wants to maintain a (possibly-temporary) fork w/ my PR, but I'm not going to maintain it myself.

@sindresorhus this is a long shot, but do you have any pull here? 😉

You can install your own patches for this package, but it still won't get rid of the yarn audit error. The security vulnerability has been flagged for all versions of the package, regardless of whether it came from npm or your own repository. So CI pipelines will still fail.
In our case, the temporary solution we're using is to install our own fork (from the @trptcolin patch), and then use the improved-yarn-audit package:
yarn improved-yarn-audit --exclude 1217

We can now "safely" ignore this advisory because we've patched it with our own fork.

Sent a mail to @kevva just now.

Thanks to everyone involved in this thread. I'm not nearly competent enough to understand the details of this issue, but I do see that it has been open for over a month, what (if any) resolution is in sight?

The maintainer is AWOL so noone here can merge PRs and make release. Last commit was 21 Aug 2017 - 3 years ago.

The maintainer is AWOL so noone here can merge PRs and make release. Last commit was 21 Aug 2017 - 3 years ago.

Alright, thanks for checking/replying to me :)

I also think that allowing by default an archive to extract anything of the destination folder to be extremely fishy.
A workaround I used is to filter files via the options:
const absoluteDestPath = await fs.realpath(destPath);
await decompress(sourcePath, absoluteDestPath, {
    /**
     * Prevent zip-slip: https://snyk.io/research/zip-slip-vulnerability
     */
    filter(file: decompress.File): boolean {
        const expectedFilePath = path.join(absoluteDestPath, file.path);
        // If dest is not found in the expected path, it means file will be unpacked somewhere else.
        if (!expectedFilePath.startsWith(path.join(absoluteDestPath, path.sep))) {
            zipSlipFiles.add([file.path, expectedFilePath]);
            return false; // only skip the exploit files, maybe the rest is fine.
        } else {
            return true;
        }
    }
});
if (zipSlipFiles.size > 0) {
    console.error(`Detected a zip-slip exploit in archive: "${sourcePath}"`);
    for (const [relativePath, expectedPath] of zipSlipFiles) {
        console.error(` - File "${relativePath}" was going to write to: "${expectedPath}"`);
    }
    console.error('See: https://snyk.io/research/zip-slip-vulnerability');
}

I think it would be better to prevent this exploit by default, but allow to disable the check via the options (would anyone want to enable this back...)

No reply from Kevva.

Edit: I had not seen the comment from jdillick where he proposed a workaround too. I am not sure if just stripping leading ../ is a nice solution. When is it that you want to pack a file such as ../../foo.txt and expect it to unpack at dest/foo.txt?

The issue is not (apparently) what you would want, but what you should be able to safely expect. The default behavior of the packaging of a parent relative path in GNU tar now is to treat ../../foo.txt as ./foo.txt in the produced tar file.
The safest route however would be simple decline any path that resolved to a parent.
e.g.
const decompress = require("decompress");
const path = require("path");
const assert = require("assert");

const allowedPath = (file, targetDir) => {
  // resolve relative path to target. e.g. file.path of a/../../b could actually be ../b
  const resolvedRelative = path.relative(
    targetDir,
    path.resolve(targetDir, file.path)
  );

  return !/^[..]{2}/.test(resolvedRelative);
};

assert.ok(allowedPath({ path: "./bar" }, "./foo"));
assert.ok(!allowedPath({ path: "../bar" }, "./foo"));
assert.ok(!allowedPath({ path: "../../bar" }, "./foo"));
assert.ok(!allowedPath({ path: "bar/../../baz" }, "./foo"));

const targetDir = path.resolve("./test-dir");

// silently filter out any path that resolves outside directory
decompress("./slip.tar.gz", targetDir, {
  filter: file => allowedPath(file, targetDir)
}).then(files => {
  files.forEach(file => console.log(file.path));
});
// no output

Why don't you use @trptcolin fix? It addresses symlinks too.
see https://github.com/kevva/decompress/pull/73

Why don't you use @trptcolin fix? It addresses symlinks too.
see #73

Current version won't allow creation of unsafe symlinks, or allow traversal of a symlink to create a file (without followSymlink option)

Ah, I see. Thanks for clarification.

Hello, is there any recommended solution for this problem: "Arbitrary File Write"?

Hi @hirvin-faria,
The long-term solution for this issue is for every package maker themselves to update their dependency to 'decompress' to another module that doesn't suffer from the same vulnerability.
In the meantime, if these errors bother you, you can manually substitute 'decompress' with '@atomic-reactor/decompress', a fork of 'decompress' where the vulnerability was corrected by @jdillick.
To do that :

Open your terminal
Change the directory to your project's folder
cd (path to your project's folder)
Install @atomic-reactor/decompress
npm install --save @atomic-reactor/decompress
Open the 'package-lock.json' file in your project's folder
open package-lock.json
Manually replace every instance where 'decompress' is listed as a dependency of another package by '@atomic-reactor/decompress' :  change (...)"requires": { "decompress": "^4.0.0"(...) to  (...)"requires": {"@atomic-reactor/decompress": "^4.0.0"(...)
You can also delete the part of package-lock.json where 'decompress' itself listed as a package, but if you don't know how to do that, you don't need to.

This is a quick fix to your issue. It might break in the future depending on how @atomic-reactor/decompress is maintained (no offense to its contributors), or on how packages that used to depend on 'decompress' will be updated themselves.

@hirvin-faria to stop getting those warnings when using packages that depend on 'decompress', these packages have to be fixed by their owners, contact them.


[Feature] Automatically label PRs with the aspect and goal labels

Problem

Automatically set the aspect and goal labels on PRs to be the same as their linked issues.
Alternatives

We can also have a default "needs labels" label that is applied on PRs by default. That way we won't have to find linked issues and just set this label.
Additional context

Since GitHub API does not offer a way to quickly find the issue linked to a PR, scraping the issue page is an inelegant but functional solution.
Implementation

[ ] 🙋 I would be interested in implementing this feature.

I'm wondering if we could overload our label check, which we know runs every time a PR is created, to apply the correct labels and pass, when the following conditions are met:

the PR does not have goal and aspect labels
there is a linked issue that has goal and aspect labels


[fix] reviewdog.yml

I fixed yml's syntax and suggest to supress reeviewdog's report.
After applying this patch, CI will no longer fail due to reviewdog.
Also, the bot will automatically perform code review based on the results of static analysis.
I think this is more suitable for our needs.
What do you think? @nickspoons

This seems like a sensible change, thanks


Look into suspicious unknowns value

Why are there Florida values ??

Backend is sending wrong states attached to counties

If we remove the counties listed with the wrong state, the values start to look more accurate

This is a known issue where the CDC restricted data contains county / state pairs which don't exist (example attached). I thought I had handled this but I think the most recent changes I made to aggregation broke this again. Will send a fix for this ASAP


Strikethrough conversion is incomplete

markdownToDraft(string, options) includes support for writing markdown with strikethrough text enclosed with  ~~ but draftToMarkdown(rawDraftObject, options) does not include support for creating a DraftJS inline style when the ~~ marker is encountered.
It's possible to add a custom entry to BlockStyles when invoking the conversion routine, but since default support is included in one direction, it makes sense to support it in the other direction as well.
I'll create a PR for this issue.

Thanks for your PR 🙂 merged it and will close this!


feat: added another option for rust integration

Problem
Currently, I struggled a bit with the current rust integration of scalar since it was to much oriented towards axum and I couldn't make it work with actix.
Solution
With this PR I propose another integration of scalar in rust that focuses on integrating scalar in any rust web framework. Don't get me wrong, tamasfe's crate is really good, but its DX is too much scoped towards axum.
Here is the crate I propose : scalar-doc.rs

hey @Courtcircuits looks like theres a prettier issue, if you run
pnpm format in root it should fix it :)
Run pnpm format:check

><EMAIL_ADDRESS>format:check /home/runner/work/scalar/scalar
> pnpm prettier --check .

Checking formatting...
[warn] documentation/integrations/rust.md
[warn] Code style issues found in the above file. Run Prettier with --write to fix.
 ELIFECYCLE  Command failed with exit code 1.

then we can get it merged ✨

It should be fine now :)


MQTT outbound connector does not use connection properties

Issue(s):
https://devtopia.esri.com/webgis/arcgis-geoevent/issues/3814
Description:
Refactored MQTT inbound/outbound transports to take advantage of GeoEvent validation framework.
Testing:

[ ] Added unit tests
[x] Full build pass
[x] Tested on your machine

GeoEvent build with deployed MQTT tranports is available for verification at https://ps0016064.esri.com:6143/geoevent/manager

@eironside , do you want me to upgrade a project release number from 2 to 3 for this transport? Here is what we currently have:

I've updated the version of this connector from 10.4.0 to 10.6.0. The release version appears as 3 on the following screenshot. I can roll it back to 2. Please let me know.