Feature/#1483 Dekorate integration

Relates to: #1483

I have not much time for a thorough review (sorry) but looks good on a first glance. Iiuc then hardening the merge strategy is a task for Dekorate, fmp just copies over its files into dekorate's input directory, right ?
I agree that dekorate needs to be on the project classpath for enabling this feature. this is necessary anyway so that the user can use the Dekorate annotations in her code.

Yes, FMP provides source manifests which are then postprocessed by Dekorate. But Dekorate has opnionated defaults too, so the most probable outcome of this "merge" is some unusable manifest that the user will have to fine-tune anyway.
I think this is a good first step for an integration. In the scope of JKube we can probably consider complete delegation to Dekorate in the future, but that would imply some serious analysis and collaboration between both projects to achieve 100% compatibility.

I tried testing this patch on Openshift(using sample spring-boot-dekorate), but somehow DeploymentConfig doesn't seem to be getting applied. It seems to be some minor issue with labeling.
[INFO] --- fabric8-maven-plugin:4.4-SNAPSHOT:deploy (default-cli) @ fabric8-maven-sample-spring-boot-dekorate ---
[INFO] F8: Using OpenShift at https://api.rh-idev.openshift.com:443/ in namespace rokumar with manifest /home/rohaan/work/repos/fabric8-maven-plugin/samples/spring-boot-dekorate/target/classes/META-INF/fabric8/openshift.yml
[INFO] OpenShift platform detected
[INFO] F8: Using project: rokumar
[INFO] F8: Using project: rokumar
[INFO] F8: Creating a Service from openshift.yml namespace rokumar name fabric8-maven-sample-spring-boot-dekorate
[INFO] F8: Created Service: target/fabric8/applyJson/rokumar/service-fabric8-maven-sample-spring-boot-dekorate-1.json
[INFO] F8: HINT: Use the command `oc get pods -w` to watch your pods start up
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  02:16 min
[INFO] Finished at: 2020-01-02T14:54:53+05:30
[INFO] ------------------------------------------------------------------------
~/work/repos/fabric8-maven-plugin/samples/spring-boot-dekorate : $ oc get all
NAME                                                        READY     STATUS      RESTARTS   AGE
pod/fabric8-maven-sample-spring-boot-dekorate-s2i-1-build   0/1       Completed   0          2m
pod/fabric8-maven-sample-spring-boot-dekorate-s2i-2-build   0/1       Completed   0          1m

NAME                                                TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/fabric8-maven-sample-spring-boot-dekorate   ClusterIP   <IP_ADDRESS>   <none>        8080/TCP   22s

NAME                                                                           TYPE      FROM      LATEST
buildconfig.build.openshift.io/fabric8-maven-sample-spring-boot-dekorate-s2i   Source    Binary    2

NAME                                                                       TYPE      FROM      STATUS     STARTED              DURATION
build.build.openshift.io/fabric8-maven-sample-spring-boot-dekorate-s2i-1   Source    Binary    Complete   2 minutes ago        23s
build.build.openshift.io/fabric8-maven-sample-spring-boot-dekorate-s2i-2   Source    Binary    Complete   About a minute ago   1m7s

NAME                                                                       DOCKER REPO                                                            TAGS      UPDATED
imagestream.image.openshift.io/fabric8-maven-sample-spring-boot-dekorate   <IP_ADDRESS>:5000/rokumar/fabric8-maven-sample-spring-boot-dekorate   latest    31 seconds ago

I tried applying the created yaml file using oc but faced this error:
Error from server (Invalid): error when creating "target/classes/META-INF/fabric8/openshift.yml": DeploymentConfig.apps.openshift.io "fabric8-maven-sample-spring-boot-dekorate" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{"app":"fabric8-maven-sample-spring-boot-dekorate", "decorated-by":"dekorate-kubernetes", "group":"rohaan", "version":"4.4-SNAPSHOT"}: `selector` does not match template `labels`

I'm not sure if I'm missing something or not. But I'm facing the same issue in the case of Kubernetes also. Seems like plugin only applies the first item in the provided Kubernetes list.
~/work/repos/fabric8-maven-plugin/samples/spring-boot-dekorate : $ ls target/classes/META-INF/fabric8/kubernetes.yml
target/classes/META-INF/fabric8/kubernetes.yml
~/work/repos/fabric8-maven-plugin/samples/spring-boot-dekorate : $ mvn fabric8:apply
[INFO] Scanning for projects...
[INFO]
[INFO] --------< io.fabric8:fabric8-maven-sample-spring-boot-dekorate >--------
[INFO] Building Fabric8 Maven :: Sample :: Spring Boot - Dekorate 4.4-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- fabric8-maven-plugin:4.4-SNAPSHOT:apply (default-cli) @ fabric8-maven-sample-spring-boot-dekorate ---
[INFO] F8: Using Kubernetes at https://<IP_ADDRESS>:8443/ in namespace default with manifest /home/rohaan/work/repos/fabric8-maven-plugin/samples/spring-boot-dekorate/target/classes/META-INF/fabric8/kubernetes.yml
[INFO] F8: Using namespace: default
[INFO] F8: Using namespace: default
[INFO] F8: Creating a Service from kubernetes.yml namespace default name fabric8-maven-sample-spring-boot-dekorate
[INFO] F8: Created Service: target/fabric8/applyJson/default/service-fabric8-maven-sample-spring-boot-dekorate.json
[INFO] F8: HINT: Use the command `kubectl get pods -w` to watch your pods start up
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  3.594 s
[INFO] Finished at: 2020-01-02T15:30:32+05:30
[INFO] ------------------------------------------------------------------------

In normal mode openshift and kubernetes directories are created in target resource directory like shown below:
~/work/repos/fabric8-maven-plugin/samples/spring-boot : $ ls target/classes/META-INF/fabric8/
kubernetes  kubernetes.yml  openshift  openshift.yml
~/work/repos/fabric8-maven-plugin/samples/spring-boot : $ ls target/classes/META-INF/fabric8/openshift/
fabric8-maven-sample-spring-boot-deploymentconfig.yml  fabric8-maven-sample-spring-boot-route.yml  fabric8-maven-sample-spring-boot-service.yml
~/work/repos/fabric8-maven-plugin/samples/spring-boot : $ ls target/classes/META-INF/fabric8/kubernetes/
fabric8-maven-sample-spring-boot-deployment.yml  fabric8-maven-sample-spring-boot-ingress.yml  fabric8-maven-sample-spring-boot-service.yml
~/work/repos/fabric8-maven-plugin/samples/spring-boot : $

But dekorate mode doesn't create resource directories. I'm not saying creating directories is necessary; I think we need to check why only first item is being applied and change behavior in ApplyService. Also I face the same issue of mismatching selectors in kuberntes mode also:
~/work/repos/fabric8-maven-plugin/samples/spring-boot-dekorate/target/classes/META-INF/fabric8 : $ kubectl create -f kubernetes.yml
service/fabric8-maven-sample-spring-boot-dekorate created
The Deployment "fabric8-maven-sample-spring-boot-dekorate" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{"app":"fabric8-maven-sample-spring-boot-dekorate", "decorated-by":"dekorate-openshift", "group":"rohaan", "version":"4.4-SNAPSHOT"}: `selector` does not match template `labels`

The Deployment "fabric8-maven-sample-spring-boot-dekorate" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{"app":"fabric8-maven-sample-spring-boot-dekorate", "decorated-by":"dekorate-openshift", "group":"rohaan", "version":"4.4-SNAPSHOT"}: `selector` does not match template `labels`

This is due to Dekorate generating wrong labels (collision between @KubernetesApplication and @OpenshiftApplication annotations) .
Regarding fabric8:apply phase ignoring the rest of list entries, is probably because the Manifest is not a list, so it's missing:
---
apiVersion: v1
kind: List
items:

Modifying the generated Manifest and converting it to a list fixes the problem. I'll investigate if I can fix the apply mojo to handle these kind of manifests.

Sorry, I forgot to test your updated PR. After pulling and testing, I see a different issue. Now all the resources are being applied but the image name is wrong in the generated Deployment. Note that generated manifest reference image rohaan/fabric8-maven-sample-spring-boot-dekorate:4.4-SNAPSHOT but image actually is fabric8/fabric8-maven-sample-spring-boot-dekorate:latest :
~/work/repos/fabric8-maven-plugin/samples/spring-boot-dekorate : $ docker images
REPOSITORY                                          TAG                 IMAGE ID            CREATED             SIZE
fabric8/fabric8-maven-sample-spring-boot-dekorate   latest              a1f42aeb4bb2        3 minutes ago       494MB
meetup/random-generator                             0.0.1               8554958dd048        21 minutes ago      475MB
k8s.gcr.io/kube-proxy                               v1.17.0             7d54289267dc        5 weeks ago         116MB
k8s.gcr.io/kube-apiserver                           v1.17.0             0cae8d5cc64c        5 weeks ago         171MB
k8s.gcr.io/kube-scheduler                           v1.17.0             78c190f736b1        5 weeks ago         94.4MB
k8s.gcr.io/kube-controller-manager                  v1.17.0             5eb3b7486872        5 weeks ago         161MB
kubernetesui/dashboard                              v2.0.0-beta8        eb51a3597525        5 weeks ago         90.8MB
k8s.gcr.io/coredns                                  1.6.5               70f311871ae1        2 months ago        41.6MB
k8s.gcr.io/etcd                                     3.4.3-0             303ce5db0e90        2 months ago        288MB
kubernetesui/metrics-scraper                        v1.0.2              3b08661dc379        2 months ago        40.1MB
k8s.gcr.io/kube-addon-manager                       v9.0.2              bd12a212f9dc        5 months ago        83.1MB
fabric8/java-centos-openjdk8-jdk                    1.5                 f68885b8dce1        9 months ago        457MB
k8s.gcr.io/pause                                    3.1                 da86e6ba6ca1        2 years ago         742kB
gcr.io/k8s-minikube/storage-provisioner             v1.8.1              4689081edb10        2 years ago         80.8MB
~/work/repos/fabric8-maven-plugin/samples/spring-boot-dekorate : $ mvn fabric8:resource
[INFO] Scanning for projects...
[INFO]
[INFO] --------< io.fabric8:fabric8-maven-sample-spring-boot-dekorate >--------
[INFO] Building Fabric8 Maven :: Sample :: Spring Boot - Dekorate 4.4-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- fabric8-maven-plugin:4.4-SNAPSHOT:resource (default-cli) @ fabric8-maven-sample-spring-boot-dekorate ---
[INFO] F8: Dekorate detected, delegating resource build
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.427 s
[INFO] Finished at: 2020-01-13T16:22:58+05:30
[INFO] ------------------------------------------------------------------------
~/work/repos/fabric8-maven-plugin/samples/spring-boot-dekorate : $ cat target/classes/META-INF/fabric8/kubernetes.yml
---
apiVersion: "v1"
kind: "Service"
metadata:
  labels:
    app: "fabric8-maven-sample-spring-boot-dekorate"
    decorated-by: "dekorate-kubernetes"
    version: "4.4-SNAPSHOT"
    group: "rohaan"
  name: "fabric8-maven-sample-spring-boot-dekorate"
spec:
  ports:
  - name: "http"
    port: 8080
    targetPort: 8080
  selector:
    app: "fabric8-maven-sample-spring-boot-dekorate"
    decorated-by: "dekorate-kubernetes"
    version: "4.4-SNAPSHOT"
    group: "rohaan"
  type: "ClusterIP"
---
apiVersion: "apps/v1"
kind: "Deployment"
metadata:
  labels:
    app: "fabric8-maven-sample-spring-boot-dekorate"
    decorated-by: "dekorate-kubernetes"
    version: "4.4-SNAPSHOT"
    group: "rohaan"
  name: "fabric8-maven-sample-spring-boot-dekorate"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: "fabric8-maven-sample-spring-boot-dekorate"
      decorated-by: "dekorate-kubernetes"
      version: "4.4-SNAPSHOT"
      group: "rohaan"
  template:
    metadata:
      labels:
        app: "fabric8-maven-sample-spring-boot-dekorate"
        decorated-by: "dekorate-kubernetes"
        version: "4.4-SNAPSHOT"
        group: "rohaan"
    spec:
      containers:
      - env:
        - name: "KUBERNETES_NAMESPACE"
          valueFrom:
            fieldRef:
              fieldPath: "metadata.namespace"
        image: "rohaan/fabric8-maven-sample-spring-boot-dekorate:4.4-SNAPSHOT"
        imagePullPolicy: "IfNotPresent"
        name: "fabric8-maven-sample-spring-boot-dekorate"
        ports:
        - containerPort: 8080
          name: "http"
          protocol: "TCP"

One more thing, in Dekorate mode Shall we expect plugin to respect Fabric8 Maven Plugin's resource configuration or shall it be Dekorate's? It seems to be always generating Service of type ClusterIP. I want it to be NodePort but it's not doing it even after adding fmp related property.

Now all the resources are being applied but the image name is wrong in the generated Deployment.
The image name is not "wrong" but according to Dekorate opinionated configuration.

The user will either have to change the image name via a Dekorate annotation/property or change FMP build configuration to name the image according to Dekorate.

One more thing, in Dekorate mode Shall we expect plugin to respect Fabric8 Maven Plugin's resource configuration or shall it be Dekorate's?

If you want FMP to respect plugin resource configuration then you have to issue the command with the "merge" flag on mvn fabric8:resource -Dfabric8.mergeWithDekorate but this will have the issues already stated in previous comments
The other option and recommended approach is to use Dekorate's specific annotations/properties to configure your resources
e.g.:
@KubernetesApplication(labels = {
    @Label(key = "decorated-by", value = "dekorate-kubernetes")
}, serviceType = ServiceType.NodePort)


mdSelect missing padding

Bug, feature request, or proposal:
Bug
What is the expected behavior?
Properly display mdSelect label when a value is selected
What is the current behavior?
Missing padding, causing the label to fall off
What are the steps to reproduce?
https://plnkr.co/edit/GPyk6vzdQKfMUtKrmJ5i?p=preview select a value
What is the use-case or motivation for changing an existing behavior?
The label falling off
Which versions of Angular, Material, OS, TypeScript, browsers are affected?
angular 4.1.3
2.0.0-beta.6
windows 10
Typescript 2.3.2
Chrome 58
Is there anything else we should know?

I think it's intentional, because it expects to be inside of a md-card and md-card-content, which has the needed padding.

I don't think so. Putting it in a form looks terrible. Also, two of them after earchother result in overlapping:
https://plnkr.co/edit/uE4w4wBCRJfgpXK9Wvep?p=preview

It's being tracked in #4755

Closing as dupe of #2124


Fork of jambonz freeswitch modules

Hello ,
thank you for this fork . Did you take the fork of  / jambonz/freeswitch-modules which is the latest one .
thanks
Abdul Rasheed

No problem, this was a fork of https://github.com/drachtio/drachtio-freeswitch-modules.git about 2 weeks before it was removed.
Is this what you are looking for?
https://github.com/lylepratt/freeswitch-modules

Hi ,
Thank you for your response... I was looking for https://github.com/jambonz/freeswitch-modules.git which was having the latest updates in it on this project.  https://github.com/drachtio/drachtio-freeswitch-modules.git   this was archived 1 year back i think .
Thanks and Regards
Abdul Rasheed

Thank you Slaney.  i got more recent forks from https://github.com/lylepratt/freeswitch-modules  and it really worked well. Thank you for your support..
Best Regards
Abdul Rasheed


Update setup.py

Python version 3.7 added.

Codecov Report

Merging #506 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #506   +/-   ##
=======================================
  Coverage   87.11%   87.11%
=======================================
  Files          57       57
  Lines        4609     4609
  Branches      501      501
=======================================
  Hits         4015     4015
  Misses        435      435
  Partials      159      159

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ff50b8b...e59b367. Read the comment docs.

Fixed by #507. @stutibiyani Thanks the PR!


mps3_an547:tests/lib/cmsis_dsp/filtering bus fault

Describe the bug
tests/lib/cmsis_dsp/filtering/libraries.cmsis_dsp.filtering.misc.fpu fails with bus fault for mps3_an547 in test case test_arm_correlate_f32_10_11. Caught in an unrelated PR: https://github.com/zephyrproject-rtos/zephyr/pull/51451/checks?check_run_id=8988244380
To Reproduce
Build and run tests/lib/cmsis_dsp/filtering/libraries.cmsis_dsp.filtering.misc.fpu on mps3_an547 board.
Expected behavior
Test pass
Impact
Blocking PR https://github.com/zephyrproject-rtos/zephyr/pull/51451
Logs and console output
START - test_arm_correlate_f32_10_11
E: ***** BUS FAULT *****
E:   Precise data bus error
E:   BFAR Address: 0x3f0f52cf
E: r0/a1:  0x30003908  r1/a2:  0x00000000  r2/a3:  0x00000000
E: r3/a4:  0x00000000 r12/ip:  0x30000298 r14/lr:  0xbdb167ce
E:  xpsr:  0x21100000
E: s[ 0]:  0x00000000  s[ 1]:  0x00000000  s[ 2]:  0x00000000  s[ 3]:  0x0000002f
E: s[ 4]:  0x00000000  s[ 5]:  0x00000000  s[ 6]:  0xffffffff  s[ 7]:  0x00000000
E: s[ 8]:  0x00000000  s[ 9]:  0x00000000  s[10]:  0x00000000  s[11]:  0x00000000
E: s[12]:  0x00000000  s[13]:  0x00000000  s[14]:  0x00000000  s[15]:  0x00000000
E: fpscr:  0x30000000
E: Faulting instruction address (r15/pc): 0x10007ad6
E: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
E: Current thread: 0x300007a0 (test_arm_correlate_f32_10_11)
E: Halting system

Environment (please complete the following information):

OS: Linux
Toolchain: Zephyr SDK 0.15.1
Commit SHA: 2af5779101f6f8201717b4ae6ccb38b89d47e3f5, branched from main 5906bad7c317a684a6d9a16fd8c73d275b509357

Additional context
N/A

@stephanosio

Hi, the same thing happens in other unrelated PR: https://github.com/zephyrproject-rtos/zephyr/pull/50352
CI: https://github.com/zephyrproject-rtos/zephyr/pull/50352/checks?check_run_id=9001159502

A regression due to https://github.com/zephyrproject-rtos/zephyr/pull/51340.
It looks like this test fails with the newlib full variant (it was previously using the nano variant due to the defaults) for some reason.  I will investigate.

This happens because the MVE arm_correlate_f32 implementation has a bug in which it writes to a negative index of the output buffer, which leads to a corrupted memory block header and hence a fault during free() of the memory block.
Upstream CMSIS-DSP bug filed: https://github.com/ARM-software/CMSIS-DSP/issues/59


feat: upgrade support

client.upgrade({
  path: '/',
  method: 'GET',
  protocol: 'Websocket'
}, (err, data) => {
  const { headers, head, socket } = data
})

@mcollina Is this something we want to have? Feedback on API?

@szmarczak @delvedor

Was thinking to do the same for connect and/or include connect here as well.

Or maybe it should throw. What about calling request if the connection has been upgraded?

What happens if there are request in flight? Or if there are a few pipelined? I don't think upgrade should throw.

No more requests should be dispatched after an upgrade until it has completed. Not implemented in this PR yet.

I don't think upgrade should throw.

Not following. We already throw on unexpected upgrade?

Any feedback on the protocol argument? Seems ok?

What about calling request if the connection has been upgraded?

What do you mean?

I think this is somewhat ready. Lacking some tests.

Notice a significant improvement over the node core API is that the head argument is not necessary.

I mean what happens if we start an upgrade to websocket or http2 and there are request in the pipeline.

I mean what happens if we start an upgrade to websocket or http2 and there are request in the pipeline.

Before or after the request?
If it's before, I would think the server doesn't start processing the upgrade until all other requests has completed. Though I'm not sure. Maybe we should change so that an upgrade is not dispatched while there are other requests at head of line.
If it's after they will stay queued until the upgrade is completed and a new socket has been assigned.


ModuleNotFoundError

Hi, I'm trying to train a lite hrnet model on a custom dataset on google colab, but when I try to start the train script:
!python tools/train.py --cgf path/to/config

I get the error:
Traceback (most recent call last):
  File "/content/lite-hrnet/tools/train.py", line 19, in <module>
    from models import build_posenet
ModuleNotFoundError: No module named 'models'

Does anybody know how to solve it?

请问这个问题解决了吗？


[BUG] 502 Bad Gateway when trying to access WebUI (Docker)

Hi,
I'm trying to install InvenTree since this morning with docker (Unraid Server), On Unraid, we don't have the docker compose, so I created two dockers, one containing InvenTree and another containing the Nginx, both are in bridge access. InvenTree was installed without too much difficulty however no way to access the web interface, I get a 502 error and the Nginx logs indicate: "[error] 31#31: *2 connect() failed (111: Connection refused) while connecting to upstream"
Concerning the docker :

InvenTree

Nginx

I've used the Nginx conf files given on the repo :
server {
# Listen for connection on (internal) port 80
listen 80;
server_name <IP_ADDRESS>;
location / {
# Change 'inventree-server' to the name of the inventree server container,
# and '8000' to the INVENTREE_WEB_PORT (if not default)
proxy_pass http://<IP_ADDRESS>:8181;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;

    proxy_redirect off;

    client_max_body_size 100M;

    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_buffering off;
    proxy_request_buffering off;

}

# Redirect any requests for static files
location /static/ {
    alias /var/www/static/;
    autoindex on;

    # Caching settings
    expires 30d;
    add_header Pragma public;
    add_header Cache-Control "public";
}

# Redirect any requests for media files
location /media/ {
    alias /var/www/media/;

    # Media files require user authentication
    auth_request /auth;
}

# Use the 'user' API endpoint for auth
location /auth {
    internal;

    proxy_pass http://<IP_ADDRESS>:8181/auth/;

    proxy_pass_request_body off;
    proxy_set_header Content-Length "";
    proxy_set_header X-Original-URI $request_uri;
}

}
Logs for the InvenTree Docker Starting :
`"Starting background tasks...
Loading action plugins...
Discovered 1 action plugins:
13:23:07 [Q] INFO Q Cluster four-east-wisconsin-pennsylvania starting.
13:23:07 [Q] INFO Process-1:1 ready for work at 18
13:23:07 [Q] INFO Process-1:2 ready for work at 19
13:23:07 [Q] INFO Process-1:3 ready for work at 20
13:23:07 [Q] INFO Process-1:4 ready for work at 21
13:23:07 [Q] INFO Process-1:5 monitoring at 22
13:23:07 [Q] INFO Process-1 guarding cluster four-east-wisconsin-pennsylvania
13:23:07 [Q] INFO Q Cluster four-east-wisconsin-pennsylvania running.
13:23:07 [Q] INFO Process-1:6 pushing tasks at 23
13:23:36 [Q] INFO Enqueued 1
13:23:36 [Q] INFO Process-1 created a task from schedule [InvenTree.tasks.delete_successful_tasks]
13:23:36 [Q] INFO Enqueued 2
13:23:36 [Q] INFO Process-1 created a task from schedule [InvenTree.tasks.check_for_updates]
13:23:36 [Q] INFO Enqueued 3
13:23:36 [Q] INFO Process-1 created a task from schedule [InvenTree.tasks.heartbeat]
13:23:36 [Q] INFO Enqueued 4
13:23:36 [Q] INFO Process-1 created a task from schedule [InvenTree.tasks.update_exchange_rates]
13:23:36 [Q] INFO Enqueued 5
13:23:36 [Q] INFO Process-1 created a task from schedule [InvenTree.tasks.delete_expired_sessions]
13:23:36 [Q] INFO Enqueued 6
13:23:36 [Q] INFO Process-1 created a task from schedule [stock.tasks.delete_old_stock_items]
13:23:36 [Q] INFO Process-1:1 processing [grey-dakota-two-hamper]
13:23:36 [Q] INFO Process-1:2 processing [floor-snake-papa-cold]
13:23:36 [Q] INFO Process-1:3 processing [red-floor-louisiana-mississippi]
13:23:36 [Q] INFO Process-1:4 processing [hawaii-sodium-lima-ten]
Could not perform heartbeat task - App registry not ready
13:23:36 [Q] INFO Process-1:3 processing [low-music-undress-magnesium]
13:23:36 [Q] INFO Process-1:1 processing [virginia-pennsylvania-seventeen-yankee]
Deleting 0 expired sessions.
Updating exchange rates from https://api.exchangerate.host/latest
Using base currency 'USD'
13:23:36 [Q] INFO Processed [red-floor-louisiana-mississippi]
13:23:36 [Q] INFO Processed [grey-dakota-two-hamper]
13:23:36 [Q] INFO Processed [low-music-undress-magnesium]
13:23:36 [Q] INFO Processed [virginia-pennsylvania-seventeen-yankee]
Latest InvenTree version: '0.5.4'
13:23:37 [Q] INFO Processed [floor-snake-papa-cold]
13:23:37 [Q] INFO Processed [hawaii-sodium-lima-ten]
13:35:08 [Q] INFO Enqueued 7
13:35:08 [Q] INFO Process-1 created a task from schedule [InvenTree.tasks.heartbeat]
13:35:08 [Q] INFO Process-1:3 processing [six-utah-sodium-seven]
Could not perform heartbeat task - App registry not ready
13:35:08 [Q] INFO Processed [six-utah-sodium-seven]"`
Deployment Method

Docker on Unraid Server

Thanks for Your Help !

Hi @Wystik-69 as far as I can see the logs are just from the worker. The InvenTree image can either be started as a webserver or as a background d worker. Maybe that is your problem.

@SchrodingersGat @nwns any idea?

Thanks @matmair first problem solved I didn't paid attention to this detail at all, my docker was alone and running with the post "Invoke Worker" only., now I can access the WebUI directly from the Inventree docker but not passing through the nginx docker, which is not a bad thing, I would have one less docker to manage, will it be a problem if I use it directly without Nginx or do I have to use Nginx? Because for the moment I still have a 502 problem.

Another question what is the default login/password, because the invoke superuser didn't work at all for me.

@Wystik-69 I don't believe InvenTree loads a default username and password unless you are loading one of the demo datasets.  The invoke superuser creates the first user on the system.
What installation instructions are you following?  Which database server do you have configured?

I've followed the docker guide install from the doc, I use MariaDB so I've configured the docker to work with mysql. Invoke user has worked, so it's ok for the login password.
For now everything is ok, Nginx keep showing the 502 gateway error.

@Wystik-69 Nginx serves as a proxy. You do not really need it but it enhances the performance and takes care of serving static content. I would not recommend exposing your instance to the internet without a proxy and some kind of ddos protection (Cloudflare for example).

I have a caddy server, I will put the instance behind it, I also have a CloudFlare plan over the Caddy, so I suppose it will be ok, I was just woried about security and stability concerning the docker and if it must run obligatory behind Nginx or not. Many Thanks for your help guys 👍

@Wystik-69 can we close this now?

Yes, I think so :) I will try the system tomorrow, I hope it will work like a charm, for now I'm translating InvenTree in French on crowdin because no one has done it lol !


Reopen if there's more demand!


[BUG] Missing Id value for custom radio button in RadioGroup

Description
On adding custom radio button, the id value which is present for default radio button isn't getting added for the custom radio button.
Steps to reproduce the behaviour:

Add Radio Group.
Delete the Second radio button option.
Add it manually using provided input box.
Check JSON schema for the same value.

Important Details

Version: Cloud
OS: MacOs 10.14.6
Browser: Firefox 80.0.1 (64-bit).

Hey I am Hetu, one of the maintainers at Appsmith. This is a great issue for anyone looking to start contributing to the repo.
The issue arises from the confusion of new options not created with id field in the options. In actuality, the id field is not needed at all, but some old config still has the id field.
You can contribute by just removing the id field in the default options config here

Need more guidance while working on this? Feel free to mention me in a comment on this issue and I'll be happy to help you out 😄

@hetunandu  I want to start contributing to this repo and I found it interesting  , there are a lot of things to contribute to . Can i take this up ? Can you assign me .

Hey @Jain-Sanchit, welcome to Appsmith! Happy to so excited about contributing here 😄
I have assigned this issue to you. Make sure you raise a PR within 2 days. You should also tag your friends in the PR to get a chance to win extra hacktober fest goodies from Appsmith. Read our hacktober fest discussion for more details

@hetunandu  Sure Sir , I will try to raise a PR within time and will share this with friends.


Consider changing resource name argument to Input[NonEmptyString]

Currently all our resource constructors take a NonEmptyString as the first argument. This however is suboptimal when batch-creating things using something like 1 to 3 map { i => because in subsequent operations you usually have Vector[Output[String]] and want to use that to create new names for new resources grouped with previous ones. You therefore end up with Output[String] and even if you interpolate it with p"" you still get an Output[String] (not even NonEmptyString but that's a separate issue (#290)) that you can't use directly with resource constructor and have to flatMap on to use the value.
The scope of this issue is to research whether we can modify the sdk so that we accept Input[NonEmptyString] for resource constructor name parameter and how many changes would that incur in sdk internals.

There's actually more to this than meets the eye, mostly in terms of how dynamic definition of resources bork up preview but not only. This in turn is closely related to the fact that we don't lie and when running a dry run Outputs that are fulfilled by remote values do not execute their maps and flatMaps because, well, there's nothing they could supply as an argument and so they short-circuit just like None in Option[A]. If we allow users to pass arbitrary Outputs as resource names it necessitates the following:
Besom programs may suddenly start being sequential because they have to wait for a previously defined resource to get its outputs resolved to even start preparing another resource that needs that property for its name. This sounds logical - you need that value to name your resource, right? The problem is - resource constructors do a part of their work on the same "thread" (monadic chain of Result) and then fork in a fire-and-forget fashion the whole "now call grpc with this request" bit and return to user's program immediately. If we wait for any of the Outputs that have values based on rpc in the first part, before the fork/fire-and-forget, we basically can't return the will-be-fulfilled-in-the-future Resource until the Output we wait for is resolved. Hence, we turn our program from parallel by default into sequential. I'm afraid a bit this already happens or can happen with new ResourceOptions API that allows anything to be passed as Output and therefore I think a lot more stuff should be moved into the after-fork part of processing.

related to #357


Refresh configuration without restart, but behind load balanced URL

I am currently using Steeltoe in an environment which is behind a load balanced URL, and looking for a means to refresh the configuration without restarting. The load balancer directs traffic to several different Cloud Foundry environments, each of which could have 1 to many instances of the app running via autoscaler.
I've read through previously reported issues and noted this issue which mentions IOptionsSnapshot and the /reload endpoint, as well as IConfigurationRoot.Reload. Based on my current understanding, Steeltoe users with a similar load balanced environment may have an issue with finding a way to reload configs across all environments, for all app instances, as a call to the /reload endpoint would only hit
a single environment/instance. Hopefully I am overlooking a solution to this issue.
If not, I noticed that perhaps the ideal solution is in the backlog. I am not intimately familiar with the Steeltoe source code yet, but is this a feasible addition, or is this a serious undertaking? I would love to help contribute to implementing this, but thought I would first check if others have looked into it already and noticed significant blockers.

@jkonicki Would your messaging abstraction support the pub/sub capabilities within Redis? That's what we used to provide this functionality so it'd be great if we could just switch to a framework supported component.

@mzakrzewski First step is us getting the Messaging abstraction framework in place.  This is no small task and is being worked on now for our 3.0 release.  At first we are going to be supporting RabbitMQ and Kafka.  We can discuss adding Redis as a third option.  Thanks for your request.

What you want is bus-refresh feature which exists in Spring. While you're waiting for necessary building blocks to be added to spring, you can find the code necessary to do what you want here, as it implements subset of needed features to make this work.
https://github.com/macsux/spa-buildpack

@TimHess do you plan on implementing this feature anytime soon? My team is using Steeltoe's refresh actuator which works good but soon we are starting a project where microservice instances will self register to a Netflix Eureka discovery server. Would you suggest an approach for refreshing the configuration of multiple instances of a microservice.
Thanks!


Enable gomobile build besides gomobile bind

gomobile bind can't be used for a lib importing golang.org/x/mobile/app so far.

CC @seebs
I started to think that it might be possible to use build tags.

Now the error seems fixed. I'll try this again.

_ "golang.org/x/mobile/app" causes runtime crash on gomobile-bind apps.
Thus, we'd need to use some build tags to distinguish gomobile-bind and gomobile-build on compile time.

As far as I remember, @fastfading wanted this feature?
I think I can implement this until this weekend.

Done https://github.com/hajimehoshi/ebiten/wiki/Mobile


e2e: Fix TA test07-mixed-allocations test

Fix the topology-aware mixed allocations test by restoring the original testing code from cri-resource-manager repo. By some miracle reason the original code works now ok and the modified one does not. This test should be made more reliable but at some later time.

Added mention of mixed-allocation test fixes to the todo file.


[spike] Investigate support for TaskFlowAPI in kedro-airflow

Description
Is your feature request related to a problem? A clear and concise description of what the problem is: "I'm always frustrated when ..."
Since Airflow 2.0, a simpler TaskFlowAPI for DAGs is released as an alternative with the Operator API. At the moment kedro-airflow supports the Operator, but it's good to keep an eye on it.
https://airflow.apache.org/docs/apache-airflow/stable/tutorial_taskflow_api.html
Context
Why is this change important to you? How would you use it? How can it benefit other users?
Possible Implementation
(Optional) Suggest an idea for implementing the addition or change.
Possible Alternatives
(Optional) Describe any alternative solutions or features you've considered.

Apparently both approaches can coexist, so TaskFlow doesn't aim to be an alternative https://www.astronomer.io/blog/apache-airflow-taskflow-api-vs-traditional-operators/

Outcome: Example of how this API would look like, plus some investigation on what's the sentiment of the community towards this new API (with respect to more modern Python operators, such as the virtual env operator, name TBC).

I attempted to update the spaceflights-pandas DAG from its current form to the TaskFlow API, as seen in this commit: https://github.com/DimedS/kedro-taskFlowAPI/commit/98b69f575a80332bc67868f8d1796346a90860e9
It works, but the result looks not so beautiful. I believe we should redesign some aspects. With the TaskFlow API, it seems we no longer need the KedroOperator, and we should also improve something in the way how we describe the execution order with TaskFlowAPI.

TaskFlow API
TaskFlow API was introduced in Apache Airflow v2 and onwards. It's an alternative way to define your DAG files for orchestration by Airflow.
Basic Structure

 Example TaskFlow DAG
from airflow.decorators import dag, task

@dag(
    schedule=None,
    start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
    catchup=False,
    tags=["example"],
)
def tutorial_taskflow_api():
    """
    ### TaskFlow API Tutorial Documentation
    This is a simple data pipeline example which demonstrates the use of
    the TaskFlow API using three simple tasks for Extract, Transform, and Load.
    Documentation that goes along with the Airflow TaskFlow API tutorial is
    located
    [here](https://airflow.apache.org/docs/apache-airflow/stable/tutorial_taskflow_api.html)
    """
    @task()
    def extract():
        """
        #### Extract task
        A simple Extract task to get data ready for the rest of the data
        pipeline. In this case, getting data is simulated by reading from a
        hardcoded JSON string.
        """
        data_string = '{"1001": 301.27, "1002": 433.21, "1003": 502.22}'

        order_data_dict = json.loads(data_string)
        return order_data_dict
    @task(multiple_outputs=True)
    def transform(order_data_dict: dict):
        """
        #### Transform task
        A simple Transform task which takes in the collection of order data and
        computes the total order value.
        """
        total_order_value = 0

        for value in order_data_dict.values():
            total_order_value += value

        return {"total_order_value": total_order_value}
    @task()
    def load(total_order_value: float):
        """
        #### Load task
        A simple Load task which takes in the result of the Transform task and
        instead of saving it to end user review, just prints it out.
        """

        print(f"Total order value is: {total_order_value:.2f}")
    order_data = extract()
    order_summary = transform(order_data)
    load(order_summary["total_order_value"])
tutorial_taskflow_api()

As compared to the same code in the traditional DAG format :

 Same Example in the traditional API
import json
from datetime import datetime

from airflow import DAG
from airflow.models.baseoperator import chain
from airflow.operators.python import PythonOperator

def extract(ti=None, **kwargs):
    """
    Pushes the estimated population (in millions) of
    various cities into XCom for the ETL pipeline.
    Obviously in reality this would be fetching this
    data from some source, not hardcoded values.
    """
    sample_data = {"Tokyo": 3.7, "Jakarta": 3.3, "Delhi": 2.9}
    ti.xcom_push("city_populations", json.dumps(sample_data))

def transform(ti=None, **kwargs):
    """
    Pulls the provided raw data from XCom and pushes
    the name of the largest city in the set to XCom.
    """
    raw_data = ti.xcom_pull(task_ids="extract", key="city_populations")
    data = json.loads(raw_data)

    largest_city = max(data, key=data.get)
    ti.xcom_push("largest_city", largest_city)

def load(ti=None, **kwargs):
    """
    Loads and prints the name of the largest city in
    the set as determined by the transform.
    """
    largest_city = ti.xcom_pull(task_ids="transform", key="largest_city")

    print(largest_city)

with DAG(
    dag_id="city_pop_etl_pythonoperator",
    schedule=None,
    start_date=datetime(2021, 1, 1),
    catchup=False,
    tags=["example"],
) as dag:

    extract_task = PythonOperator(
        task_id="extract",
        python_callable=extract,
    )

    transform_task = PythonOperator(
        task_id="transform",
        python_callable=transform,
    )

    load_task = PythonOperator(
        task_id="load",
        python_callable=load,
    )

    chain(
        extract_task,
        transform_task,
        load_task,
    )

You can decorate python functions with @task decorators
The DAG file is decorated with @dag decorator

Kedro Spaceflights DAGs example
Consider the Spaceflights tutorial (only the data processing part to keep the code snippets short):
Just python code
If we wanted to perform data processing steps on the companies, shuttles, and the reviews data from scratch (without using Kedro), we could simply define python functions as @tasks and pass data between them -

 Data processing DAG without Kedro
from airflow.decorators import dag, task

import pandas as pd
import logging

def _is_true(x: pd.Series) -> pd.Series:
    return x == "t"

def _parse_percentage(x: pd.Series) -> pd.Series:
    x = x.str.replace("%", "")
    x = x.astype(float) / 100
    return x

def _parse_money(x: pd.Series) -> pd.Series:
    x = x.str.replace("$", "").str.replace(",", "")
    x = x.astype(float)
    return x

@task
def preprocess_companies(companies: pd.DataFrame) -> pd.DataFrame:
    """Preprocesses the data for companies.

    Args:
        companies: Raw data.
    Returns:
        Preprocessed data, with `company_rating` converted to a float and
        `iata_approved` converted to boolean.
    """
    companies["iata_approved"] = _is_true(companies["iata_approved"])
    companies["company_rating"] = _parse_percentage(companies["company_rating"])
    logging.info("Preprocessing companies data")
    return companies

@task
def preprocess_shuttles(shuttles: pd.DataFrame) -> pd.DataFrame:
    """Preprocesses the data for shuttles.

    Args:
        shuttles: Raw data.
    Returns:
        Preprocessed data, with `price` converted to a float and `d_check_complete`,
        `moon_clearance_complete` converted to boolean.
    """
    shuttles["d_check_complete"] = _is_true(shuttles["d_check_complete"])
    shuttles["moon_clearance_complete"] = _is_true(shuttles["moon_clearance_complete"])
    shuttles["price"] = _parse_money(shuttles["price"])
    logging.info("Preprocessing shuttles data")
    return shuttles

@task
def create_model_input_table(
    shuttles: pd.DataFrame, companies: pd.DataFrame, reviews: pd.DataFrame
) -> pd.DataFrame:
    """Combines all data to create a model input table.

    Args:
        shuttles: Preprocessed data for shuttles.
        companies: Preprocessed data for companies.
        reviews: Raw data for reviews.
    Returns:
        Model input table.

    """
    logging.info("Preprocessing model input table data")
    rated_shuttles = shuttles.merge(reviews, left_on="id", right_on="shuttle_id")
    rated_shuttles = rated_shuttles.drop("id", axis=1)
    model_input_table = rated_shuttles.merge(
        companies, left_on="company_id", right_on="id"
    )
    model_input_table = model_input_table.dropna()
    return model_input_table

shuttle_data = pd.read_excel("shuttles.xlsx")
company_data = pd.read_csv("companies.csv")
reviews = pd.read_csv("reviews.csv")

@dag(dag_id="space_no_kedro")
def space_dag():
    companies = preprocess_companies(company_data)
    shuttles = preprocess_shuttles(shuttle_data)
    create_model_input_table(shuttles, companies, reviews)

space_dag()

Airflow can infer the order of these tasks from the inputs and outputs of the tasks -
@dag(dag_id="space_no_kedro")
def space_dag():
    companies = preprocess_companies(company_data)
    shuttles = preprocess_shuttles(shuttle_data)
    create_model_input_table(shuttles, companies, reviews)

With Kedro with the traditional API
Now, if we were already using Kedro, the steps to deploy the Spaceflights tutorial are currently as follows -

Make sure your project doesn't have any MemoryDatasets - add explicit entries for the memory datasets or add a default dataset factory pattern to the catalog
Generate a DAG using kedro-airflow with kedro airflow create
Move the generated DAG to <AIRFLOW_HOME>/dags
Package and install the spaceflights project in the airflow environment
Point to the conf_source (in the DAG) or move/copy the conf/ folder to airflow directory

This is the DAG generated by kedro airflow create -

 Traditional DAG generated by `kedro-airflow`
from __future__ import annotations

from datetime import datetime, timedelta
from pathlib import Path

from airflow import DAG
from airflow.models import BaseOperator
from airflow.utils.decorators import apply_defaults

from kedro.framework.session import KedroSession
from kedro.framework.project import configure_project

class KedroOperator(BaseOperator):
    @apply_defaults
    def __init__(
        self,
        package_name: str,
        pipeline_name: str,
        node_name: str | list[str],
        project_path: str | Path,
        env: str,
        *args, **kwargs
    ) -> None:
        super().__init__(*args, **kwargs)
        self.package_name = package_name
        self.pipeline_name = pipeline_name
        self.node_name = node_name
        self.project_path = project_path
        self.env = env

    def execute(self, context):
        configure_project(self.package_name)
        with KedroSession.create(self.project_path, env=self.env) as session:
            if isinstance(self.node_name, str):
                self.node_name = [self.node_name]
            session.run(self.pipeline_name, node_names=self.node_name)

# Kedro settings required to run your pipeline
env = "local"
pipeline_name = "__default__"
project_path = Path.cwd()
package_name = "space"

# Using a DAG context manager, you don't have to specify the dag property of each task
with DAG(
    dag_id="space-project",
    start_date=datetime(2023,1,1),
    max_active_runs=3,
    # https://airflow.apache.org/docs/stable/scheduler.html#dag-runs
    schedule_interval="@once",
    catchup=False,
    # Default settings applied to all tasks
    default_args=dict(
        owner="airflow",
        depends_on_past=False,
        email_on_failure=False,
        email_on_retry=False,
        retries=1,
        retry_delay=timedelta(minutes=5)
    )
) as dag:
    tasks = {
        "preprocess-companies-node": KedroOperator(
            task_id="preprocess-companies-node",
            package_name=package_name,
            pipeline_name=pipeline_name,
            node_name="preprocess_companies_node",
            project_path=project_path,
            env=env,
        ),
        "preprocess-shuttles-node": KedroOperator(
            task_id="preprocess-shuttles-node",
            package_name=package_name,
            pipeline_name=pipeline_name,
            node_name="preprocess_shuttles_node",
            project_path=project_path,
            env=env,
        ),
        "create-model-input-table-node": KedroOperator(
            task_id="create-model-input-table-node",
            package_name=package_name,
            pipeline_name=pipeline_name,
            node_name="create_model_input_table_node",
            project_path=project_path,
            env=env,
        ),

    }

    tasks["create-model-input-table-node"] >> tasks["split-data-node"]
    tasks["preprocess-companies-node"] >> tasks["create-model-input-table-node"]
    tasks["preprocess-shuttles-node"] >> tasks["create-model-input-table-node"]

To run Kedro nodes on Airflow, we define a KedroOperator which is a subclass of the BaseOperator. It creates a Session and runs specific nodes for each call -
class KedroOperator(BaseOperator):
    @apply_defaults
    def __init__(
        self,
        package_name: str,
        pipeline_name: str,
        node_name: str | list[str],
        project_path: str | Path,
        env: str,
        *args, **kwargs
    ) -> None:
        super().__init__(*args, **kwargs)
        self.package_name = package_name
        self.pipeline_name = pipeline_name
        self.node_name = node_name
        self.project_path = project_path
        self.env = env

    def execute(self, context):
        configure_project(self.package_name)
        with KedroSession.create(self.project_path, env=self.env) as session:
            if isinstance(self.node_name, str):
                self.node_name = [self.node_name]
            session.run(self.pipeline_name, node_names=self.node_name)

And then we define tasks as a dict of node name to it's KedroOperator object:
 tasks = {
        "preprocess-companies-node": KedroOperator(
            ...
        ),
        "preprocess-shuttles-node": KedroOperator(
            ...
        ),
        "create-model-input-table-node": KedroOperator(
            ...
        ),

    }

And finally, we define the order of execution of these tasks -
tasks["create-model-input-table-node"] >> tasks["split-data-node"]
tasks["preprocess-companies-node"] >> tasks["create-model-input-table-node"]
tasks["preprocess-shuttles-node"] >> tasks["create-model-input-table-node"]

Few things to note:

We assume that the intermediate datasets are saved and loaded from file because each Session runs independently
We need to define the task order

With Kedro using the TaskFlow API
I tried to convert the traditional spaceflights DAG to the TaskFlow API

 TaskFlow format DAG
from airflow.decorators import dag, task
from datetime import datetime, timedelta
from pathlib import Path

from kedro.framework.session import KedroSession
from kedro.framework.project import configure_project

def run_kedro_node(package_name, pipeline_name, node_name, project_path, env):
    configure_project(package_name)
    with KedroSession.create(project_path, env=env) as session:
        return session.run(pipeline_name, node_names=[node_name])

# Kedro settings required to run your pipeline
env = "local"
pipeline_name = "__default__"
project_path = Path.cwd()
package_name = "space"

@dag(
    dag_id="space_tf",
    start_date=datetime(2023,1,1),
    max_active_runs=3,
    # https://airflow.apache.org/docs/stable/scheduler.html#dag-runs
    schedule_interval="@once",
    catchup=False,
    # Default settings applied to all tasks
    default_args=dict(
        owner="airflow",
        depends_on_past=False,
        email_on_failure=False,
        email_on_retry=False,
        retries=1,
        retry_delay=timedelta(minutes=5)
    )
)
def kedro_dag():
    @task(task_id="preprocess-companies-node")
    def preprocess_companies_node():
        run_kedro_node(package_name, pipeline_name, "preprocess_companies_node", project_path, env)

    @task(task_id="preprocess-shuttles-node")
    def preprocess_shuttles_node():
        run_kedro_node(package_name, pipeline_name, "preprocess_shuttles_node", project_path, env)

    @task(task_id="create-model-input-table-node")
    def create_model_input_table_node():
        run_kedro_node(package_name, pipeline_name, "create_model_input_table_node", project_path, env)

    preprocess_shuttles_node()  >> create_model_input_table_node()
    preprocess_companies_node() >> create_model_input_table_node()

kedro_dag()

Key Differences + Similarities

Get rid of KedroOperator for a function that does the same thing as KedroOperator.execute()

def run_kedro_node(package_name, pipeline_name, node_name, project_path, env):
    configure_project(package_name)
    with KedroSession.create(project_path, env=env) as session:
        return session.run(pipeline_name, node_names=[node_name])

Each task is a function which calls run_kedro_node with the specific node name:

@task(task_id="preprocess-companies-node")
def preprocess_companies_node():
    run_kedro_node(package_name, pipeline_name, "preprocess_companies_node", project_path, env)

Still have to define a task order because we don't pass MemoryDatasets between tasks

preprocess_shuttles_node()  >> create_model_input_table_node()
preprocess_companies_node() >> create_model_input_table_node()

There is a call to the dag function at the end

kedro_dag()

Benefits of TaskFlow DAGs
These are the benefits that the new TaskFlow API offers -
Reduce boilerplate code
TaskFlow API greatly benefits you if you've been using the PythonOperator in your code, you can just decorate your python functions with @task. In the case of Kedro projects, it can still reduce the boilerplate code.

We don't need a KedroOperator just to create and run a session, it can be a simple python function
The lines of code go down from 119 to 67 for the spaceflights tutorial for example
Not much of a difference though, each task still needs a function + we still need to define the task order

Intuitive data transfer between DAGs
If you are working with python functions as tasks, TaskFlow API makes it super easy to pass data between them. It would have to be done through xcom_push or xcom_pull earlier with the traditional DAGs.
This is not directly relevant to the way Kedro works right now since we expect users to not have MemoryDatasets in their project - they can add explicit entries to their catalog or add a default catch-all dataset factory pattern.
NOTE (Out of scope) : It is possible to pass datasets between different tasks. But it isn't very easy to inject data into a KedroSession (related ticket https://github.com/kedro-org/kedro/issues/2169) See the comment below 👇🏾
No need to define task order
This again is not relevant to the way Kedro works right now, this is because in the TaskFlow API infers the orders of the tasks from the inputs and outputs between the tasks. For Kedro, we create individual KedroSessions for each node, there is no data that passes between them. We still need to define task orders
Conclusion

The benefits of the TaskFlow API are not very relevant to the way we define Kedro DAGs, it does reduce the boilerplate code a little bit.
It is possible to use the traditional + the TaskFlow API together
It is possible to leverage the TaskFlow API to pass MemoryDatasets between tasks - with some modifications to the KedroSession. See the comment following this for further discussion.

Decisions

Should we offer this at all?: TaskFlow API is not replacing the traditional DAGs, it's just an alternative way to define the DAGs

If yes ^:

Fully?

Replace the current DAG jinja with the TaskFlow one? Yes/No

Partially?

Modify the current DAG jinja to incorporate some elements of the TaskFlow API (like @dag decorator)
Offer it as an option

If option, what would be the design? --api=<traditional/taskflow>

Extra: Leveraging the TaskFlow API
TaskFlow API makes it simpler to pass data between tasks without relying on XCom.
Currently we recommend users to make sure their projects do not rely on MemoryDatasets - either put in explicit entries for the memory datasets or a catch-all pattern in their catalog files.
We can leverage TaskFlowAPI to make it possible to pass along intermediate datasets between tasks without the overhead of saving and loading them from file.
[COMMENT WRITING IN PROGRESS]

Fantastic! This is a very detailed and high-quality analysis, thank you @ankatiyar ! I believe this topic would be excellent for discussion in our Technical Design session.
In my opinion:

for the first step, we should partially implement the TaskFlow API: remove the KedroOperator and use the @task and @dag decorators, while maintaining the task order from the traditional API. This approach, as I understand, significantly reduces the size of the code in the generated DAG and should be relatively easy to implement.
In the second step, we could consider supporting transitions in Memory Datasets using the TaskFlow API.
I think we should retain the task order from the traditional API for tasks that do not involve Memory Datasets

Based on the discussions, this is not worth pursuing with the current state of things. Closing this.

Upon further discussion, we will adopt certain aspects of the TaskFlow API like @dag decorators etc without transitioning to it fully #705


Updated Ansilove URL per issue #25

Merging Frederic Cambus's update to Ansilove URL

Thanks, Frederic! This change is going in the next release.


Question: 3D Secure

Hello,
Have you by any chance worked on 3D Secure with this bridge?

See #74 and #75


Checks performed in check_public seem insufficient

The check_public function nominally checks "that the public key is well formed and has an exponent within acceptable bounds."
https://github.com/RustCrypto/RSA/blob/master/src/key.rs#L642
But it also accepts even exponents, as long as they are within the defined range.
And it does not perform any checks on the modulus, accepting values including zero, even numbers, or modulus smaller than the public exponent.
Is accepting such keys intentional?

no, this is likely a carry over from the fact thah originally it was not possible to specify a custom exponent. So this should be fixed

In fact, the checks should be carried out in a couple of places. For instance, passing an even exponent to the function RsaPrivateKey::new_with_exp will hang forever, since the computation of the modular inverse in src/algorithms.rs:128 will never succeed. This simple test showcases the issue:
    #[test]
    fn test_new_with_exp() {
        let mut rng = rand::thread_rng();
        let bits = 2048;
        let _priv_key = RsaPrivateKey::new_with_exp(&mut rng, bits, &BigUint::from_u64(65536).unwrap());
    }

I think this should be given some thought to make sure checks are in place where needed and not much duplicated code is introduced. Maybe one could also set a lower bound to the modulus size (e.g., 1024), and add to the checks. NIST SP 800-56B Revision 2 (section 6.4) may provide some guidance.
If you give me some time to get familiarized with the code, I can propose some changes.

We added an initial set of public key checks in #170 and #176.
There currently aren't any such checks on private keys, beyond checking that there are at least two primes.
Checks on both public and private keys could definitely be further improved. A minimum modulus size definitely seems like a good idea (with the ability to bypass it through an _unchecked API, which seems important to @dignifiedquire's goals of supporting exotic legacy OpenPGP keys).
I think the private key APIs currently permit a number of potential misuses (see above-mentioned exponent-related DoS) which it would be good to mitigate.

Thanks for your feedback @tarcieri. I'd be happy to know more about those exotic/legacy use cases. @dignifiedquire: Any pointer where this has been discussed or documented?

The original reason why I wrote this implementation was to support https://github.com/rpgp/rpgp which is a full rust implementation of OpenPGP, which is used with a variety of existing keys out there in the wild. There is a set of keys in the tests there, but in general I have to follow roughly "if Gnupg accepts the key, I have to accept it"

We can still make the default APIs misuse resistant, with *_unchecked APIs that bypass the checks.
See the existing RsaPublicKey::new_unchecked API.

Hi guys. I took a look at this issue in a bit more detail, and have a couple of considerations:

I find the "unchecked" approach dangerous. It leads to inconsistencies such as accepting a key of length outside the bounds that are accepted in key::check_public_with_max_size. This would currently clash with the check in oaep::encrypt and oaep::decrypt. Moreover, it may lead to duplicating code all over the place to allow a "checked" and an "unchecked" interface for all algorithms/schemes.
From what I see at GnuPG's website, they support keys from 1k to 4k (up to 8k/16k  with a compile-time flag):

RSA is the world’s premier asymmetric cryptographic algorithm, and is built on the difficulty of factoring extremely large composites. GnuPG supports RSA with key sizes of between 1024 and 4096 bits.

Since version 2.0.27 and 1.4.19 GnuPG can be compiled with --enable-large-secmem to offer an --enable-large-rsa option that can create keys up to 8 KiB. Some elder versions supported creating of keys up to 16 KiB.

In light of this, a reasonable approach to satisfy all parties may be to

accept keys >= 1k at RsaPrivateKey::from_components and public keys (since these are built "from components" too), and
generate keys >= 2k only, since this is the best practice these days.

Then the checks simply ensure the least restrictive of these conditions.
According to the above, I would suggest:
For RsaPublicKey::new*:

3 <= e <= upper_bound, and odd
1024 <= bits(n) <= 4096 (or larger), and odd
e < n

I'm not sure why  RsaPublicKey::new_with_max_size exists. Maybe one should rather find a reasonable upper bound and stick to it for the checks. Also, as I mentioned above, I would drop RsaPublicKey::new_unchecked.
For RsaPrivateKey::{new*,from_components}:
At most, I would consider checking for primality of the primes. Everything else I can think of is already there.
What do you think?

I understand the desire to avoid the unchecked versions, but it will reduce the value of this library. There is still work happening today, eg for using RSA for accumulators, and other experiments, where being able to widely vary the modulus is quite beneficial, from tests to benchmarks, etc. So I think the we need to keep unchecked versions available.

Understood. In that case, I guess we can leave the key bounds, the unchecked versions, etc., aside for now, and address the minimum requirements that originated this issue. That is, adding the following checks to key::check_public_with_max_size:

n odd
e odd and e < n


Add way to send extra_params to _BucketIterator and _BlobIterator

PRs #802 and #806 should address this issue.


Update interactive notebooks to switch to S2 C3 data

Proposed changes
In a recent PR we excluded two interactive notebooks from being updated until NRT and final maturity data were contained in the same Sentinel-2 products:
https://github.com/GeoscienceAustralia/dea-notebooks/pull/976#pullrequestreview-1176832173
Because we now have a working dataset maturity workflow including both final and NRT, I have updated these remaining notebooks to use only the ga_s2am_ard_3 and ga_s2bm_ard_3 products.
(I also added Landsat 9 to both of these interactive notebooks to avoid us having to come back and add it later)

Hey @uchchwhash @geoscience-aman, this is a quick PR to remove Sentinel-2 Provisional data from the last two remaining interactive notebooks - I unfortunately formatted the data using Black for the first time so the diff is big, but I've tested both of the notebooks and they work nicely.
@JM-GA @ohehir FYI, apart from Aman's S2 C3 notebook PR here that's awaiting approval from Maree, this PR removes the last trace of S2 C1 data from the DEA Notebooks repo, so we can probably tick off that section of the decomissioning plan once these PRs are merged!


Error message scary but not great

> git pull --tags origin main
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256: <removed>
Please contact your system administrator.
Add correct host key in /Users/kieferrm/.ssh/known_hosts to get rid of this message.
Offending RSA key in /Users/kieferrm/.ssh/known_hosts:1
Host key for github.com has changed and you have requested strict checking.
Host key verification failed.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

While I agree that this is not the best error message, the error was a result of an extraordinary event, that as far as I know has only happened once. For the time being I will go ahead and close the issue as "won't fix".


Payment via newSubscriptionViaMollieCheckout when registering through Fortify

In our application we want to redirect to Mollie to create an subscription after the user has redirected. When using the  Features::emailVerification() option from Fortify the request needs to implement the interface Authenticatable, but when redirecting to Mollie it creates a RedirectToCheckoutResponse resulting in the following error.
Argument 1 passed to Illuminate\Auth\SessionGuard::login() must implement interface Illuminate\Contracts\Auth\Authenticatable, instance of Laravel\Cashier\SubscriptionBuilder\RedirectToCheckoutResponse given, called in /var/www/html/vendor/laravel/fortify/src/Http/Controllers/RegisteredUserController.php on line 56
Is there any way around this except disabling the emailVerification?

Hi @Joezoefhalihoza ,
Can you share some example code to reproduce this issue?

@sandervanhooft
use App\Models\User;
use Illuminate\Support\Facades\Hash;
use Illuminate\Support\Facades\Validator;
use Laravel\Fortify\Contracts\CreatesNewUsers;
use Laravel\Jetstream\Jetstream;

class CreateNewUser implements CreatesNewUsers {
use PasswordValidationRules;

public function create(array $input)
{
    Validator::make($input, [
        'name' => ['required', 'string', 'max:255'],
        'email' => ['required', 'string', 'email', 'max:255', 'unique:users'],
        'password' => $this->passwordRules(),
        'subscription' => ['string', 'nullable', 'max:15'],
        'terms' => Jetstream::hasTermsAndPrivacyPolicyFeature() ? ['required', 'accepted'] : '',
    ])->validate();

    $user =  User::create([
        'name' => $input['name'],
        'email' => $input['email'],
        'password' => Hash::make($input['password']),
        'subscription' => $input['subscription'],
    ]);

    return $user->newSubscription('main', $input['subscription'])
        ->trialDays(30)
        ->create();

        //return $user; this is the standard action that contains the authenticable object.
}
}


Database design

The database needs to be designed to efficiently support queries such as:

Get all non-archived links sent to a user
Get all non-archived links sent to a user tagged with listen
Get information about the link, such as who shared it, it's title and URL
Get users by hashed phone number
Sharing the same link with multiple users
Store different kinds of meta-data for different links (e.g: artist for music links)

Less priority but still of value would be the possibility to:

Get all links a user shared with other users
Sort archived links after time of archival
Add and see comments/reactions for a link (in the scope of the users friends)

Relational database
Suggested tables for a relational database
User

id
phoneNumberHash

Share

id
timestamp
fromUser
linkId

User_Share

id
shareId
toUser
category

Archive

id
shareId
toUser
category
archivedTime

Link

id
url
title
suggestedCategory
hits

AudioLink

id
type
artist
duration
provider

VideoLink

id
type
channel
duration
provider

ArticleLink

id
type
author
readTime
provider

NoSQL
Suggested structure for JSON documents.
User
{ id: 1,
  phoneNumberHash: 'a7fc6d998e889acf779',
  links: [
     {linkId: '54', sharedByUser: 5, timestamp: 4657894},
     {linkId: '37', sharedByUser: 8, timestamp: 1325489}
   ],
   archivedLinks: [
     {linkId: '12', sharedByUser: 5, timestamp: 4657894, archivedTime: 7894563},
     {linkId: '78', sharedByUser: 9, timestamp: 8798453, archivedTime: 4654213}
   ],
   sharedLinks: {
      {linkId: '12', timestamp: 4657894, sentToUsers: [{userId: 7}, {userId: 9}, {userId=23}]}
   ]
}
...

Link
{ id: 12,
  title: "Epic fails of summer 2016",
  url; "https://youtube.com/watch?v=4a5a3c45d",
  videoDuration: 64,
  provider: 'YouTube',
  suggestedCategory: 'watch'
  hits: 455
}
...


Fix deprecation notice in package's entry point

This commit resolves a deprecation warning raised by including the TextEditor class, which is no longer a public interface.

Thank you!


feat(transformer_conformance): skip some tests that are known to fail

I don't think we need to be completely consistent with Babel's output. Because Babel's output doesn't always make sense.

#3293  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @Dunqing and the rest of your teammates on  Graphite

As usual, I miss the chance to comment before the PR is merged, but...
Rather than disabling these tests entirely, could we convert them into "exec" tests which test that behavior of the code after transform is what it should be with assertions? If we make sure those assertions pass on both Babel's output and Oxc's, then we know we're fulfilling the spirit of Babel's tests, even if we're not producing the exact same output.
It's great that we don't have to write tons of tests because we can just use Babel's, but when we can't do that, we do still want some tests, don't we?

As usual, I miss the chance to comment before the PR is merged, but...
Rather than disabling these tests entirely, could we convert them into "exec" tests which test that behavior of the code after transform is what it should be with assertions? If we make sure those assertions pass on both Babel's output and Oxc's, then we know we're fulfilling the spirit of Babel's tests, even if we're not producing the exact same output.
It's great that we don't have to write tons of tests because we can just use Babel's, but when we can't do that, we do still want some tests, don't we?

Yes, But to convert to exec tests we need to patch the code. It's going to be a lot of work. I think we can just make sure the results are as consistent as possible for now. Once the transformer is almost complete we can optimize it and write our own tests.


Add set/getAttribute

btw this overlaps w/ an existing npm module; we should probably pick a different name so we can publish it there

@justinfagnani fixed my own nits :smile:
Also made getAttributeIndex always return an int for consistency.


Lower Java version to 11 for OpenSearch 1.3.0.

Signed-off-by: dblock<EMAIL_ADDRESS>Description
We've lowered the JDK for OpenSearch to 11 in https://github.com/opensearch-project/OpenSearch/pull/940.
Issues Resolved
Part of https://github.com/opensearch-project/opensearch-build/issues/74.
Check List

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #981 (aaad4fb) into main (d095963) will increase coverage by 0.16%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main     #981      +/-   ##
==========================================
+ Coverage   92.78%   92.95%   +0.16%
==========================================
  Files          89       89
  Lines        2385     2385
==========================================
+ Hits         2213     2217       +4
+ Misses        172      168       -4

Impacted Files
Coverage Δ

src/system/process.py
100.00% <0.00%> (+7.27%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d095963...aaad4fb. Read the comment docs.

cc: @reta

This is awesome, thanks @dblock ! :+1: :+1: :+1:


Allow running files and adding a Zig example

Issue - Redirecting of error doesn't work with Zig, giving this error

@

@franko Any way to get the redirection working？

Users say it doesn't work in other languages as well

I will be glad to help with this but I am confused. This is a PR that does something meaningful and at the same time you are rising an issue with redirecting standard error.
I will begin by first fixing the issue if I can. Could you please precise a little bit better the problem ? It is to be sure I am going to work in the right direction.
Just to add a little bit to the context: I am not the author of the console plugin, rxi is the author and I just made some adaptations for lite-xl. I don't use myself the console plugin but I will be glad to work on some improvements, just give please more details about the problem you encountered and what you want to accomplish.

@franko Thanks for reaching out

As given here, I expect the the cursor to the line where the error occurs
Instead, I get the error from the image in the issue in my log


What's the version of the open-sora model reported in Report 1.2?

I would like to know the version of the open-sora's vae.
I guess it would be open-sora v1.2 since your report v1.2.0 is more resent than their's.
I will wait for your response.

See https://github.com/hpcaitech/Open-Sora/blob/main/docs/report_03.md#video-compression-network


New enableIterativeGsiUpdates feature fails with Rate exceeded Error

Describe the bug
A clear and concise description of what the bug is.
I have set enableIterativeGsiUpdates to true because I need to make about 300 GSI modifications spread across 48 models.
I do it in 2 parts - first step to compile and it does so successfully. Long story about why I have to do it in 2 parts.
$ amplify api gql-compile --minify
Next step is the push
$ amplify push --no-gql-override --yes
Unfortunately it fails with the following error
✖ An error occurred when pushing the resources to the cloud
Rate exceeded
An error occurred during the push operation: Rate exceeded
=======================================
Appreciate any suggestions on how I can do this throttling.
Amplify CLI Version
4.41.2
To Reproduce
Steps defined above
Expected behavior
Successful push

@sacrampton that is a lot of updates ;-) even with support for iterative updates I would suggest to reach out to customer support and try to increase the rate limits if possible. Throttling is not something we control as I assume - please confirm - that this Rate limitation comes from CF itself from the underlying DynamoDB update? or CF itself?
As a workaround you could phase your modifications of the schema to get a successful push.
If you can give some background on how you ended up with 300 updates that could help to understand more if we can do something around it. Is is happening during your regular dev flow of a project or a major refactoring or something else?

@sacrampton that is a lot of updates ;-) even with support for iterative updates I would suggest to reach out to customer support and try to increase the rate limits if possible. Throttling is not something we control as I assume - please confirm - that this Rate limitation comes from CF itself from the underlying DynamoDB update? or CF itself?
As a workaround you could phase your modifications of the schema to get a successful push.
If you can give some background on how you ended up with 300 updates that could help to understand more if we can do something around it. Is is happening during your regular dev flow of a project or a major refactoring or something else?

Hi @attilah - I'll reach out to support to see if CloudFormation can change throttling. I'll try a few work arounds.
The reason for this is that I'm trying to get DataStore to work with an existing application. This has been an off again / on again process for at least a year - every time we get into it we run into a major road block. We've been working through them and the latest one is how the GSI's are defined.
Our app has been developed over time and connections are declared like this.
assetVisit: [AssetVisit] @connection (name: "AssetAssetVisit", sortField: "lastPhotoTaken" )

However, if you do that then DataStore doesn't work. You have to use the @key directive to create all GSI's and then declare the connection like this.
assetVisit: [AssetVisit] @connection(keyName: "byAssetVisitAssetId", fields: ["id"])

Which on the back end means the original GSI has to be deleted and a new GSI created - so with about 150 connections then that means 300 updates.

Hi @attilah - I'll reach out to support to see if CloudFormation can change throttling. I'll try a few work arounds.
The reason for this is that I'm trying to get DataStore to work with an existing application. This has been an off again / on again process for at least a year - every time we get into it we run into a major road block. We've been working through them and the latest one is how the GSI's are defined.
Our app has been developed over time and connections are declared like this.
assetVisit: [AssetVisit] @connection (name: "AssetAssetVisit", sortField: "lastPhotoTaken" )

However, if you do that then DataStore doesn't work. You have to use the @key directive to create all GSI's and then declare the connection like this.
assetVisit: [AssetVisit] @connection(keyName: "byAssetVisitAssetId", fields: ["id"])

Which on the back end means the original GSI has to be deleted and a new GSI created - so with about 150 connections then that means 300 updates.

@sacrampton did you see any throttling value in CloudWatch or CloudTrail(?) which shows how long are you throttled? One uber-advanced thing came to my mind, that if we would maximize resources within update steps AND add a delay (how long?) between steps...perhaps that could help with throttling...but it depends on the length of the throttling...

@sacrampton did you see any throttling value in CloudWatch or CloudTrail(?) which shows how long are you throttled? One uber-advanced thing came to my mind, that if we would maximize resources within update steps AND add a delay (how long?) between steps...perhaps that could help with throttling...but it depends on the length of the throttling...

Hi @attilah - sadly the only options for CloudFormation service limit increase are:

Cloudformation Stack Limit
Module Limit per Account
Version Limit per Module

There is no option to request any service limit increase for CloudTrail
There is no mention of any throttling value on CloudFormation - I'm assuming that is what is happening given that the amplify push error says "An error occurred during the push operation: Rate exceeded" and various searches said this was a throttling issue.
Some online suggestions were to add DependsOn statements. So I also edited cloudformation-template.json to add DependsOn after every model such that it waited for the previous model. I've done that before when amplify push crashes when creating lots of DynamoDB resources simultaneously - but that made no difference this time.

Hi @attilah - sadly the only options for CloudFormation service limit increase are:

Cloudformation Stack Limit
Module Limit per Account
Version Limit per Module

There is no option to request any service limit increase for CloudTrail
There is no mention of any throttling value on CloudFormation - I'm assuming that is what is happening given that the amplify push error says "An error occurred during the push operation: Rate exceeded" and various searches said this was a throttling issue.
Some online suggestions were to add DependsOn statements. So I also edited cloudformation-template.json to add DependsOn after every model such that it waited for the previous model. I've done that before when amplify push crashes when creating lots of DynamoDB resources simultaneously - but that made no difference this time.

Hi @attilah  - I just halved the number of updates and got exactly the same error. Rather than continuous trial and error it would be good to understand the threshold that is causing this to choke. I can't see anything in CloudFormation that impacts this.
Obviously enableIterativeGsiUpdates is a long sought after addition, but if all that does is cause CloudFormation to choke then its not really helping. But I'm also now sure how to progress this, or precisely where the choke point it.
Appreciate anything you can do to progress this as it is a really important feature to move beyond experimental status.

Hi @attilah  - I just halved the number of updates and got exactly the same error. Rather than continuous trial and error it would be good to understand the threshold that is causing this to choke. I can't see anything in CloudFormation that impacts this.
Obviously enableIterativeGsiUpdates is a long sought after addition, but if all that does is cause CloudFormation to choke then its not really helping. But I'm also now sure how to progress this, or precisely where the choke point it.
Appreciate anything you can do to progress this as it is a really important feature to move beyond experimental status.

The non-ideal solution is to sleep(x) between iterations...

The non-ideal solution is to sleep(x) between iterations...

Hi @GeorgeBellTMH - just want to check with you - is this something that I can configure, or is it a modification that needs to happen within the amplify framework

Hi @GeorgeBellTMH - just want to check with you - is this something that I can configure, or is it a modification that needs to happen within the amplify framework

I'm marking it as an enhancement to make sure we are tracking it.

I'm marking it as an enhancement to make sure we are tracking it.

I've just gotten this with one index update, almost instantly as well.  It doesn't even start running the cloudformation.
Comment out the index and it works 🤷♂
Interestingly I could not get feature working on my demo branch at all (constantly complaining about multiple GSI updates) but when I switch to a fresh environment I get this just making one index change (new environment since adding the flag instead of the demo one which existed before enabling)

We've got a new logic to wait for the index to be created. Please create a new issue if this still happens.


Ticket validation error for embedded entity-id ending in double equal signs ("==")

We have our ShibCas configured to embed the entity-id in the URL sent to CAS.  We're having a problem with an entity-id that ends in == the full value is:: https://login.adaptiveinsights.com:443/samlsso/S0FOU0FTVU1DMg==.  Users are able to login with CAS, but when the ticket is validated, the service url is encoded and appears to have dropped the == at the end of the url.
ERROR [net.unicon.idp.externalauth.ShibcasAuthServlet:109] - Ticket validation failed, returning InvalidTicket
org.jasig.cas.client.validation.TicketValidationException: Ticket 'ST-60-PDiukKqi-CD-e-CcUT8902hF2V4cas2-t' does not match supplied service. The original service was 'https://castest.kumc.edu/idp/Authn/ExtCas?conversation=e1s1&entityId=https://login.adaptiveinsights.com:443/samlsso/S0FOU0FTVU1DMg==' and the supplied service was 'https://castest.kumc.edu/idp/Authn/ExtCas?conversation=e1s1&entityId=https%3A%2F%2Flogin.adaptiveinsights.com%3A443%2Fsamlsso%2FS0FOU0FTVU1DMg'.
I haven't located any configurations to change this.  Changing back to appending the entity-id works, but we need the entity-id embedded to apply rules in CAS.

FYI, that's not a valid entityID. EntityIDs are defined by the SAML standard as being a URI, either a URL or a URN. Equals sign is a reserved character in a URL, only for use in a query argument. So no entityID should include an equals sign in it. other than as an encoded character (which would be strange for an entityID.)
As to whether anything can be done to work around that, short of changing the entityID, I'll leave it to others to respond.

On Apr 24, 2020, at 1:40 PM, ghosler-kumc<EMAIL_ADDRESS>wrote:
We have our ShibCas configured to embed the entity-id in the URL sent to CAS. We're having a problem with an entity-id that ends in == the full value is:: https://login.adaptiveinsights.com:443/samlsso/S0FOU0FTVU1DMg== https://login.adaptiveinsights.com/samlsso/S0FOU0FTVU1DMg==. Users are able to login with CAS, but when the ticket is validated, the service url is encoded and appears to have dropped the == at the end of the url.
ERROR [net.unicon.idp.externalauth.ShibcasAuthServlet:109] - Ticket validation failed, returning InvalidTicket
org.jasig.cas.client.validation.TicketValidationException: Ticket 'ST-60-PDiukKqi-CD-e-CcUT8902hF2V4cas2-t' does not match supplied service. The original service was 'https://castest.kumc.edu/idp/Authn/ExtCas?conversation=e1s1&entityId=https://login.adaptiveinsights.com:443/samlsso/S0FOU0FTVU1DMg== https://castest.kumc.edu/idp/Authn/ExtCas?conversation=e1s1&entityId=https://login.adaptiveinsights.com:443/samlsso/S0FOU0FTVU1DMg==' and the supplied service was 'https://castest.kumc.edu/idp/Authn/ExtCas?conversation=e1s1&entityId=https%3A%2F%2Flogin.adaptiveinsights.com%3A443%2Fsamlsso%2FS0FOU0FTVU1DMg https://castest.kumc.edu/idp/Authn/ExtCas?conversation=e1s1&entityId=https%3A%2F%2Flogin.adaptiveinsights.com%3A443%2Fsamlsso%2FS0FOU0FTVU1DMg'.
I haven't located any configurations to change this. Changing back to appending the entity-id works, but we need the entity-id embedded to apply rules in CAS.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub https://github.com/Unicon/shib-cas-authn3/issues/44, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAG5BCDYF47SIHICLCAEO33ROHMJ3ANCNFSM4MQKK7MQ.
This email has been scanned for spam and viruses by Proofpoint Essentials. Click here https://us2.proofpointessentials.com/index01.php?mod_id=11&mod_option=logitem&mail_id=1587753630-GFBuvKO-pWee&r_address=mgrady%40unicon.net&report=1 to report this email as spam.

--
Michael A. Grady
IAM Architect, Unicon, Inc.


Improve support for writing custom assertions

Currently custom assertions can't delegate to existing assertions like equal() without messing up the stack trace we display for failed assertions. Always having to use QUnit.push isn't the best API for writing custom assertions.
Ideas so far:

at least expose the traversal of QUnit.equiv, related to #466
maybe expose an API for custom assertions that fixes stack traces when delegating to other assertions?
or try to have the assertion wrapper/constructor keep track, so that it doesn't matter on what level you call assertions, it'll unwrap correctly
or try and see what happens if we simplify the stack trace handling - a few more lines might be worth removing the complexity

Perhaps a separate QUnit.compare object that houses re-usable functions that return boolean.

Based on my own usage and from reported issues and implementations of custom assertions, I don't see a practical need for this.
Regarding stack traces, we're now exposing QUnit.stack(), which might be of use in custom assertions.
I don't see a need to keep this ticket open any longer.


Reconnect WebRTC-based RPC clients

If the server ends the connection (say through a restart), we want to be able to reconnect the client. This requires reconnecting via signaling again.

Easy way to test this is two viam managed RDKs where one is a main and one is a sub-part. connect both, bring down the sub-part and bring it back up after a minute. you will see the main get stuck thinking it's still connected.


Updating Known Issues, adding WPF known issue for gRPC

Added a known issue for WPF.

@vatsan-madhavan FYI, documented the gRPC issue as a starting point.


Removing an element’s PHP class breaks the CMS editor

TypeError: Cannot read property 'title' of undefined
    at t.value (bundle.js?m=1557373499:1)
    at Ln (vendor.js?m=1560140517:1)
    at Dn (vendor.js?m=1560140517:1)
    at Nn (vendor.js?m=1560140517:1)
    at Jn (vendor.js?m=1560140517:1)
    at Xn (vendor.js?m=1560140517:1)
    at _r (vendor.js?m=1560140517:1)
    at hr (vendor.js?m=1560140517:1)
    at dr (vendor.js?m=1560140517:1)
    at rr (vendor.js?m=1560140517:1)

graphql response looks like:

Manually deleting the records (e.g. id 117 from above) fixes it
Acceptance criteria

[x] The elemental area still renders
[x] Blocks of deleted classes are not rendered on the front end
[x] Blocks of deleted classes are displayed with a warning in the CMS. Users can not edit them, but can delete them.
[x] We seek feedback from designers and bespoke teams on the best way to communicate the situation to the end user.

PRs

https://github.com/silverstripe/silverstripe-elemental/pull/991

What should we do here? The CMS says "this page type is obsolete" and I guess changes its class to something more generic on save (whatever is populated first in the class type dropdown). Do we need to do something similar for elements?
We need to handle this error gracefully anyway, maybe a "couldn't render this element: [error message here]" or something, assuming there is an error message.
cc @silverstripeux re: "How do we display elements in the editor that have had their PHP classes removed in code"

I like the idea of the error message but being able to extract the raw data so it could be copypasted into another block, and for the frontend we don't show anything.
My thought would be to show a collapsed block with the error message, expand it to show the table data (doesn't need to be pretty at all) with any additional error info if available.

Looking at what the graphql endpoint returns, I wonder if the original intention was for the block editor to display a generic “Block” as the block type... so maybe this a regression?
If it helps, the error was coming from this line specifically: https://github.com/dnadesign/silverstripe-elemental/blob/8a2b9db84c7c4b0ef51c216514613476c223bc10/client/src/components/ElementEditor/Element.js#L192
Thankfully this doesn’t cause any errors on the frontend, the affected blocks are just missing.

Hi @kinglozzer @robbieaverill ! I was wondering if you guys have any updates on this bug? It's still happening in v4.3.1, and it's making the deprecation of a component a bit difficult.

I think we need some sort of task that runs on flush which can re-map obsolete elemental classes in the DB to something like ElementContent that always exists. Then we could show in the UI that the old element type is gone, like we do for pages. This would allow us to implement the UI in the way @clarkepaul described above.
In terms of updates, there's been no progress made on this issue. Pull requests would be welcomed if you wanted to give it a crack in the meantime!

Had this issue today as well
Aim: Rename a component i.e from SomethingAbigious -> SlideShow
Repercussions: Whole page doesn't display elemental Block section (Where you can add blocks and click into them) - No errors
Tried:
Dev/build
Solution:
Renaming component back to SomethingAbigious removing it and renaming back to SlideShow
I feel as though it needs to check if that element exists any more and not try to show it?

I run into it several times. Usually if the whole ElementalArea breaks this or wrong provideBlockSchema()in one of the used Elements is the cause. The "no error" part makes it a bummer :(

Solution was manually tested in local environment. Everything works perfect.
Close issue.

Hi @sabina-talipova . Do you know in which version of Silverstripe this fix would be released?

@gdelavil Released as 4.9.2


Create a Workflow Project from a Template

Tutorial URL: https://developers.sap.com/tutorials/cp-workflow-create-project.html
Step 2.4: there is no Category "Business Process Management"

Hi Robert,
Have you enabled the services and extensions as described in the first tutorial "Prepare to Create Workflows in SAP Cloud Platform"?
Best regards,
Astrid

Hi Astrid,
yes i did.
For me also the Workflow Service comes within Orchestration, not Integration

Best regards
Robert

Hi Robert,
I see that the Workflow service itself is enabled. There is an option to enable the workflow extension inside the Web IDE. Have you set that one, too?

Regards,
Astrid

Hi Astrid,
thanks, that's it - complicated but works....

Thank you for your feedback. The issue seems to be resolved, so I am closing the issue.
If you still have questions, feel free to reopen the issue.


add validation for instance create/delete req

What this PR does / why we need it:
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #448
Special notes for your reviewer:
Documentation:

Does this PR introduce a user-facing change?:

NONE

/retest
/lgtm

/retest

/retest

/test all


chore: remove upper bound from Python version specifier

I've removed the upper bound of the Python version specifier in pyproject.toml, i.e. I've replaced the version specifier ^ by >=.
Capping the Python version (e.g. <4) is a bad practice.

https://github.com/pypa/packaging.python.org/pull/850
https://iscinumpy.dev/post/bound-version-constraints/#pinning-the-python-version-is-special
https://discuss.python.org/t/requires-python-upper-limits/12663

This seems sensible. Thanks @sisp!
PRs also welcome in other snok libraries if you've got extra time on your hands 👍


Feature/game mode common component

Pull Request Template
Description
Add a summary that address the following:
motivation/purpose for the feature
To build a button bar for the your mission click. When they want to write or draw they click the button and it activates those props as true or false. Then it will style the opacity to make it look like they are enabled or disabled. Each props function will be activated with the each button's onClick event but **note** the buttons do not yet work because the dispatcher has not been added yet.
What were those changes
Changes were made from the pages/gamemode/ThrashBar.jsx to common/ButtonBar.js. Also new testing was done on h1 text within that component it first worked when testing with the gamemode folder but when I deleted it and only kept common buttonbar in there the test did not work so they are now not added to the pr.
What issues were fixed
The button bar was changed into an industry standard design as a common component for the child to choose draw or write.
Provide the link to the user story it is referencing in Trello board or other supporting document
https://trello.com/c/72I6DGBE/51-as-a-child-player-i-want-to-be-able-to-chose-whether-to-draw-or-and-write-for-single-player-mode

Commits Checklist

[ ] commits have clear title names
[ ] commit has descriptive message of what has changed
[ ] more small commits than less big commits

Code Checklist

[ ] code is formatted appropriately
[ ] code meets DRY standards
[ ] no dead code (check for logs and print statements)
[ ] no TODOS instead necessary comments
[ ] code follows Story Squad standards for:

Language
Framework
Libraries

PRs needs to be reviewed by at least 2 team members

TPL is the 3rd reviewer
Lots of comments and suggestions, please!

Pull Request Rubric

The buttons look good, but it looks like github is annoying us with checks.

So like as I stated in the slack chat I'm pretty sure the import { useOktaAuth } from '@okta/okta-react/dist/OktaContext'; is what is breaking the build but if we short it down to import { useOktaAuth } from '@okta/okta-react/'; it seems to build or at least locally.

I wonder if there are merge conflict that got messed up and are breaking the build? It might be worth starting a new branch from main and copy your changes over to see if the tests pass. If so, kill this PR/branch and move on with the new branch/PR.

Thank you so much, Jake, for going through this PR. It was extremely helpful to see the comments you made where you pointed out certain code that wasn't necessary or needed more explanation on.  Going through the gitignore file and addressing the major version changes was also extremely helpful especially when working on projects involving other team members.

Because this PR is over a week old there have been recent changes made since then that may have possibly created merge conflicts and failing tests. There has been another PR created that has the changes from this PR so we will no longer use this one we will go with the most recent PR.


V1.0.6

Coverage remained the same at 91.477% when pulling 853ea22695b416d938de3ee4cc72a032cd7f95d0 on v1.0.5fix into 8b15d9c18861804273add1901c0a61e54210f2e0 on master.

Coverage remained the same at 91.477% when pulling 853ea22695b416d938de3ee4cc72a032cd7f95d0 on v1.0.5fix into 8b15d9c18861804273add1901c0a61e54210f2e0 on master.


Github.com/master -abebetekil-explore/Data automation-index

RUpdate index of data

Please replace this line with an explanation of why you think these changes

[ ] I've formatted my changes as https://github.com/collections/[worldbank/health/api])
My collectin contain All required fields in my index.md conform to the Style Guide and API docs: https://github.com/github/explore/tree/master/docs

Please replace this line with an explanation of why you think this topic or collection should be curated.
i don't know the reason

Please note: we will close your PR without comment if you do not check the boxes above and provide ALL requested information.

HI @abebetekil 👋
I've closed this PR since it doesn't seem to propose a realistic change. If you need help, please see our contribution guidelines for how to contribute. Thank you!


Add default matcher to pluginLess

This PR fixes this issue: https://github.com/fuse-box/angular2-example/pull/36
Problem was that LESS plugin failed if no matcher was defined (like in docs), so I guess we would create a default matcher like in SASS plugin for this to work:
pluginLess({ asText: true, useDefault: false })

Altho the workaround is to use matcher
pluginLess('*.less', { asText: true, useDefault: false })

I think this should be fixed as it was possible to define plugin without a matcher and then it's just a potato that does nothing.

👍 thanks a million!


E5E2: As a user in a deal I can chat, so I can communicate about the deal.

Abstract:
Once users agree to engage in a deal, they can chat with each other. That way they can plan in detail how the deal will be executed.
How it could work:
In a deal's detail-view, a chat button appears.
By tapping the chat button, the view changes to a chat window where the Seeker and Provider can communicate about the deal.
What it looks like:
As seen by Seeker and Provider

When an item is a deal, the detail-view shows a chat-button directly above the green area (both seen as Provider and Seeker).
When the chat button is tapped, the view changes to chat-view.
The chat-view contains:

(left-top) avatars of the involved Seeker and Provider in that order
a blue X-mark (close-button - right top)
scrollable chat area containing the chat-items. The chat-items of the current user are aligned left, the chat-items of the other party are aligned right. Each chat-item contains:

Datetime of chat-item
chat-msg
avatar of msg-sender

a horizontal divider line, under which is a chat-msg input area. It includes:

avatar of current user
input field for the chat-msg
a white button with black V-mark (submit-button)

When the user inputs a chat-msg and submits, it appears (left-aligned) in the chat area, where all chat-msgs are visible. The input area is sticky, meaning it will always be visible.
When the blue X-mark (close-button) is tapped by the user, the chat-view changes back to the detail-view of the item.

Desktop view:

Done!