How does kubeless achieve parallelism for function execution?

Kubeless provides the following cmd to deploy a function
$ kubeless function deploy test --runtime python2.7 \
                                --handler test.foobar \
                                --from-file test.py \
                                --trigger-topic test-topic

This cmd tells k8s to create a corresponding deployment for the function. However, this cmd doesn't specify the replica, so only one consumer will be created for the function handler. kafka cluster uses multi-consumer to achieve parallelism, does kubeless support multi-consumer?
thanks!

We just merged an auto-scaling feature, the runtimes are instrumented via prometheus so the number of replicas will scale depending on the number of function calls.
So yes "parallelism" will be achieved. We are not testing it currently, so there might be some kafka tuning to do

I notice that when typing kubectl get hpa, the output is
No resources found.
Should I have to setup auto-scaling manually by using kubectl autoscale deployment test --min=2 --max=10?
or kubeless can setup auto-scale by itself? and which code does this job?
thanks!

Hi @FrankWalter, so the auto-scaling feature in kubeless now requires few manual steps to setup:

First, you must be on 1.7 and follow this guideline to setup the "infrastructure" https://github.com/kubeless/kubeless/tree/master/manifests/autoscaling. Stop at the last step (or you might want to try the sample app)

As it's not included in the lastest release, at this moment you must build the binary from master to have autoscale command. Then follow kubeless autoscale --help. There are two metrics supported: qps and cpu. You might want to try qps - number of function calls per second.

I am happy to see you're going to try the feature. Feedbacks are very welcome :-)

@ngtuna , thanks for your reply, I will try it!

I followed the instruction in https://github.com/kubeless/kubeless/tree/master/manifests/autoscaling
Everything went fine, but when I tried to deploy a test function, I found there was no pod created for this function.
root@instance-2:~# vim test.py
root@instance-2:~# kubeless function deploy get-python --runtime python2.7 \
>                                 --from-file test.py \
>                                 --handler test.foobar \
>                                 --trigger-http
root@instance-2:~# kubeless function ls
NAME            NAMESPACE       HANDLER         RUNTIME         TYPE    TOPIC           DEPENDENCIES
get-python      default         test.foobar     python2.7       HTTP
test            default         test.foobar     python2.7       PubSub  test-topic
root@instance-2:~# kubectl get pods
NAME                                  READY     STATUS    RESTARTS   AGE
prometheus-operator-815607840-z3x71   1/1       Running   0          52m
prometheus-sample-metrics-prom-0      2/2       Running   0          51m
sample-metrics-app-1614459654-22dfh   1/1       Running   0          49m
sample-metrics-app-1614459654-hffjk   1/1       Running   0          49m
root@instance-2:~# kubectl get pods -n kubeless
NAME                                  READY     STATUS    RESTARTS   AGE
kafka-0                               1/1       Running   0          18m
kubeless-controller-915192800-m7zbm   1/1       Running   0          18m
zoo-0                                 1/1       Running   0          18m

Is this caused by dind? I didn't see this problem when deploying a function in google cloud.

I don't think it's caused by dind. It worked fine with dind in my setup. Can you paste the kubeless-controller log ?

Sorry, I made a mistake, I should use the RBAC-enabled version.
I successfully create a autoscale for test using the following cmd
kubeless autoscale create test --metric qps --value 1000

And I called this function by using
for((i=0; i < 100; i++)); do kubeless topic publish --topic test-topic --data "Hello World!" ; done

But I still didn't see a replica for pod test-xxxxxx
root@instance-2:~# kubectl get pods
NAME                                  READY     STATUS    RESTARTS   AGE
prometheus-operator-815607840-z3x71   1/1       Running   0          4h
prometheus-sample-metrics-prom-0      2/2       Running   0          4h
sample-metrics-app-1614459654-22dfh   1/1       Running   0          4h
sample-metrics-app-1614459654-hffjk   1/1       Running   0          4h
test-3023077429-2pj5k                 1/1       Running   0          54m

What qps value should I assign?

Cool!!! The qps value is number of incoming requests per second. So you set 1000 requests per second. Can you just check kubectl get hpa and paste the output here in order to see it's working properly ?

root@instance-2:~# kubeless autoscale create test --min 1 --max 4  --metric qps --value 1000
root@instance-2:~# kubectl get hpa
NAME                     REFERENCE                       TARGETS           MINPODS   MAXPODS   REPLICAS   AGE
sample-metrics-app-hpa   Deployment/sample-metrics-app   <unknown> / 100   2         10        2          4h
test                     Deployment/test                 <unknown> / 1k    1         4         1          10m

There are other things I care about.
I read the developer guild of AWS lambda, and it seems lamdba support event based async invokes.

Does kubeless support this?
Is there a mechanism to coordinate different functions with dependence?

@FrankWalter Basically the HPA controller takes a bit time to catch up the metrics from the custom api server, but it should work. I hope you can see it after few minutes. I want to do the optimization at that point (PR is welcome 😃 ) I post here the recorded video I made just in case you need a worked example: https://www.youtube.com/watch?v=Vywu9dBRh0U
Regarding the event based async invoke, we are having a plan to support it. Please stay tuned it will be there soon.

@ngtuna thanks for your help! I will follow the kubeless community, and I'm glad to see kubeless becomes better and better.
Great work by the way:smiley:

One more question....
If I want to achieve serial execution for a set of functions with dependence, should I do the synchronization by myself?

@FrankWalter it would be terrific if you could confirm that the HPA integration works, a blog would even been super helpful to get the word out. There is also no documentation yet.
As for the function dependency there is an issue filed:
https://github.com/kubeless/kubeless/issues/342
Which we all need to discuss.

ok, I'll try. thanks @sebgoa

I modified the test.py file into this to block the function:
import time
def foobar(context):
    print "start"
    time.sleep(1000)
    print "end"
    return context

I expected this could cause low qps which would lead to pod autoscale. However, it didn't work, and the number of test pod remained to be 1.
I notice that there was something abnormal in the output of kubectl get hpa
root@instance-2:/home/liuzhe452769109# kubectl get hpa
NAME      REFERENCE         TARGETS          MINPODS   MAXPODS   REPLICAS   AGE
test      Deployment/test   <unknown> / 1k   1         4         1          42m

The TARGETS is <unknown>, does it matter?

You need to call the function in a loop to increase the "query per second" rate. Putting a sleep in there without putting some load on the function will not do anything.

I used
for((i=0; i < 1020; i++)); do kubeless topic publish --topic test-topic --data "Hello World!" ; done

to increase qps previously. But it didn't work either. :disappointed:

@ngtuna does the autoscaling work for pubsub functions ?

If the qps metric is used, autoscale will monitor number of "hit" to the function service. Thus, it currently doesn't work for pubsub unfortunately.
So in order to support pubsub function, we need to count the number of messages that the runtime consumes from kafka. I assume we have to define a new prometheus metric accordingly.

Ok so the correct answer is yes! Two of my team mates confirmed that the metric function_call_total can also count the number of messages consumed from kafka, then the autoscaling for pubsub function should work.
@FrankWalter Let me try to represent the test environment and do a test quickly but in the meantime may I ask you to represent the test on your setup and give the clue about kubectl get hpa ? Checking the prometheus target also gives some useful info

thanks! I will redo a quick test now, and please contact me if you need any information.
I will do my test on google cloud compute engine. @ngtuna

I just redo the test, and the logs are shown below. (google cloud compute instance with 2 vcpus and 4G mem)
What I'm really concern is that the result of  kubectl get hpa shows the TARGETS is unknown
root@instance-2:/home/liuzhe452769109# kubectl api-versions
admissionregistration.k8s.io/v1alpha1
apiextensions.k8s.io/v1beta1
apiregistration.k8s.io/v1beta1
apps/v1beta1
authentication.k8s.io/v1
authentication.k8s.io/v1beta1
authorization.k8s.io/v1
authorization.k8s.io/v1beta1
autoscaling/v1
autoscaling/v2alpha1
batch/v1
batch/v2alpha1
certificates.k8s.io/v1beta1
custom-metrics.metrics.k8s.io/v1alpha1
extensions/v1beta1
k8s.io/v1
monitoring.coreos.com/v1alpha1
networking.k8s.io/v1
policy/v1beta1
rbac.authorization.k8s.io/v1alpha1
rbac.authorization.k8s.io/v1beta1
settings.k8s.io/v1alpha1
storage.k8s.io/v1
storage.k8s.io/v1beta1
v1

kubeless autoscale create test --metric qps --value 1000

root@instance-2:/home/liuzhe452769109# kubeless autoscale list
NAME    NAMESPACE       TARGET  MIN     MAX     METRIC          VALUE
test    default         test    1       10      function_calls  1k

root@instance-2:/home/liuzhe452769109# kubectl get hpa
NAME      REFERENCE         TARGETS          MINPODS   MAXPODS   REPLICAS   AGE
test      Deployment/test   <unknown> / 1k   1         10        1          12m

for((i=0; i < 10; i++)); do kubeless topic publish --topic test-topic --data "Hello World!" ; done &

root@instance-2:/home/liuzhe452769109# kubectl get pods
NAME                                  READY     STATUS    RESTARTS   AGE
prometheus-operator-815607840-ph8fd   1/1       Running   0          25m
prometheus-sample-metrics-prom-0      2/2       Running   0          25m
test-3023077429-d7c6f                 1/1       Running   0          13m

I am a little late to this party, but this is of a lot of interest to us as well. Just so I get this right

Autoscaling is not the default configuration for Kubeless but can be used if the cluster is set up for HPA
Autoscaling is supported for HTTP via the "qps" metric and for Pub-sub via the  "function_call_total" metric and both of these are implemented and ready to be used
If Autoscaling is not enabled, and lets say I am running an HTTP method, if the method is called 100 times in 1 second, but each function execution takes say 10 seconds, then the requests start getting backed up instead of being all served at the same time because there is only one pod deployed by default
If Autoscaling is not enabled and let's say I am running a PubSub method, then I am not leveraging the multiple replicas that the Kafka consumer would be set up with but would instead work off just 1 replica that corresponds to the 1 function pod that got deployed. Is that correct?

Just pointing out that calling a function 100 times/s that returns in 0.01s is the same "qps" value as calling a function 100 times/s that takes 10s to complete - even though the number of simultaneous requests is now 1000x greater.  I would suggest that qps is not a universally applicable proxy for "load"...

Hi @arjunrao87
in the kube-controller-manager, how do you config --master=apiserver-address:port?
does <IP_ADDRESS>:8080 right?

The autoscaling  worked for http functions, but as for pubsub functions, when I tried to create a autoscale by using:
kubeless autoscale create pubsub-python --max 10 --metric function_call_total --min 1 --value 5
FATA[0000] only supported metrics: cpu, qps.

Things went wrong.
It seems that the function_call_total metric doesn't exist.
https://github.com/kubeless/kubeless/blob/8187b22f42c9f3c20c1b7d031dbba8065664be6a/cmd/kubeless/autoscaleCreate.go#L39
So I just use the qps metrics
kubeless autoscale create pubsub-python --max 10 --metric qps--min 1 --value 10000

and use
for i in {1..10000}; do kubeless topic publish --topic s3-python --data "hello"; done

to trigger autoscaling.
The hpa log showed that:
NAME                 REFERENCE               TARGETS             MINPODS           MAXPODS         REPLICAS       AGE
pubsub-python  Deployment/pubsub-python       500m / 10000          1                10                1            4m

But the autoscaling didn't happen. Apparently the hpa worked, did the pressure not reach to the target?

@FrankWalter Sorry for the delay response. TBH I have missed this thread for couple days :-( And the usage should be documented. I will spend time to write it today.
So currently there are two supported metrics: qps and cpu. They are listed in the help: kubeless autoscale create --help. If you use qps, the rate of incoming requests to function per second will be collected and monitored for autoscaling and the inline function_calls metric will be added to the corresponding HPA object. (the function_calls metric is consumed from HPA perspective and it means qps from UX design). There is no different, both http and pubsub functions use the same metric.
The cpu one is supported by default so I won't mention it here.
Thanks to your log, I can see the auto-scaling works with pubsub function.
root@instance-1:~# kubectl describe hpa
Name:                                           pubsub-python
Namespace:                                      default
Labels:                                         <none>
Annotations:                                    <none>
CreationTimestamp:                              Mon, 23 Oct 2017 01:38:35 +0000
Reference:                                      Deployment/pubsub-python
Metrics:                                        ( current / target )
  "function_calls" on Service/pubsub-python:    566m / 5
Min replicas:                                   1
Max replicas:                                   10
Conditions:
  Type                  Status  Reason                  Message
  ----                  ------  ------                  -------
  AbleToScale           True    ReadyForNewScale        the last scale time was sufficiently old as to warrant a new scale
  ScalingActive         True    ValidMetricFound        the HPA was able to succesfully calculate a replica count from Service metric function_calls
  ScalingLimited        False   DesiredWithinRange      the desired replica count is within the acceptible range
Events:
  FirstSeen     LastSeen        Count   From                            SubObjectPath   Type            Reason                          Message
  ---------     --------        -----   ----                            -------------   --------        ------                          -------
  16m           15m             6       horizontal-pod-autoscaler                       Warning         FailedGetObjectMetric           unable to get metric function_calls: Service on default pubsub-python/unable to fetch metrics from API: the server could not find the metric function_calls for services
  16m           15m             6       horizontal-pod-autoscaler                       Warning         FailedComputeMetricsReplicas    failed to get object metric value: unable to get metric function_calls: Service on default pubsub-python/unable to fetch metrics from API: the server could not find the metric function_calls for services
  15m           15m             2       horizontal-pod-autoscaler                       Warning         FailedGetObjectMetric           unable to get metric function_calls: Service on default pubsub-python/unable to fetch metrics from API: the server could not find the metric function_calls for services pubsub-python
  15m           15m             2       horizontal-pod-autoscaler                       Warning         FailedComputeMetricsReplicas    failed to get object metric value: unable to get metric function_calls: Service on default pubsub-python/unable to fetch metrics from API: the server could not find the metric function_calls for services pubsub-python

The problem here is this line:
  ScalingLimited        False   DesiredWithinRange      the desired replica count is within the acceptible range

I guess the calculated replicas from autoscaling is over the range. Could you just adjust the max number to bigger (let's just try 1000) ?

The warning message happened at the initial time when hpa controller was trying to connect prometheus system and retrieved the metric. You might want to optimize the hpa controller a bit by adjusting the sync period described here: https://github.com/kubeless/kubeless/pull/374

@ngtuna thanks for your detailed reply, I will try to adjust the max number. :smile:

adjusting the max number to bigger didn't seem to work either. :disappointed:
root@instance-1:/# kubeless autoscale create pubsub-python --max 100000 --metric qps --min 1 --
value 100

root@instance-1:/# for i in {1..10000}; do kubeless topic publish --topic s3-python --data "hello"; done

root@instance-1:/home/liuzhe452769109# kubectl get hpa -w
NAME            REFERENCE                  TARGETS           MINPODS   MAXPODS   REPLICAS   AGE
pubsub-python   Deployment/pubsub-python   <unknown> / 100   1         100000    1          32s
pubsub-python   Deployment/pubsub-python   <unknown> / 100   1         100000    1         1m
pubsub-python   Deployment/pubsub-python   566m / 100   1         100000    1         1m
pubsub-python   Deployment/pubsub-python   533m / 100   1         100000    1         1m
pubsub-python   Deployment/pubsub-python   566m / 100   1         100000    1         2m
pubsub-python   Deployment/pubsub-python   533m / 100   1         100000    1         3m
pubsub-python   Deployment/pubsub-python   566m / 100   1         100000    1         3m
pubsub-python   Deployment/pubsub-python   533m / 100   1         100000    1         4m
pubsub-python   Deployment/pubsub-python   566m / 100   1         100000    1         5m
pubsub-python   Deployment/pubsub-python   533m / 100   1         100000    1         6m
pubsub-python   Deployment/pubsub-python   566m / 100   1         100000    1         6m
pubsub-python   Deployment/pubsub-python   533m / 100   1         100000    1         7m
pubsub-python   Deployment/pubsub-python   566m / 100   1         100000    1         8m
pubsub-python   Deployment/pubsub-python   533m / 100   1         100000    1         9m
pubsub-python   Deployment/pubsub-python   566m / 100   1         100000    1         10m
pubsub-python   Deployment/pubsub-python   533m / 100   1         100000    1         11m
pubsub-python   Deployment/pubsub-python   566m / 100   1         100000    1         11m

root@instance-1:/home/liuzhe452769109# kubectl describe hpa
Name:                                           pubsub-python
Namespace:                                      default
Labels:                                         <none>
Annotations:                                    <none>
CreationTimestamp:                              Tue, 24 Oct 2017 06:41:02 +0000
Reference:                                      Deployment/pubsub-python
Metrics:                                        ( current / target )
  "function_calls" on Service/pubsub-python:    566m / 100
Min replicas:                                   1
Max replicas:                                   100000
Conditions:
  Type                  Status  Reason                  Message
  ----                  ------  ------                  -------
  AbleToScale           True    ReadyForNewScale        the last scale time was sufficiently old as to warrant a new scale
  ScalingActive         True    ValidMetricFound        the HPA was able to succesfully calculate a replica count from Service metric function_calls
  ScalingLimited        False   DesiredWithinRange      the desired replica count is within the acceptible range
Events:
  FirstSeen     LastSeen        Count   From                            SubObjectPath   Type            Reason                          Message
  ---------     --------        -----   ----                            -------------   --------        ------                          -------
  12m           12m             6       horizontal-pod-autoscaler                       Warning         FailedGetObjectMetric           unable to get metric function_calls: Service on default pubsub-python/unable to fetch metrics from API: the server could not find the metric function_calls for services
  12m           12m             6       horizontal-pod-autoscaler                       Warning         FailedComputeMetricsReplicas    failed to get object metric value: unable to get metric function_calls: Service on default pubsub-python/unable to fetch metrics from API: the server could not find the metric function_calls for services
  11m           11m             1       horizontal-pod-autoscaler                       Warning         FailedGetObjectMetric           unable to get metric function_calls: Service on default pubsub-python/unable to fetch metrics from API: the server could not find the metric function_calls for services pubsub-python
  11m           11m             1       horizontal-pod-autoscaler                       Warning         FailedComputeMetricsReplicas    failed to get object metric value: unable to get metric function_calls: Service on default pubsub-python/unable to fetch metrics from API: the server could not find the metric function_calls for services pubsub-python

Hi @arjunrao87 ,

Autoscaling is not the default configuration for Kubeless but can be used if the cluster is set up for HPA

Yes and no. Actually it's default configuration for cpu metric. If you want to use custom metrics (qps) then the cluster is set up for HPA controller can retrieve that metric.

Autoscaling is supported for HTTP via the "qps" metric and for Pub-sub via the "function_call_total" metric and both of these are implemented and ready to be used

No. Autoscaling is supported for both HTTP and Pubsub via the qps metric and ready to be used. The function_call_total metric is added into HPA object and it represents qps.

If Autoscaling is not enabled, and lets say I am running an HTTP method, if the method is called 100 times in 1 second, but each function execution takes say 10 seconds, then the requests start getting backed up instead of being all served at the same time because there is only one pod deployed by default

Yes.

If Autoscaling is not enabled and let's say I am running a PubSub method, then I am not leveraging the multiple replicas that the Kafka consumer would be set up with but would instead work off just 1 replica that corresponds to the 1 function pod that got deployed. Is that correct?

I am not pretty sure. WDYT @anguslees @sebgoa ?

to the latest. Correct, there would be only one Kafka consumer.

ping @ahriy if increasing the max number doesn't work out, can you just add more requests to the function and adjust the hpa sync period? It should work. If you need a screen capture from my setup I will do

Look at the targets above:

"function_calls" on Service/pubsub-python:    566m / 100

Your current "load" is 0.566/100 - there's no way it should scale to more than one replica with that configuration!
I suggest you try an autoscale target that is several orders of magnitude smaller, and less than the load you are currently generating (just to watch it work), eg:
kubeless autoscale create pubsub-python --max 10 --metric qps --min 1 --value 100m

(ie: scale up if the per-pod load exceeds "100m" qps == 0.1 qps == 1 query every 10 seconds)

I followed @anguslees 's instruction, and it works!
Thanks for your kindness help! @absoludity @ngtuna @sebgoa
root@instance-1:/home/liuzhe452769109# kubectl get hpa -w
NAME            REFERENCE                  TARGETS       MINPODS   MAXPODS   REPLICAS   AGE
pubsub-python   Deployment/pubsub-python   533m / 500m   1         10        1          6m

pubsub-python   Deployment/pubsub-python   466m / 500m   1         10        1         7m
pubsub-python   Deployment/pubsub-python   566m / 500m   1         10        1         7m
pubsub-python   Deployment/pubsub-python   566m / 500m   1         10        2         7m
pubsub-python   Deployment/pubsub-python   566m / 500m   1         10        2         7m
pubsub-python   Deployment/pubsub-python   433m / 500m   1         10        2         8m
pubsub-python   Deployment/pubsub-python   433m / 500m   1         10        2         8m

terrific, it would be great to read a blog about it , that's really cool

According to kafka docs, kafka distributes messages across partition, and partitions are assigned to consumers.
Some related discussion here https://github.com/kubeless/kubeless/issues/418.
So I'd really like to know what is the story of partition behind this auto scaling feature.

@ccll we configure the basic kafka setup before we had HPA working, so it is time to review our we use partitions and consumer groups...
short answer, let's review that setup.

Hello
Please i'm trying to configure my kubernetes cluster to have the autoscaling to use kubeless , i followed the tutorials but it doens't work for me , i can't see the autoscaling/v2alpha1when executing the command kubectl api-versions any suggestions please??

What version of kubernetes are you using and how did you deploy it ?

I'm using kubernetes 1.9 i deploy it with kubeadm and each time i do the steps to have the autoscaling when i list down the api version i can see only autoscaling/v1 and autoscaling/v2beta1.

@white-cloud we still don't provide support for Kubernetes 1.9, the current supported versions are 1.7 and 1.8. Since now the default GKE is 1.8 we will probably move soon to support 1.8 and 1.9.

@andresmgot   so please in this case what do you suggest for me because i really need it ..??

@white-cloud you can still create autoscale rules directly using kubectl. For example kubectl autoscale deployment get-python --min=2 --max=5 --cpu-percent=80 would create an autoscale rule for the function get-python.

You think that would work ? so i wont use the step mentioned in the kubeless tuto i can only use this command directly  ? @andresmgot

Yes, the output should be the same. AFAIK the only difference will be that for cleaning up you need to manually delete the autoscale rule. In any case note that the guide was written for Kubernetes 1.7 so it may require to be updated.

i will try it and let you know ; thank you for your answer .


Adding selection field for keypair type

Description
Adding selection for key pair type in profile manager

Changes

Added selection for  key pair type
Added a warning for the user
Added key pair type in profile interface
Updated get grid method

Related Issues
#1113
Checklist

[ ] Tests included
[x] Build pass
[ ] Documentation
[x] Code format and docstrings
[x] Screenshots/Video attached (needed for UI changes)

Validation should be reloaded.
In that case, I've added mnemonic of type ed25519 and then chose it.

Screencast from 12-19-2023 03:00:31 PM.webm

not sure if we should move the new warning message to the top or keep it. But if we keep it, I think it should have some spacing after @ehab-hassan

Validation should be reloaded.
In that case, I've added mnemonic of type ed25519 and then chose it.

Screencast.from.12-19-2<IP_ADDRESS>.PM.webm

Please consider the same for Activate Account and Create Account buttons.

Also as long as the user can't create account with ed25519 (just import pre existing account), I think w should say that somewhere.

not sure if we should move the new warning message to the top or keep it. But if we keep it, I think it should have some spacing after @ehab-hassan

yes please let's keep it down and make sure to have the same margin's first warning message had.


CSS

CSS
User story
As a boot camp student
I want the prework notes to be structured on a webpage
So that I can easily find and read the information
A.C.
GIVEN a Prework Study Guide website
WHEN I view the study guide
THEN I see a dark blue header and footer, and four boxes with a shadow

CSS added with .card element


Update labels as required for RHTAP

Why do we need this change? :thought_balloon:
If we don't have LABELS defined then by default image creation inherits it from parent image which is UBI. This is not acceptable by RHTAP. So defining the LABELS for ROS image.
Documentation update? :memo:

[ ] Yes
[ ] No

Security Checklist :lock:
Upon raising this PR please go through RedHatInsights/secure-coding-checklist
:guardsman: Checklist :dart:

[ ] Bugfix
[ ] New Feature
[ ] Refactor
[ ] Unittests Added
[ ] DRY code
[ ] Dependency Added
[ ] DB Migration Added

Additional :mega:
Feel free to add any other relevant details such as links, notes, screenshots, here.

/ok-to-test

/retest

/retest