Host Site From Home

TODO:

[x] Host a simple site from home.
[ ] Setup https server from home.
[ ] Setup up analytics for the system.
[x] Containerize the setup.

RESOURCES:
https://www.digitalocean.com/community/questions/can-i-run-two-domain-on-single-ip-address
https://support.comodo.com/index.php?/Knowledgebase/Article/View/304/17/if-i-provide-two-domain-names-for-the-same-server-ip-address-can-i-get-two-different-ssl-certificates-one-for-each-domain-name

[ ] https://github.com/thadd/nuc-cluster Building a Light-weight OpenShift Cluster
[ ] https://blog.container-solutions.com/bootstrap-as-code-part-1-hardware-is-here-to-stay Setup home system
[ ] https://www.digitalocean.com/community/tutorials/how-to-configure-bind-as-a-caching-or-forwarding-dns-server-on-ubuntu-14-04

Basic implementation is done.
Steps:

At home router, port forwarded all the requests to from 80 to one of the computer.
At the domain register, mapped the domain to the public IP.

TODO:

Setup https to serve the content from Docker

Came to know about Traefik which automates the HTTPS and other processes to run website from locally. Created repo for practice.

Tried to implement from another source

https://medium.com/@containeroo/traefik-2-0-docker-a-simple-step-by-step-guide-e0be0c17cfa5
https://medium.com/@containeroo/traefik-2-0-docker-an-advanced-guide-d098b9e9be96

Could not make it workable and the last error is
too many failed authorizations recently
I think I need to look into Traefik documents a bit more.

https://docs.traefik.io/v2.0/user-guides/docker-compose/basic-example for basics on docker with Tarefik.

Docker with LetsEncrypt and NGINX
https://medium.com/@pentacent/nginx-and-lets-encrypt-with-docker-in-less-than-5-minutes-b4b8a60d3a71
https://gist.github.com/ezhulkov/355fff964e54ffed541ed09e80341fd0
https://github.com/wmnnd/nginx-certbot
https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-18-04
https://docs.docker.com/compose/install
https://github.com/wmnnd/nginx-certbot

https://www.darkreading.com/network-and-perimeter-security/most-home-routers-are-full-of-vulnerabilities/d/d-id/1332987

Cups seems to be running. Stopped and disabled.
sudo systemctl stop cups
sudo systemctl disable cups

CUPS - Print Server
The primary mechanism for Ubuntu printing and print services is the Common UNIX Printing System (CUPS). This printing system is a freely available, portable printing layer which has become the new standard for printing in most Linux distributions.
CUPS manages print jobs and queues and provides network printing using the standard Internet Printing Protocol (IPP), while offering support for a very large range of printers, from dot-matrix to laser and many in between. CUPS also supports PostScript Printer Description (PPD) and auto-detection of network printers, and features a simple web-based configuration and administration tool.

Ran a multi-node Kubernetes cluster.
Coredns pods were running but were not ready.
Logged message was [INFO] plugin/ready: Still waiting on: "kubernetes"
So I deleted the pods, and the pods were running fine.
kubectl get pods -n kube-system -oname |grep coredns |xargs kubectl delete -n kube-system
I think the other way would be just to redeploy CoreDNS pods
kubectl rollout restart -n kube-system deployment/coredns
kubectl logs --tail=100 -f Tail last 100 lines
kubectl logs -f
https://github.com/kubernetes/kubernetes/blob/master/cluster/kube-down.sh
https://stackoverflow.com/questions/37016546/kubernetes-how-do-i-delete-clusters-and-contexts-from-kubectl-config
https://linkerd.io/2.10/tasks/exposing-dashboard
https://linkerd.io/2.10/tasks/troubleshooting
To confirm things are working.
The simplest way to test our setup is to create Apache HTTP Server with a pod http-page and expose it via service named http-service with port 80 and NodePort type:
kubectl run http-page --image=httpd --port=80
kubectl expose pod http-page --name=http-service --port=80 --type=NodePort
Verify that it works:
kubectl get service http-service
kubectl get pods http-page -o wide
Set host IP address to expose it from outerworld.
kubectl patch svc kubernetes -p '{"spec":{"externalIPs":["<IP_ADDRESS>"]}}'
kubectl exec --stdin --tty http-page -- /bin/bash
Run other workload:
kubectl create deployment nginx --image=nginx
kubectl expose deploy nginx --port 80 --target-port 80 --type NodePort
kubectl exec --stdin --tty ngninx -- sh
And other workload with deployment and add service to it:
kubectl apply -f https://k8s.io/examples/service/load-balancer-example.yaml // Deployment name is hello-world
kubectl expose deployment hello-world --type=LoadBalancer --name=my-service
kubectl scale --replicas=1 deployment my-service
kubectl delete services my-service
kubectl delete deployment hello-world
The commands:
kubectl get pods
kubectl delete pod  &
kubectl get services
kubectl delete service nginx
kubectl get deployments
kubectl delete deployment nginx

Setup Linkerd
Installed and enabled dashboard
linkerd viz install | kubectl apply -f -
Tried to access the dashboard but it's only accessible in the admin server. And it is not accessible from the admin server's IP. So listed the pods, the dashboard services runs on web pod. Then checked if there are any services that exposes it.
NAMESPACE     NAME                        TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                  AGE
linkerd-viz   web                         ClusterIP      <IP_ADDRESS>    <none>          8084/TCP,9994/TCP        170m

There was no external IP set, so set it with the admin's IP. While trying to access it, gave error:
It appears that you are trying to reach this service with a host of 'IP:8084'.
This does not match /^(localhost|127\.0\.0\.1|web\.linkerd-viz\.svc\.cluster\.local|web\.linkerd-viz\.svc|\[::1\])(:\d+)?$/ and has been denied for security reasons.
Please see https://linkerd.io/dns-rebinding for an explanation of what is happening and how to fix it.

The page suggests redeployment of deployment - https://linkerd.io/2.10/tasks/exposing-dashboard/#tweaking-host-requirement
So changed the config
Check : kubectl get deployment web -n linkerd-viz -o=json | jq '.spec.template.spec.containers[0].args'
Get the deployment config: kubectl getkubectl get deployment web -n linkerd-viz -o json > linkerd-dashboard.json
Update host config by adding IP address
Applied the changes: kubectl apply -f  linkerd-dashboard.json
Set external with the same IP added in the host config.
Access it : IP:8084
My previous resources(pods, deployments and services, and default cluster) were not injected. So injected Linkerd:
kubectl get deploy -o yaml | linkerd inject - | kubectl apply -f -
To test few things generated load with ab.
ab -n 100000 -c 10 -H "Accept-Encoding: gzip, deflate; Hello:world" -rk http://IP/
-n: Number of requests
-c: Number of concurrent requests
-H: Add header
—r: flag to not exit on socket receive errors
-k: Use HTTP KeepAlive feature
-p: File containing data to POST
-T: Content-type header to use for POST/PUT data,

Resource
https://linkerd.io/2.10/reference/cli/viz
https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#updating-a-deployment
https://stackoverflow.com/questions/49860181/how-do-i-update-the-args-for-a-kubernetes-deployment
https://linkerd.io/2.10/reference/cli/inject
https://diamantidis.github.io/2020/07/15/load-testing-with-apache-bench
https://www.sumologic.com/blog/kubectl-logs
https://medium.com/tegess/getting-started-with-linkerd-service-mesh-on-private-k8s-cloud-371b532f47c6

Kubectl command cheatsheet

Command
Description

kubectl get
used to view and find resourcescan output JSON, YAML, or be directly formatted

kubectl describe
retrieve extra information about a resourcesneeds a resource type and (optionally) a resource name

kubectl create
create a resource from a file or standard input

kubectl delete
can delete resources across multiple namespaces

kubectl label
can add/remove/update labels across multiple namespaces

kubectl logs
view container logs for debugging

kubectl exec
execute a command in a running container

Issue: Linkerd Dashboard
Dashboard has some issue. Metrics and Tap Injector pods were failing.
NAME                            READY   STATUS             RESTARTS          AGE
grafana-5c47556496-pn4vw        2/2     Running            0                 16h
metrics-api-86b88fddc-jrdgb     0/1     CrashLoopBackOff   175 (3m34s ago)   10h
prometheus-554c6cc89f-wml4r     1/1     Running            0                 10h
tap-7948bf84c5-plkt6            2/2     Running            0                 16h
tap-injector-5fc98dc7c5-ct548   0/1     CrashLoopBackOff   175 (75s ago)     10h
web-dcb9c6975-cg5fs             2/2     Running            0                 12h

Tap log

Events:
  Type     Reason     Age                   From     Message
  ----     ------     ----                  ----     -------
  Warning  Unhealthy  41m (x1652 over 10h)  kubelet  Readiness probe failed: Get "http://<IP_ADDRESS>:9995/ready": dial tcp <IP_ADDRESS>:9995: connect: connection refused
  Normal   Created    26m (x170 over 10h)   kubelet  Created container tap-injector
  Warning  Unhealthy  11m (x520 over 10h)   kubelet  Liveness probe failed: Get "http://<IP_ADDRESS>:9995/ping": dial tcp <IP_ADDRESS>:9995: connect: connection refused
  Warning  BackOff    84s (x2102 over 10h)  kubelet  Back-off restarting failed container

Metrics log

Events:
  Type     Reason     Age                    From     Message
  ----     ------     ----                   ----     -------
  Warning  Unhealthy  36m (x1677 over 10h)   kubelet  Readiness probe failed: Get "http://<IP_ADDRESS>:9995/ready": dial tcp <IP_ADDRESS>:9995: connect: connection refused
  Warning  Unhealthy  6m32s (x527 over 10h)  kubelet  Liveness probe failed: Get "http://<IP_ADDRESS>:9995/ping": dial tcp <IP_ADDRESS>:9995: connect: connection refused
  Warning  BackOff    85s (x2110 over 10h)   kubelet  Back-off restarting failed container

Ping works

PING <IP_ADDRESS> (<IP_ADDRESS>) 56(84) bytes of data.
64 bytes from <IP_ADDRESS>: icmp_seq=1 ttl=64 time=0.194 ms
64 bytes from <IP_ADDRESS>: icmp_seq=2 ttl=64 time=0.097 ms
64 bytes from <IP_ADDRESS>: icmp_seq=3 ttl=64 time=0.095 ms

IP route
default via <IP_ADDRESS> dev enp3s0 proto dhcp metric 100
<IP_ADDRESS>/24 dev cni0 proto kernel scope link src <IP_ADDRESS>
<IP_ADDRESS>/24 via <IP_ADDRESS> dev flannel.1 onlink
<IP_ADDRESS>/16 dev cni0 scope link metric 1000
<IP_ADDRESS>/16 dev docker0 proto kernel scope link src <IP_ADDRESS> linkdown
<IP_ADDRESS>/16 dev br-940c3874b064 proto kernel scope link src <IP_ADDRESS> linkdown
<IP_ADDRESS>/16 dev br-847829250b8c proto kernel scope link src <IP_ADDRESS> linkdown
<IP_ADDRESS>/16 dev br-6f507f240f54 proto kernel scope link src <IP_ADDRESS> linkdown
<IP_ADDRESS>/24 dev enp3s0 proto kernel scope link src <IP_ADDRESS> metric 100
<IP_ADDRESS>/24 dev virbr0 proto kernel scope link src <IP_ADDRESS> linkdown

IP neigh
<IP_ADDRESS> dev cni0 lladdr 22:e8:b1:87:f9:c1 REACHABLE
<IP_ADDRESS> dev cni0 lladdr f6:3d:7a:45:8c:38 REACHABLE
<IP_ADDRESS> dev cni0 lladdr e2:1a:a6:b3:02:f7 DELAY
<IP_ADDRESS> dev enp3s0 lladdr 28:d2:44:f3:aa:98 REACHABLE
<IP_ADDRESS> dev cni0 lladdr ee:05:a5:fb:f0:04 REACHABLE
<IP_ADDRESS> dev cni0 lladdr 3a:9f:2a:bf:d2:4e REACHABLE
<IP_ADDRESS> dev cni0 lladdr 96:aa:28:03:6d:bb REACHABLE
<IP_ADDRESS> dev cni0 lladdr 2e:7d:d6:9e:0e:4e REACHABLE
<IP_ADDRESS> dev cni0 lladdr 46:75:ae:95:79:b3 REACHABLE
<IP_ADDRESS> dev cni0 lladdr de:99:d3:1b:2f:dd REACHABLE
<IP_ADDRESS> dev cni0 lladdr b2:c6:55:3d:8f:c4 STALE
<IP_ADDRESS> dev cni0 lladdr 56:96:b4:ba:1e:fa REACHABLE
<IP_ADDRESS> dev flannel.1  FAILED
<IP_ADDRESS> dev flannel.1 lladdr 22:27:1c:99:b6:52 PERMANENT
<IP_ADDRESS> dev cni0 lladdr c2:5c:bc:c1:6b:75 REACHABLE
<IP_ADDRESS> dev flannel.1  FAILED
<IP_ADDRESS> dev cni0 lladdr be:8d:0b:f3:86:33 REACHABLE
<IP_ADDRESS> dev cni0 lladdr e6:45:5a:1b:ea:19 STALE
<IP_ADDRESS> dev flannel.1  FAILED
<IP_ADDRESS> dev cni0 lladdr 1a:f0:3e:6f:15:cd REACHABLE
<IP_ADDRESS> dev enp3s0 lladdr 88:b3:62:24:2d:60 REACHABLE
<IP_ADDRESS> dev cni0 lladdr 72:1f:44:6e:26:53 REACHABLE
<IP_ADDRESS> dev enp3s0 lladdr d0:53:49:57:e3:57 REACHABLE
fe80::c05c:bcff:fec1:6b75 dev cni0 lladdr c2:5c:bc:c1:6b:75 STALE
fe80::e01a:a6ff:feb3:2f7 dev cni0 lladdr e2:1a:a6:b3:02:f7 STALE
fe80::94aa:28ff:fe03:6dbb dev cni0 lladdr 96:aa:28:03:6d:bb STALE
fe80::dc99:d3ff:fe1b:2fdd dev cni0 lladdr de:99:d3:1b:2f:dd STALE
fe80::18f0:3eff:fe6f:15cd dev cni0 lladdr 1a:f0:3e:6f:15:cd STALE
fe80::bc8d:bff:fef3:8633 dev cni0 lladdr be:8d:0b:f3:86:33 STALE
fe80::5496:b4ff:feba:1efa dev cni0 lladdr 56:96:b4:ba:1e:fa STALE
fe80::389f:2aff:febf:d24e dev cni0 lladdr 3a:9f:2a:bf:d2:4e STALE
fe80::4475:aeff:fe95:79b3 dev cni0 lladdr 46:75:ae:95:79:b3 STALE
fe80::2c7d:d6ff:fe9e:e4e dev cni0 lladdr 2e:7d:d6:9e:0e:4e STALE
fe80::1 dev enp3s0 lladdr 88:b3:62:24:2d:60 router STALE
fe80::44bf:ff:fee5:312e dev cni0 lladdr 46:bf:00:e5:31:2e STALE
fe80::701f:44ff:fe6e:2653 dev cni0 lladdr 72:1f:44:6e:26:53 STALE
fe80::e445:5aff:fe1b:ea19 dev cni0 lladdr e6:45:5a:1b:ea:19 STALE
fe80::f43d:7aff:fe45:8c38 dev cni0 lladdr f6:3d:7a:45:8c:38 STALE
fe80::20e8:b1ff:fe87:f9c1 dev cni0 lladdr 22:e8:b1:87:f9:c1 STALE
fe80::ec05:a5ff:fefb:f004 dev cni0 lladdr ee:05:a5:fb:f0:04 STALE
fe80::b0c6:55ff:fe3d:8fc4 dev cni0 lladdr b2:c6:55:3d:8f:c4 STALE

ip a

2: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 3c:4a:92:55:34:84 brd ff:ff:ff:ff:ff:ff
    inet <IP_ADDRESS>/24 brd <IP_ADDRESS> scope global dynamic noprefixroute enp3s0
       valid_lft 71015sec preferred_lft 71015sec
    inet6 2403:3800:322b:603:b91d:9ee4:977f:a62c/64 scope global temporary dynamic
       valid_lft 86215sec preferred_lft 71148sec
    inet6 2403:3800:322b:603:5ba0:d7d5:3e24:b976/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 86215sec preferred_lft 86215sec
    inet6 2403:3800:322b:49a:b91d:9ee4:977f:a62c/64 scope global temporary deprecated dynamic
       valid_lft 70925sec preferred_lft 0sec
    inet6 2403:3800:322b:49a:cd66:2849:154f:3080/64 scope global temporary deprecated dynamic
       valid_lft 70925sec preferred_lft 0sec
    inet6 2403:3800:322b:49a:bc4a:f344:181e:281f/64 scope global temporary deprecated dynamic
       valid_lft 70925sec preferred_lft 0sec
    inet6 2403:3800:322b:49a:4a95:b9ba:ab0b:bd3f/64 scope global deprecated dynamic mngtmpaddr noprefixroute
       valid_lft 70925sec preferred_lft 0sec
    inet6 fe80::21f2:2000:9a2a:2e4d/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
3: wlp2s0b1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether e0:2a:82:fb:d5:cc brd ff:ff:ff:ff:ff:ff
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:91:be:07:3f brd ff:ff:ff:ff:ff:ff
    inet <IP_ADDRESS>/16 brd <IP_ADDRESS> scope global docker0
       valid_lft forever preferred_lft forever
5: br-6f507f240f54: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:ed:92:bd:ab brd ff:ff:ff:ff:ff:ff
    inet <IP_ADDRESS>/16 brd <IP_ADDRESS> scope global br-6f507f240f54
       valid_lft forever preferred_lft forever
    inet6 fc00:f853:ccd:e793::1/64 scope global tentative
       valid_lft forever preferred_lft forever
    inet6 fe80::1/64 scope link tentative
       valid_lft forever preferred_lft forever
6: br-847829250b8c: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:a0:ce:67:5d brd ff:ff:ff:ff:ff:ff
    inet <IP_ADDRESS>/16 brd <IP_ADDRESS> scope global br-847829250b8c
       valid_lft forever preferred_lft forever
7: br-940c3874b064: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:33:0a:ed:c9 brd ff:ff:ff:ff:ff:ff
    inet <IP_ADDRESS>/16 brd <IP_ADDRESS> scope global br-940c3874b064
       valid_lft forever preferred_lft forever
8: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:37:80:14 brd ff:ff:ff:ff:ff:ff
    inet <IP_ADDRESS>/24 brd <IP_ADDRESS> scope global virbr0
       valid_lft forever preferred_lft forever
9: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:37:80:14 brd ff:ff:ff:ff:ff:ff
11: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
    link/ether ea:e0:0d:4c:85:67 brd ff:ff:ff:ff:ff:ff
    inet <IP_ADDRESS>/32 brd <IP_ADDRESS> scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::e8e0:dff:fe4c:8567/64 scope link
       valid_lft forever preferred_lft forever
12: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether 5a:e4:c4:12:f7:4a brd ff:ff:ff:ff:ff:ff
    inet <IP_ADDRESS>/24 brd <IP_ADDRESS> scope global cni0
       valid_lft forever preferred_lft forever
    inet6 fe80::58e4:c4ff:fe12:f74a/64 scope link
       valid_lft forever preferred_lft forever
18: veth58578a24@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
    link/ether 86:65:e4:05:d0:7c brd ff:ff:ff:ff:ff:ff link-netns cni-9d8ed6c9-9048-2b39-b799-04e828c9de25
    inet6 fe80::8465:e4ff:fe05:d07c/64 scope link
       valid_lft forever preferred_lft forever
19: veth83d654aa@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
    link/ether 02:4e:d9:bb:d7:3e brd ff:ff:ff:ff:ff:ff link-netns cni-bf6a6602-462f-d47a-db7e-4b1fbff87f73
    inet6 fe80::4e:d9ff:febb:d73e/64 scope link
       valid_lft forever preferred_lft forever
20: veth8e382bf1@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
    link/ether ee:4d:03:3c:cf:27 brd ff:ff:ff:ff:ff:ff link-netns cni-4a69d3aa-02b3-3d2d-c082-d04d26e99c26
    inet6 fe80::ec4d:3ff:fe3c:cf27/64 scope link
       valid_lft forever preferred_lft forever
21: veth77f4365a@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
    link/ether 2a:2f:c0:b4:84:6d brd ff:ff:ff:ff:ff:ff link-netns cni-bc4e3554-db66-48a7-6ce5-0927c589f27d
    inet6 fe80::282f:c0ff:feb4:846d/64 scope link
       valid_lft forever preferred_lft forever
22: vetha82cf484@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
    link/ether fe:d7:b7:56:b0:ee brd ff:ff:ff:ff:ff:ff link-netns cni-06885625-eb9b-97b3-4b3a-217597a61263
    inet6 fe80::fcd7:b7ff:fe56:b0ee/64 scope link
       valid_lft forever preferred_lft forever
23: vethbe7db6af@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
    link/ether 42:b9:25:5f:56:db brd ff:ff:ff:ff:ff:ff link-netns cni-99b484b1-c75a-dba2-5229-f42992af32da
    inet6 fe80::40b9:25ff:fe5f:56db/64 scope link
       valid_lft forever preferred_lft forever
24: vetheb84669e@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
    link/ether 36:a6:ad:f5:64:38 brd ff:ff:ff:ff:ff:ff link-netns cni-24276917-be09-891b-40cb-6d93e6f814f5
    inet6 fe80::34a6:adff:fef5:6438/64 scope link
       valid_lft forever preferred_lft forever
25: veth7963ccfe@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
    link/ether 7a:57:8a:19:3c:54 brd ff:ff:ff:ff:ff:ff link-netns cni-6c2ab0f3-d875-ed27-417a-90ef1d054a00
    inet6 fe80::7857:8aff:fe19:3c54/64 scope link
       valid_lft forever preferred_lft forever
26: vethc04ca050@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
    link/ether e6:16:e8:fe:15:90 brd ff:ff:ff:ff:ff:ff link-netns cni-a8537697-481c-c5f6-116a-6565dcfa9c12
    inet6 fe80::e416:e8ff:fefe:1590/64 scope link
       valid_lft forever preferred_lft forever
27: veth5267eb7b@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
    link/ether f2:bd:11:ca:70:f9 brd ff:ff:ff:ff:ff:ff link-netns cni-cad70dd8-55a7-1cd3-c9b1-9af5f4279d88
    inet6 fe80::f0bd:11ff:feca:70f9/64 scope link
       valid_lft forever preferred_lft forever
28: veth6e1d8373@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
    link/ether c6:68:e2:3f:a0:7a brd ff:ff:ff:ff:ff:ff link-netns cni-18e5b9a4-2fde-f8dd-7620-e76fe4e3be47
    inet6 fe80::c468:e2ff:fe3f:a07a/64 scope link
       valid_lft forever preferred_lft forever
29: veth65ffc762@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
    link/ether 86:c4:e1:f3:44:4d brd ff:ff:ff:ff:ff:ff link-netns cni-97c921cb-3ded-027a-5019-b2a628a61583
    inet6 fe80::84c4:e1ff:fef3:444d/64 scope link
       valid_lft forever preferred_lft forever
30: veth242d4505@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
    link/ether 96:c1:12:eb:55:1b brd ff:ff:ff:ff:ff:ff link-netns cni-fb15b669-6eba-a90f-b4f2-fc071b9509d0
    inet6 fe80::94c1:12ff:feeb:551b/64 scope link
       valid_lft forever preferred_lft forever
31: veth11d4a14d@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
    link/ether ca:64:ef:e2:6a:77 brd ff:ff:ff:ff:ff:ff link-netns cni-9eed5228-0d7c-4225-e5d6-d67734f812bd
    inet6 fe80::c864:efff:fee2:6a77/64 scope link
       valid_lft forever preferred_lft forever
32: vethe806cfd8@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
    link/ether 22:7c:ac:09:90:5e brd ff:ff:ff:ff:ff:ff link-netns cni-e75517f4-45f3-1b8e-b598-263cbd6742be
    inet6 fe80::207c:acff:fe09:905e/64 scope link
       valid_lft forever preferred_lft forever
33: veth1a143748@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
    link/ether 22:34:a8:bb:a4:d3 brd ff:ff:ff:ff:ff:ff link-netns cni-cb09e88b-3251-ce51-084a-52eb570c4dec
    inet6 fe80::2034:a8ff:febb:a4d3/64 scope link
       valid_lft forever preferred_lft forever
34: veth104a6ba8@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
    link/ether 3e:17:f2:25:8d:23 brd ff:ff:ff:ff:ff:ff link-netns cni-3538fc5a-13c5-53ae-c24a-81b0c789c3e1
    inet6 fe80::3c17:f2ff:fe25:8d23/64 scope link
       valid_lft forever preferred_lft forever

linkerd check

× viz extension self-checkeck
    Post "http://localhost:40009/api/v1/SelfCheck": EOF
    see https://linkerd.io/2.10/checks/#l5d-viz-metrics-api for hints
‼ linkerd-viz pods are injected
    could not find proxy container for metrics-api-86b88fddc-jrdgb pod
    see https://linkerd.io/2.10/checks/#l5d-viz-pods-injection for hints
‼ viz extension pods are running
    container metrics-api-86b88fddc-jrdgb in pod metrics-api is not ready
    see https://linkerd.io/2.10/checks/#l5d-viz-pods-running for hints

This is also not that helpful.

A bit about Readiness and Liveness

The `Readiness` and `Liveness` probes serve slightly different purposes.

The readiness probe controls whether the pod IP is included in the list of endpoints for a service, and so also whether a target for a route when it is exposed via an external URL.

The liveness probe determines whether a pod is still running normally or whether it should be restarted.

Technically an application could still be running fine, but is perhaps backlogged, and so you want to use the readiness probe to temporarily remove it from the set of endpoints for a service to avoid further requests being routed its way and simply being blocked in the request queue for that specific pod when another pod could handle it.

So I personally would agree the duplication seems strange, but it is that way so the different situations can be distinguished.

kubectl -n linkerd-viz get all : List all resources in the namespace
Nothing is else comes to mind so uninstall and reninstall.

linkerd viz uninstall | kubectl delete -f -
linkerd viz install | kubectl apply -f -

Everything looks good
kubectl -n linkerd-viz get all | less
NAME                                READY   STATUS    RESTARTS   AGE
pod/grafana-5c47556496-vnsd6        2/2     Running   0          102s
pod/metrics-api-86b88fddc-g9jnv     2/2     Running   0          102s
pod/prometheus-554c6cc89f-4vcxz     2/2     Running   0          101s
pod/tap-b5f699dcb-7d46j             2/2     Running   0          100s
pod/tap-injector-7784dbcfb8-kfv67   2/2     Running   0          98s
pod/web-6b8894468b-ghfgk            2/2     Running   0          98s

NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
service/grafana        ClusterIP   <IP_ADDRESS>   <none>        3000/TCP            102s
service/metrics-api    ClusterIP   <IP_ADDRESS>   <none>        8085/TCP            103s
service/prometheus     ClusterIP   <IP_ADDRESS>   <none>        9090/TCP            101s
service/tap            ClusterIP   <IP_ADDRESS>    <none>        8088/TCP,443/TCP    101s
service/tap-injector   ClusterIP   <IP_ADDRESS>    <none>        443/TCP             100s
service/web            ClusterIP   <IP_ADDRESS>     <none>        8084/TCP,9994/TCP   99s

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/grafana        1/1     1            1           102s
deployment.apps/metrics-api    1/1     1            1           103s
deployment.apps/prometheus     1/1     1            1           101s
deployment.apps/tap            1/1     1            1           100s
deployment.apps/tap-injector   1/1     1            1           99s
deployment.apps/web            1/1     1            1           99s

NAME                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/grafana-5c47556496        1         1         1       102s
replicaset.apps/metrics-api-86b88fddc     1         1         1       103s
replicaset.apps/prometheus-554c6cc89f     1         1         1       101s
replicaset.apps/tap-b5f699dcb             1         1         1       100s
replicaset.apps/tap-injector-7784dbcfb8   1         1         1       99s
replicaset.apps/web-6b8894468b            1         1         1       99s

Config now has been reset
kubectl get deployment web -n linkerd-viz -o=json | jq '.spec.template.spec.containers[0].args'
[
  "-linkerd-controller-api-addr=linkerd-controller-api.linkerd.svc.cluster.local:8085",
  "-linkerd-metrics-api-addr=metrics-api.linkerd-viz.svc.cluster.local:8085",
  "-cluster-domain=cluster.local",
  "-grafana-addr=grafana.linkerd-viz.svc.cluster.local:3000",
  "-controller-namespace=linkerd",
  "-viz-namespace=linkerd-viz",
  "-log-level=info",
  "-enforced-host=^(localhost|127\\.0\\.0\\.1|web\\.linkerd-viz\\.svc\\.cluster\\.local|web\\.linkerd-viz\\.svc|\\[::1\\])(:\\d+)?$"
]

Set the external IP kubectl patch -n linkerd-viz svc web -p '{"spec":{"externalIPs":["<IP_ADDRESS>"]}}'
Everything is working fine.
Resource

https://stackoverflow.com/questions/44309291/kubernetes-readiness-probe-execution-after-container-started

Deploying Emojivoto and use Linkerd CLI tap, top, edges, and stat commands.
Emojivoto is a gRPC application that has three services:

web: the frontend that users interact with
emoji: provides API endpoints to list emoji
voting: provides API endpoints to vote for emoji
There is also a fourth component, vote-boti, that generates simulated traffic to the application

Deploy the Emojivoto application to your cluster:
kubectl apply -f https://run.linkerd.io/emojivoto.yml
Review the yaml file and saw ServiceAccount.
kubectget all -n emojivoto
NAME                            READY   STATUS    RESTARTS   AGE
pod/emoji-66ccdb4d86-dgq5v      1/1     Running   0          13m
pod/vote-bot-69754c864f-lb9rf   1/1     Running   0          13m
pod/voting-f999bd4d7-2g8hz      1/1     Running   0          13m
pod/web-79469b946f-b82rv        1/1     Running   0          13m

NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP     PORT(S)             AGE
service/emoji-svc    ClusterIP   <IP_ADDRESS>   <none>         8080/TCP,8801/TCP   13m
service/voting-svc   ClusterIP   <IP_ADDRESS>    <none>          8080/TCP,8801/TCP   13m
service/web-svc      ClusterIP   <IP_ADDRESS>      <none>          80/TCP              13m

NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/emoji      1/1     1            1           13m
deployment.apps/vote-bot   1/1     1            1           13m
deployment.apps/voting     1/1     1            1           13m
deployment.apps/web        1/1     1            1           13m

NAME                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/emoji-66ccdb4d86      1         1         1       13m
replicaset.apps/vote-bot-69754c864f   1         1         1       13m
replicaset.apps/voting-f999bd4d7      1         1         1       13m
replicaset.apps/web-79469b946f        1         1         1       13m

Default setup to access is to port forward but as are accessing NOT from the host so we need to update the External IP to the service but before that lets check the current service status.
kubectl -n emojivoto port-forward svc/web-svc 8080:80
kubectl describe service web-svc -n emojivoto
Name:              web-svc
Namespace:         emojivoto
Labels:            <none>
Annotations:       <none>
Selector:          app=web-svc
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                <IP_ADDRESS>
IPs:               <IP_ADDRESS>
Port:              http  80/TCP
TargetPort:        8080/TCP
Endpoints:         <IP_ADDRESS>:8080
Session Affinity:  None
Events:            <none>

kubectl patch service web-svc -n emojivoto -p '{"spec":{"externalIPs":["<IP_ADDRESS>"]}}'

Injecting the Emojivoto deployments with the Linkerd proxy.
First, annotate the emojivoto namespace:
kubectl annotate ns emojivoto linkerd.io/inject=enabled

This annotation is all we need to inform Linkerd to inject the proxies into pods in this namespace. However, simply adding the annotation won't affect existing resources. We'll also need to restart the Emojivoto deployments: kubectl rollout restart deploy -n emojivoto
kubectl scale deploy/web -n emojivoto --replicas=2
Tap all the traffic from the web deployment:
linkerd tap deploy/web -n emojivoto
tap traffic from deploy/web and output as JSON:
linkerd tap deployment/web --namespace emojivoto --to deployment/voting --path /emojivoto.v1.VotingService/VoteDoughnut -o json
Use linkerd top to view traffic sorted by the most popular paths:
linkerd top deploy/web -n emojivoto
See the golden metrics, the latencies, success/error rates, and requests per second that you saw in the dashboard, except this time, you want to see them in the terminal.
linkerd stat deploy -n emojivoto
Let's narrow the query to focus in on the traffic from the web deployment to the emoji deployment:
linkerd viz stat -n emojivoto deploy/web --to deploy/emoji -o wide
Let's take a look at the traffic between the web and voting deployments to investigate
linkerd stat -n emojivoto deploy/web --to deploy/voting -o wide
View the metrics for traffic to all deployments in the emojivoto namespace that comes from the web deployment:
linkerd viz stat -n emojivoto deploy --from deploy/web
The Linkerd control plane is not only "aware" of the services that are meshed, but it also is aware of which services communicate with each other.
Get the edges for all the deployments in the emojivoto namespace:
linkerd edges -n emojivoto deploy
Zoom in on the service graph by looking at the edges between pods rather than deployments, like this:
linkerd edges -n emojivoto po
Notes

ServiceAccount
A service account provides an identity for processes that run in a Pod.
When you (a human) access the cluster (for example, using kubectl), you are authenticated by the apiserver as a particular User Account (currently this is usually admin, unless your cluster administrator has customized your cluster). Processes in containers inside pods can also contact the apiserver. When they do, they are authenticated as a particular Service Account (for example, default)/

Ports
Kubernetes does not manage loose containers but manages them in units called a Pod. A Pod could be a single container or set of containers launched as a service.

“containerPort” defines the port on which app can be reached out inside the container.
targetPort and containerPort must be identical most of the time because whatever port is open for your application in a container that would be the same port you will wish to send traffic from service via targetPort.
The Service object of Kubernetes exposes a Pod/s to the outer world with nodePort via static IP.  Furthermore, it also load-balances among the Pods and also acts as a firewall.

Port exposes the Kubernetes service on the specified port within the cluster. Other pods within the cluster can communicate with this server on the specified port.
TargetPort is the port on which the service will send requests to, that your pod will be listening on. Your application in the container will need to be listening on this port also.
NodePort exposes a service externally to the cluster by means of the target nodes IP address and the NodePort. NodePort is the default setting if the port field is not specified.

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    app: hello-world
spec:
  containers:
  - name: nginx
    image: nginx
    ports:
      - containerPort: 80
 ---
apiVersion: v1
kind: Service
metadata:
  name: hello-world
spec:
  type: NodePort
  selector:
    app: hello-world
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 80
      nodePort: 30036

Create a pod running nginx that listens at port 80.
Create a service which will be exposed internally to other applications in the same cluster at port 8080 and externally to the cluster at port 30036. It will also forward requests to pods with the label “app: hello-world” on port 80.
Resource

https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account
https://github.com/linkerd/linkerd2/issues/2585
https://sysdig.com/blog/debug-kubernetes-crashloopbackoff
https://linkerd.io/2.10/tasks/troubleshooting

How to get per-route metrics using service profiles
By creating a ServiceProfile for a Kubernetes service, we can specify the routes available to the service and collect data.
Use the Linkerd dashboard and linkerd routes command to view per-route metrics for services in the Emojivoto application.
A Linkerd service profile is implemented by instantiating a Kubernetes Custom Resource Definitions (CRD) called, as you might expect, ServiceProfile. The ServiceProfile enumerates the routes that Linkerd should expect for that service.
Use the linkerd profile command to generate ServiceProfile definitions. If the services in your application have OpenAPI or protobuf definitions, you can use those to create ServiceProfile definitions. Alternatively, Linkerd can observe requests in real time to generate the service profile for you. Finally, you can write the profile by hand.
We create a per-route metric for UI with linkerd profile -h
Viewing Per-Route Metrics with the Linkerd CLI
linkerd routes deploy/emoji -n emojivoto
linkerd routes deploy/web -n emojivoto
linkerd routes deploy/voting -n emojivoto
linkerd viz top deploy/voting -n emojivoto

Linkerd can help ensure reliability through features like traffic splitting, load balancing, retries, and timeouts. Each of these plays an important role in enhancing the overall reliability of the system.
But what kind of reliability can these features add, exactly? The answer is that, ultimately, what Linkerd (or any service mesh!) can help protect against, is transient failures. If a service is completely down, or consistently returns a failure, or consistently hangs, no amount of retrying or load balancing can help. But if one single instance of a service is having an issue, or if the underlying issue is just a temporary one, well, that's where Linkerd can be useful. Happily—or unhappily?—these sorts of partial, transient failures are endemic to distributed systems!
There are two important things to understand when it comes to Linkerd's core reliability featureset of load balancing, retries, and timeouts (traffic splitting, also a reliability feature, is a bit different—we'll be addressing that in a later chapter). They are:

All these techniques happen on the client side: the proxy on the side that makes the call is the one that enacts these features, not the one on the server side. One implication is that if your server is meshed, but your client isn't, these features will not be enabled for calls between the two!
These three techniques work best together.

From the last setup, further research was done on Flannel, Linkerd, security of Kubernetes overall.
Some conclusions made were:

Flannel needs to be enhanced further to provide better end-to-end encryption security.
Flannel does not work service meshes, so need to research further on Flannel and Linkerd usages.
There was no end-to-end logging and monitoring, so need to setup a Security Information and Event Management (SIEM). And configure clusters to forward all audit and log events to it.

The next I thought to move ahead with to use the setup and make it more practical would be run example Microservices  applications completely with full build and deploy process. And for that a CI/CD system would be needed. Also, as I am trying to move away from Docker, I looked into tools to build images. Kaniko, Buildah/Podman seems to be the best options, with Kaniko fitting more with it's native feature to build image inside container, it will be used.
Then, there was the case of local image registry was looked for and Harbor seems to be a good tool. Quay also seems to be good tool, Podman and Docker as well. But for now Harbor will be tried at first as an image registry. For trial purpose, Dockerhub public image repository will be used.
For CI, Jenkins running on Kubernetes will implemented, and probably ArgoCD for CD. So the first step was to run Jenkins in Kubernetes. To run Jenkins on Kubernetes, requires Persistent Volume Claim. A better understanding of PVC was needed.
The steps would be:

Run Jenkins in Kubernetes
Setup CI to be fired with Github's webhook
Run Kaniko in Kubernetes to build image
Pushes image to Dockerhub
The application should be deployed on the respective Kubernetes cluster

Note
https://goharbor.io
Harbor is an open source registry that secures artifacts with policies and role-based access control, ensures images are scanned and free from vulnerabilities, and signs images as trusted. Harbor, a CNCF Graduated project, delivers compliance, performance, and interoperability to help you consistently and securely manage artifacts across cloud native compute platforms like Kubernetes and Docker.
https://www.projectquay.io
Quay is a container image registry that enables you to build, organize, distribute, and deploy containers. Quay gives you security over your repositories with image vulnerability scanning and robust access controls. Project Quay provides a scalable open source platform to host container images across any size organization.
https://argoproj.github.io/argo-cd
Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes.
 Found a good tool: Kubescape
Kubescape is the first tool for testing if Kubernetes is deployed securely as defined in Kubernetes Hardening Guidance by to NSA and CISA Tests are configured with YAML files, making this tool easy to update as test specifications evolve.
Installed and did a security scan:
kubescape scan framework nsa --exclude-namespaces kube-system,kube-public

Summary - Passed:18   Failed:0   Total:18
+-------------------------------------------------+------------------+---------------+-----------+
|                  CONTROL NAME                   | FAILED RESOURCES | ALL RESOURCES | % SUCCESS |
+-------------------------------------------------+------------------+---------------+-----------+
| Allow privilege escalation                      | 0                | 18            | 100%      |
| Allowed hostPath                                | 0                | 18            | 100%      |
| Applications credentials in configuration files | 11               | 26            | 57%       |
| Automatic mapping of service account            | 20               | 20            | 0%        |
| Cluster-admin binding                           | 14               | 149           | 90%       |
| Control plane hardening                         | 0                | 18            | 100%      |
| Dangerous capabilities                          | 0                | 18            | 100%      |
| Exec into container                             | 2                | 149           | 98%       |
| Exposed dashboard                               | 0                | 35            | 100%      |
| Host PID/IPC privileges                         | 0                | 18            | 100%      |
| Immutable container filesystem                  | 18               | 18            | 0%        |
| Insecure capabilities                           | 0                | 18            | 100%      |
| Linux hardening                                 | 18               | 18            | 0%        |
| Non-root containers                             | 17               | 18            | 5%        |
| Privileged container                            | 0                | 18            | 100%      |
| Resource policies                               | 18               | 18            | 0%        |
| hostNetwork access                              | 0                | 18            | 100%      |
+-------------------------------------------------+------------------+---------------+-----------+
|                       17                        |       118        |      595      |    80%    |
+-------------------------------------------------+------------------+---------------+-----------+

The summary listed out many issues.
The configuration used to access the was admin's which was created while creating clusters using kubeadmin
, i.e. /etc/kubernetes/admin.conf.  I wanted to know if the result would change if another user with lower privileged would do the scan.
So the question - How can I create a user and restrict user to access only one namespace in Kubernetes?
Kubernetes gives us a way to regulate access to Kubernetes clusters and resources based on the roles of individual users through a feature called Role-based access control (RBAC) via rbac.authorization.k8s.io/v1 API.
Therefore to achieve a complete isolation in Kubernetes, the concepts on namespaces and role based access control is used.

Role: A role contains rules that represent a set of permissions. A role is used to grant access to resources within a namespace.
RoleBinding: A role binding is used to grant the permissions defined in a role to a user or set of users. It holds a list of subjects (users, groups, or service accounts), and a reference to the role being granted.
Service Account: Account meant for processes, which run in pods. The idea behind service accounts is based on the principle of least privilege. An account is created for specific tasks.

Create and Limit Service account to a namespace in Kubernetes:

Create a demo namespace
kubectl create namespace demo
Create a service account

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: demo-user
  namespace: demo
EOF

Create a role that will give full access to the demo namespace

cat <<EOF | kubectl apply -f -
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: demo-admin
  namespace: demo
rules:
- apiGroups: ["", "extensions", "apps"]
  resources: ["*"]
  verbs: ["*"]
- apiGroups: ["batch"]
  resources:
  - jobs
  - cronjobs
  verbs: ["*"]
EOF

Confirm it kubectl get roles -n demo
An example of role with limited permissions:
cat <<EOF | kubectl apply -f -
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: demo
  name: deployment-admin
rules:
- apiGroups: ["", "extensions", "apps"]
  resources: ["deployments", "replicasets", "pods", "services", "ingresses"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] # You can also use ["*"]
EOF

Bind the role to a user

Resource

https://github.com/armosec/kubescape

[] Setup Jenkins in Kubernetes Cluster

Issue: Pods/Deployment not ready, and log from deployment unavailable
`k logs deployment/jenkins -n jenkins
error: a container name must be specified for pod jenkins-79cf9b4b8f-hhnkx, choose one of: [jenkins linkerd-proxy] or one of the init containers: [linkerd-init]
'''
The reason is because Istio adds a second container to deployments via the istio-sidecar-injector.
The resources created by default after new namespace is created are:
k apply -f namespace.yaml
kubectl get all,cm,secret,ing -n jenkins
NAME                         DATA   AGE
configmap/kube-root-ca.crt   1      2m23s

NAME                         TYPE                                  DATA   AGE
secret/default-token-xg2j2   kubernetes.io/service-account-token   3      2m23s

Create ServiceAccount
k apply -f serviceAccount.yaml
serviceaccount/jenkins created
clusterrole.rbac.authorization.k8s.io/jenkins created
clusterrolebinding.rbac.authorization.k8s.io/jenkins created

Create storage volume
k apply -f volume.yaml
storageclass.storage.k8s.io/local-storage unchanged
persistentvolume/jenkins-pv-volume unchanged
persistentvolumeclaim/jenkins-pv-claim created

Create deployment
k apply -f deployment.yaml
deployment.apps/jenkins created

Conclusion
First attempt worked. I wanted to learned each steps more and redid the steps which did not work out.
Resource

[ ] https://devopscube.com/setup-jenkins-on-kubernetes-cluster
[ ] https://www.jenkins.io/doc/book/installing/kubernetes
[ ] https://raw.githubusercontent.com/jenkins-infra/jenkins.io/master/content/doc/tutorials/kubernetes/installing-jenkins-on-kubernetes/jenkins-sa.yaml
[ ] https://raw.githubusercontent.com/jenkins-infra/jenkins.io/master/content/doc/tutorials/kubernetes/installing-jenkins-on-kubernetes/jenkins-volume.yaml
[ ] https://raw.githubusercontent.com/jenkins-infra/jenkins.io/master/content/doc/tutorials/kubernetes/installing-jenkins-on-kubernetes/jenkins-deployment.yaml
[ ] https://devopscube.com/kubernetes-ingress-tutorial
[ ] https://devopscube.com/setup-ingress-kubernetes-nginx-controller

[ ] Setup Jenkins in Kubernetes Cluster
Re-installation worked. Default manifests used from https://devopscube.com/setup-jenkins-on-kubernetes-cluster.
An issue with volume mount was as the volume was on created on node and given RW permission.

Objectives

[ ] Understand more Kubernetes native CI/CD solution.

https://tekton.dev, https://github.com/tektoncd/pipeline
Tekton is a cloud-native solution for building CI/CD systems. It consists of Tekton Pipelines, which provides the building blocks, and of supporting components, such as Tekton CLI and Tekton Catalog, that make Tekton a complete ecosystem.

https://jenkins-x.io
Uses Tekton's Pipeline.

https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions

https://landscape.cd.foundation

https://github.com/ortelius/ortelius
Ortelius simplifies the implementation of microservices. By providing a central catalog of services with their deployment specs, application teams can easily consume and deploy services across cluster. Ortelius tracks application versions based on service updates and maps their service dependencies eliminating confusion and guess work.

Next Steps

[ ] Implement Tekton and move to JenkinsX.

[ ]  Setup CI/CD pipeline in Jenkins

[ ] Run Tests, Linters, Unit, Functional
[ ] Notify if tests fails, email, slack
[ ] Build image with Kaniko
[ ] Deploy image to Docker registry
[ ] Apply in environment, production

Continuing from the last installed state of Jenkins.
Change namespace to jenkins
kubectl config set-context --current --namespace=jenkins
List the resources in the namespace
kubectl get all,cm,secret,ing,pvc
NAME                           READY   STATUS    RESTARTS   AGE
pod/jenkins-559d8cd85c-wgj8z   1/1     Running   0          2d6h

NAME                      TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
service/jenkins-service   NodePort   <IP_ADDRESS>   <none>        8080:32000/TCP   2d6h

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/jenkins   1/1     1            1           2d6h

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/jenkins-559d8cd85c   1         1         1       2d6h

NAME                         DATA   AGE
configmap/kube-root-ca.crt   1      2d6h

NAME                               TYPE                                  DATA   AGE
secret/default-token-wvg5h         kubernetes.io/service-account-token   3      2d6h
secret/jenkins-admin-token-tqfhk   kubernetes.io/service-account-token   3      2d6h

NAME                                     STATUS   VOLUME              CAPACITY   ACCESS MODES   STORAGECLASS    AGE
persistentvolumeclaim/jenkins-pv-claim   Bound    jenkins-pv-volume   10Gi       RWO            local-storage   2d6h

Set external IP to access UI
kubectl patch service/jenkins-service -p '{"spec":{"externalIPs":["<IP_ADDRESS>"]}}'
Build image with Kaniko
Build image with Kaniko and deployed to DockerHub. For demo the process was implemented as a Kubernetes Job.
To send image to Dockerhub, authentication is required.
kubectl create secret docker-registry regcred --docker-server=https://index.docker.io/v2/ --docker-username=codeanit --docker-password=TOKEN<EMAIL_ADDRESS>apiVersion: batch/v1
kind: Job
metadata:
  name: kaniko
spec:
  template:
    spec:
      containers:
      - name: kaniko
        image: gcr.io/kaniko-project/executor:latest
        args: ["--dockerfile=Dockerfile",
               "--context=git://github.com/kubernetes-utilities/kubernetes-kaniko.git#refs/heads/master",
               "--destination=codeanit/demo-cicd-go:v0.0.1"]
        volumeMounts:
        - name: kaniko-secret
          mountPath: "/kaniko/.docker"
      restartPolicy: Never
      volumes:
      - name: kaniko-secret
        secret:
          secretName: regcred
          items:
          - key: .dockerconfigjson
            path: config.json

Although there wasn't any configuration set for AWS, an error was logged. But this did not stop from
k logs  pod/kaniko--1-q4xfx
Enumerating objects: 62, done.
Counting objects: 100% (15/15), done.
Compressing objects: 100% (12/12), done.
Total 62 (delta 5), reused 12 (delta 3), pack-reused 47
INFO[0001] GET KEYCHAIN
INFO[0001] running on kubernetes ....
E0831 10:44:58.578978       1 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
        For verbose messaging see aws.Config.CredentialsChainVerboseErrors
INFO[0010] Retrieving image manifest nginx
INFO[0010] Retrieving image nginx from registry index.docker.io
INFO[0010] GET KEYCHAIN
INFO[0014] Built cross stage deps: map[]
INFO[0014] Retrieving image manifest nginx
INFO[0014] Returning cached image manifest
INFO[0014] Executing 0 build triggers
INFO[0014] Unpacking rootfs as cmd RUN echo 'This is version v0.0.1' > /usr/share/nginx/html/index.html requires it.
INFO[0029] RUN echo 'This is version v0.0.1' > /usr/share/nginx/html/index.html
INFO[0029] Taking snapshot of full filesystem...
INFO[0033] cmd: /bin/sh
INFO[0033] args: [-c echo 'This is version v0.0.1' > /usr/share/nginx/html/index.html]
INFO[0033] Running: [/bin/sh -c echo 'This is version v0.0.1' > /usr/share/nginx/html/index.html]
INFO[0033] Taking snapshot of full filesystem...
INFO[0033] GET KEYCHAIN
INFO[0033] Pushing image to codeanit/demo-cicd-go:v0.0.1
INFO[0038] Pushed image to 1 destinations

The tutorial, https://betterprogramming.pub/how-to-build-containers-in-a-kubernetes-cluster-with-kaniko-2d01cd3242a7, used FluxCD for deployment. FluxCD uses a push based GitOps, i.e. No Github webhooks are required, FluxCD pulls or checks the changes like cron jobs.
Resource

[ ] https://betterprogramming.pub/how-to-build-containers-in-a-kubernetes-cluster-with-kaniko-2d01cd3242a7
[ ] https://betterprogramming.pub/how-to-continuously-deliver-kubernetes-applications-with-flux-cd-502e4fb8ccfe
[ ] https://github.com/codeanit/til/issues/668 GitOps
[ ] https://blog.csanchez.org/2020/09/15/building-docker-images-with-kaniko-pushing-to-docker-registries/
[ ] https://devopscube.com/build-docker-image-kubernetes-pod/
[ ] https://dev.to/ankitbansal/building-custom-image-in-kubernetes-cluster-with-kaniko-11km
[ ] https://cloud.google.com/blog/products/containers-kubernetes/introducing-kaniko-build-container-images-in-kubernetes-and-google-container-builder-even-without-root-access
[ ] https://blog.alexellis.io/building-containers-without-docker/
[ ] https://medium.com/hootsuite-engineering/building-docker-images-inside-kubernetes-42c6af855f25
[ ] https://www.ibm.com/cloud/blog/build-a-container-image-inside-a-kubernetes-cluster-and-push-it-to-ibm-cloud-container-registry
[ ] https://bmuschko.com/blog/go-on-jenkins/
[ ] https://anupamgogoi.medium.com/writing-a-kubernetes-operator-from-zero-to-hero-8ca5dc2462b7
[ ] https://medium.com/artisans-of-tech/building-a-ci-system-for-go-with-jenkins-4ab04d4bacd0
[ ] https://www.jenkins.io/doc/book/getting-started/
[ ] https://bmuschko.com/blog/go-on-jenkins/
[ ] https://devopscube.com/setup-jenkins-on-kubernetes-cluster
[ ] https://plugins.jenkins.io/kubernetes/
[ ] https://betterprogramming.pub/how-to-continuously-deliver-kubernetes-applications-with-flux-cd-502e4fb8ccfe

After further reading on DevOps practices based on Kubernetes, JenkinsX seemed more relevant. Most example of service mesh included Istio, also in Google example: https://github.com/GoogleCloudPlatform/microservices-demo. So using Istio for this purpose.
Uninstall Linkerd: https://linkerd.io/2.10/tasks/uninstall, https://linkerd.io/2.10/reference/cli/uninject

To remove Linkerd Viz

linkerd viz uninstall | kubectl delete -f -

To remove Linkerd Jaeger

linkerd jaeger uninstall | kubectl delete -f -

To remove Linkerd Multicluster

linkerd multicluster uninstall | kubectl delete -f -

Uninject from pods/deploymnets

kubectl get deploy -o yaml | linkerd uninject - | kubectl apply -f -
A pod was stuck on terminating state so delete it forcefully: kubectl delete pod xxx --now/force
https://stackoverflow.com/questions/35453792/pods-stuck-in-terminating-status, https://github.com/kubernetes/kubernetes/issues/51835,
Also deleted tekton-pipelines ns as it was not useful but it was also stuck in terminating status. Tired another approach for that.
kubectl get namespace tekton-pipelines -o json > tekton-pipelines.json
Removed kubernetes value from
"spec": {
        "finalizers": [
        ]
    },

And reapplied:kubectl replace --raw "/api/v1/namespaces/tekton-pipelines/finalize" -f tekton-pipelines.json

[ ] https://tech.ebayinc.com/research/kubernetes-secrets-a-secure-credential-store-for-jenkins, https://github.com/eBay/kube-credentials-plugin
[ ] https://tech.ebayinc.com/engineering/how-ebay-leverages-kubernetes-helm-charts-and-jenkins-pipelines-to-deliver-high-quality-software
[ ] https://tech.ebayinc.com/engineering/grit-a-protocol-for-distributed-transactions-across-microservices
[ ] https://itnext.io/k8s-native-jenkins-x-and-tekton-pipelines-e2b5a61a1d22, https://itnext.io/jenkins-x-istio-flagger-canary-deployment-9d5e187c2334

Flagger https://docs.flagger.app
Flagger is a progressive delivery tool that automates the release process for applications running on Kubernetes. It reduces the risk of introducing a new software version in production by gradually shifting traffic to the new version while measuring metrics and running conformance tests.
Flagger implements several deployment strategies (Canary releases, A/B testing, Blue/Green mirroring) using a service mesh (App Mesh, Istio, Linkerd, Open Service Mesh) or an ingress controller (Contour, Gloo, NGINX, Skipper, Traefik) for traffic routing. For release analysis, Flagger can query Prometheus, Datadog, New Relic, CloudWatch or Graphite and for alerting it uses Slack, MS Teams, Discord and Rocket.
Flagger can be configured with Kubernetes custom resources and is compatible with any CI/CD solutions made for Kubernetes. Since Flagger is declarative and reacts to Kubernetes events, it can be used in GitOps pipelines together with tools like Flux, JenkinsX, Carvel, Argo, etc.

State
The laptops with K8s installed were turned off. Kubelet was not active probably due to swapp was on. Re-instantiated Kubelet by executing shell script:
https://github.com/SystemUtilities/utils/blob/main/k8s-init.sh
Kubelet active. Node joined the cluster.
Resource

[ ] https://phoenixnap.com/kb/how-to-install-kubernetes-on-a-bare-metal-server

Note: This issue is has become too long. Need to make a note from this and close it.

Setting up systems with primary domain and subdomains

https://blog.logrocket.com/how-to-build-web-app-with-multiple-subdomains-nginx
https://srimalfernando.medium.com/how-to-host-multiple-servers-with-different-sub-domains-inside-one-digitalocean-droplet-f6d5887538fa


Update 01-Quick Start.md docker compose

Adds Network_MODE: host which enables python wifi connect to work out of the box :)

This issue has been suggested as a fix twice now, so I believe that adding it to the docs would fix future questions.
Past issues:

https://github.com/balena-labs-research/starter-interface/discussions/138
https://github.com/balena-labs-research/python-wifi-connect/issues/6

Ah sorry to hear that @maggie44!
Does anyone from the balena team know who owns this now?

LGTM


Preload with union query

I am raising this PR to illustrate a scenario that involves preloading and union or intersect queries that I can't quite figure out.
The CI will fail with the error below.
In this PR, I've got two integration tests.  Each test builds the same query on comments which is the union of two other queries:
q1 = from c in Comment, where: c.text == "a"
q2 = from c in Comment, where: c.text == "b"
query = union(q1, ^q2)

When I use this query to fetch all comments that match, using Repo.all(query), I get an empty list, as expected. This means the query was properly accepted by the database and did execute successfully.
However, if I use the same query to preload a specific post with some comments, using Repo.preload([p1], [{:comments, query}], []), then I get the error:
** (Postgrex.Error) ERROR 42601 (syntax_error) each UNION query must have the same number of columns

and if I take a look at the generated sql:
query: SELECT c0."id", c0."text", c0."lock_version", c0."post_id", c0."author_id", c0."post_id" FROM "comments" AS c0 WHERE (c0."text" = 'a') AND (c0."post_id" = $1) UNION (SELECT c0."id", c0."text", c0."lock_version", c0."post_id", c0."author_id" FROM "comments" AS c0 WHERE (c0."text" = 'b')) ORDER BY c0."post_id"

I can see indeed there is a difference in the number of columns selected by the two queries. The first query is selecting c0.post_id twice (not expected?), while the second is selecting it just once (expected).
The same happens with intersect queries. It is highly possible that I am abusing the preloader, or maybe this is expected, but I was curious to know why this behaviour?
And most importantly, is there a way I could workaround it ? :)
Thanks in advance!

There are for sure a couple issues when using combinations to preload, but I'm not sure it makes sense to use them in the first place. Here is my thinking:
A preload is done against a single table. There isn't a need to use intersections on a single table because it can be done using filters. Unions correspond to OR, intersections to AND, set differences AND NOT.
If ^ makes sense, maybe raising when a preload query is a combination is the way to go.

Hey thank you for your response!
Yeah, rewriting my query also crossed my mind, but I am not sure this would be doable in 100% of the cases.
Also came across https://github.com/elixir-ecto/ecto/issues/3839 which hits the same problem. The workaround described there works well:
q1 = from c in Comment, where: c.text == "a"
fields = Comment.__schema__(:fields) ++ [:post_id]
q2 = from c in Comment, select: map(c, fields), where: c.text == "b"
query = union(q1, ^q2)

Oh I'm surprised that works. Preloading a has_many association will also try to order the preload query, which I thought would break with a union. When you run this test it passes?
test "should preload comments into posts" do
      p1 = TestRepo.insert!(%Post{title: "p1"})
      TestRepo.insert!(%Comment{text: "c1", post: p1})

      q1 = from c in Comment, where: c.text == "a"
      fields = Comment.__schema__(:fields) ++ [:post_id]
      q2 = from c in Comment, select: map(c, ^fields), where: c.text == "b"
      query = union(q1, ^q2)

      p1 = TestRepo.preload([p1], [{:comments, query}], [])

      assert [] == p1.comments
    end

Actually, you are right.
The new error I get is:
** (Postgrex.Error) ERROR 42P01 (undefined_table) missing FROM-clause entry for table "c0"

 query: SELECT c0."id", c0."text", c0."lock_version", c0."post_id", c0."author_id", c0."post_id" FROM "comments" AS c0 WHERE (c0."text" = 'a') AND (c0."post_id" = $1) UNION (SELECT c0."id", c0."text", c0."lock_version", c0."post_id", c0."author_id", c0."post_id" FROM "comments" AS c0 WHERE (c0."text" = 'b')) ORDER BY c0."post_id"

I had tested this with a belongs_to relationship, and I assumed that would also work with has_many.
Good catch!

I just remembered you can wrap it a subquery to make both issues go away.
q1 = from c in Comment, where: c.text == "a"
q2 = from c in Comment, where: c.text == "b"
query = from s in subquery(union(q1, ^q2))

[p1] = TestRepo.preload([p1], [{:comments, query}], [])

But it's still a bit dicey because you have to make sure the foreign key is in all the union queries otherwise the subquery won't be able to select it. So the logic that adds the field to ensure it's there is kind of neutered. All this is also pretty unintuitive unless you've studied the Ecto code or have had it explained to you.
I think there are 2 options:

Ecto handles combinations with special logic.
Ecto detects combinations in preloads and raises in error suggesting to wrap it in a subquery

I'm still pretty skeptical that combinations are needed for preloads so leaning towards 2. @josevalim Do you have a preference?

I would go with 2 too!

Thank you guys for the insight !

Thanks for the report @pedro-gutierrez! Are you interested in submitting the fix? If not I can do it so no worries.

Thanks for the report @pedro-gutierrez! Are you interested in submitting the fix? If not I can do it so no worries.

Hey I'd love to contribute, but I also feel you're gonna spend a lot less time than me in this, also cos you really know the library inside out, and I don't.
We can't close this PR too.


Questions about evaluation performance

Thank you for the active sharing of your valuable research.
I am deeply impressed by your research and tested a few things, but I have some questions.

I am curious about which configuration you used to obtain the DS values you reported.
https://github.com/opendilab/LMDrive?tab=readme-ov-file#lmdrive-weights

Here are my configurations:
leaderboard/teamcode/lmdrive_config.py
import os

class GlobalConfig:
    """base architecture configurations"""

    # Controller
    turn_KP = 1.25
    turn_KI = 0.75
    turn_KD = 0.3
    turn_n = 40  # buffer size

    speed_KP = 5.0
    speed_KI = 0.5
    speed_KD = 1.0
    speed_n = 40  # buffer size

    max_throttle = 0.75  # upper limit on throttle signal value in dataset
    brake_speed = 0.1  # desired speed below which brake is triggered
    brake_ratio = 1.1  # ratio of speed to desired speed at which brake is triggered
    clip_delta = 0.35  # maximum change in speed input to logitudinal controller

    llm_model = 'weights/llava-v1.5-7b'
    preception_model = 'memfuser_baseline_e1d3_return_feature'
    preception_model_ckpt = 'weights/LMDrive-vision-encoder-r50-v1.0/vision-encoder-r50.pth.tar'
    lmdrive_ckpt = 'weights/LMDrive-llava-v1.5-7b-v1.0/llava-v1.5-checkpoint.pth'

    agent_use_notice = False # True
    sample_rate = 2

    def __init__(self, **kwargs):
        for k, v in kwargs.items():
            setattr(self, k, v)

leaderboard/scripts/run_evaludation.sh
export ROUTES=langauto/benchmark_long.xml
export TEAM_AGENT=leaderboard/team_code/lmdriver_agent.py # agent
export TEAM_CONFIG=leaderboard/team_code/lmdriver_config.py # model checkpoint, not required for expert
export CHECKPOINT_ENDPOINT=results/sample_result.json # results file
#export SCENARIOS=leaderboard/data/scenarios/no_scenarios.json #town05_all_scenarios.json
export SCENARIOS=leaderboard/data/official/all_towns_traffic_scenarios_public.json

Are these configurations accurate? If these configurations are incorrect, then the following questions may not be necessary.

The results of the evaluation using above settings are as follows. However, a higher performance was achieved when agent_use_notice = False. What could be the reason for this? Also, the DS (LangAuto) reported by you was obtained from the "score_route" value. Does DS (LangAuto) not consider infraction penalties?

agent_use_notice = True
"scores": {
                "score_composed": 22.576320522257177,
                "score_penalty": 0.7903388993546836,
                "score_route": 27.799761292913487
},

agent_use_notice = False
"scores": {
                "score_composed": 29.6279197611852,
                "score_penalty": 0.8052244697640636,
                "score_route": 36.35275911099924
},

Hi!
I have attached the log file for your reference.
long_vicuna2_with_notice_t1.json
Could you run it with more times? the variance of the performance scores is large. Also, you can check your Carla version and the start command. If you still encounter the problem, I will check the agent code. Thanks for your attention.

Same here, I cannot reproduce reported results either. Could you please give me some suggestions for this? Thanks for your sincere help!
lmdrive_config.py
import os

class GlobalConfig:
    """base architecture configurations"""

    # Controller
    turn_KP = 1.25
    turn_KI = 0.75
    turn_KD = 0.3
    turn_n = 40  # buffer size

    speed_KP = 5.0
    speed_KI = 0.5
    speed_KD = 1.0
    speed_n = 40  # buffer size

    max_throttle = 0.75  # upper limit on throttle signal value in dataset
    brake_speed = 0.1  # desired speed below which brake is triggered
    brake_ratio = 1.1  # ratio of speed to desired speed at which brake is triggered
    clip_delta = 0.35  # maximum change in speed input to logitudinal controller

    llm_model = './huggingface/models--liuhaotian--llava-v1.5-7b/snapshots/12e054b30e8e061f423c7264bc97d4248232e965/'
    preception_model = 'memfuser_baseline_e1d3_return_feature'
    preception_model_ckpt = 'models/vision-encoder-r50.pth.tar'
    lmdrive_ckpt = 'models/llava-v1.5-checkpoint.pth'

    agent_use_notice = True
    sample_rate = 2

    def __init__(self, **kwargs):
        for k, v in kwargs.items():
            setattr(self, k, v)

run_evaluation.sh
export LEADERBOARD_ROOT=leaderboard
export CHALLENGE_TRACK_CODENAME=SENSORS
export PORT=$PT # same as the carla server port
export TM_PORT=$(($PT+500)) # port for traffic manager, required when spawning multiple servers/clients
export DEBUG_CHALLENGE=0
export REPETITIONS=1 # multiple evaluation runs
export ROUTES=langauto/benchmark_long.xml
export TEAM_AGENT=leaderboard/team_code/lmdriver_agent.py # agent
export TEAM_CONFIG=leaderboard/team_code/lmdriver_config.py # model checkpoint, not required for expert
export CHECKPOINT_ENDPOINT=results/sample_result.json # results file
#export SCENARIOS=leaderboard/data/scenarios/no_scenarios.json #town05_all_scenarios.json
export SCENARIOS=leaderboard/data/official/all_towns_traffic_scenarios_public.json
export SAVE_PATH=data/eval # path for saving episodes while evaluating
export RESUME=False

Final results, agent_use_notice=True
"scores": {
                "score_composed": 25.870979763631585,
                "score_penalty": 0.7629596874513518,
                "score_route": 35.450098903630824
            },

agent_use_notice=False
"scores": {
                "score_composed": 26.58510262333422,
                "score_penalty": 0.7749396906256292,
                "score_route": 35.97223471729377
            },

By the way, the results look similar after multiple times evaluation, the variance is acceptable.

I encountered the same issue. I tried to reproduce the experimental results using the official code and model weights on an RTX 4090. The CARLA version was <IP_ADDRESS>, but the results I obtained were lower than those reported in the paper. I got DS=25.044, RC=36.526.


How to continue training from previous checkpoints?

Hi guys I'm new to nlp and summarization in particular. I'm running the BERT+RNN model on Google Colab. I understand that the param "-save_checkpoint_steps 1000" in the training script helps save the model every 1000 train steps. However I don't know how to continue from that checkpoint (All I know is to run the training script: "python train.py ....." and each time, it starts from the very beginning). In detail, I have mounted my google drive to the notebook and the checkpoints are saved as in the below image. Can someone tell me how to do that please (Is there a param I should add to the script to make it work?). Many thanks in advance!

use the paramater args.train_from

use the paramater args.train_from

Yeah I just found out this afternoon :v Thanks anw!

I want to use this model but I'm new to this and not sure about whether my loss trend is correct, can we talk?

use the paramater args.train_from

Yeah I just found out this afternoon :v Thanks anw!

I want to use this model but I'm new to this and not sure about whether my loss trend is correct, can we talk?

I'm a newbie too, man. I'm just attempting to complete a short course at my school and I'm currently not so interested in this field.


groups are not idempotent when multiple classes are used

Migrated from: https://github.com/puppetlabs/prosvcs-node_manager/issues/46
Because the classes are expressed in a hash puppet will compare the hash as a whole instead of doing a deep inspection of the items that might have changed in the hash.  So each puppet run is no longer idempotent because the order of the hash is always different thus triggering a change.
We will need to inspect the change_to_hash and the current_hash for the classes and args so that the data is inspected and not just the order of contents.
(hash1.to_a - hash2.to_a).empty?  -- possible solution ?
The PE puppet master group is a good example case since it comes with many classes by default.

The way the provider compares current and desired.  Unfortunately, I'm not sure of a way to tell it to compare other than ==.  So what I've tried to do is tell the RAL and the DSL to deep-sort the classes Hash, so at least they appear equal.  Added this method:
  def self.sort_hash(data)
    newhash = Hash.new
    if data.is_a?(Hash)
      # .to_h method doesn't exist until Ruby 2.1.x
      data.sort.flatten(1).each_slice(2) { |a,b| newhash[a] = b }
    end
    newhash.each do |k,v|
      if v.is_a?(Hash)
        newhash[k] = sort_hash(v)
      else
        newhash[k] = v
      end
    end
    newhash
  end

It looks much more elegant with the to_h method, but that's not available in Ruby 2.0.0.  To so favor compatibility with PE 3.8.x, wrote a substitute.

Merged https://github.com/WhatsARanjit/prosvcs-node_manager/commit/ee3b4a80b11f119a43d87609ebed32cd772ba4d7


Fix database error handling in malfeasance syncer

Motivation
Database errors were being swallowed in the malfeasance syncer
Description
Fix error handling

bors merge


Add support for deployment to Kubernetes

Add description @aguschin

closed by https://github.com/iterative/mlem/pull/374


@OneToOne not cascade delete

model Consignment:
    @OneToOne(cascade = {CascadeAction.SAVE})
    public RedirectIneligibility redirectIneligibility;

model RedirectIneligibility
@OneToOne(mappedBy = "redirectIneligibility", cascade = {CascadeAction.DELETE, CascadeAction.SAVE})
    public Consignment consignment;

insertion works fine. However when I call
dataStore.delete(Consignment.class)

only consignment table gets cleared, not RedirectIneligibility.

dataStore.delete(Consignment.class) only deletes all of Consignment types. The other types would only get deleted if they had a cascading foreign key, or you deleted an entity individually e.g. dataStore.delete(consignment). You can mark one side with @ForeignKey to define the side with the foreign key. Hope that helps.

@npurushe Thanks. It works now.


Allow invoke to send requests to a different server address

Added invoke_host and ipv4 validation as a util.

I think this needs another attempt. Parsing looks more consistent if all errors are from miette. Otherwise different errors show up in different formatting.


feat: Add support for pnpm-lock.yaml

Before
Only supports shrinkwrap.yaml for pnpm <=2
After
Supports pnpm-lock.yaml for pnpm >=3
Details
https://github.com/pnpm/pnpm/issues/1635

v0.4.3 release is cut and published, thanks for the contribution!


[docs] Fix hooks codesandbox broken

Fix hooks codesandbox and resolves issue https://github.com/mui-org/material-ui/issues/14491

[x] I have followed (at least) the PR section of the contributing guide.

Closes #14491

@Abbo44 It's a great first pull request on Material-UI 👌🏻. Thank you for working on it!

I'm applying the same fix on master.


[Feature Request] loadmore component add move event

I need to update ui when the loadmore move end or start.
For example, when a user starts the move, fade out some div.
events:

movestart
moveend

PR welcome

get it, i will do it.


Paginate posts which are displayed with Ajax

In my rails app, I am rendering posts with ajax:
<%= link_to explore_posts_path, remote: true %>

In my controller:
def explore
@posts = Post.all
respond_to do |format|
format.js
end
end
In explore.js.erb:
$('.explore-wrapper').append('\

\
		<%= j render @posts %>\

\
  ');

However, I'd like to paginate those posts with Kaminari. I managed to have Kaminari to work in my app on other pages, but posts weren't displayed with AJAX, they were displayed on page load.
I changed my explore.js.erb file:
$('.explore-wrapper').append('\

\
		<%= j render @posts %>\
		<%= paginate @posts %>\

\
  ');

But now there aren't any posts displayed, so it "broke" the callback.
How can I display with AJAX paginated posts ?

Thank you for your question. However, we only accept bug reports and pull requests on the GitHub issues. If you have a support question about how to use kaminari, please ask on StackOverflow.


Resolves #214: Restrict some characters from node names

Solves issue #214
Added name-validating functions in /src/mount/linux.rs and /src/encryptedfs.rs.
Use said functions in multiple methods to check for '/' and '\' characters.

@Ropler6 any updates on this?

Will start working on it tomorrow


IPEX changes the return type of the API sycl_queue and sycl_device. Triton XPU backend need to align these changes.

The IPEX changes the sycl_queue return type at commit.
https://github.com/intel-innersource/frameworks.ai.pytorch.ipex-gpu/commit/6182b5eba7b46230fe2631677dc8aba13703389e
Changes the sycl_device return type at commit.
https://github.com/intel-innersource/frameworks.ai.pytorch.ipex-gpu/commit/f65c45e75fa52209dc174a6b56e87e107c702371
Both returns the capsule of the sycl::queue and sycl::device pointer refer to the corresponding object inside IPEX internally.
The capsule names are torch.xpu.Stream.sycl_queue and torch.xpu.device.sycl_device.
The Triton XPU backend runtime need to align the new API in IPEX to support torch DL complier use cases.

@chengjunlu we already use PyCapsule_GetPointer(cap, PyCapsule_GetName(cap)), not hardcode name. https://github.com/intel/intel-xpu-backend-for-triton/blob/5a0d78f36a3a941fcbb92a195774b61a574677ed/third_party/intel/backend/driver.c#L185

The merged PR.
https://github.com/intel/intel-xpu-backend-for-triton/pull/239

Fixed by https://github.com/intel/intel-xpu-backend-for-triton/pull/239


chore: update labels to remove ec policy warnning

Update the DOCKER file to align with the latest requirement and update the service name

typo in commit title. s/warnning/warning/


Fix typo in GCP PSC datasource documentation

Description
Typo fix in GCP PSC datasource documentation. It currently incorrectly references Azure.
Related Issues
N/A
Motivation and Context
It's a typo.
How Has This Been Tested?
N/A
Types of Changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Refactoring (improves code quality but has no user-facing effect)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[x] Documentation

Readiness Checklist

[x] My code follows the code style of this project
[x] My change requires a change to the documentation
[x] I have updated the documentation accordingly
[ ] I have added tests to cover my changes
[ ] All new and existing tests passed

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

jenkins test this

Hi @tobio @alaudazzi ! Thanks for the review.
The CLA was signed by my employer, but the status does not seem to be automatically updated. Is there anything I'm still missing?


Unauthorised API Access, Account Not PCI Certified

get error when try to payment with live credentials

Hi @harshad26
This usually occurs when you use the Direct method to send un-encrypted card information. If your system is PCI certified you can contact eWAY to allow this method for your account. Otherwise you can make use of Client Side Encryption to protect the card data, or use an alternate Rapid method.


make repeating "inject" nodes and non-repeating "inject" nodes visually distinct

Does that make sense?  Some are just for debugging; others are for cron-like functionality

Please use the mailing list for feature suggestions/discussions

Have thrown this together, but not 100% sold on it yet.

Uses a unicode symbol in the label to indicate it's a repeating node - this was the best symbol I could find. Currently doesn't show it if you set the name field - but it could...
Will chew it over some more.

Have put the icon on the right, and included it even if you've set the name field.
Pushed to the 0.14.0 branch - this issue will get closed when we merge to master for the release.

Change of plan - closing now so I don't keep thinking there is more work to do on this

@knolleary Thanks!


Improper escaping in JSON output

Running gas on the following file produces invalid JSON output:
package main

func main() {
	password := "repro for escaping bug < > & ;"
}

Produces:
$ gas -fmt json main.go
{
        "metrics": {
            "files": 1,
            "lines": 5,
            "nosec": 0,
            "issues": 1
        },
        "issues": [
        {
          "file": "main.go",
          "line": "4",
          "details": "Potential hardcoded credentials",
          "confidence": "LOW",
          "severity": "HIGH",
          "code": "password := \"repro for escaping bug \x3C \x3E & ;\""
        }
        ]
}

Note that < and > are escaped in a way that's not valid JSON.
$ gas -fmt json main.go | jq .
parse error: Invalid escape at line 15, column 72

good spot, thanks @csstaub 👍

lgtm - closing


libproj9 on ubuntu 15.10

deps needs updating on ubuntu15.10, looks like the proj package has been renamed libproj9 (?!)

Now that we have a from-source version this should be obsolete.


Setting a stl_customTabTextLayoutId or stl_defaultTabBackground hides the indicator

Hi
If I set stl_defaultTabBackground to any color, say Blue, it draws on top of the indicator bar.
Same way if I apply a stl_customTabTextLayoutId and set background of textview to blue color, that draws on top of indicator. In all these cases the indicator gets hidden. It looks like these properties are painted after painting the tab indicator which ends up hiding the indicator
Is this a known issue?
Thanks
Vibhor

Hi, @vibhorB
Oh,,, sorry.
The current version is to be able to specify the background for touch feedback.
Background do you change for each tab?
If the same background, please specify in the background of the TabLayout.

Hi @ogaclejapan
I'm having the same issue. I set stl_defaultTabBackground to a selector for touch feedback app:stl_defaultTabBackground="@drawable/bg_selector", and the indicator still gets hidden.
Here's my bg_selector:
<?xml version="1.0" encoding="utf-8"?>
<selector xmlns:android="http://schemas.android.com/apk/res/android">
    <item android:drawable="@color/blue" android:state_pressed="true"/>
    <item android:drawable="@color/red"/>
</selector>

Is there something missing?
In my case, specify the background of the TabLayout isn't something I need. CMIIW, it won't work for touch feedback anyway.
Thanks

Hi again @ogaclejapan
So, I managed to fix my problem. I changed my bg_selector:
<?xml version="1.0" encoding="utf-8"?>
<selector xmlns:android="http://schemas.android.com/apk/res/android">
    <item android:state_pressed="true" android:state_selected="true">
        <layer-list>
            <item android:bottom="5dp" android:drawable="@color/blue"/>
        </layer-list>
    </item>
    <item android:state_selected="true">
        <layer-list>
            <item android:bottom="5dp" android:drawable="@color/red"/>
        </layer-list>
    </item>
    <item android:state_pressed="true">
        <layer-list>
            <item android:bottom="1dp" android:drawable="@color/blue"/>
        </layer-list>
    </item>
    <item>
        <layer-list>
            <item android:bottom="1dp" android:drawable="@color/red"/>
        </layer-list>
    </item>
</selector>

Well, that's not an ideal solution, but at least fix my problem for now. The idea is to put offset at the bottom in order to make the bottom indicator visible. I have app:stl_underlineThickness="1dp" and app:stl_indicatorThickness="5dp".
Anyway, I suspect the root cause is in SmartTabLayout#populateTabStrip, specifically . If the tabView (which is TextView) background was set to specific color, it would drawn above the tabStrip, so the tabStrip's indicator would never visible. Maybe you should specify margin for the tabView.
Really hope you could fix this issue with the best solution.
Thanks

Hi, @vibhorB @apradanas
Right now, 1.5.0 released : )
stl_drawDecorationAfterTab="true"

So, this issue it will close.


Extract IPC ArrayReader struct

Which issue does this PR close?

Relates to #4255
Rationale for this change

Whilst investigating #4255 I realised this code has naturally grown to the point of it now being quite hard to follow. This is the first of what will probably be a series of PRs to
What changes are included in this PR?

Are there any user-facing changes?

I'll review this soon.


GTM - Copy changes from old repo

Leaving in draft until we can review further

These GTM changes were deployed to the sandbox environment to ensure it was communicating with Google.


Sharing models through Hugging Face

Hi there! Congrats on the launch!
I see you currently save your model checkpoints through a link to a FB server (https://dl.fbaipublicfiles.com/fair-esm/models/esm2_t48_15B_UR50D.pt) . Would you be interested in sharing the model in the Hugging Face Hub? There's an existing FaceBook org with over 300 models where these could be hosted.
The Hub offers free hosting of over 100K models, and it would make your work more accessible and visible to the rest of the ML community. Creating the repos and adding new models should be relatively straightforward if you've used Git before. This is a step-by-step guide explaining the process in case you're interested. Please let us know if you would be interested, and if you have any questions, we're happy to assist.
Happy to hear your thoughts!
cc @NimaBoscarino

We're working now with the huggingface team to get ESM-1x and ESM-2 models into 🤗 transformers. I guess then it'll be no Hub automatically?
For ESMFold, is Transformers a better place or Hub?

I missed we had an ongoing collab :hugs: Once this is supported in transformers, it will indeed be in the Hub automatically, and be much better supported, so let's keep going that way :rocket:

Hey, this would be awesome to have! Thank you so much for taking this on 🚀. Do you think it's possible to give a broad timeline by when the models could be available? 😄

if esmFold can be released soon, i think many kagglers are eager to use it at protein stability prediction:
Novozymes Enzyme Stability Prediction: Help identify the thermostable mutations in enzymes
https://www.kaggle.com/competitions/novozymes-enzyme-stability-prediction

Very happy to share that - thanks to the amazing work of @Rocketknight1 and 🤗 team - this is now supported:

Classification tasks with proteins, just like BERT: https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/protein_language_modeling.ipynb
Fold proteins in Colab or your local GPU and export PDB files: https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/protein_folding.ipynb


Clean up a bunch of unused stuff in e2e

Remove all the various e2e local setups, and document the one true local setup: kind. There isn't much reason to keep around docs to tell people how to run in environments different than how CI runs. Additionally, cleanup an unused tool.

@howardjohn: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

lint_istio
eec0ca862c207758ca000e5f8bdfa6f90a36a03c
link
/test lint_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@howardjohn: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

pilot-multicluster-e2e_istio
c33362485f89aa51d49fa44150c1c7c938ff6fb9
link
/test pilot-multicluster-e2e_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest

@howardjohn: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

lint_istio
c33362485f89aa51d49fa44150c1c7c938ff6fb9
link
/test lint_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest


Job Queue Datetime bug when using the MariaDB Database driver

Laravel Version
11.0.5
PHP Version
8.3.3
Database Driver & Version
MariaDB 10.11.2 on Ubuntu 22.04 LTS with the MariaDB driver
Description
SQLSTATE[22007]: Invalid datetime format: 1292 Incorrect datetime value: '1710393058' for column laravel_test.jobs.available_at at row 1
I get the error mentioned above when running jobs on the PHP 8.3.3. The database I use has not been changed from the one migrated via Laravel 7 with the MySQL driver.

Column
DataType
Null
Default

reserved_at
timestamp
YES
NULL

available_at
timestamp
NO
0000-00-00 00:00:00

created_at
timestamp
NO
current_timestamp()

Steps To Reproduce

Run migrations for the jobs and job_batches tables from an existing Laravel 7 or earlier application.
Upgrade sequentially to Laravel 11
Switch to the new mariadb driver.
Attempt to push a job to the queue (from any source, controller or artisan command etc.) while using the database as the QUEUE_DRIVER.

Hi @apoorvpal01,
Did you upgrade from Laravel 7 to Laravel 11 just now or have you been running Laravel 10 for a while?

Hi @staudenmeir
I have been running on Laravel 10 for the past year without any issues, with the same config (except the DB driver change)
I switched back to the mysql driver on Laravel 11 and it worked fine. The problem occurs only on the mariadb driver.

cc @hafezdivandari

@apoorvpal01 It's maybe related to strict mode, can you try mariadb driver with strict => false?

@hafezdivandari You're right. Disabling the strict mode fixed it. If this is the default requirement for queues, is it in the docs somewhere? Did I miss it? If not, maybe I could add a PR for this to be included in the docs for jobs.

@apoorvpal01 actually it's not about strict mode. The problem is that you're using an incompatible jobs table structure, the available_at column is not a timestamp but unsignedInteger (even in Laravel 7).
Laravel 7:
https://github.com/laravel/framework/blob/7.x/src/Illuminate/Queue/Console/stubs/jobs.stub#L21-L23
Laravel 11:
https://github.com/laravel/framework/blob/11.x/src/Illuminate/Queue/Console/stubs/jobs.stub#L19-L21

The disabled strict mode just hid the error from you all this time.

Oh, thanks for the clarification. I am not sure how I got this schema on the database then. I'll check that out. Sorry for the trouble, and thanks for your help!


Cleanup: Rename IOBufferReader of Http2ClientSession

No logic change. Just renaming 2 IOBufferReader(s).
-  IOBufferReader *_reader             = nullptr;
+  IOBufferReader *_read_buffer_reader = nullptr;

-  IOBufferReader *sm_writer           = nullptr;
+  IOBufferReader *_write_buffer_reader = nullptr;

Cherry-picked to v9.1.x branch.


Add support for eslint flat config

https://eslint.org/docs/latest/use/configure/configuration-files-new
https://eslint.org/docs/latest/use/configure/migration-guide#start-using-flat-config-files

Hi @TheElegantCoding, Thanks for the report!. Is there anything else we need to do to support flat config?

thanks that work and also i find an error in my config, in order to work inside all the repo i have to put files: [ '**/*.html' ]
Other problem i have with eslint flat config is the type in typescript my config is in typescript
import { htmlDisabledRule } from '@module/html/rule/html_disabled_rule';
import { htmlGeneralRule } from '@module/html/rule/html_general_rule';
import pluginHtml from "@html-eslint/eslint-plugin";
import parserHtml from "@html-eslint/parser";

const html =
{
  files: [ '**/*.html' ],
  languageOptions:
  {
    parser: parserHtml
  },
  plugins:
  {
    html: pluginHtml
  },
  rules: {
    ...htmlGeneralRule,
    ...htmlDisabledRule
  }
};
export { html };

in the pluginHtml import i get this error Could not find a declaration file for module '@html-eslint/eslint-plugin'.

in the pluginHtml import i get this error Could not find a declaration file for module '@html-eslint/eslint-plugin'.

@TheElegantCoding Thank you. I think this should be fixed..!

@yeonjuan  Dear sir, I got this warning when import recommended setting.

Hi @TheElegantCoding, Thanks for the report!. Is there anything else we need to do to support flat config?
I've tested the config below and it seems to be working fine, did you run into any issues?
const Plugin = require("@html-eslint/eslint-plugin");
const Parser = require("@html-eslint/parser");

module.exports = [
  {
    plugins: { "@html-eslint": Plugin },
    languageOptions: {
      parser: Parser,
    },
    files: ["*.html"],
    rules: {
      //...
    },
  },
];

Eslint still working fine even error output.

Please help me 😢

Hi @LatenPath , Could you try pluginHtml.configs.recommended.rules instead of pluginHtml.configs.recommended?
const pluginHtml = require("@html-eslint/eslint-plugin");
const parserHtml = require("@html-eslint/parser");

module.exports = [
  {
    files: ["**/*.html"],
    languageOptions: {
      parser: parserHtml,
    },
    plugins: {
      "@html-eslint": pluginHtml,
    },
    rules: pluginHtml.configs.recommended.rules,
  },
];

@yeonjuan It's work, sir!

Disable rule/ Auto-fix,... also work fine!

Thank you so much!

Hi @TheElegantCoding @LatenPath
new version is released. html-eslint supports flat config now! Thanks for the reporting issue!

https://html-eslint.org/docs/getting-started#flat-config


Issue with babel-preset-react-native plugin on v0.56

Environment
React Native Environment Info:
System:
OS: macOS Sierra 10.12.6
CPU: x64 Intel(R) Core(TM) i5 CPU       M 540  @ 2.53GHz
Memory: 302.29 MB / 8.00 GB
Shell: 3.2.57 - /bin/bash
Binaries:
Node: 9.9.0 - /usr/local/bin/node
Yarn: 1.3.2 - /usr/local/bin/yarn
npm: 5.6.0 - /usr/local/bin/npm
Watchman: 4.7.0 - /usr/local/bin/watchman
SDKs:
iOS SDK:
Platforms: iOS 11.2, macOS 10.13, tvOS 11.2, watchOS 4.2
Android SDK:
Build Tools: 23.0.1, 23.0.2, 23.0.3, 24.0.0, 25.0.0, 25.0.2, 25.0.3,26.0.0, 26.0.1, 26.0.2, 27.0.0, 27.0.1, 27.0.2, 27.0.3
API Levels: 23, 24, 25, 26, 27
IDEs:
Xcode: 9.2/9C40b - /usr/bin/xcodebuild
npmPackages:
react: 16.4.1 => 16.4.1
react-native: 0.56.0 => 0.56.0
npmGlobalPackages:
react-native-cli: 2.0.1
react-native-git-upgrade: 0.2.7
react-native-rename: 2.2.2
Description
I have a babel-preset-react-native depedency in my package.json. I am using v0.56 of react native.
Issue
If I use v5.0.0 of babel-preset-react-native the jest cases do not run and throw an error as follows  :-
Cannot read property 'throwIfClosureRequired' of undefined (Wh
ile processing preset: "/Users/harkirat/wealthy/thrift/node_modules/babel-pre
set-react-native/index.js")

If I use v4.0.0 of babel-preset-react-native the react native application does not start as seen in https://github.com/facebook/react-native/issues/20042
Can someone please help me resolving this?

For issues related to jest/testing refer to #19859

thanks a lot @kelset


Movie app overview's overflow is cut-off

As you can see in the above Screenshot, the overview of the hovered movie is pretty large and thus needs to overflow. But it's hidden within the card.

I'd also like to align the cards to center so that it doesn't look like this


sim: fix code

What was broken because of this, just the formatting?

yup, nothing was broken, this is just formatting.


Test platform: Remove unused strings.h inclusion

The header was not used and not cross platform =>Remove it.

This pull request is now managed with CodeReviewHub. Refresh.

:+1: @dawagner @OznOg

:shipit:

:shipit:


feat(caa upload): add AllMusic provider

Closes #31
We're extractign the covers from a JSON object embedded in the HTML source. No types though, unfortunately.
Depends on #209 for the new shared behaviours in tests.

fantastic, thank you! :)