eBPF-based host-routing and tunnel mode

while reading https://cilium.io/blog/2021/05/11/cni-benchmark and https://docs.cilium.io/en/latest/operations/performance/tuning/#ebpf-host-routing
my understanding is that host routing also support tunnel mode and packet egress from container, and encapsulated overlay packet   should skip the netfilter/iptables processing, but I still see the overlay packet to destination VXLAN port 8472 seems still processed by netfilter chains, maybe my understanding is incorrect or my observation is not accurate, please correct me if I am wrong.
Here is my test:
two nodes runs cilium in tunnel mode,  kubeproxy strict
cilium status to show host routing BPF

kubectl  exec -it $CA2 -n kube-system -- cilium status
KVStore:                Ok   Disabled
Kubernetes:             Ok   1.20 (v1.20.2) [linux/amd64]
Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement:   Strict   [ens192 <IP_ADDRESS> (Direct Routing)]
Cilium:                 Ok   1.10.0-rc2 (v1.10.0-rc2-d65e51a)
NodeMonitor:            Listening for events on 4 CPUs with 64x4096 of shared memory
IPAM:                   IPv4: 3/254 allocated from <IP_ADDRESS>/24,
BandwidthManager:       EDT with BPF   [ens192]
Host Routing:           BPF
Masquerading:           BPF   [ens192]   <IP_ADDRESS>/24 [IPv4: Enabled, IPv6: Disabled]
Controller Status:      23/23 healthy
Proxy Status:           No managed proxy redirect
Hubble:                 Ok   Current/Max Flows: 4095/4095 (100.00%), Flows/s: 7.21   Metrics: Disabled
Encryption:             Disabled
Cluster health:               Warning   cilium-health daemon unreachable

from busybox pod to ping a remote nginx pod,

root@cilium-bigip:/home/vincent# kubectl  get po -o wide
NAME                             READY   STATUS    RESTARTS   AGE   IP           NODE               NOMINATED NODE   READINESS GATES
busybox-worker-ccb4ffc6c-ww8hd   1/1     Running   0          15m   <IP_ADDRESS>    cilium-fl-worker
nginx-master-6698c4ccdc-x6vw5    1/1     Running   0          15m   <IP_ADDRESS>   cilium-bigip
root@cilium-bigip:/home/vincent# kubectl exec -it $BUSYBOX -- /bin/sh
/ # ping -c 1  <IP_ADDRESS>
PING <IP_ADDRESS> (<IP_ADDRESS>): 56 data bytes
64 bytes from <IP_ADDRESS>: seq=0 ttl=63 time=1.275 ms

--- <IP_ADDRESS> ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 1.275/1.275/1.275 ms

run iptables trace to trace packet to destination port 8472 on the node  cilium-fl-worker busybox pod is on, I see following iptables trace:

May 24 01:52:13 cilium-fl-worker kernel: [10511.914227] TRACE: raw:OUTPUT:policy:2 IN= OUT=ens192 SRC=<IP_ADDRESS> DST=<IP_ADDRESS> LEN=134 TOS=0x00 PREC=0x00 TTL=64 ID=51649 PROTO=UDP SPT=41595 DPT=8472 LEN=114
May 24 01:52:13 cilium-fl-worker kernel: [10511.914248] TRACE: mangle:OUTPUT:policy:1 IN= OUT=ens192 SRC=<IP_ADDRESS> DST=<IP_ADDRESS> LEN=134 TOS=0x00 PREC=0x00 TTL=64 ID=51649 PROTO=UDP SPT=41595 DPT=8472 LEN=114
May 24 01:52:13 cilium-fl-worker kernel: [10511.914258] TRACE: nat:OUTPUT:policy:2 IN= OUT=ens192 SRC=<IP_ADDRESS> DST=<IP_ADDRESS> LEN=134 TOS=0x00 PREC=0x00 TTL=64 ID=51649 PROTO=UDP SPT=41595 DPT=8472 LEN=114
May 24 01:52:13 cilium-fl-worker kernel: [10511.914265] TRACE: filter:OUTPUT:rule:1 IN= OUT=ens192 SRC=<IP_ADDRESS> DST=<IP_ADDRESS> LEN=134 TOS=0x00 PREC=0x00 TTL=64 ID=51649 PROTO=UDP SPT=41595 DPT=8472 LEN=114
May 24 01:52:13 cilium-fl-worker kernel: [10511.914272] TRACE: filter:KUBE-FIREWALL:return:3 IN= OUT=ens192 SRC=<IP_ADDRESS> DST=<IP_ADDRESS> LEN=134 TOS=0x00 PREC=0x00 TTL=64 ID=51649 PROTO=UDP SPT=41595 DPT=8472 LEN=114
May 24 01:52:13 cilium-fl-worker kernel: [10511.914276] TRACE: filter:OUTPUT:policy:2 IN= OUT=ens192 SRC=<IP_ADDRESS> DST=<IP_ADDRESS> LEN=134 TOS=0x00 PREC=0x00 TTL=64 ID=51649 PROTO=UDP SPT=41595 DPT=8472 LEN=114
May 24 01:52:13 cilium-fl-worker kernel: [10511.914282] TRACE: mangle:POSTROUTING:policy:1 IN=cilium_vxlan OUT=ens192 MAC=b2:7a:2a:88:d6:73:ca:0a:89:ad:b0:6e:08:00 SRC=<IP_ADDRESS> DST=<IP_ADDRESS> LEN=134 TOS=0x00 PREC=0x00 TTL=64 ID=51649 PROTO=UDP SPT=41595 DPT=8472 LEN=114
May 24 01:52:13 cilium-fl-worker kernel: [10511.914288] TRACE: nat:POSTROUTING:rule:1 IN=cilium_vxlan OUT=ens192 MAC=b2:7a:2a:88:d6:73:ca:0a:89:ad:b0:6e:08:00 SRC=<IP_ADDRESS> DST=<IP_ADDRESS> LEN=134 TOS=0x00 PREC=0x00 TTL=64 ID=51649 PROTO=UDP SPT=41595 DPT=8472 LEN=114
May 24 01:52:13 cilium-fl-worker kernel: [10511.914296] TRACE: nat:KUBE-POSTROUTING:rule:1 IN=cilium_vxlan OUT=ens192 MAC=b2:7a:2a:88:d6:73:ca:0a:89:ad:b0:6e:08:00 SRC=<IP_ADDRESS> DST=<IP_ADDRESS> LEN=134 TOS=0x00 PREC=0x00 TTL=64 ID=51649 PROTO=UDP SPT=41595 DPT=8472 LEN=114
May 24 01:52:13 cilium-fl-worker kernel: [10511.914300] TRACE: nat:POSTROUTING:policy:3 IN=cilium_vxlan OUT=ens192 MAC=b2:7a:2a:88:d6:73:ca:0a:89:ad:b0:6e:08:00 SRC=<IP_ADDRESS> DST=<IP_ADDRESS> LEN=134 TOS=0x00 PREC=0x00 TTL=64 ID=51649 PROTO=UDP SPT=41595 DPT=8472 LEN=114
May 24 01:52:13 cilium-fl-worker kernel: [10511.914978] TRACE: raw:PREROUTING:policy:2 IN=ens192 OUT= MAC=00:50:56:86:66:45:00:50:56:86:48:98:08:00 SRC=<IP_ADDRESS> DST=<IP_ADDRESS> LEN=134 TOS=0x00 PREC=0x00 TTL=64 ID=59241 PROTO=UDP SPT=37135 DPT=8472 LEN=114
May 24 01:52:13 cilium-fl-worker kernel: [10511.914989] TRACE: mangle:PREROUTING:policy:1 IN=ens192 OUT= MAC=00:50:56:86:66:45:00:50:56:86:48:98:08:00 SRC=<IP_ADDRESS> DST=<IP_ADDRESS> LEN=134 TOS=0x00 PREC=0x00 TTL=64 ID=59241 PROTO=UDP SPT=37135 DPT=8472 LEN=114
May 24 01:52:13 cilium-fl-worker kernel: [10511.914996] TRACE: nat:PREROUTING:rule:1 IN=ens192 OUT= MAC=00:50:56:86:66:45:00:50:56:86:48:98:08:00 SRC=<IP_ADDRESS> DST=<IP_ADDRESS> LEN=134 TOS=0x00 PREC=0x00 TTL=64 ID=59241 PROTO=UDP SPT=37135 DPT=8472 LEN=114
May 24 01:52:13 cilium-fl-worker kernel: [10511.915000] TRACE: nat:DOCKER:return:2 IN=ens192 OUT= MAC=00:50:56:86:66:45:00:50:56:86:48:98:08:00 SRC=<IP_ADDRESS> DST=<IP_ADDRESS> LEN=134 TOS=0x00 PREC=0x00 TTL=64 ID=59241 PROTO=UDP SPT=37135 DPT=8472 LEN=114
May 24 01:52:13 cilium-fl-worker kernel: [10511.915004] TRACE: nat:PREROUTING:policy:2 IN=ens192 OUT= MAC=00:50:56:86:66:45:00:50:56:86:48:98:08:00 SRC=<IP_ADDRESS> DST=<IP_ADDRESS> LEN=134 TOS=0x00 PREC=0x00 TTL=64 ID=59241 PROTO=UDP SPT=37135 DPT=8472 LEN=114
May 24 01:52:13 cilium-fl-worker kernel: [10511.915010] TRACE: mangle:INPUT:policy:1 IN=ens192 OUT= MAC=00:50:56:86:66:45:00:50:56:86:48:98:08:00 SRC=<IP_ADDRESS> DST=<IP_ADDRESS> LEN=134 TOS=0x00 PREC=0x00 TTL=64 ID=59241 PROTO=UDP SPT=37135 DPT=8472 LEN=114
May 24 01:52:13 cilium-fl-worker kernel: [10511.915038] TRACE: filter:INPUT:rule:1 IN=ens192 OUT= MAC=00:50:56:86:66:45:00:50:56:86:48:98:08:00 SRC=<IP_ADDRESS> DST=<IP_ADDRESS> LEN=134 TOS=0x00 PREC=0x00 TTL=64 ID=59241 PROTO=UDP SPT=37135 DPT=8472 LEN=114
May 24 01:52:13 cilium-fl-worker kernel: [10511.915043] TRACE: filter:KUBE-FIREWALL:return:3 IN=ens192 OUT= MAC=00:50:56:86:66:45:00:50:56:86:48:98:08:00 SRC=<IP_ADDRESS> DST=<IP_ADDRESS> LEN=134 TOS=0x00 PREC=0x00 TTL=64 ID=59241 PROTO=UDP SPT=37135 DPT=8472 LEN=114
May 24 01:52:13 cilium-fl-worker kernel: [10511.915046] TRACE: filter:INPUT:policy:2 IN=ens192 OUT= MAC=00:50:56:86:66:45:00:50:56:86:48:98:08:00 SRC=<IP_ADDRESS> DST=<IP_ADDRESS> LEN=134 TOS=0x00 PREC=0x00 TTL=64 ID=59241 PROTO=UDP SPT=37135 DPT=8472 LEN=114
May 24 01:52:13 cilium-fl-worker kernel: [10511.915049] TRACE: nat:INPUT:policy:1 IN=ens192 OUT= MAC=00:50:56:86:66:45:00:50:56:86:48:98:08:00 SRC=<IP_ADDRESS> DST=<IP_ADDRESS> LEN=134 TOS=0x00 PREC=0x00 TTL=64 ID=59241 PROTO=UDP SPT=37135 DPT=8472 LEN=114

my understanding is that host routing also support tunnel mode and packet egress from container, and encapsulated overlay packet should skip the netfilter/iptables processing

Not quite, before the support Pod-to-Pod packets had to go twice through the stack: 1) UDP stack for overlay for the outer one, 2) upper host for netfilter/routing for the inner one. With BPF host routing, the 2nd part is removed as well as the case when you access Pods from the outside not going through the tunnel (e.g. NodePort svc - also here it went to upper stack before the change, and now it goes directly to the Pod). The latter benefits from the BPF host routing in particular as shown in the benchmark.

@borkmann
Thank you very much for your clarification!  just curious, does it make sense to skip netfilter for outer one too or not worth the effort and not much gain there

Just add more lab observation, I downgraded my kernel to 5.8 which does not support eBPF host routing, so cilium fall back to legacy host routing

kubectl  exec -it $CA2 -n kube-system -- cilium status
KVStore:                Ok   Disabled
Kubernetes:             Ok   1.20 (v1.20.7) [linux/amd64]
Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement:   Strict   [ens192 <IP_ADDRESS> (Direct Routing)]
Cilium:                 Ok   1.10.0-rc2 (v1.10.0-rc2-d65e51a)
NodeMonitor:            Listening for events on 4 CPUs with 64x4096 of shared memory
IPAM:                   IPv4: 2/254 allocated from <IP_ADDRESS>/24,
BandwidthManager:       EDT with BPF   [ens192]
Host Routing:           Legacy
Masquerading:           BPF   [ens192]   <IP_ADDRESS>/24 [IPv4: Enabled, IPv6: Disabled]
Controller Status:      19/19 healthy
Proxy Status:           No managed proxy redirect
Hubble:                 Ok   Current/Max Flows: 4095/4095 (100.00%), Flows/s: 0.61   Metrics: Disabled
Encryption:             Disabled
Cluster health:               Warning   cilium-health daemon unreachable

and I ran iptables trace in host namespace, I saw similar iptables trace for outer overlay packet, but no iptables trace for inner pod-to-pod packet, my understanding is that the veth interface in host side for container has ingress "from-container"  that does the encapsulation and redirect to cilium_vxlan interface, it appears netfilter in hostname space does not process pod-to-pod for cilium tunnel mode. if I ran kubernetes with kube-proxy and flannel, no cilium, indeed pod-to-pod traffic and overlay traffic both processed by netfilter in hostname space. but I see no difference for cilium tunnel mode between legacy routing and ebpf host routing for inner pod-to-pod processing in host namespace, of course I could be missing something, just my observation :)
for example, run both iptables trace in container and in hostname space with following rules:

 iptables -t raw -j TRACE -I PREROUTING 1
iptables -t raw -j TRACE -I OUTPUT 1

ping remote pod from netshoot pod

 kubectl  exec -it netshoot -- /bin/bash
bash-5.1# ping -c 1 <IP_ADDRESS>
PING <IP_ADDRESS> (<IP_ADDRESS>) 56(84) bytes of data.
64 bytes from <IP_ADDRESS>: icmp_seq=1 ttl=63 time=0.888 ms

--- <IP_ADDRESS> ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.888/0.888/0.888/0.000 ms
bash-5.1# iptables -t raw -n -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
TRACE      all  --  <IP_ADDRESS>/0            <IP_ADDRESS>/0

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
TRACE      all  --  <IP_ADDRESS>/0            <IP_ADDRESS>/0

iptables trace inside container:

May 26 14:53:21 cilium-fl-worker kernel: [55613.473004] TRACE: raw:OUTPUT:policy:2 IN= OUT=eth0 SRC=<IP_ADDRESS> DST=<IP_ADDRESS> LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=24469 DF PROTO=ICMP TYPE=8 CODE=0 ID=21198 SEQ=1 UID=0 GID=0

May 26 14:53:21 cilium-fl-worker kernel: [55613.473866] TRACE: raw:PREROUTING:policy:2 IN=eth0 OUT= MAC=f2:92:2b:4d:18:fc:6e:26:08:e4:56:64:08:00 SRC=<IP_ADDRESS> DST=<IP_ADDRESS> LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=990 PROTO=ICMP TYPE=0 CODE=0 ID=21198 SEQ=1

iptables trace in hostname space:

May 26 14:53:21 cilium-fl-worker kernel: [55613.473148] TRACE: raw:OUTPUT:policy:2 IN= OUT=ens192 SRC=<IP_ADDRESS> DST=<IP_ADDRESS> LEN=134 TOS=0x00 PREC=0x00 TTL=64 ID=62790 PROTO=UDP SPT=52588 DPT=8472 LEN=114
May 26 14:53:21 cilium-fl-worker kernel: [55613.473155] TRACE: mangle:OUTPUT:policy:1 IN= OUT=ens192 SRC=<IP_ADDRESS> DST=<IP_ADDRESS> LEN=134 TOS=0x00 PREC=0x00 TTL=64 ID=62790 PROTO=UDP SPT=52588 DPT=8472 LEN=114
May 26 14:53:21 cilium-fl-worker kernel: [55613.473160] TRACE: nat:OUTPUT:policy:2 IN= OUT=ens192 SRC=<IP_ADDRESS> DST=<IP_ADDRESS> LEN=134 TOS=0x00 PREC=0x00 TTL=64 ID=62790 PROTO=UDP SPT=52588 DPT=8472 LEN=114
May 26 14:53:21 cilium-fl-worker kernel: [55613.473163] TRACE: filter:OUTPUT:rule:1 IN= OUT=ens192 SRC=<IP_ADDRESS> DST=<IP_ADDRESS> LEN=134 TOS=0x00 PREC=0x00 TTL=64 ID=62790 PROTO=UDP SPT=52588 DPT=8472 LEN=114
May 26 14:53:21 cilium-fl-worker kernel: [55613.473166] TRACE: filter:KUBE-FIREWALL:return:3 IN= OUT=ens192 SRC=<IP_ADDRESS> DST=<IP_ADDRESS> LEN=134 TOS=0x00 PREC=0x00 TTL=64 ID=62790 PROTO=UDP SPT=52588 DPT=8472 LEN=114
May 26 14:53:21 cilium-fl-worker kernel: [55613.473168] TRACE: filter:OUTPUT:policy:2 IN= OUT=ens192 SRC=<IP_ADDRESS> DST=<IP_ADDRESS> LEN=134 TOS=0x00 PREC=0x00 TTL=64 ID=62790 PROTO=UDP SPT=52588 DPT=8472 LEN=114
May 26 14:53:21 cilium-fl-worker kernel: [55613.473172] TRACE: mangle:POSTROUTING:policy:1 IN=cilium_vxlan OUT=ens192 MAC=6e:26:08:e4:56:64:f2:92:2b:4d:18:fc:08:00 SRC=<IP_ADDRESS> DST=<IP_ADDRESS> LEN=134 TOS=0x00 PREC=0x00 TTL=64 ID=62790 PROTO=UDP SPT=52588 DPT=8472 LEN=114
May 26 14:53:21 cilium-fl-worker kernel: [55613.473175] TRACE: nat:POSTROUTING:rule:1 IN=cilium_vxlan OUT=ens192 MAC=6e:26:08:e4:56:64:f2:92:2b:4d:18:fc:08:00 SRC=<IP_ADDRESS> DST=<IP_ADDRESS> LEN=134 TOS=0x00 PREC=0x00 TTL=64 ID=62790 PROTO=UDP SPT=52588 DPT=8472 LEN=114
May 26 14:53:21 cilium-fl-worker kernel: [55613.473181] TRACE: nat:KUBE-POSTROUTING:rule:1 IN=cilium_vxlan OUT=ens192 MAC=6e:26:08:e4:56:64:f2:92:2b:4d:18:fc:08:00 SRC=<IP_ADDRESS> DST=<IP_ADDRESS> LEN=134 TOS=0x00 PREC=0x00 TTL=64 ID=62790 PROTO=UDP SPT=52588 DPT=8472 LEN=114
May 26 14:53:21 cilium-fl-worker kernel: [55613.473186] TRACE: nat:POSTROUTING:policy:3 IN=cilium_vxlan OUT=ens192 MAC=6e:26:08:e4:56:64:f2:92:2b:4d:18:fc:08:00 SRC=<IP_ADDRESS> DST=<IP_ADDRESS> LEN=134 TOS=0x00 PREC=0x00 TTL=64 ID=62790 PROTO=UDP SPT=52588 DPT=8472 LEN=114

but I see no difference for cilium tunnel mode between legacy routing and ebpf host routing for inner pod-to-pod processing in host namespace, of course I could be missing something, just my observation :)

That observation is correct. For Pod-Pod over tunnel in legacy mode only the outer one goes through the host stack. And the BPF prog from cilium_vxlan/geneve will redirect at tc layer to the Pod. From the Pod in bpf_lxc, it will redirect back to the tunnel dev. The one difference here for Pod-Pod case is that due to the use of redirect_peer() it will decrease the latency slightly due to not having to switch to the different netns via net_rx_action.

The one difference here for Pod-Pod case is that due to the use of redirect_peer() it will decrease the latency slightly due to not having to switch to the different netns via net_rx_action.

Is it menas that, if we use the legacy mode, it needs send the packet to the lxc-xxxxxxxx interface. but with the BPF mode, it will be redirected to the pod's eth0 interface by redirect_peer() helpers. if wrong, please correct me here, thanks.

 Payload=[..86..] SrcMAC=be:2c:08:54:47:03 DstMAC=72:e7:d4:b4:30:c1 EthernetType=IPv4 Length=0}
IPv4    {Contents=[..20..] Payload=[..64..] Version=4 IHL=5 TOS=0 Length=84 Id=32240 Flags=DF FragOffset=0 TTL=64 Protocol=ICMPv4 Checksum=42469 SrcIP=<IP_ADDRESS> DstIP=<IP_ADDRESS> Options=[] Padding=[]}
ICMPv4  {Contents=[..8..] Payload=[..56..]  Payload=[..86..] SrcMAC=72:e7:d4:b4:30:c1 DstMAC=be:2c:08:54:47:03 EthernetType=IPv4 Length=0}
IPv4    {Contents=[..20..] Payload=[..64..] Version=4 IHL=5 TOS=0 Length=84 Id=34574 Flags= FragOffset=0 TTL=63 Protocol=ICMPv4 Checksum=56775 SrcIP=<IP_ADDRESS> DstIP=<IP_ADDRESS> Options=[] Padding=[]}
ICMPv4  {Contents=[..8..] Payload=[..56..] TypeCode=EchoReply Checksum=17296 Id=45 Seq=0}
  Failed to decode layer: No decoder for layer type Payload
CPU 04: MARK 0x0 FROM 809 to-endpoint: 98 bytes (98 captured), state reply, interface lxcb527e53a7df7, , identity 29735->29735, orig-ip <IP_ADDRESS>, to endpoint 809

About this part:
CPU 04: MARK 0x0 FROM 464 DEBUG: Conntrack lookup 1/2: src=<IP_ADDRESS>:43200 dst=<IP_ADDRESS>:8472
CPU 04: MARK 0x0 FROM 464 DEBUG: Conntrack lookup 2/2: nexthdr=17 flags=1

We can see that this CT is deal by cilium_host interface. is it mean that the VxLAN packet need deal with cilium_host interface?
root@cilium-vxlan-host-routing-control-plane:~# cilium endpoint get 464 | grep -C 1 "interface-name"
        "host-mac": "d2:b6:fc:31:39:75",
        "interface-name": "cilium_host",
        "mac": "d2:b6:fc:31:39:75"
root@cilium-vxlan-host-routing-control-plane:~#


How to assign a right click to a key?

I looked for it but can't seem to find the RightClick option in "Simple modifications".

I have tried with button2, can you try it?

It works. Thank you.
With this, the right click contextual menu opens where the mouse is, not where the folder, file is selected. Which is what I was looking for.
I think I will keep using Alfred actions as the right click contextual menu has more or less the same items.  But I can launch Alfred actions via a keyboard shortcut.


[test] Fix and re-enable test that tries to build the Foundation overlay

Checking determinism is important, so it's okay that the test is a little brittle. Still better on than off.
rdar://problem/28450391

@swift-ci Please smoke test and merge

Hey, @DougGregor, you didn't by chance introduce non-determinism into serialization of conformances, did you?

Filed a bug for Doug to fix the newly-introduced non-determinism.


update to Swift 2

Sorry I already had a working version before I noticed your pull request. I just pushed an update supporting Swift 2.0. Feel free to contribute in the future, I welcome it.


Add tar.gz as new output option.

Resolves #283

ℹ At this point this PR is still a draft/proposal for further discussion before finalizing it.

In this proposal some new options would allow the writing of tar.gz files as output. Essential changes:

New TarOutputDirectory parameter which would activate the tar.gz writing.
New OutputMode which allows more specific control of which output flow to follow (local daemon, remote push, file writing).

Pending change until approval of approach: writing tests.
Alternative proposal: Use the new "local daemon" feature to have something like a fake "tar" Daemon. Unfortunately at this point I am lacking an option to then pass in the folder into which to write the tarballs. And also the output property on the Task would be missing in this case unless we extend it. Would need some confirmation on the desired public API for this feature.

Noting that there's some conflict here with https://github.com/dotnet/sdk-container-builds/pull/350 - we'll need to discuss approaches.

We'd like to go forward with this approach @Danielku15 - the team likes the explicitness of the OutputMode flag and the overall approach of elevating each output mechanism into a distinct case. We'd love to see tests for this, or we can help out to get this over the finish line.

@baronfel Things are currently quite busy on my side. I will try to continue on this topic soon but I plan to wrap everything up and extend it with tests.

No pressure at all - please let us know if there's anything the team can do to lighten the load :)

Shouldn't be a problem. I made those kind of changes already 2-3 times on top of different states. One more time should go very smooth 😉

Hi Daniel - do you want to pick this up still? If you don't have time that's ok too :)

@baronfel Sure. Things are getting a bit more relaxed currently and I could take this up. I am just wondering if it makes sense to implement first https://github.com/dotnet/sdk-container-builds/issues/379 to have only one single tar.gz output. Currently due to the different metadata generated in the archive we otherwise would have the need for multiple outputs.
What do you think?

Closing the PR here. We can continue here: https://github.com/dotnet/sdk/pull/32132
The design of the solution changed quite a bit. It made more sense to rework things in a bigger scale to solve multiple problems at once.


Add mcpu pattern for Morello and related tests in buildbot

Added needed mpu argument pattern to support bitcode generation in morello. Removed (no longer needed) test for sqlite and added lua tests for riscv64 and morello-purecap.

bors try

bors try

Please squash while also thinking whether the commit message should say something like "Add -mcpu flag" rather than "argument pattern" (I don't know what an "argument pattern" is!).,

Please squash while also thinking whether the commit message should say something like "Add -mcpu flag" rather than "argument pattern" (I don't know what an "argument pattern" is!).,

Sure. I've just repeated what the code says. I'll follow your suggestion, though.

Squashed.


HFlow changes layout behaviour of TextField

consider:
VStack {
    HFlow {
        TextField("Test", text: .constant(""))
            .border(Color.red)
    }
    TextField("Test", text: .constant(""))
        .border(Color.red)
}

We notice that HFlow is affecting the default layout behaviour of TextField:

A TextField wants to expand horizontally to adopt all available space, but the HFlow is disallowing this.
Unless there is some mechanism in place to restore the default behaviour of the TextField component (and likely others?), HFlow should be adjusted to restore it.

Try setting the alignment in the VStack explicitly.
VStack(alignment: .leading) {
    HFlow {
        TextField("Test", text: .constant(""))
            .border(Color.red)
    }
    TextField("Test", text: .constant(""))
        .border(Color.red)
}

Thanks, I am curious - does this not take away some agency of the individual items in the flow? Some views are naturally content-wrapping, while some views are naturally expanding. It would appear this behaviour overrides these natural preferences and either makes all items wrapping or all expanding, regardless of their individual nature.
One might expect, eg. Text("Test") to be naturally content-wrapped, but TextField("Test", _) to be horizontally-expanding. As designed, these two items with their natural preferences cannot be honoured within the same flow.
Is there a reason we cannot honour the views' default preferences?

Thank you for the thoughtful question @lhunath! This opens up an interesting exploration of layout behavior.
HFlow is a size-constrained layout that prompts its child views for their ideal sizes and arranges them in rows based on that information. It already respects each view’s sizing preferences by proposing constrained sizes, and it’s up to the child views to respond accordingly.
Regarding flexible views like TextField, consider a case with multiple of them in the same Flow. Should they all fit in one row, or should they wrap into separate rows? This decision often depends on the layout context, and HFlow leaves that choice to the user rather than enforcing assumptions about flexibility.
The default design of HFlow prioritizes common use cases like arranging tags, texts, or other small items in rows. For flexible views, there’s ambiguity: where do they fit best within a constrained flow layout? While respecting flexibility might seem desirable in certain cases, it could lead to unintuitive layouts, especially when multiple flexible views are involved.
(Additionally, the library provides an option to evenly distribute items using the Knuth-Plass line-breaking algorithm. Dynamically handling flexible subview constraints in this mode involves a lot of ambiguity, potential conflicts and significant complexity. Layout priorities can possibly reduce some of those ambiguities.)
That said, I’d love to see an example where preserving the flexible nature of views (in the default case) would lead to a better outcome. How would you handle cases with (potentially multiple) flexible views?
Looking forward to your thoughts! This could help shape the future direction of the library.

Appreciate expanding on this, @tevelee. What you say makes sense.
Given that HFlow sizes the items based on their own sizing preferences within a constrained context, might I inquire as to whether enough information is passed to the child in this case for it to elect to size itself to the full available (remaining) width in the row? How might I implement such a custom view?
Or is it the case that within this constrained sizing environment, the child has no information on the parent's width, nor does it have an option to size itself non-content-wrapping?
Certainly one option on the table would be a custom modifier which can be selectively applied to children in order to indicate to the HFlow that this particular child wishes to be measured differently (though I am not sure about the complexity of implementing such).

I've been thinking about this issue in the past 2 days, and right now I think what you're proposing is possible but very complex to implement. I've been trying to prototype the idea of arranging the flexible items in either compact, natural, or expanded modes. In compact mode (which is essentially current behavior) items are squished. Natural is what you are proposing, adding the flexible item and letting it span however possible. Expanded makes room for flexible items to spread even full rows.
The implementation is pretty complex at this point, but I'm trying to wrap my head around it. The current API has to change to adopt a new mode like this, since this will redefine the justified behavior and the line breaking. (I've already considered some breaking changes for 2.0).

Best of luck, and much appreciation for the consideration. I'm around to feedback into your thought process or think along with some of the concepts should that be beneficial.
Unfortunately I've struggled to find reliable in-depth information from Apple on how the measuring process truly works under the hood, or even a conceptual overview.

I just published version 3.0.0 of the library. Now the default mode of flexibility is .natural, the behavior you suggested (with other options to specify). Please try it out if you have a chance

Seems to work beautifully with my initial tests, thanks for the energy!


Add ability to plot any variable against any other variable

The user should be able to investigate the raw relationship between any of the variables included in the analysis. This functionality should be added to the Data Explorer tab.

This feature has been added and will be merged in an upcoming PR.

Closed in #14


Add latest-release plugin

I have a simple plugin that lists the latest git releases for k8s projects. I have tested it locally and works well for me. It would be great if we can add it to krew
Also, do let me know if you think I need to rename anything here.

I think this tool is suitable as a generic tool that's not specific to kubernetes ecosystem. You can probably repurpose this tool to just search the repo with that name that has highest stars, and then look up its releases, like:
$ latest-release git
$ latest-release kubectx
$ latest-release fortio
$ latest-release ahmetb/kubectl-aliases
...

I think this is a super cool idea, but I doubt its usefulness as a kubectl plugin. I'd be happy to accept it anyway, but I suggest you re-evaluate if you want to limit this to be a kubectl plugin.  It would be way more useful as a generic to me personally.

@ahmetb thanks for the feedback. I agree with the idea of making this tool generic. I will rework this soon but for now I will go ahead and close the PR! :+1:


[Bug]: connectivity_plus v6.0.0 fails to build on iOS

Platform
iOS 17.4
Plugin
connectivity_plus
Version
6.0.0
Flutter SDK
3.19.3
Steps to reproduce

install connectivity_plus: 6.0.0
compile iOS app in debug mode (did not check prod mode)
compilation fails

Code Sample
No response
Logs
Failed to build iOS app
Could not build the precompiled application for the device.
Error (Xcode): ../../.pub-cache/hosted/pub.dev/connectivity_plus-6.0.0/lib/src/connectivity_plus_linux.dart:22:36: Error: The return type of the method 'ConnectivityPlusLinuxPlugin.checkConnectivity' is 'Future<List<ConnectivityResult>>', which does not match the return type, 'Future<ConnectivityResult>', of the overridden method, 'ConnectivityPlatform.checkConnectivity'.

Flutter Doctor
[✓] Flutter (Channel stable, 3.19.3, on macOS 14.4 23E214 darwin-arm64, locale en-GB)
    • Flutter version 3.19.3 on channel stable at /Users/techouse/fvm/versions/3.19.3
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision ba39319843 (13 days ago), 2024-03-07 15:22:21 -0600
    • Engine revision 2e4ba9c6fb
    • Dart version 3.3.1
    • DevTools version 2.31.1

[✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
    • Android SDK at /Users/techouse/Library/Android/sdk
    • Platform android-34, build-tools 34.0.0
    • ANDROID_HOME = /Users/techouse/Library/Android/sdk
    • Java binary at: /Users/techouse/Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 17.0.9+0-17.0.9b1087.7-11185874)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 15.3)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 15E204a
    • CocoaPods version 1.15.2

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2023.2)
    • Android Studio at /Users/techouse/Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 17.0.9+0-17.0.9b1087.7-11185874)

[✓] Connected device (4 available)
    • Pixel 6 (mobile)           • xxxxxxxxxxxxxx            • android-arm64  • Android 14 (API 34)
    • techouse’s iPhone (mobile) • xxxxxxxxxxxxxx            • ios            • iOS 17.4 21E219
    • macOS (desktop)            • macos                     • darwin-arm64   • macOS 14.4 23E214 darwin-arm64
    • Chrome (web)               • chrome                    • web-javascript • Google Chrome 122.0.6261.129

[✓] Network resources
    • All expected network resources are available.

• No issues found!

Checklist before submitting a bug

[X] I searched issues in this repository and couldn't find such bug/problem
[X] I Google'd a solution and I couldn't find it
[X] I searched on StackOverflow for a solution and I couldn't find it
[X] I read the README.md file of the plugin
[X] I'm using the latest version of the plugin
[X] All dependencies are up to date with flutter pub upgrade
[X] I did a flutter clean
[X] I tried running the example project

Aha, I see :) https://github.com/fluttercommunity/plus_plugins/pull/2735
Thanks :)


Use Helm to Install Ingress Controller

The lab as written does not work because the git clone command is invalid. It's easier to use helm repo add instead.
Also, the ingress yaml file is updated to use the networking.k8s.io/v1 API.
This PR closes https://github.com/confluentinc/operator-earlyaccess/issues/47

@arodoni @rohit2b This PR is ready for merge. I don't appear to have the permissions to request reviewers. Maybe could you also add me as a collaborator on the repo?

@chuck-confluent I made you a Maintainer on the repo. Let me know if that works.

@yangpei1214 This is ready for your review and merge! https://github.com/confluentinc/operator-earlyaccess/pull/53 depends on this PR, but this PR doesn't depend on that one.


Create a design document

I need to start this as a good habit and i have been putting this off for way to long

I found out this app called Obsidian.md. I will most likely use this for all the documentation in this project as i like the way it handles everything.


add pictures to readme

Readme is already updated. I updated it before this pull request. If the PR goes through, the images will show the updated UI.

Ah i see it now


[chat] add eos token to generate

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


Adding multi language support system✨

I want to add multi language support system to the website which can help us understand in their native language.
@DeadmanAbir can you assign me this issue,

I want to add multi language support system to the website which can help us understand in their native language. @DeadmanAbir can you assign me this issue,

We use contentlayer for writing and rednering .mdx files. What is your approach to implement multi language system in contentlayer?


client.IndexExists return false, but index exist  Definitely!  thanks for you help !

Please use the following questions as a guideline to help me answer
your issue/question without further inquiry. Thank you.
Which version of Elastic are you using?
[ ✔] elastic.v7 (for Elasticsearch 7.x)
[ ] elastic.v6 (for Elasticsearch 6.x)
[ ] elastic.v5 (for Elasticsearch 5.x)
[ ] elastic.v3 (for Elasticsearch 2.x)
[ ] elastic.v2 (for Elasticsearch 1.x)
Please describe the expected behavior
Please describe the actual behavior
Any steps to reproduce the behavior?

sorry， i made a mistake, so sorry !


Exptected rename to Expected

BatchAccumulator Exptected rename to Expected
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@showuon hi, cloud you help me.
PTAL

Thanks @Kvicii
For such small changes you don't need to create a JIRA. Just prefix your commit with MINOR: .

@mimaison  OK


Model(s) overhaul: ExplicitPermInc and PersistentShock

We should overhaul / redo the ExplicitPermInc and PersistentShock solvers/models to represent the consumption function in terms of normalized variables.  That is, cFunc will take in mNrm values and spit out cNrm values, rather than the current system of taking in mLvl and spitting out cLvl.  This will simplify the simulation methods, as fewer need to be overwritten.
Moreover, this will likely result in more accurate solutions that are more easily understood graphically.  At high levels of permanent income, far above the center of the distribution, adjacent 1D "layers" of the consumption function diverge more and more from each other, as the differences between consecutive pLvlGrid values become large.  The shape of the "above" and "below" 1D cFuncs for a pLvl between grid values might be significantly different, reducing accuracy of the interpolation.
Making this switch will make it easy to graphically verify that ExplicitPermInc is solving correctly: plotting the cFunc at various pLvl values should perfectly align with each other.  They might not perfectly align because the pLvlGrid is bounded, so the top and bottom 1D cFuncs are subject to extrapolation error.  In the PersistentShock model, we can more easily see that high pLvls are associated with lower normalized consumption, as consumers expect their productivity to revert to the mean (and vice versa for low pLvls).
Most broadly, the inheritance structure of these models should not be ConsIndShock --> ExplicitPermInc --> PersistentShock, but rather ConsIndShock --> ConsGenIncProcess, with ExplicitPermInc and PersistentShock being child models of the new ConsGenIncProcess.  All of these will use the same solver, with the only difference being how the attribute (and solver input) PermIncNextFunc is constructed.  This function is a generalized income process that will be used as:
pLvlNext = PermIncNextFunc(pLvlNow)*PermShkNext
The ExplicitPermIncConsumerType class will construct its PermIncNextFunc(s) as linear functions (using primitive attribute PermGroFac), while PersistentShock will construct AR1-type functions based on PermGroFac, PermIncCorr, and pLvlInitMean.  The baseline GenIncProcessConsumerType will have no constructor for PermIncNextFunc.
The ConsGenIncProcess solver will let us better model how the drop in permanent income at retirement depends on pLvl, i.e. the SS replacement rate (approximating the AIME calculation).  As is, we use Cagetti's method of imposing different permanent income drops based on education.  The new way will better reflect reality.

I'm doing this over the next couple days, as I need it for a project.  Will hopefully have a PR by Saturday.

The changes described in the last four paragraphs of the original post have been implemented in the branch GeneralIncomeProcess, in a new module called ConsGenIncProcessModel.py, which will replace ConsPersistentShockModel.py.    I still need to change inheritance in ConsMedModel to use the new module (and test), and I might rename PermIncNextFunc to pLvlNextFunc.
The new agent classes have a method called installRetirementFunc that looks for an attribute called PermIncNextFuncRet and puts it in self.PermIncNextFunc[self.T_retire] if it exists and the model has retirement.  This is supposed to represent the "retirement permanent income function", as in the last paragraph of the original post.  As yet untested, because the example models are infinite horizon.

Forgot to close this issue after the branch was merged.


sparse conv support

hi，now I can see the libxsmm support the sprase matrix multiply dense matrix 。When can we get the sparse conv kernel。I see the libxsmm_sconvfunction function，but it's a little different from the libxsmm_xmmfunction function，It's too hard for me to finish the sparse conv kernel.can anyone supply some help?thks.

We are not planning right now for implementing a sparse convolution kernel. libxsmm_sconvfunction is a dense small convolution kernel: RS(16)C*16(K). We can also bring more C inside. The kernel assumes following memory layout: [N][Cblock][H][W][16], [N][Kblock][P][Q][16], [Kblock][Cblock][R][S][16][16].
details can be found here: https://arxiv.org/abs/1808.05567

Please feel free to open an issue if further assistence is needed.


New API for ORM Adapter gems

Motivation
Since v2.0.0 will be the first time we're actually allowing external adapter gems to be created (the list of recognized adapters is hard-coded in the v1 line), this API addition is technically only a backwards-compatible semver-minor change. However, we will be committing to supporting whatever ORM Adapter API we release in v2.0.0 until at least v3.0.0, so I think its really important that we get this right before the first v2 release. Therefore I'd really appreciate your feedback!
To test out this new API, I have also opened two companion PRs for the two most popular ORM adapter gems: ActiveRecord and Redis. These also happen to be particularly good candidates for evaluating this proposal, because they are the most complex adapter and the simplest adapter, respectively.
https://github.com/DatabaseCleaner/database_cleaner-active_record/pull/41
https://github.com/DatabaseCleaner/database_cleaner-redis/pull/8
I'll start by describing the current state of affairs in the Status Quo section, and then afterwards, I'll describe the changes proposed in these PRs in the Proposal section.
Status Quo
ORM Adapter
The ORM Adapter namespace module must define two module functions, e.g.:
# lib/database_cleaner/active_record.rb
require "database_cleaner/active_record/version"
require "database_cleaner/core"
require "database_cleaner/active_record/transaction"
require "database_cleaner/active_record/truncation"
require "database_cleaner/active_record/deletion"

module DatabaseCleaner::ActiveRecord
  def self.available_strategies
    %i[transaction truncation deletion]
  end

  def self.default_strategy
    :transaction
  end
end

# explicit registry, now that ORM autodetection is out
DatabaseCleaner[:active_record].strategy = :transaction

Strategies
Strategies get their functionality in bits and pieces by including various modules (which themselves include other modules). In core, there are three different modules in three different files under DatabaseCleaner::Generic:

DatabaseCleaner::Generic::Base
DatabaseCleaner::Generic::Transaction
DatabaseCleaner::Generic::Truncation

Understanding which to include where is surprisingly tricky, leading to the code in one single strategy being spread across four files and two gems! For example, here is the v1.8 module inclusion heirarchy of the truncation strategies from the ActiveRecord and Redis adapters.
class DatabaseCleaner::ActiveRecord::Truncation
├--> module DatabaseCleaner::ActiveRecord::Base
|    └--> module DatabaseCleaner::Generic::Base
└--> module DatabaseCleaner::Generic::Truncation

class DatabaseCleaner::Redis::Truncation
├--> module DatabaseCleaner::Redis::Base
|    └--> module DatabaseCleaner::Generic::Base
└--> module DatabaseCleaner::Generic::Truncation

Proposal
Elucidated in this PR and companion PRs:
https://github.com/DatabaseCleaner/database_cleaner-active_record/pull/41
https://github.com/DatabaseCleaner/database_cleaner-redis/pull/8
ORM Adapter
Both module functions are no longer necessary! The default strategy is already set explicitly when the adapter registers itself, and the available strategies are now determined through introspection of the adapter module namespace. One downside to this is that internal adapter constants are now falsely interpreted as strategies, but those can be privatized to avoid this. See DC::AR PR for an example of this.
Honestly, I'm not 100% on this exact strategy, and I wouldn't be surprised if this particular bit saw some iteration before final release. For example, we could alternatively do the detection by checking for descendence from DC::Strategy, etc. Despite this uncertainty, I think the goal of narrowing the Adapter API to its minimum is really worth shooting for, even perhaps to the point of erring a bit too far in v2.0! Because we can always change our minds and add something back in v2.1 or v2.2, etc, but we cannot take anything away until at least v3.0, which I hope is a long way off.
# lib/database_cleaner/active_record.rb
require "database_cleaner/active_record/version"
require "database_cleaner/core"
require "database_cleaner/active_record/transaction"
require "database_cleaner/active_record/truncation"
require "database_cleaner/active_record/deletion"

DatabaseCleaner[:active_record].strategy = :transaction

Strategies
All three modules in DatabaseCleaner::Generic are superseded by a new single DatabaseCleaner::Strategy abstract class, and the concrete strategies simply inherit from that, now. The examples above become:
class DatabaseCleaner::ActiveRecord::Truncation
└--> class DatabaseCleaner::ActiveRecord::Base
     └--> class DatabaseCleaner::Strategy

class DatabaseCleaner::Redis::Truncation
└--> class DatabaseCleaner::Strategy

This can be done because the differences between the original three generic modules were largely superficial, with the exception of the DatabaseCleaner::Generic::Truncation module doing some work to enable the :only and :except options, as those are likely to be useful in every truncation strategy. To address this I have refactored this option handling into a private DatabaseCleaner::Strategy#tables_to_clean method that is included in every strategy. If those two options are useful to a strategy, then this method can be used internally to provide them. If not, it can simply be ignored! You can see this method being used in both of the truncation strategies in the two adapter PRs.
Conclusion
I think this PR is an substantial improvement over the status quo, particularly with regards to simplifying the developer experience of creating an ORM adapter gem, and minimizing the surface area of the API. However, there are certainly lots of alternative designs to what I have proposed here, both in the large, and in the details, and I want to get this right in v2.0.0, so I'd love to receive your feedback!

I've pushed a few more commits which make two minor changes to make the ORM Adapter API even simpler, still.

Strategies now must be a descendant of DatabaseCleaner::Strategy. As mentioned above, this cuts down on false positives in the detection of available strategies, which mostly means that adapter authors probably won't have to know how it works, or need to muck with private_constant. Simpler!

Remove the private #tables_to_clean  method. After looking more closely at this method, it seemed to me that it wasn't at all pulling its weight. It only does two things:

Throws an error if both only and except were passed, explaining that it doesn't really make sense. IMO, this is of dubious value. I actually disagree with the assertion that that situation is nonsensical; saying "clean these, but not those" makes perfect sense to me! So I've simply removed it. If someone manages to come up with a scenario that demonstrates this kind of check is actually needed, it can be put into place at the adapter level, instead.
Performs a single array subtraction.
IMO, neither of those things are worth the cost of adding that method to the adapter API, nor the cost of the indirection that using that method would entail. Removing the check, and inlining the subtraction it into both adapter PRs shows an immediate improvement in grokkability, and surprisingly without an increase in complexity. Strategy, now, is dead simple. It seems like an obvious move now, in hindsight.

With these two changes, I'm personally very happy with the state of the ORM Adapter API, and think its ready to merge. Since there have been no issues raised here, I'm planning on merging this tomorrow, unless someone chimes in with a concern. The next (and hopefully final) beta release will be forthcoming soon after that!


Next Section button linked incorrectly.

In Next Steps section, the text says "To learn how to deploy Azure resources across multiple regions, and how to use safe deployment practices, see".. The button shows/targets "Use Azure Deployment Manager".
The button should be "Safe Deployment Practices" and link to respective article.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 6fa260de-4d34-678d-71cb-a33005bc9e99
Version Independent ID: 26b43abc-0b22-3e70-e965-40dc1f21cb02
Content: Continuous integration with Azure Pipelines
Content Source: articles/azure-resource-manager/resource-manager-tutorial-use-azure-pipelines.md
Service: azure-resource-manager
GitHub Login: @mumian
Microsoft Alias: jgao

@venkatx5 Thanks for the comment. We are actively investigating on the issue and will get back to you shortly with an update.

@venkatx5 Verified and updated the document as required. Thanks.


Fix RDPWinst.exe -w in first time

Fix RDPWinst.exe -w in first time after #2
Previously, it only worked the second time.

thanks


Replace favicons

Hi @gomzyakov, I have an assignment in my school to solve any issue labeled "good-first-issue" and I found the issue Replace favicons on the site in your repository. Could you review whether my pull request suits your requirements?

@gomzyakov, my school in Vietnam ^^


Syntax for generic reply function

Similar to https://github.com/node-modbus/stream/issues/34 I'm confused about the different signatures that need to be used for reply. It seems to depend on the function code, that is, reply seems to assume the data it receives is byte-for-byte identical to the data section of the response frame, even if that includes additional fields like start address, number of elements to read/write, etc. Is that right?
For example, in my application I define a "modbus map" to associate handler functions with ROB & ROR read operations (function code 2 & 4) as well as RWB & RWR read/write operations (codes 1, 3, 5, 6, 15, and 16). However, I am not certain how I should call reply with the resulting data.
modbus.tcp.server(serverOptions, (connection: any) => {
    // ...
    connection.transport.on('request', (eventName, transaction, reply) => {
        const request = transaction.request;
        const size = request.quantity;
        const firstAddr = request.address;
        const lastAddr = firstAddr + size - 1;
        // (call appropriate handlers)
        // ...
        // This is the part that seems weird / opaque to me.
        // Why isn't there a function that will just accept `result` and format it appropriately
        // (since the transaction contains the information needed)?
        switch(eventName) {
            case 'read-coils':               // fcode 1: response format = [ID][FC][BC][DATA, 1 byte per 8 coils]
            case 'read-discrete-inputs':     // fcode 2: response format = [ID][FC][BC][DATA, 1 byte per 8 coils]
                reply(null, result);
                break;
            case 'read-holding-registers':   // fcode 3: response format = [ID][FC][BC][DATA, 2 bytes per register]
            case 'read-input-registers':     // fcode 4: response format = [ID][FC][BC][DATA, 2 bytes per register]
                result = result.map(x => (x instanceof Buffer) ? x : createUInt16BufferFromNumber(x));
                reply(null, result);
                break;
            case 'write-single-coil':        // fcode 5: response format = [ID][FC][ADDR][DATA] (mirrors request exactly)
            case 'write-single-register':    // fcode 6: response format = [ID][FC][ADDR][DATA] (mirrors request exactly)
                reply(null, firstAddr, request.value);
                break;
            case 'write-multiple-coils':     // fcode 15: response format = [ID][FC][ADDR][NUM]
            case 'write-multiple-registers': // fcode 16: response format = [ID][FC][ADDR][NUM]
                reply(null, firstAddr, size);
                break;
            default:
                warn(`Replying with default format -- probably wrong!`, result);
                reply(null, result);
        }
    })
    // ...
}

Yes, the reply assumes a first (fixed) argument for an error, and then the remaining arguments depend on the function code and what the standard defines for a response.

Yes, the reply assumes a first (fixed) argument for an error, and then the remaining arguments depend on the function code and what the standard defines for a response.

OK, thank you for clarifying that. Is it the same regardless of whether reply is obtained from a specific event like the read-coils event vs. from the lower level request event?
connection.on("request", (request, reply) => {
    // ...
});
connection.on("read-coils", (request, reply) => {
    // ...
    // does this `reply` have the same signature/contract as the one above?
});

Is it the same regardless of whether reply is obtained from a specific event like read-coils vs. from the lower level request event?

I believe yes. But perhaps, in that case, it makes more sense to be able to send a raw response. You could just ignore the reply and write a response yourself, but it would be easier.

OK, thanks for the reply. I will plan to just keep using the request event and it's reply function as I have it (like above -- seems to be working OK). It just feels a little non-intuitive because the library code does some of the protocol work (e.g. ID & function code / exception code) but not all of it (e.g. start address & number of coils/registers), so I have to dig into the library code to see what I need to do. For example, should I be sending "one big buffer" or separate arguments for, say, start address and array of data.

You could just ignore the reply and write a response yourself

Would that look something like this: transport.write(<Buffer>)?

Yes.

Closing due to inactivity


Plugin does not appear to work

Only saving/restoring zoom distances and the four orthogonal directions (North/West/South/East) seems to work, but arbitrary pitch/yaw does not seem to work.

Sadly RuneLite didn't allow for the pitch/yaw to be set to arbitary values so only the functionality you see now made it to the plugin.


Additional plugins syncying between repos

After doing a review on plugins, repos and related stuff (docker images), we need to see if the documentation on plugins match for the documentation project, the docker container that will map the plugins, and the plugins repos.

Are plugins “compatible” with older CRS versions? (I see "Preparation for older installations") in the docs, e.g. https://github.com/coreruleset/phpmyadmin-rule-exclusions-plugin
how is the proposed flow for installing plugins? using git? getting a zip file? none of the above?
depending on the previous, things may vary. if we use git, we can’t clone everything to the same directory, so people will need to use a subdirectory on the plugins folder for each plugin.
we can use the https://github.com/coreruleset/plugin-registry repo to create a yaml file with the list of plugins so it can be consumed by scripts. I foresee a script that gets the plugins, for example. Just simple bash + curl, but we need to verify things probably (gpg?).
documentation should match the efforts in the crs docker container (https://github.com/coreruleset/modsecurity-crs-docker/pull/66). People should have a way of installing plugins there (even if using volume mount).

So, we need to check that:

plugin registry
plugins doc
coreruleset documentation
crs docker documentation
all matches and have the same description and usage.

Otherwise it is going to be caos.

I'm tagging you both because you did most of the work on plugins setup @RedXanadu @azurit

As far as I know, the latest and greatest documentation we have regarding plugins is here:
https://coreruleset.org/docs/configuring/plugins/
This is based on the CRS blog post announcing plugins and how to install them.
If we want to have a "single source of truth" for plugin documentation, I propose that should be it.

Plugin Installation:
I have described two methods. The copy method or the symlink method. The symlink method would also work with separate git folders.
Providing a script that allows for easy plugin installation (-> like enabling / disabling apache modules) sounds like a plan. Maybe for the future once we are sure things have settled down (we also provided a CRS installation / update script back in the day and scrapped it again).
I think zero thought went into docker integration so far, so any useful method is welcome.

So we agree on:

source of truth: coreruleset.org/docs..
every plugin has a small intro on what it does, and points ☝ for installing/configuring
we can adapt the containers based on the documentation

I agree.

@fzipi Is there anything more which needs to be done here?

I think we need to do a massive review on the plugins and see if everything matches this:

every plugin has a small intro on what it does, and points ☝ for installing/configuring
we can adapt the containers based on the documentation

@RedXanadu @azurit Is there some text that can/should be copied to every plugin out there? Or we close and ignore this one?

Based on previous answers from the team, we close this one.


Add Python support for sequence mode

Added an eval_sequence function to RWKVModel that evaluates a list of tokens at once using sequence mode.

@LoganDark plans to rewrite the whole Python API, but I see no problem in having these functions until the rewrite is done.


Processing data using DeepProfiler

Juan asked this:
I need to process the LINCS dataset to proceed with the plan we discussed for LUAD. I'm going to need access to the images, which is a lot of data! Here is my plan to make it efficient, and I would like to get your feedback and recommendations:

Get all the plates back from Glacier for a few days. If I remember correctly, the entire set is 21TB or so.
Use EC2 instances to compress the images using DeepProfiler. I did this in the past and we can get down to 800GB or so.
Save the compressed dataset in S3 to work with it during the next couple of months, then send it to Glacier.  In fact, with that size we can probably keep it in the DGX and the GPU-cluster too.

Does this make sense? Do you have any recommendations for me before moving forward?
By the way, the compression should take roughly 4 hours per plate (pessimistic estimate), and can be run in parallel, with multiple plates per machine (one per CPU). So using 30 cheap instances (with 4 cores each) in spot mode should do the trick in one day, including the operator's time :)

Lets try a small set first, maybe just 1 plate, to make sure this process works

Restore the tarball from Glacier, following instructions here. And while you are at it, it would be great if you could help with the TODO in that file :D
Download the files to an EBS volume, instructions are above
Upload the uncompressed files back to S3, at imaging-platform instead of imaging-platform-cold. E.g. SQ00015096 would go to s3://imaging-platform/projects/2015_10_05_DrugRepurposing_AravindSubramanian_GolubLab_Broad/2016_04_01_a549_48hr_batch1/images/SQ00015096, instructions are above. The reason is that going forward, we will no longer be tarballing and archiving, but instead setting lifecycle rules that will automatically move any object that is >60 days to Glacier state. So we might as well put these files back at imaging-platform
Delete the files from EBS (expensive to have larger EBS volumes sitting around)
Now run your compressions in EC2 by directly accessing images from s3 (if you DeepProfiler can't do that yet, I'd recommend adding that functionality)
Save your images back to imaging-platform e.g. s3://imaging-platform/projects/2015_10_05_DrugRepurposing_AravindSubramanian_GolubLab_Broad/2016_04_01_a549_48hr_batch1/images/SQ00015096_compressed/Images
Add an automatically generated README in s3://imaging-platform/projects/2015_10_05_DrugRepurposing_AravindSubramanian_GolubLab_Broad/2016_04_01_a549_48hr_batch1/images/SQ00015096_compressed/ explaining how this was generated

@jccaicedo

Sounds great! Thanks for these details and suggestions. Will report the results of compressing one plate soon.

@jccaicedo I changed some paths, please note:
s3://imaging-platform/projects/2015_10_05_DrugRepurposing_AravindSubramanian_GolubLab_Broad/2016_04_01_a549_48hr_batch1_compressed/images/SQ00015096/
and not
s3://imaging-platform/projects/2015_10_05_DrugRepurposing_AravindSubramanian_GolubLab_Broad/2016_04_01_a549_48hr_batch1/images/SQ00015096_compressed/

Restore the tarball from Glacier, following instructions here. And while you are at it, it would be great if you could help with the TODO in that file :D

This is now done

@shntnu here is an update from following the plan above, with a couple of questions:
Steps 1 to 4 were run by @MarziehHaghighi. She used EFS and then deleted the unarchived files from glacier. So no EBS volumes were needed in the process and no extra storage is being used after retrieving plates. The original images are back in their original locations.
Step 5: I run DeepProfiler in an EC2 instance (m5d.xlarge 4CPUs and 16GB of RAM). I have an script that configures the environment and runs everything given the plate name. Question: should I put this script in this repository?
Step 6: This is easy and will be part of the script mentioned in the previous step. I noted the new path that you suggest for this.
Step 7: Question: what exactly this file should have? Is it enough with the script that I mentioned above?
The process is currently running so I will update this thread again with the relevant statistics of the resulting compression.

Plate compression is done. Here are the stats:

Copying data from S3 to local machine for compression:
15 minutes, 162 GB, 17,280 individual TIFF files.
Running the compression methods:
8.5 hours computing and applying illumination correction, rescaling and PNG compression.
Uploading compressed data to S3:
1 minute, 6.8 GB, 17,280 individual PNG files.
Compression rates:
Up to 95% of space savings.
Computing cost:
Approx. $0.7 per plate, using a m5n.xlarge spot instance ($0.07/hour x 10 hours).

Note that this instance can potentially process 4 plates at a time because it has 4 cores. However, I/O may slow down overall performance. I think it's safe to assume that we can compress two plates for approximately $1.
The data has been uploaded here: s3://imaging-platform/projects/2015_10_05_DrugRepurposing_AravindSubramanian_GolubLab_Broad/2016_04_01_a549_48hr_batch1_compressed/images/SQ00015096/

Steps 1 to 4 were run by @MarziehHaghighi. She used EFS and then deleted the unarchived files from glacier. So no EBS volumes were needed in the process and no extra storage is being used after retrieving plates. The original images are back in their original locations.

Nice! When doing the whole dataset, please keep these numbers in mind:

The image dataset is ~22TB.
EFS is 0.30 $/GB-month, pay what you use (unlike EBS where you pay for what you block)
22,528 GB per month x 0.30 USD = 6,758.40 USD / month or $225/day
S3 Standard is 0.023 $/GB-month

So you'd want to batch it appropriately so that storage costs don't pile up.

Step 5: I run DeepProfiler in an EC2 instance (m5d.xlarge 4CPUs and 16GB of RAM). I have an script that configures the environment and runs everything given the plate name. Question: should I put this script in this repository?

Yes please make a PR

Step 7: Question: what exactly this file should have? Is it enough with the script that I mentioned above?

A link to the URL of the file on GitHub would be great

Copying data from S3 to local machine for compression:
15 minutes, 162 GB, 17,280 individual TIFF files.

Sounds good. I assume it is not easy to modify so that you can read directly from s3 via http? skimage.io.imread can do it. But if its not trivial, feel free to stick with this approach

Running the compression methods:
8.5 hours computing and applying illumination correction, rescaling and PNG compression.
Nice

Uploading compressed data to S3:
1 minute, 6.8 GB, 17,280 individual PNG files.
8-bit PNG?

Compression rates:
Up to 95% of space savings.

Wow!

Computing cost:
Approx. $0.7 per plate, using a m5n.xlarge spot instance ($0.07/hour x 10 hours).

That's great! Do keep in mind that the unarchived uncompressed TIFFs sitting on EFS (during Step 2 to Step 3) may drive the cost depending on how you do it, so as long as you have a reasonable plan for that, we are all set.

@shntnu will check whether some of these images are still available on /cmap/imaging/

Here is the plan after discussing with Shantanu, unless we find data on /cmap/imaging, for unarchiving and placing the images to S3, @MarziehHaghighi will download restored data to EFS in 10 plates chunks to avoid costs caused by keeping huge data on EFS. I will start tomorrow and will be done in two weeks.

@MarziehHaghighi I found several plates, possibly all. I've started uploading from /cmap/imaging/. So no need to unarchive etc. Please ping me on this thread in a few days to get status.

@shntnu Pinging you here and also two questions regarding the data you are transferring:

Do we have cp profiles on /cmap/imaging/ as well or we just have images? I think we will need profiles (single cells) for at least cell center information. @jccaicedo can you confirm that? Also I personally need cell outlines to create cell masks.
Location of the uploaded images in DGX?

Do we have cp profiles on /cmap/imaging/ as well or we just have images? I think we will need profiles (single cells) for at least cell center information. @jccaicedo can you confirm that? Also I personally need cell outlines to create cell masks.

Only images, but @gwaygenomics knows where to find the SQLite files (Level 2) because he just finished creating Level 3 from that.

Location of the uploaded images in DGX?

I thought the plan was to transfer them to S3, and has started that, but looks like the process stopped and only 7 plates were transferred.
But I now realize you wanted them transferred to DGX-1.
The good news is that I've verified that all the images you need are indeed available on /cmap/imaging.
After learning more about the DeepProfiler configuration today, I think it will be relatively easily to set it up on the Broad cluster and prepare the images right there, avoiding the need to transfer the 22TB out of /cmap/imaging. IIUC @jccaicedo is able to reduce it to ~1TB with 8-bit png and 2x downsampling.
Have a look at this https://broad.service-now.com/sp?id=kb_article&sys_id=63354be4137f0b00449fb86f3244b049 to get started with UGER.
If you'd rather not figure that out, you can transfer the images DGX-1 by connecting to Broad login node and then scp from /cmap/imaging to DGX1.

Only images, but @gwaygenomics knows where to find the SQLite files (Level 2) because he just finished creating Level 3 from that.
👉  https://github.com/broadinstitute/lincs-cell-painting/blob/master/profiles/profiling_pipeline.py#L16-L20

@shntnu I assume I can't use compressed images for creating new CP profiles (with mito features)  and need to run CellProfiler on original images. Also, for generating single cell images my initial experiments show results based on uncompressed images are more promising.

@shntnu I assume I can't use compressed images for creating new CP profiles (with mito features) and need to run CellProfiler on original images. Also, for generating single cell images my initial experiments show results based on uncompressed images are more promising.

Ah so you need the images for two different tasks

Extract new sets of features using CellProfiler  https://github.com/broadinstitute/2016_08_01_RadialMitochondriaDistribution_donna/issues/3#issuecomment-533186206 using this pipeline.
Extract deep learning features using DeepProfiler, which is what we want to do in this repo.

Is that correct?
If so, then I agree you need uncompressed images for 1. and sounds like you need them for 2. as well.
You have instructions for doing so here. Does that address what you need to do next?

After learning more about the DeepProfiler configuration today, I think it will be relatively easily to set it up on the Broad cluster and prepare the images right there, avoiding the need to transfer the 22TB out of /cmap/imaging. IIUC @jccaicedo is able to reduce it to ~1TB with 8-bit png and 2x downsampling.

Yes, this is correct. We can reduce the size to less than 1TB with the compression implemented in DeepProfiler.

If you'd rather not figure that out, you can transfer the images DGX-1 by connecting to Broad login node and then scp from /cmap/imaging to DGX1.

Remember that the DGX-1 is invisible to the rest of the Broad network. We can't copy data from/to UGER, unless there is a sort of tunnel that goes through the web.

@shntnu Considering Juan's comment about transfer to DGX from /cmap/imaging, Do you have any objection with transfering all the plates from /cmap/imaging to S3 and then transfer from S3 to DGX?

Remember that the DGX-1 is invisible to the rest of the Broad network. We can't copy data from/to UGER, unless there is a sort of tunnel that goes through the web.

Pretty sure we can still do this https://github.com/broadinstitute/2016_08_01_RadialMitochondriaDistribution_donna/blob/issues/3/process-repurposing-data/1.download-image-files.sh#L4-L6
because we did that before
i.e. log in to a Broad login node, then rsync data from /cmap/imaging to DGX-1
You'd need ssh keys to access the DGX-1 set up on the Broad login of course.
Does that make sense?

Considering Juan's comment about transfer to DGX from /cmap/imaging, Do you have any objection with transfering all the plates from /cmap/imaging to S3 and then transfer from S3 to DGX?

Pretty sure that what I have suggested above should work. Can you test that out?

Pretty sure that what I have suggested above should work. Can you test that out?

Yes we have done that before :D It seems that my memory restarts every night! Sorry Shantanu.
@jccaicedo I started the transfer. Will update you as soon as it is finished.

@shntnu and @jccaicedo 136 of 142 plates are transferred now and NAS is full. @jccaicedo Should we apply compresion on these images now and delete a couple of plates and move the rest?

The dataset has only 136 plates; not sure what the extra ones are
On Fri, Jun 5, 2020 at 2:01 PM MarziehHaghighi<EMAIL_ADDRESS>wrote:

@shntnu https://github.com/shntnu and @jccaicedo
https://github.com/jccaicedo 136 of 142 plates are transferred now and
NAS is full. @jccaicedo https://github.com/jccaicedo Should we apply
compresion on these images now and delete a couple of plates and move the
rest?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/broadinstitute/lincs-cell-painting/issues/2#issuecomment-639671338,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJHQPFLTX5KWOYE3EZQYDLRVEXIVANCNFSM4KX4D3PA
.

@shntnu where can I find the name of 136 plates? Since there are 140 plates on /cmap/imaging and also in an old metadata file I have.

https://github.com/broadinstitute/lincs-cell-painting/blob/master/metadata/platemaps/2016_04_01_a549_48hr_batch1/barcode_platemap.csv
On Fri, Jun 5, 2020 at 6:14 PM MarziehHaghighi<EMAIL_ADDRESS>wrote:

@shntnu https://github.com/shntnu where can I find the name of 136
plates? Since there are 140 plates on /cmap/imaging and also in an old
metadata file I have.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/broadinstitute/lincs-cell-painting/issues/2#issuecomment-639860458,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJHQPE3SYGFBCOE6D2IVA3RVFU6HANCNFSM4KX4D3PA
.

https://github.com/broadinstitute/lincs-cell-painting/blob/master/metadata/platemaps/2016_04_01_a549_48hr_batch1/barcode_platemap.csv
…
On Fri, Jun 5, 2020 at 6:14 PM MarziehHaghighi @.***> wrote: @shntnu https://github.com/shntnu where can I find the name of 136 plates? Since there are 140 plates on /cmap/imaging and also in an old metadata file I have. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJHQPE3SYGFBCOE6D2IVA3RVFU6HANCNFSM4KX4D3PA .

This one also has 140 plates.

These are missing
https://github.com/broadinstitute/lincs-cell-painting/blob/460f69052934f27a56e1e3de3b7bab62dea190cd/metadata/clue_manifest/clue_data_library.Rmd#L77-L82
On Fri, Jun 5, 2020 at 6:24 PM MarziehHaghighi<EMAIL_ADDRESS>wrote:

https://github.com/broadinstitute/lincs-cell-painting/blob/master/metadata/platemaps/2016_04_01_a549_48hr_batch1/barcode_platemap.csv
… <#m_3724885139149466672_>
On Fri, Jun 5, 2020 at 6:14 PM MarziehHaghighi @.***> wrote: @shntnu
https://github.com/shntnu https://github.com/shntnu where can I find
the name of 136 plates? Since there are 140 plates on /cmap/imaging and
also in an old metadata file I have. — You are receiving this because you
were mentioned. Reply to this email directly, view it on GitHub <#2
(comment)
https://github.com/broadinstitute/lincs-cell-painting/issues/2#issuecomment-639860458>,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJHQPE3SYGFBCOE6D2IVA3RVFU6HANCNFSM4KX4D3PA
.
This one also has 140 plates.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/broadinstitute/lincs-cell-painting/issues/2#issuecomment-639865619,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJHQPCUO3OWGO42UDVEMCTRVFWCBANCNFSM4KX4D3PA
.

Great! I removed the plates mentioned by Shantanu in previous comment and transferred the remained 4 plates. And Nas is full now:)
@jccaicedo data is ready!

The first batch of 65 plates has been compressed and it's available in the following path in the DGX server:
/dgx1nas1/cellpainting-datasets/2015_10_05_DrugRepurposing_AravindSubramanian_GolubLab_Broad/DP-project/outputs/compressed/
The total space being used so far is 450 GBs.

The second batch of plates has been successfully compressed and has been added to the same directory above. This results in a total of 136 plates available in compressed format. The total amount of space used for storing the images is 891GB, which makes it easier to handle in a single server for machine learning research. I will make a back up in S3 as well.
Thanks @MarziehHaghighi and @shntnu for your help with this process.

@jccaicedo Super! Can you point to the parameters used for compression?

The compression procedure for this dataset included the following steps:

Compute one illumination correction function for each channel-plate. The illumination correction function is computed at 25% of the width/height of the original images.
Apply the illumination correction function to images before any of the following compression steps
Resize the images to half the width and height, effectively reducing the total number of pixels to 25% of the original (4X compression). Images in this dataset were 2160x2160, resulting in 1080x1080. This makes sense for this specific dataset because it did not have binning, resulting in double the resolution w.r.t other datasets taken at the same magnification.
Change the pixel depth from 16 bits to 8 bits. This results in 2X compression.
Save the resulting image in PNG lossless format. This results in approximately 3X compression.

The total compression factor is (4X)(2X)(3X) = 24X, resulting in 21TB being compressed in 891GB (about 96% compression rate). Do we lose any relevant information? Our experiments in other smaller Cell Painting datasets have shown zero loss of precision in downstream analysis applications (MoA or Pathway connections), indicating no harm to the biologically meaningful data. We are just re-encoding redundant imaging information in a more efficient way.
DeepProfiler has built in functionalities to compress high-throughput images in parallel.

This is great 💯 @jccaicedo - I do not doubt that compression results in biological information loss according to those metrics, but do you know of a citation to support this? (I am curious)

@jccaicedo Thanks for documenting this!
IIUC, you also rescale after step 3, correct?
https://github.com/cytomining/DeepProfiler/blob/8be5994805df92479d030c19f186350aa31a0173/deepprofiler/dataset/compression.py#L90

@gwaygenomics I don't know of references that report results like this. However, we have data from DeepProfiler experiments in other datasets that I can share if you are interested. We hope to make those results public soon as well.
@shntnu Line 90 corresponds to step 4 (rescale intensities from 16 bits to 8 bits). We rescale images to half the resolution in step 3, according to line 82 in the code: https://github.com/cytomining/DeepProfiler/blob/master/deepprofiler/dataset/compression.py#L82
We set the scaling_factor=1.0 for most projects. For this dataset we used scaling_factor=0.5, which reduces the image to half the resolution: https://github.com/cytomining/DeepProfiler/blob/master/deepprofiler/dataset/compression.py#L62

Line 90 corresponds to step 4 (rescale intensities from 16 bits to 8 bits).

Got it. I think I meant to say that before mapping 16-bit to 8-bit, you clip the intensities to lie between 5th and 99.5th percentiles.

Your totally right! I forgot to document that step in the list. Just edited the procedure to include this.

Thanks @shntnu

Note that 2016_04_01_a549_48hr_batch1 is ~22TB (same as LCKP)

Awesome, I updated https://github.com/broadinstitute/lincs-cell-painting/issues/54#issuecomment-791481261 - do you know if, roughly, there are the same amount of tifs? There are ~2.7 million in batch 2.

We can instead ask BITS for help to transfer them from /cmap/imaging

I love this idea. Who is the point person?

Awesome, I updated #54 (comment) - do you know if, roughly, there are the same amount of tifs? There are ~2.7 million in batch 2.

There are 136 plates in Pilot 1, and 135 plates in Pilot 2. So there should be nearly (135/136) as many tiffs. Oh but wait, we didn't have brightfield in Pilot 1 but do in Pilot 2, so there should be 6/5*135/136 = 19% more tiffs in Pilot 2

I love this idea. Who is the point person?

Once you know the destination, you can just email help@broad and someone from scientific computing should be able to help.

The compression procedure for this dataset included the following steps:

Snippet from https://www.biorxiv.org/content/10.1101/2<IP_ADDRESS>3783v2.full.pdf
3.1 Image preprocessing
The original Cell Painting images in all the datasets used in this work are encoded and stored in 16-bit TIFF format. To facilitate image loading from disk to memory during training of deep learning models, we used image compression. This is only required for training, which requires repeated randomized loading of images for minibatch-based optimization.
The compression procedure is as follows:

Compute one illumination correction function for each channel-plate 35. The illumination correction function is computed at 25% of the width/height of the original images.
Apply the illumination correction function to images before any of the following compression steps.
Stretch the histogram of intensities of each image by removing pixels that are too dark or too bright (below 0.05 and above 99.95 percentiles). This expands the bin ranges when changing pixel depth and prevents having dark images as a result of saturated pixels.
Change the pixel depth from 16 bits to 8 bits. This results in 2X compression.
Save the resulting image in PNG lossless format. This results in approximately 3X compression.
The resulting compression factor is approximately (2X)(3X) = 6X.

This preprocessing pipeline is implemented in DeepProfiler and can be run with a metadata file that lists the images in the dataset that require compression, together with plate, well and site (image or field of view) information.


Added Licence, contributions, ioshields, Changelog and version tag

I added a LICENSE FILE, a contribution files some iobadges a logo and a CHANGELOG. The changelog can be automatically updated using the auto-changelog npm plugin. The version can be automatically updated using the versionbump python plugin.

@rickstaa , nice improvements. Having a logo is awesome. Some comments:

I am guessing that the logo-2 svg is intended to be the final one; it looks nice. However, it looks like the wrong one is being linked in the README.md.
In general, do avoid checking in binaries, in this case the pngs. svgs are fine. In case you developed the pngs and then converted them to svgs and want to keep track of the source images, you can simply create an issue and upload the images there. This way there is a backup copy on GitHub, but not in the repo itself.
Would be nice if you could include 1 or 2 line instructions for updating the changelog and versioning in the README or contributing.md. Would be useful for future contributors too.

@ChanderG, good points!

1. I am guessing that the logo-2 SVG is intended to be the final one; it looks nice. However, it looks like the wrong one is being linked in the README.md.

I added two logos to the repository so you can choose which one you liked the most. As I assume logo-2 has your preference I will remove the other logos and update the readme.

2. In general, do avoid checking in binaries, in this case the `png`s. `svg`s are fine. In case you developed the `png`s and then converted them to `svg`s and want to keep track of the source images, you can simply create an issue and upload the images there. This way there is a backup copy on GitHub, but not in the repo itself.

Ah, I see Goodthanks for the tip. I will remove all the png's and only keep the final .svg.

3. Would be nice if you could include 1 or 2 line instructions for updating the changelog and versioning in the README or contributing.md. Would be useful for future contributors too.

Good suggestion I will add this to the documentation!
I will send you a new pull request somewhere later this week.


Reuse a single TreeWalker in nested templates

Creating TreeWalkers isn't very optimized, so reusing a single one in the nested templates can speed it up a bit.
https://jsbench.github.io/#d0363ded1753dec3e060e61e1794896b

I don't care that much about optimizing nested templates, since that should be a pretty rare case and we added support mainly for correctness so lit-html templates didn't blow up with nested templates...
But, if reusing a TreeWalker is that much faster, what if we reused one across the all templates? That could have an impact on real-world code (though I do think we're at the point where DOM mutation operations are dominating costs already).
The Template constructor isn't re-entrant. TemplateResult#_clone shouldn't be either, but I'd want some tests of scenarios like templates with custom elements that synchronously create ShadowRoots under the polyfill, etc.

But, if reusing a TreeWalker is that much faster, what if we reused one across the all templates? That could have an impact on real-world code

That was the end-goal. 😉

The Template constructor isn't re-entrant. TemplateResult#_clone shouldn't be either, but I'd want some tests of scenarios like templates with custom elements that synchronously create ShadowRoots under the polyfill, etc.

Unfortunately, we're not there yet. https://github.com/Polymer/lit-html/pull/831 has to come first, then we need to fix the reentrancy behavior in insertNodeIntoTemplate, which can fire upgrade/connectedCallback from the insertBefore.


databinding newsapi java

The sample is mostly done. I consider a few changes/additions but I want to send this like this, so that you can see what we have up to here and we can discuss on further points.

Looks good to me, @OyaCanli . I'll be merging the PR now and then by the next evening, I'll be checking both Java and Kotlin version to see if there is any sync required, though I think it's going perfectly. Thanks a lot for working on this continuously. And now, may I request you that please try writing the article to be published on Medium? Maybe both of you and Greta can collaborate again to do that part. As this is holidays so I hope you enjoy that and do this part after you get some time. Hoping to see first article published on Enpassio soon. Thanks again and stay Enpassious :D I'm also trying to do architecture parts and should be able to get some solid progress on that very soon.

Thanks! There is something I consider changing. But I want to ask your ideas:
-I have put the network checking logic in the ArticleListFragment: https://github.com/enpassio/Databinding/blob/dev-databinding-newsapi/DatabindingWithNewsApi/app/src/main/java/com/enpassio/databindingwithnewsapi/ui/ArticleListFragment.java
If you don't have connection it shows a snack with a button to retry which checks again etc..
On the other hand, if you have the connection and get the news, go to details, then cut internet and come back to list, it checks the connection again and show no network warning. In the news app we have done in ABND that was the expected behaviour. But since we're using a repository here, we still have the data we fetched. So it shows no network warning(and an empty list) although the data is already in our repository. So I consider changing it. May be by moving the network checking logic to repository ( but it will require to pass a context and probably a callback to inform ui) Or I can change the point where we start fetching data, move it away from the constructor, so ArticleListFragment can subscribe to ViewModel even if there is no network.. I'll experiment with this..
But what do you think is the correct behaviour? @rajtheinnovator @GretaGri

@OyaCanli the idea behavior is that, the UI should care only of the contents to be shown on the screen and so the network checking part could be moved from here. The best place obviously would be either repository(it can be given the task to check for network but will do so only in separate class) or utils(better this).

Ok, thanks @rajtheinnovator . I'll try like that then.
I'm less enthousiastic for the writing part, but I know, we should do that part as well.
As a reminder: For kotlin repo(recyclerview), don't merge it yet. There is a problem with package name. @GretaGri said she will fix it when she has time.

@OyaCanli  the first PR was merged but yes, I'll wait for that to be fixed before making the final merge with master. I can understand that part about writing but hopefully that'll also be helpful for the community  and all of us :smile: as well


Public instance is being used for lots of traffic

Recently (in the past month) I've gotten bills for 1.5 TB of data transfer from the public instance of bore, at bore.pub.
If I had deployed this on AWS, there would be problems because it would be costing me between $100-$200. However I anticipated that there would be some traffic, so I deployed it on DigitalOcean specifically to take advantage of their low pricing on network egress at $0.01/GB, which is 10x cheaper than other cloud offerings.
However, 1.5 TB is way more than I was expecting for usage of this service. So this is a gentle reminder that if you're using the public instance of bore, try to use it for its intended purpose, which is as a short-lived way to expose ports to the Internet. Please respect other users and be reasonable with how much data you transfer.
By the way, if you'd like to help me keep bore running, any amount of contribution through GitHub Sponsors would be very much appreciated. Even $1/month lets me support 100 GB of network transfer through the public instance.

Thanks for making bore.pub public.
Have you considered hosting the VPS on Hetzner? I did an analysis a while back, they have some of the lowest bandwidth costs that I could find. I've been hosting my website with them for years and have no complaints.

I'd also suggest OVH Cloud, which has no bandwidth costs.

Another bump, last month we suddenly saw a usage of 2.8 TB per month. If you're seeing this and sending a lot of traffic over bore, please reduce your usage! :)


[intel-mkl] Lookup oneapi installation for mkl headers

Describe the pull request

What does your PR fix?
Scenarios where oneMKL is installed but vcpkg's intel-mkl port cannot locate MKL.

Which triplets are supported/not supported? Have you updated the CI baseline?
NA

Does your PR follow the maintainer guide?

I don't think port changed per say, but I am not sure how empty packages are handled when lookup logic changes. Please advice.

If you are still working on the PR, open it as a Draft: https://github.blog/2019-02-14-introducing-draft-pull-requests/

Could you please also update CONTROL file?
Add Port-Version: 1 below Version: 2020.0.0.

Thank you for help. I pushed the commit, please check.

My bad, this requires the x-add- command run. give me a moment.

Run vcpkg x-add-version intel-mkl, then submit the changes.


feat(localization): translate deleteOrganization and domainSection objects to Korean

Type of change

[ ] 🐛 Bug fix
[ ] 🌟 New feature
[ ] 🔨 Breaking change
[ ] 📖 Refactoring / dependency upgrade / documentation
[x] other: localization

Packages affected

[ ] @clerk/clerk-js
[ ] @clerk/clerk-react
[ ] @clerk/nextjs
[ ] @clerk/remix
[ ] @clerk/types
[ ] @clerk/themes
[x] @clerk/localizations
[ ] @clerk/clerk-expo
[ ] @clerk/backend
[ ] @clerk/clerk-sdk-node
[ ] @clerk/shared
[ ] @clerk/fastify
[ ] @clerk/chrome-extension
[ ] gatsby-plugin-clerk
[ ] build/tooling/chore

Description

[ ] npm test runs as expected.
[ ] npm run build runs as expected.

translate deleteOrganization and domainSection objects to Korean

Great stuff, thank you @JungHoonGhae
Merging so we can get this out soon.

This PR has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.


Use new StreamWriterBuilder/CharReaderBuilder

I see these changes in the conformance profile:
@@ -2,21 +2,6 @@

 ## 1. Parse Validation

-* `../data/jsonchecker/fail07.json` is invalid but was mistakenly deemed valid.
-~~~js
-["Comma after the close"],
-~~~
-
-* `../data/jsonchecker/fail08.json` is invalid but was mistakenly deemed valid.
-~~~js
-["Extra close"]]
-~~~
-
-* `../data/jsonchecker/fail10.json` is invalid but was mistakenly deemed valid.
-~~~js
-{"Extra value after close": true} "misplaced quoted value"
-~~~
-

@@ -33,8 +18,18 @@
 break"]
+* `../data/jsonchecker/fail29.json` is invalid but was mistakenly deemed valid.
+~~~js
+[0e]
+~~~
+
+* `../data/jsonchecker/fail30.json` is invalid but was mistakenly deemed valid.
+~~~js
+[0e+]
+~~~
+

-Summary: 28 of 34 are correct.
+Summary: 29 of 34 are correct.

Thanks


[BUG]  Command: powershell.exe -NoProfile -NonInteractive "new-object System.Security.Cryptography.X509Certificates.X509Certificate2

:information_source: Info
Version: v3.7.0
:speech_balloon: Description
I followed the steps described here to create a self signed pfx: [Flutter Desktop Documentation](https://docs.flutter.dev/development/platform-integration/windows/building#create-a-self-signed-pfx-certificate-for-local-testing: create self signed pfx)
Now when I run the command flutter pub run msix:create it throws this error:
Enter a description of your problem here.
Any logs, error and assets you have that can help to find this bug.
To run the tool with extended logs use the -v argument, for example: flutter pub run msix:create -v
[ +194 ms] executing: C:\Dev\flutter\bin\cache\dart-sdk\bin\dart __deprecated_pub --color run msix:create -v
[     0.025s] parsing cli arguments
[     0.236s] validating config values
[     0.252s] running "flutter build windows"...
[    34.715s] building msix files...
[    34.716s] validating build files
[    34.731s] cleaning temporary files
[    34.789s] create app icons
[    34.790s] generating icons
[    54.996s] copying VC libraries
[    55.018s] getting certificate publisher
[    55.019s] getting certificate "Subject" by file certificate
Unhandled exception:
ProcessException: The system cannot find the file specified.

  Command: powershell.exe -NoProfile -NonInteractive "new-object System.Security.Cryptography.X509Certificates.X509Certificate2(\"C:\Users\admin\projects\dvhdesktopapp\CERTIFICATE.pfx\",
        \"J#######\") | select -expandproperty Subject -First 1"
#0      _ProcessImpl._start (dart:io-patch/process_patch.dart:401:33)
#1      Process.start (dart:io-patch/process_patch.dart:38:20)
#2      _runNonInteractiveProcess (dart:io-patch/process_patch.dart:578:18)
#3      Process.run (dart:io-patch/process_patch.dart:49:12)
#4      SignTool._executePowershellCommand (package:msix/src/sign_tool.dart:81:21)
#5      SignTool._getCertificateSubjectByFile (package:msix/src/sign_tool.dart:136:53)
#6      SignTool.getCertificatePublisher (package:msix/src/sign_tool.dart:36:23)
#7      Msix._buildMsixFiles (package:msix/msix.dart:114:24)
<asynchronous suspension>
#8      Msix._createMsix (package:msix/msix.dart:98:5)
<asynchronous suspension>
#9      Msix.create (package:msix/msix.dart:57:5)
<asynchronous suspension>
#10     main (file:///C:/Users/admin/AppData/Local/Pub/Cache/hosted/pub.dev/msix-3.7.0/bin/create.dart:4:3)
<asynchronous suspension>
[+61147 ms] "flutter run" took 61,671ms.
[   +8 ms] pub finished with exit code 255
[   +2 ms]
           #0      throwToolExit (package:flutter_tools/src/base/common.dart:10:3)
           #1      _DefaultPub.interactively (package:flutter_tools/src/dart/pub.dart:586:7)
           <asynchronous suspension>
           #2      PackagesForwardCommand.runCommand (package:flutter_tools/src/commands/packages.dart:250:5)
           <asynchronous suspension>
           #3      FlutterCommand.run.<anonymous closure> (package:flutter_tools/src/runner/flutter_command.dart:1257:27)
           <asynchronous suspension>
           #4      AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:150:19)
           <asynchronous suspension>
           #5      CommandRunner.runCommand (package:args/command_runner.dart:209:13)
           <asynchronous suspension>
           #6      FlutterCommandRunner.runCommand.<anonymous closure> (package:flutter_tools/src/runner/flutter_command_runner.dart:283:9)
           <asynchronous suspension>
           #7      AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:150:19)
           <asynchronous suspension>
           #8      FlutterCommandRunner.runCommand (package:flutter_tools/src/runner/flutter_command_runner.dart:229:5)
           <asynchronous suspension>
           #9      run.<anonymous closure>.<anonymous closure> (package:flutter_tools/runner.dart:64:9)
           <asynchronous suspension>
           #10     AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:150:19)
           <asynchronous suspension>
           #11     main (package:flutter_tools/executable.dart:91:3)
           <asynchronous suspension>

[ +140 ms] ensureAnalyticsSent: 134ms
[        ] Running 0 shutdown hooks
[        ] Shutdown hooks complete
[        ] exiting with code 255

:scroll: Pubspec.yaml

name: DVH
description: DVH APP

publish_to: "none" # Remove this line if you wish to publish to pub.dev

version: 1.0.0+1

environment:
  sdk: ">=2.12.0 <3.0.0"

dependencies:
  flutter:
    sdk: flutter

  camera: ^0.10.0+1
  country_code_picker: ^2.0.2
  cupertino_icons: ^1.0.3
  dart_ipify: ^1.1.0
  device_info_plus: ^3.0.1
  double_back_to_close_app: ^2.0.1
  easy_mask: ^2.0.1
  email_validator: ^2.0.1
  encrypt: ^5.0.0
  faker: ^2.0.0
  firebase_auth: ^3.6.3
  firebase_core: ^1.20.1
  flutter_font_icons: ^2.2.4
  flutter_spinkit: ^5.0.0
  fuzzywuzzy: ^0.1.7
  get: ^4.6.5
  google_fonts: ^3.0.1
  google_sign_in: ^5.0.4
  hive: ^2.0.5
  hive_flutter: ^1.1.0
  http: ^0.13.4
  image: ^3.0.4
  intl: ^0.17.0
  livechatt: ^1.2.0
  package_info_plus: ^1.3.0
  path: ^1.8.0
  path_provider: ^2.0.2
  permission_handler: ^8.3.0
  provider: ^6.0.3
  shared_preferences: ^2.0.6
  sign_in_with_apple: ^3.3.0
  system_info_plus: ^0.0.5
  url_launcher: ^6.0.6
  webview_flutter: ^3.0.0
  sklite:
    git: https://github.com/SebghatYusuf/SkLite-dart.git
  csv: ^5.0.1
  connectivity_plus: ^2.3.5
  sqflite: ^2.0.2+1
  pay: ^1.0.10
  flutterwave_standard: ^1.0.6
  uuid: ^3.0.6
  google_mlkit_text_recognition: ^0.4.0
  google_mlkit_commons: ^0.2.0
  dropdown_search: ^5.0.2
  shimmer: ^2.0.0
  firebase_analytics: ^9.3.3
  syncfusion_flutter_pdfviewer: ^20.2.48-beta
  auto_size_text: ^3.0.0
  firebase_in_app_messaging: ^0.6.0+27
  share_plus: ^4.5.3
  firebase_core_platform_interface: 4.5.1
  in_app_update: ^4.0.1
  in_app_review: ^2.0.6
  firebase_core_desktop: ^1.0.2
  firebase_auth_desktop: ^1.0.2
  flutter_svg: ^1.0.3
  cool_dropdown: ^1.4.2
  overlayment: ^0.2.1
  google_sign_in_dartio: ^0.2.2
  number_paginator: ^0.2.3
  in_app_purchase: ^3.1.5
  window_manager: ^0.3.1
  uni_links_desktop: ^0.1.6
  uni_links: ^0.5.1
  bot_toast: ^4.0.3

dependency_overrides:
  modal_bottom_sheet: ^3.0.0-pre

dev_dependencies:
  flutter_driver:
    sdk: flutter
  flutter_native_splash: ^1.2.4
  flutter_test:
    sdk: flutter
  integration_test:
    sdk: flutter
  hive_generator: ^1.1.1
  build_runner: ^2.1.5
  msix: ^3.7.0

flutter:
  uses-material-design: true

  assets:
    - assets/
    - images/
    - images/icons/
    - images/svg/

msix_config:
  display_name: DVH
  publisher_display_name: Vehicle Databases
  identity_name: vehicledatabases.dvh.app
  msix_version: <IP_ADDRESS>
  logo_path: C:\Users\admin\projects\dvhdesktopapp\images\AppLogo.png
  start_menu_icon_path: C:\Users\admin\projects\dvhdesktopapp\images\AppLogo.png
  tile_icon_path: C:\Users\admin\projects\dvhdesktopapp\images\AppLogo.png
  capabilities: internetClient, location
  certificate_path: C:\Users\admin\projects\dvhdesktopapp\CERTIFICATE.pfx
  certificate_password: #######
  certificate_subject: CN=MY_CN

Are you sure the path to the certificate is current?
Log: The system cannot find the file specified

@YehudaKremer Sorry for the late reply,
Yes the certificate exists in same the directory as logo (icon) and they're all using absolute paths.

I just experienced the same issue and I found out why it happens and how to fix it:
The file it can't find is powershell.exe, which is not on always on PATH apparently, so the fix is simply adding the directory to PATH.
In my case it's in
C:\Windows\System32\WindowsPowerShell\v1.0
To fix this in the library I would

Test if powershell.exe is available
If not, try to find it in common paths
If not found exit with a proper error message

@iAndyHD3 Thank you for your instructions, actually powershell was on PATH, issue was having anti virus on, and it was blocking the process.


add healthz endpoint for liveness probing

Hey,
i just added your container to my helm chart for kubernetes deployments.
ill create a PR late to add it to the upstream helm chart here: SQLJames/factorio-server-charts
for liveness checks it is really useful to have probing endpoints like /healthz as described here:
https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

Ah, thanks, I will integrate them ASAP I finish my travel (2 days left)

PR: https://github.com/SQLJames/factorio-server-charts/pull/57

Released https://github.com/nekomeowww/factorio-rcon-api/releases/tag/v1.1.0


Delete all unused AB3 classes

Delete Person classes & test classes

Deleted for storage

I think I have deleted most (if not all) of them in #54.
Do help to check if anything is missed out. Thanks guys!


Implement checks for non existing object

@dasnotme Please, take a look. Here I implemented following:

Replaced minitest with rspec (yes, no love to minitest anymore :( )
Added optional attribute render_when_empty
Implemented checks to make sure that it doesn't render empty objects to avoid missing method errors

@SamyRai Please take code styles files (.rubocop, .reek, .codeclimate) from github.com/distribusion/supa
I will continue after the fixes

All changes will be implemented under distribusion organization.