Pod-to-pod networking broken

From my tests, pod-to-pod networking through service IPs doesn't work. I'm pretty sure it's because the packet are not SNATed:
$ time curl -i mail.master
HTTP/1.1 404 Not Found
Content-Type: application/octet-stream
Content-Length: 58
Server: Jetty(9.2.9.v20150224)

Could not find resource for full path: http://mail.master/
real	0m0.019s
user	0m0.008s
sys	0m0.000s
$ kubectl --namespace=master exec keystone-zdr02 -- wget mail
--2015-07-11 00:08:44--  http://mail/
Resolving mail (mail)... <IP_ADDRESS>
Connecting to mail (mail)|<IP_ADDRESS>|:80... failed: Connection timed out.
Retrying.

I'll try to add a MASQERADE rule. I know this will hide the real source IP from the pod, but it's not worse than with the original proxy ;)

("opened" in #2)
I used MASQUERADE because otherwise the source IP is the real one, thus the destination pod replies with its own IP instead of the service IP, which causes the packet to be rejected.

IMHO, what we need is a way to have the reply packet catched by the kernel's connection tracking, and apply a rule to set the service IP as the source.
I used MASQUERADE to do that as a side effect, since it replaces the source IP with the outgoing interface IP, which make the target pod to reply to the host, which then catches it with the conntrack, and replaces the reply's source IP to the service IP.
There may be a way (with ebtables?) to catch the packet as it goes through the pod's bridge (flannel.1 for me). This would make the target pod see "source pod IP -> target pod IP" and the source pod see "source pod IP -> service IP". The problem is... I don't know enough of networking to do that.

That makes sense.
I don't know how to do the last part either but I'll look into it as that sounds like what we would want ideally, but MASQUERADE seems like a sane solution in the meantime.

MASQUERADEcould be limited to traffic going from the pod to the service IPs, that would allow us to solve the problem with only one rule like -j MASQUERADE -s pod_network -d service_network.

I found this, I'm having a look: http://ebtables.netfilter.org/br_fw_ia/br_fw_ia.html

I think the limited MASQUERADE rule seems like the best solution for now.

Yes, I'm trying to implement it on my "development production" single-node cluster.

Thank you. I'm still reading through the options for manipulating Source Addresses but so far that sounds ideal.

The MASQUERADE rule I gave is not ok, POSTROUTING gets the pods IPs, so it works with this ugly thing:
iptables -t nat -I POSTROUTING -j MASQUERADE -s <IP_ADDRESS>/16 -d <IP_ADDRESS>/16

I may use the recent pattern finally ;)

The following additionnal rules make it work for me:
iptables -t nat -I PREROUTING -s pod_network -d service_network -m state --state NEW -m recent --name KUBE-MASQUERADE --set
iptables -t nat -I POSTROUTING -m recent --name KUBE-MASQUERADE --rcheck --seconds 1 --reap -j MASQUERADE

Great!
I guess we'll have to move those out to more chains so that we can update these in syncProxyRules() and set the chains up in iptablesinit.
As I read the snippet you posted, service_network is the target ip's (?); then is the pod_network then the proxyIP (host/listening ip) on each node or?

Yes, I think at least one more chain is required since no kubernetes' specific chain is targeted from POSTROUTING.
The service network is the --service-cluster-ip-range of apiserver. The pod network is provided by flannel in my setup; it's the --bip of the docker daemon. I don't how you can get this information though... an alternative way of achieve the same result would be to use an ipset or to put as many rules as there are pods on the host.

Hmm, ok well we'll have to figure that out then. Thanks again. I'll look into it more tomorrow.

@Symmetric, any thoughts on this? I know you were looking forward to the source ip preservation but this seems necessary, we will probably need a MASQUERADE rule.
Discussion started here: https://github.com/BenTheElder/kubernetes/pull/2#issuecomment-120555518

This is puzzling, I'll have to dig deeper here -- when I tested Tim's basic rules, and a previous iteration of the PR, things seemed to work fine. My understanding of DNAT is that the return packet should be un-DNAT'd so the src and dst IPs at the originator will be correct (i.e. the response packet should have the Service IP as the source once iptables is finished with it).
 Sounds like Mikeal is using Flannel, which might be complicating things.
I should have time to dig in a day or two. But yes, it would be desirable to keep the true src IP of service packets, since that allows us to do proper firewalling at the destination.

I have not, so far in my own testing all of the problems i've found appear to have been related to vagrant not the proxy. I'm experimenting with the current source now, but as of yet have not reproduced the issue.
I'm not very familiar with Flannel, perhaps it is an issue specific to using Flannel.

Latest full e2e results:
Summarizing 3 Failures:

[Fail] Kubectl client Simple pod [It] should support port-forward
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/kubectl.go:197

[Fail] Services [It] should be able to create a functioning NodePort service
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/service.go:1036

[Fail] ServiceAccounts [It] should mount an API token into pods
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/util.go:915

Ran 51 of 98 Specs in 2844.044 seconds
FAIL! -- 48 Passed | 3 Failed | 2 Pending | 45 Skipped --- FAIL: TestE2E (2849.25s)

The first and last failures appear to be unrelated to the proxy, the second one however...
• Failure [330.666 seconds]
Services
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/service.go:798
  should be able to create a functioning NodePort service [It]
  /go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/service.go:327

  Error waiting for the url http://<IP_ADDRESS>:31673 to be reachable
  Expected error:
      <*errors.errorString | 0x209345230>: {
          s: "timed out waiting for the condition",
      }
      timed out waiting for the condition
  not to have occurred

  /go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/service.go:1036

I will switch to master and compare results.
It should be noted that a number of tests are now skipped on vagrant because they don't make sense, so "full" is actually missing a few tests that would be run on e.g. GCE.

For reference, the currently comparable master gives only one failure (the first).
Summarizing 1 Failure:

[Fail] Kubectl client Simple pod [It] should support port-forward
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/kubectl.go:197

Ran 51 of 98 Specs in 2225.808 seconds
FAIL! -- 50 Passed | 1 Failed | 2 Pending | 45 Skipped --- FAIL: TestE2E (2231.01s)

@Symmetric I think it is not going through the conntrack code because it's at IP level, and replying on a local bridge doesn't go through IP code because it doesn't have to be routed. See here:

The bridge's decision for a frame (as seen on Figure 2b) can be one of these:

bridge it, if the destination MAC address is on another side of the bridge;
flood it over all the forwarding bridge ports, if the position of the box with the destination MAC is unknown to the bridge;
pass it to the higher protocol code (the IP code), if the destination MAC address is that of the bridge or of one of its ports;
ignore it, if the destination MAC address is located on the same side of the bridge.

So, something may be possible at the bridge (ebtables) level, or somewhere unknown to me. I will try to upgrade my flannel, just in case, but as per the 3rd rule quoted here, they would need to do something very special. The standard configuration as per kubernetes getting-started guide for CoreOS creates a vxlan to allow containers to communicate on a multi-host bridge.

I completely stopped and removed flannel (including the vxlan interface it created). Masquerading is still required for pod-to-pod networking to work.
I also tried to set the reply's MAC address to the docker bridge's IP using DNAT/SNAT rules at the ebtables level without success.
For SNAT I used:
ebtables -t broute -A BROUTING --ip-destination <IP_ADDRESS>/24 -p 0x0800 -j mark --mark-set 1
ebtables -t nat -A POSTROUTING --mark 1 -j snat --to-source 02:0e:e4:cc:0b:cf

For DNAT I used:
ebtables -t nat -A PREROUTING --in-interface docker0 --ip-destination <IP_ADDRESS>/16 -p 0x0800 -j redirect

Having any of these, or both, didn't solve the problem :(

Finally got a chance to look at this. As stated above, the problem is that traffic on the bridge isn't going through iptables. This is configurable behaviour (see bridge-nf for details).
I set this on the node:
sudo  sysctl -w net.bridge.bridge-nf-call-iptables=1
sudo  sysctl -w net.bridge.bridge-nf-call-ip6tables=1

And the rules work again. @MikaelCluseau, can you give this a try and see if it fixes things in your environment?
bridge-nf seems to have been introduced in kernel 2.6.1, which is satisfied by the k8s requirement for RHEL 2.6.32-431/other 3.10, so I think it's safe to assume this flag is present.

Thanks for looking into it @Symmetric.
I'm in the process of updating to the latest master and re-running everything and looking at those e2e tests, since I am still seeing failure on proxy related (NodePort) tests which suggests that the rules may not be fully functioning as kubernetes expects still. I'll check in with findings later.
Hopefully it's just a flake and the configuration change fixes @MikaelCluseau's issues.

@Symmetric ok, that was somewhere unknown to me :-) Under CoreOS, the nf-bridge module isn't loaded by default, and loading it is enough to have the sysctls set to 1:
# modprobe br-netfilter
# cat /proc/sys/net/bridge/bridge-nf-call-iptables
1

After that, you are right, the bridge traffic goes through the IP stack and pod-to-pod networking works as expected, this is really great, thanks! :-)
The getting-started for CoreOS will have to be updated so that node.yaml has the following added:
write_files:
  - path: /etc/modules-load.d/kubernetes.conf
    content: br_netfilter
coreos:
  units:
    - name: systemd-modules-load.service
      command: restart

For reference, the output of my test:
$ kubectl --namespace=master exec mail-9j4y6 -- curl --connect-timeout0 keytone:35357 >/dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   765  100   765    0     0  88357      0 --:--:-- --:--:-- --:--:--  106k
$ kubectl --namespace=master logs keystone-ea7b4 |tail -n1
2015-07-15 19:07:02.884 20 INFO eventlet.wsgi.server [-] <IP_ADDRESS> - - [15/Jul/2015 19:07:02] "GET / HTTP/1.1" 300 932 0.001088

This is not CoreOS but Linux >=3.18 specific, as can be seen here:

Since Linux kernel 3.18-rc1, you have to modprobe br_netfilter to enable bridge-netfilter.

@MikaelCluseau great news, glad we didn't have to get into the gnarly iptables rules!
@BenTheElder, I think we need to make sure that this flag is set on all kubernetes clusters when upgrading to the version that includes this patch. Not sure how to go about that; are you aware of a mechanism in kubelet to set sysctls? Or do we need to set this in the salt files (and every other getting started guide)? Obviously the former is vastly preferable.
There's also a question about how to handle upgrades; before upgrading an existing k8s cluster to the version that introduces this patch, operators will need to make sure the module is loaded and the sysctls set. Perhaps one to follow up with the k8s guys, as I don't know if there is a precedent for this issue that we can copy.

@MikaelCluseau excellent!
@Symmetric I'm not aware of one, as far as I can tell all of the that is done in the salt files. A quick search suggests that sysctl does not exist in any of the go source but some is in the guides and salt files.
I think we may have to open an issue upstream with the k8s guys.

Huho, I have last thing that will very probably require MASQUERADING, for a very specific case: if a pod connects to a service that it serves, it will its own IP from the source side, will answer to that directly, and is not catched by connection tracking. We have that on a replicated service that tries to check if it is a duplicate by connecting to its advertised host name, which is bound to the service IP: the connection stay in SYN_SENT state. netstat in the pod gives:
Proto Recv-Q Send-Q Local Address           Foreign Address         State
[...]
tcp6       0      1 <IP_ADDRESS>:49381      <IP_ADDRESS>:9092      SYN_SENT

<IP_ADDRESS> is the service IP bound to <IP_ADDRESS>.
This will require a masquerading rule in the form iptables -t nat -I POSTROUTING -s  -d  -j MASQUERADE or something like that. I still have to test that though, but I wanted to share if anyone here have an idea like "hey, you need the br_netfilter module" :)

Hmm. Perhaps the rule can be added separately/externally to the proxy? That would still be less than ideal though. I'm not sure of a more elegant solution but it would be nice to avoid falsifying the source address in the general case.

Erf... again I used <> without previewing... but anyway this rules doesn't seems to work. I'm still looking for the best solution. The rule was intented exactly for the case --source pod_IP --destination pod_IP, so very specific (will not falsify anything but podA to podA traffic).

Ah, yes that does seem like a very reasonable rule. @Symmetric would this be problematic for calico?

I tried many ways... this kind of rule doesn't work. From what I could understand using -j TRACE on the connection packet, the packet stops traversing the rules after the matching DNAT rule, because it sets a conntrack entry. Which mean the MASQUERADE rules are not applied after the DNAT, and if we don't DNAT we don't set the right target for the connection.
The only solution I could find was to fallback to some userland proxy and a REDIRECT rule, which is a quite disappointing fallback :( I continue to look for a better solution.

Hmm, sorry about that. I'm not really sure what would be a better solution. I'm studying for my summer finals right now but i'll be back on the case by tuesday afternoon in theory.

Hi @BenTheElder, I hope you finals went good :)
I still haven't found a better solution than either using userspace proxying or making the pod connect to himself by mapping the expected name to localhost in /etc/hosts (defeating the purpose of the "self connection check", if I understand it correctly; I may ask people of Apache Kafka about that, but it's another story).
Also, I think the interpretation I read somewhere on the Internets about "it's because a conntrack has been set" is wrong. I used a TRACE rule in mangle's PREROUTING table, and the life of the packet doesn't end after the DNAT rule in the normal case. The packet stops appearing in iptables after it has been DNATed specifically to the same IP than the source one. I reformated the trace logs below, using "___" when the value hasn't changed.
pod2-to-pod1 (the normal case):
Position                                |      IN |     OUT |      PHYSIN |                                       MAC |           SRC |            DST | LEN |  TOS | PREC | TTL |    ID | DF | PROTO |   SPT |  DPT |        SEQ | ACK | WINDOW |  RES | SYN | URGP | OPT |<PHONE_NUMBER>02080A19FA54340000000001030307) |     PHYSOUT
--------------------------------------- | ------- | ------- | ----------- | ----------------------------------------- | ------------- | -------------- | --- | ---- | ---- | --- | ----- | -- | ----- | ----- | ---- | ---------- | --- | ------ | ---- | --- | ---- | --- | ------------------------------------------ | -----------
raw:PREROUTING:policy:3                 | docker0 |         | vethb63f6b7 | 02:47:40:98:8f:97:02:42:0a:f4:25:da:08:00 | <IP_ADDRESS> | <IP_ADDRESS> |  60 | 0x00 | 0x00 |  64 | 30198 |    |   TCP | 34925 | 9092 |<PHONE_NUMBER> |   0 |  28200 | 0x00 |     |    0 |     |                                            |
mangle:PREROUTING:policy:1              | _______ | _______ | ___________ | _________________________________________ | _____________ | ______________ | ___ | ____ | ____ | ___ | _____ | __ | _____ | _____ | ____ | __________ | ___ | ______ | ____ | ___ | ____ | ___ | __________________________________________ | ___________
nat:PREROUTING:rule:1                   | _______ | _______ | ___________ | _________________________________________ | _____________ | ______________ | ___ | ____ | ____ | ___ | _____ | __ | _____ | _____ | ____ | __________ | ___ | ______ | ____ | ___ | ____ | ___ | __________________________________________ | ___________
nat:KUBE-PORTALS-CONTAINER:rule:197     | _______ | _______ | ___________ | _________________________________________ | _____________ | ______________ | ___ | ____ | ____ | ___ | _____ | __ | _____ | _____ | ____ | __________ | ___ | ______ | ____ | ___ | ____ | ___ | __________________________________________ | ___________
nat:KUBE-SVC-9xs1gp9Yi0xCMiVqbUL:rule:1 | _______ | _______ | ___________ | _________________________________________ | _____________ | ______________ | ___ | ____ | ____ | ___ | _____ | __ | _____ | _____ | ____ | __________ | ___ | ______ | ____ | ___ | ____ | ___ | __________________________________________ | ___________
nat:KUBE-SVC-iWFvW7fh34r6zYRIvv0:rule:2 | _______ | _______ | ___________ | _________________________________________ | _____________ | ______________ | ___ | ____ | ____ | ___ | _____ | __ | _____ | _____ | ____ | __________ | ___ | ______ | ____ | ___ | ____ | ___ | __________________________________________ | ___________
mangle:FORWARD:policy:1                 | _______ | docker0 | ___________ | 02:42:0a:f4:25:7d:02:42:0a:f4:25:da:08:00 | _____________ |  <IP_ADDRESS> | ___ | ____ | ____ | ___ | _____ | __ | _____ | _____ | ____ | __________ | ___ | ______ | ____ | ___ | ____ | ___ | __________________________________________ | veth0fc21f5
filter:FORWARD:rule:1                   | _______ | _______ | ___________ | _________________________________________ | _____________ | ______________ | ___ | ____ | ____ | ___ | _____ | __ | _____ | _____ | ____ | __________ | ___ | ______ | ____ | ___ | ____ | ___ | __________________________________________ | ___________
filter:DOCKER:return:2                  | _______ | _______ | ___________ | _________________________________________ | _____________ | ______________ | ___ | ____ | ____ | ___ | _____ | __ | _____ | _____ | ____ | __________ | ___ | ______ | ____ | ___ | ____ | ___ | __________________________________________ | ___________

pod1-to-pod1 (the problematic case):
Position                                |      IN | OUT |      PHYSIN |                                       MAC |           SRC |            DST | LEN |  TOS | PREC | TTL |    ID | DF | PROTO |   SPT |  DPT |       SEQ | ACK | WINDOW |  RES | SYN | URGP | OPT |<PHONE_NUMBER>02080A19F404C60000000001030307)
--------------------------------------- | ------- | --- | ----------- | ----------------------------------------- | ------------- | -------------- | --- | ---- | ---- | --- | ----- | -- | ----- | ----- | ---- | --------- | --- | ------ | ---- | --- | ---- | --- | ------------------------------------------
raw:PREROUTING:policy:2                 | docker0 |     | veth0fc21f5 | 02:47:40:98:8f:97:02:42:0a:f4:25:7d:08:00 | <IP_ADDRESS> | <IP_ADDRESS> |  60 | 0x00 | 0x00 |  64 | 14813 |    |   TCP | 58587 | 9092 | 414465307 |   0 |  28200 | 0x00 |     |    0 |     |
mangle:PREROUTING:policy:1              | _______ | ___ | ___________ | _________________________________________ | _____________ | ______________ | ___ | ____ | ____ | ___ | _____ | __ | _____ | _____ | ____ | _________ | ___ | ______ | ____ | ___ | ____ | ___ | __________________________________________
nat:PREROUTING:rule:1                   | _______ | ___ | ___________ | _________________________________________ | _____________ | ______________ | ___ | ____ | ____ | ___ | _____ | __ | _____ | _____ | ____ | _________ | ___ | ______ | ____ | ___ | ____ | ___ | __________________________________________
nat:KUBE-PORTALS-CONTAINER:rule:197     | _______ | ___ | ___________ | _________________________________________ | _____________ | ______________ | ___ | ____ | ____ | ___ | _____ | __ | _____ | _____ | ____ | _________ | ___ | ______ | ____ | ___ | ____ | ___ | __________________________________________
nat:KUBE-SVC-9xs1gp9Yi0xCMiVqbUL:rule:1 | _______ | ___ | ___________ | _________________________________________ | _____________ | ______________ | ___ | ____ | ____ | ___ | _____ | __ | _____ | _____ | ____ | _________ | ___ | ______ | ____ | ___ | ____ | ___ | __________________________________________
nat:KUBE-SVC-iWFvW7fh34r6zYRIvv0:rule:1 | _______ | ___ | ___________ | _________________________________________ | _____________ | ______________ | ___ | ____ | ____ | ___ | _____ | __ | _____ | _____ | ____ | _________ | ___ | ______ | ____ | ___ | ____ | ___ | __________________________________________

nat:KUBE-SVC-9xs1gp9Yi0xCMiVqbUL:rule:1 is -A KUBE-SVC-9xs1gp9Yi0xCMiVqbUL -m comment --comment "master/kafka1:" -j KUBE-SVC-iWFvW7fh34r6zYRIvv0
nat:KUBE-SVC-iWFvW7fh34r6zYRIvv0:rule:1 is -A KUBE-SVC-iWFvW7fh34r6zYRIvv0 -p tcp -m comment --comment "master/kafka1:" -m recent --set --name KUBE-SVC-iWFvW7fh34r6zYRIvv0 --mask <IP_ADDRESS> --rsource -j DNAT --to-destination <IP_ADDRESS>:9092

Hope this helps.

A solution could be to manage the pod's iptable, if it's possible, and use a REDIRECT rule there.
If it's worth it, we could even handle the LB case this way, if we set up rules like this (for 3 hosts LB):

in the pod, a probability rule, like -j REDIRECT --probability 0.333
on the host, use a subset of the normal rules in this case, starting a probability 0.5, to the other replicas (so -j DNAT --probability 0.5 --to-destination replica2:port plus -j DNAT --to-destination replica3:port
This will require a specific set of rule per endpoint when the service has more than 1 endpoint, though.

@MikaelCluseau thanks for updating me!
They went well (I think).
Is this still with flannel?
Hmm; With those rules, that seems to kind of break the expected round robin aspect, particularly for other services trying to access something that happens to be running on the same host. Also How would we handle single endpoint services?

This is still with flannel, but I don't see where it could interact, especially when the packet is already traversing the rules. The only thing it does in iptables is:
-A POSTROUTING -s <IP_ADDRESS>/16 -j FLANNEL
-A FLANNEL -d <IP_ADDRESS>/16 -j ACCEPT
-A FLANNEL ! -d <IP_ADDRESS>/4 -j MASQUERADE

The -j FLANNEL is the last rule in POSTROUTING.
I meant those rules only when the packet comes from one the endpoints. To be more verbose, what I'm thinking about, outside the pod, is a service rule (KUBE-SVC-{{service}}) like:
-s {replica1-ip} -j KUBE-SVC-{to-replica2} --probability 0.5
-s {replica1-ip} -j KUBE-SVC-{to-replica3} --probability 1

-s {replica2-ip} -j KUBE-SVC-{to-replica1} --probability 0.5
-s {replica2-ip} -j KUBE-SVC-{to-replica3} --probability 1

-s {replica3-ip} -j KUBE-SVC-{to-replica1} --probability 0.5
-s {replica3-ip} -j KUBE-SVC-{to-replica2} --probability 1

-j KUBE-SVC-{to-replica1} --probability 0.33
-j KUBE-SVC-{to-replica2} --probability 0.5
-j KUBE-SVC-{to-replica3} --probability 1

Which means that a packet to this service from replica would traverse this sequence:

on the pod: -j REDIRECT --to... --probability 0.33
on the host: -s {replica1-ip} -j KUBE-SVC-{to-replica2} --probability 0.5
on the host: -s {replica1-ip} -j KUBE-SVC-{to-replica3} --probability 1

The problem here is the O(n**2) rule complexity (a 100 replicas will produce 100 times 99 rules set for the specific case plus 1 time the 100 rule set for the normal case). But well, in this case, you have a service that requires a 100 replicas so it's probably not that a big deal. Also, we can lower the rule count by filtering the specific rule sets to the only endpoint that are local to the host we manage.

Hmm. I'm not sure how reliably we could handle filtering though as we'd need to know all of the host addresses, and otherwise it would be very costly.
I'm still interested in why it won't work from pod1->pod as well, it seems odd that it works pod1->pod2 and then stalls pod1->pod1.

What is different between raw:PREROUTING:policy:2 and raw:PREROUTING:policy:3 ?

I'm not sure how reliably we could handle filtering though as we'd need to know all of the host addresses, and otherwise it would be very costly.

I'm not sure as I haven't worked a lot in kubernetes, but the set of pods to handle the the set of pod that (1) are endpoints of the service and (2) have Status.HostIP = to our host IP (as given by the kubelet).

I'm still interested in why

I'm too, but I don't have this answer (it's somewhere in Linux's sources of course).

What is different between raw:PREROUTING:policy:2 and raw:PREROUTING:policy:3 ?

One to match pod1-to-pod1 and one to match pod2-to-pod1 (tracing everything is not possible :)).

Hmm. I'm going to see if I can ping @thockin on this one, this is a bit beyond my limited iptables knowledge, in the meantime I want to keep digging around to see if I can find anything but if not then I suppose we may have to fall back on something like the rule filtering you suggested. It'd be a shame to increase the complexity so much though.

Yes, this why we need to see if it is worth the complexity. IMHO it's not and we should just add a rule in the container so it loops back to itself when it wants to use a service it provides, but I don't have the broad view to assert it's true in every use case.
Envoyé depuis un mobile.
-------- Message d'origine --------
De : Benjamin Elder<EMAIL_ADDRESS>Date : 29/07/2015  11:24  (GMT+11:00)
À : BenTheElder/kubernetes<EMAIL_ADDRESS>Cc : MikaelCluseau<EMAIL_ADDRESS>Objet : Re: [kubernetes] Pod-to-pod networking broken (#3)
Hmm. I'm going to see if I can ping @thockin on this one, this is a bit beyond my limited iptables knowledge, in the meantime I want to keep digging around to see if I can find anything but if not then I suppose we may have to fall back on something like the rule filtering you suggested. It'd be a shame to increase the complexity so much though.
—
Reply to this email directly or view it on GitHub.

I think that a rule to loop back to itself is pretty reasonable, I've still messaged thockin on freenode and started looking around some more to try and pin down a cause. I do wish we could get some more discussion going with other users, but I suppose they can just rewrite it how they want later if not.

@MikaelCluseau have you seen @Symmetric's proposal? https://github.com/GoogleCloudPlatform/kubernetes/pull/9210#issuecomment-125794747
Currently I've been planning to look into just adding the redirect rule and working on the iptables-save stuff but I wanted to get some more discussion going and the proposal there seems like an interesting possible change.
I also most certainly do not have the broad view to assert what we should be doing in all cases, perhaps if we can get more devs discussing the rules then we can assemble the broader view.

I think these cases do not overlap (my suggestion is for nodePort, rather than hitting a service VIP from a pod in the service).
I've thought about this a bit, and I can't see a clean way to get round this problem. The trouble with NAT is that it's hard to tell a perfect lie...
Did you try something like iptables -t nat -I POSTROUTING -s <pod IP> -d <pod IP> -j SNAT --to-source <svc IP>? I suspect that might work for a single service, but it won't work in general since the pod could be referenced by multiple services, and you can't disambiguate which service the traffic was DNATed by. You might be able to get round this by MARKing packets before DNATing them, to specify which service's IP you need to SNAT in the src=dst=podIP case.
Anything that causes O(n^2) complexity in the number of iptables rules is very problematic -- n could be hundreds or conceivably thousands if you're running a big DC.

Hi there, sorry to be late, I had issues here as the DC hosting us had multiple powerlosses...

I think these cases do not overlap

They do not, although they can both require userland proxying unless a better iptables solution is found.

Did you try something like iptables -t nat -I POSTROUTING -s  -d  -j SNAT --to-source ?

Yes, that did not work because rule traversal stop after the DNAT did set the same source and destination IPs on the packet. Which means this...

You might be able to get round this by MARKing packets before DNATing them

... did not work either (it was my second guess too ;)), because the rules are not traversed anymore (not even filter's FORWARD table), as can be seen in the trace tables I posted earlier.

Anything that causes O(n^2) complexity in the number of iptables rules is very problematic

For sure, especially when it's for a corner case.

n could be hundreds or conceivably thousands if you're running a big DC

I'm not sure, here n is the number of replicas for the service; if you have m services with an average of n replicas, the rule count will be O(mn**2) instead of O(mn). And since a host do not have to create rules for pods running outside of him, n will 1 if the replicas are perfectly spread over the cluster (at most 1 pod per service per host). We will write twice as much rules, but this stays O(m*n). Even will userspace proxying, we will end up writing twice as much rules (1 REDIRECT rule per host pod). The only way to stay closer to the current rule count is to put a REDIRECT in the source pod than will unconditionnaly loop the service request back to him, which I think is completely acceptable: the case I hit is a service checking its "advertised hostname", and the more think about that case, the more I feel like its a mistake anyway (I will have to talk to Kafka people about that).

@MikaelCluseau, noting that I'm following all of this discussion and have been working on getting/parsing and properly handling the results of iptables-save when syncing rules, but this is next on my todo list after that. I hit a snag with increased memory requirements bugging the build when I rebased against master again earlier but I'm back on track now.
I currently think the redirect in source pod seems pretty reasonable, if it's trying to connect to itself anyhow then we can just redirect to it, but I'm not sure about the other rule changes or alternate solutions.
I'm hoping to have those two checked off soon (by Monday perhaps?), and once I at least have the iptables-save stuff complete I'll ping thockin again and see if I can get the review started. If we decide we need more rule changes that should help get more discussion going on which tradeoffs to make etc...

@BenTheElder no worries, I solved the specific problem here by adding a /etc/hosts entry (side note: not having any userspace proxy dropped our average "full loop" response time from 4-5s to <100ms, so its pretty huge). So, monday or anytime later is ok to me. I just want to make clear that I'm not putting any pressure here. In fact, I'm even pretty happy that it broke the Kafka service because I was wondering why it was still using the (userland) kube-proxy before. I'm putting myself in standby mode until you have the redirect rule for me test, but I can't promise I won't try some things in the mean time, and report here :-)

Thanks.
That sounds like an impressive improvement!
I don't feel under any pressure, but I would like to get the PR tidied up
for review before GSoC ends.
On Jul 31, 2015 8:22 PM, "MikaelCluseau"<EMAIL_ADDRESS>wrote:

@BenTheElder https://github.com/BenTheElder no worries, I solved the
specific problem here by adding a /etc/hosts entry (side note: not having
any userspace proxy dropped our average "full loop" response time from 4-5s
to <100ms, so its pretty huge). So, monday or anytime later is ok to me. I
just want to make clear that I'm not putting any pressure here. In fact,
I'm even pretty happy that it broke the Kafka service because I was
wondering why it was still using the (userland) kube-proxy before. I'm
putting myself in standby mode until you have the redirect rule for me
test, but I can't promise I won't try some things in the mean time, and
report here :-)
—
Reply to this email directly or view it on GitHub
https://github.com/BenTheElder/kubernetes/issues/3#issuecomment-126839850
.

I'm finally caught up a bit, but this is a long issue - can someone summarize for me?

In @MikaelCluseau's testing, a pod connecting back to itself will have the packet hang.
This comment has trace information from doing pod2 -> pod1 and pod1 -> pod1.

In further summary, he's currently falling back on a host entry and the plan to try and solve it is to handle pod->self with a redirect rule, but I don't really know what the root of the problem is yet so that is tentative.
I'm in the process of moving atm but Sunday I'll start trying to debug this further after I finish up making use of the iptables-save state to preserve counters etc.

regarding the syscall: You could have the kube-proxy exec() modprobe for
the module, ignore failures, then blast the value into the that sysctl if
it is not correct.  And log that you did that.
I'm trying to repro the self-endpoint problem.  I see it (strangely) hang
for a second then try another endpoint, so it work but with some latency.
Will keep investigating.
On Fri, Jul 31, 2015 at 9:17 PM, Benjamin Elder<EMAIL_ADDRESS>wrote:

In further summary, he's currently falling back on a host entry and the
plan to try and solve it is to handle pod->self with a redirect rule, but
I don't really know what the root of the problem is yet so that is
tentative.
I'm in the process of moving atm but Sunday I'll start trying to debug
this further after I finish up making use of the iptables-save state to
preserve counters etc.
—
Reply to this email directly or view it on GitHub
https://github.com/BenTheElder/kubernetes/issues/3#issuecomment-126859452
.

Thanks for looking into this! I'll put the sysctl stuff on my todo list for
this week.
Have not reproduced the other issue yet as my local branch has been in a
dirty state, but if you or anyone else doesn't get to it before my next
commit I'll start trying to reproduce and debug well.
On Mon, Aug 3, 2015 at 12:25 AM, Tim Hockin<EMAIL_ADDRESS>wrote:

regarding the syscall: You could have the kube-proxy exec() modprobe for
the module, ignore failures, then blast the value into the that sysctl if
it is not correct. And log that you did that.
I'm trying to repro the self-endpoint problem. I see it (strangely) hang
for a second then try another endpoint, so it work but with some latency.
Will keep investigating.
On Fri, Jul 31, 2015 at 9:17 PM, Benjamin Elder<EMAIL_ADDRESS>wrote:

In further summary, he's currently falling back on a host entry and the
plan to try and solve it is to handle pod->self with a redirect rule, but
I don't really know what the root of the problem is yet so that is
tentative.
I'm in the process of moving atm but Sunday I'll start trying to debug
this further after I finish up making use of the iptables-save state to
preserve counters etc.
—
Reply to this email directly or view it on GitHub
<
https://github.com/BenTheElder/kubernetes/issues/3#issuecomment-126859452>
.

—
Reply to this email directly or view it on GitHub
https://github.com/BenTheElder/kubernetes/issues/3#issuecomment-127119038
.

The retry I observed seems to be inside wget.  Sigh.  I can repro easily with a single specific rule to cause it.  I have not found an answer yet.
Anything involving putting iptables rules into pods is sketchy at best - a pod can be in multiple services at once, and that can change dynamically.  We really don't want to be in the business of switching namespaces underneath docker if we can avoid it.
As for the rest of this work - I apologize for largely ignoring it most of the summer - 1.0 launch was nuts.  It is my intent to give this as much attaention as I can henceforth until it is merged or at least I could take it over to iron out last wrinkles.  I'm deferring all other kube-proxy work until then.  Have time to chat this week?

Aside: the note aboutthe new kernels requiring a modprobe will (I think) fix an entirely unrelated problem I am having!  Thanks for the clue!!  We'll need to load that modula and set that sysctl for kubernetes in general.

Ah, too bad. I'm not really sure what to do then, my iptables knowledge is
pretty limited in comparison to the three of you.
And understandable, If I sounded antsy it's because I wanted to make sure
things were headed in the right direction, but I could see how swamped k8s
has been in pull requests, issues, and the launch etc.
I definitely have time to chat this week.
On Mon, Aug 3, 2015 at 12:51 AM, Tim Hockin<EMAIL_ADDRESS>wrote:

The retry I observed seems to be inside wget. Sigh. I can repro easily
with a single specific rule to cause it. I have not found an answer yet.
Anything involving putting iptables rules into pods is sketchy at best - a
pod can be in multiple services at once, and that can change dynamically.
We really don't want to be in the business of switching namespaces
underneath docker if we can avoid it.
As for the rest of this work - I apologize for largely ignoring it most of
the summer - 1.0 launch was nuts. It is my intent to give this as much
attaention as I can henceforth until it is merged or at least I could take
it over to iron out last wrinkles. I'm deferring all other kube-proxy work
until then. Have time to chat this week?
—
Reply to this email directly or view it on GitHub
https://github.com/BenTheElder/kubernetes/issues/3#issuecomment-127123154
.

A solution.  We have to enable hairpin mode on bridge interfaces and we
have to masquerade access-to-self.  Observe:
thockin@kubernetes-minion-6b59:~$ sudo bash

root@kubernetes-minion-6b59:/home/thockin# iptables-save | grep <IP_ADDRESS>

root@kubernetes-minion-6b59:/home/thockin# ID=$(docker run -d
thockin/serve_hostname:thtest)
Unable to find image 'thockin/serve_hostname:thtest' locally
thtest: Pulling from thockin/serve_hostname
cf2616975b4a: Already exists
6ce2e90b0bc7: Already exists
8c2e06607696: Already exists
109f47b829e0: Already exists
26fa00b034bd: Already exists
452f40184a2a: Already exists
748b0d776d7f: Already exists
4aab7f9c61b7: Already exists
4aab7f9c61b7: Already exists
Digest:
sha256:d7e0d8ac048db344981ec2005b8d9e1e72868fa1a1e2ba513e45811b942b235a
Status: Image is up to date for thockin/serve_hostname:thtest

root@kubernetes-minion-6b59:/home/thockin# echo $ID
df4cc3092b3817ec202259f06464e5b87d6b8b4db3654143662b56fb9d71c154

root@kubernetes-minion-6b59:/home/thockin# IP=$(docker exec $ID hostname -i)

root@kubernetes-minion-6b59:/home/thockin# echo $IP
<IP_ADDRESS>

root@kubernetes-minion-6b59:/home/thockin# iptables -t nat -A PREROUTING -d
<IP_ADDRESS>/32 -p tcp -m state --state NEW -m tcp --dport 80 -j DNAT
--to-destination $IP:9376

root@kubernetes-minion-6b59:/home/thockin# iptables -t nat -A OUTPUT -d
<IP_ADDRESS>/32 -p tcp -m state --state NEW -m tcp --dport 80 -j DNAT
--to-destination $IP:9376

root@kubernetes-minion-6b59:/home/thockin# curl <IP_ADDRESS>; echo
df4cc3092b38

root@kubernetes-minion-6b59:/home/thockin# docker exec $ID wget -qO-
<IP_ADDRESS>; echo
^C

root@kubernetes-minion-6b59:/home/thockin# iptables -t nat -A POSTROUTING
-s $IP/32 -d $IP/32 -p tcp -m tcp --dport 9376 -j MASQUERADE

root@kubernetes-minion-6b59:/home/thockin# docker exec $ID wget -qO-
<IP_ADDRESS>; echo
^C

root@kubernetes-minion-6b59:/home/thockin# brctl showstp cbr0 | grep -i
hairpin

root@kubernetes-minion-6b59:/home/thockin# for intf in $(ip link list |
grep veth | cut -f2 -d:); do brctl hairpin cbr0 $intf on; done

root@kubernetes-minion-6b59:/home/thockin# brctl showstp cbr0 | grep -i
hairpin
 hairpin mode   1
 hairpin mode   1
 hairpin mode   1
 hairpin mode   1
 hairpin mode   1

root@kubernetes-minion-6b59:/home/thockin# docker exec $ID wget -qO-
<IP_ADDRESS>; echo
^C

root@kubernetes-minion-6b59:/home/thockin# cat
/proc/sys/net/ipv4/conf/cbr0/route_localnet
0

root@kubernetes-minion-6b59:/home/thockin# echo 1 >
/proc/sys/net/ipv4/conf/cbr0/route_localnet

root@kubernetes-minion-6b59:/home/thockin# cat
/proc/sys/net/ipv4/conf/cbr0/route_localnet
1

root@kubernetes-minion-6b59:/home/thockin# docker exec $ID wget -qO-
<IP_ADDRESS>; echo
df4cc3092b38

Can you guys repro to confirm, please?
Now how to actually deliver this?  Adding the masquerade on access-to-self
is easy and can be part of this patch set.  Adding the brctl change is a
little trickier.  We need to do that for every interface, which really
means it needs to be a network plugin or something.  I think we could just
add it in the network plugin path.
Assuming you can confirm this finding, we can work it into this patch set,
too.
On Sun, Aug 2, 2015 at 9:55 PM, Benjamin Elder<EMAIL_ADDRESS>wrote:

Ah, too bad. I'm not really sure what to do then, my iptables knowledge is
pretty limited in comparison to the three of you.
And understandable, If I sounded antsy it's because I wanted to make sure
things were headed in the right direction, but I could see how swamped k8s
has been in pull requests, issues, and the launch etc.
I definitely have time to chat this week.
On Mon, Aug 3, 2015 at 12:51 AM, Tim Hockin<EMAIL_ADDRESS>wrote:

The retry I observed seems to be inside wget. Sigh. I can repro easily
with a single specific rule to cause it. I have not found an answer yet.
Anything involving putting iptables rules into pods is sketchy at best -
a
pod can be in multiple services at once, and that can change dynamically.
We really don't want to be in the business of switching namespaces
underneath docker if we can avoid it.
As for the rest of this work - I apologize for largely ignoring it most
of
the summer - 1.0 launch was nuts. It is my intent to give this as much
attaention as I can henceforth until it is merged or at least I could
take
it over to iron out last wrinkles. I'm deferring all other kube-proxy
work
until then. Have time to chat this week?
—
Reply to this email directly or view it on GitHub
<
https://github.com/BenTheElder/kubernetes/issues/3#issuecomment-127123154>

.

—
Reply to this email directly or view it on GitHub
https://github.com/BenTheElder/kubernetes/issues/3#issuecomment-127123358
.

@thockin is it an issue that you needed to use route_localnet (since that's not included in Red Hat's 2.6 kernel)?
I'll need to test this fix with Calico (which doesn't use the bridge); I think that the masquerade rule alone should work in our case since we don't have the bridge in the path.

Hi @thockin, I can confirm it solves the problem, the packet does the expected traversal just after I enabled the hairpin mode (I tried setting route_localnet before enabling hairpin to be sure hairpin was required). So, the POSTROUTING show this:
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

    1    60 MASQUERADE  all  --  *      *       <IP_ADDRESS>        <IP_ADDRESS>

and the self-connection works as expected:
$ kubectl --namespace=master exec keystone-yukci -- curl --connect-timeout 1 -Is keystone:35357
HTTP/1.1 300 Multiple Choices
Vary: X-Auth-Token
X-Distribution: Ubuntu
Content-Type: application/json
Content-Length: 753
Date: Mon, 03 Aug 2015 20:13:17 GMT

That's great, thanks :-)

FWIW, I found it easy to understand what hairpin does and how it relates to this issue as I found this:
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/net/bridge/br_forward.c#n30

So to recap:

For the nf-bridge modprobe and sysctl we will add this somewhere kubernetes wise to fix the first issue.
For the route to self, we will enable hairpin mode on bridges and set:
-A POSTROUTING -s <hostIP>/32 -d <hostIP>/32 -p tcp -m tcp --dport 9376 -j MASQUERADE on the host?
Do we also need this for udp?

I think that kube v1.1 will either not support CentOS or will have a
fallback mode for kube-proxy.  We need to figure out a position on this.
On Mon, Aug 3, 2015 at 1:10 PM, Paul Tiplady<EMAIL_ADDRESS>wrote:

@thockin https://github.com/thockin is it an issue that you needed to
use route_localnet (since that's not included in Red Hat's 2.6 kernel)?
I'll need to test this fix with Calico (which doesn't use the bridge); I
think that the masquerade rule alone should work in our case since we don't
have the bridge in the path.
—
Reply to this email directly or view it on GitHub
https://github.com/BenTheElder/kubernetes/issues/3#issuecomment-127393149
.

I currently select the new proxy or the old proxy depending on the iptables
version (this needs discussing), so right now it should fall back on the
userspace proxy.
On Mon, Aug 3, 2015 at 7:57 PM, Tim Hockin<EMAIL_ADDRESS>wrote:

I think that kube v1.1 will either not support CentOS or will have a
fallback mode for kube-proxy. We need to figure out a position on this.
On Mon, Aug 3, 2015 at 1:10 PM, Paul Tiplady<EMAIL_ADDRESS>wrote:

@thockin https://github.com/thockin is it an issue that you needed to
use route_localnet (since that's not included in Red Hat's 2.6 kernel)?
I'll need to test this fix with Calico (which doesn't use the bridge); I
think that the masquerade rule alone should work in our case since we
don't
have the bridge in the path.
—
Reply to this email directly or view it on GitHub
<
https://github.com/BenTheElder/kubernetes/issues/3#issuecomment-127393149>
.

—
Reply to this email directly or view it on GitHub
https://github.com/BenTheElder/kubernetes/issues/3#issuecomment-127436934
.

Another approach would be to use the pure proxy and install a userspace
proxy JUST for that one case.
On Mon, Aug 3, 2015 at 4:59 PM, Benjamin Elder<EMAIL_ADDRESS>wrote:

I currently select the new proxy or the old proxy depending on the iptables
version (this needs discussing), so right now it should fall back on the
userspace proxy.
On Mon, Aug 3, 2015 at 7:57 PM, Tim Hockin<EMAIL_ADDRESS>wrote:

I think that kube v1.1 will either not support CentOS or will have a
fallback mode for kube-proxy. We need to figure out a position on this.
On Mon, Aug 3, 2015 at 1:10 PM, Paul Tiplady<EMAIL_ADDRESS>wrote:

@thockin https://github.com/thockin is it an issue that you needed
to
use route_localnet (since that's not included in Red Hat's 2.6 kernel)?
I'll need to test this fix with Calico (which doesn't use the bridge);
I
think that the masquerade rule alone should work in our case since we
don't
have the bridge in the path.
—
Reply to this email directly or view it on GitHub
<

https://github.com/BenTheElder/kubernetes/issues/3#issuecomment-127393149>

.

—
Reply to this email directly or view it on GitHub
<
https://github.com/BenTheElder/kubernetes/issues/3#issuecomment-127436934>

.

—
Reply to this email directly or view it on GitHub
https://github.com/BenTheElder/kubernetes/issues/3#issuecomment-127437147
.

Won't the iptables rules for the userspace mode conflict with pure-iptables version rules?
Also, if I can avoid having the userspace proxy for just one case that doesn't affect me as a CoreOS user, it would be better (though not critical of course).

I've been trying to handle the MASQUERADE rule but running into this has made spinning up a cluster impossible for the moment. So unfortunately while I think I've had it working, I still can't verify it (yet).
@thockin, otherwise though, the PR should be essentially ready for review, since you've said we will handle the hairpin and nf-bridge stuff elsewhere.

@thockin @MikaelCluseau Do we / should we have upstream issues for the hairpin mode and nf-bridge?
We should probably get those taken care of along with the PR; I plan to go ahead add a masquerade rule to the new proxy's iptablesInit along the lines of:
-A POSTROUTING -s <hostIP>/32 -d <hostIP>/32 -j MASQUERADE
So that we can have this taken care of when the PR goes in.

I think we can safely say this is being taken care of upstream. Closing this here.


Update jsoniter-scala-core to 2.13.37

Updates com.github.plokhotnyuk.jsoniter-scala:jsoniter-scala-core from 2.10.4 to 2.13.37.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "com.github.plokhotnyuk.jsoniter-scala", artifactId = "jsoniter-scala-core" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "@monthly" },
  dependency = { groupId = "com.github.plokhotnyuk.jsoniter-scala", artifactId = "jsoniter-scala-core" }
}]

labels: library-update, early-semver-minor, semver-spec-minor, commit-count:1

Superseded by #950.


Better handle first-time opening of workspaces with explicit typescript.tsdk specified

See #26046 for context.
The reason why VS Code does not implicitly use the workspace TS version is claimed to be for security reasons: if you clone a repo that has a malicious TS version in its node_modules, the current behaviour prevents that code from being run implicitly.
This is a nonsensical argument because it assumes that nothing should be done implicitly when the user performs an explicit action - if we follow said argument to its conclusion, then nothing should happen when you open a VS Code workspace, or when you press the power button on your computer. An explicit user action always has implicit elements.

Regardless, the current behaviour of having to explicitly select the workspace TS version the first time you open a workspace is both irritating and barely a step above just implicitly using that version. After all, there is no "hey this could pwn your machine are you sure?" prompt when you actually select the workspace version.
A global setting "preferWorkspaceTypeScriptVersion" would essentially cover this scenario. By default it's false which means do the current behaviour. If a user sets it to true (implicit action, therefore safe!) then the workspace TS version is always used.
If that's not suitable (because a user could copy someone else's global VS Code config with "preferWorkspaceTypeScriptVersion" = true and then they would not explicitly assent to running the TS code they downloaded, oh the humanity!) then the least-effort best-experience way of making this better is to simply add a Yes/No dialog that pops up the first time opening a workspace with typescript.tsdk specified, that asks the user if they want to use the workspace TS version (with appropriate "you could get pwned and Microsoft won't be held responsible" warnings), or the VS Code builtin version.

No need to make it so complicated. The reason we don't load the workspace tsdk automatically without prompting is simple: opening a folder/file should never be able to automatically run code from that workspace. That's seems like a reasonable goal
Having said that, this is a duplicate of #65546


[Bug] Cannot run validator locally via docker

🐛 Bug
I have been trying unsuccessfully to run validator nodes locally on Ubuntu 18.04 with docker by running the docker/validator-dynamic /run.sh script.
Seems that there is an error in the loop
To reproduce
docker/validator-dynamic/build-dynamic.sh
(because validator folder was not working: https://github.com/libra/libra/issues/2155)
This is where it fails.
when I try to run  run.sh I get:
 + image=libra_validator_dynamic:latest
 + nodes=5
 + base_ip=172.18.0
 + ip_offset=10
 + bootstrap=<IP_ADDRESS>
 + docker network create --subnet <IP_ADDRESS>/24 testnet
 b8f56d7828b1a507cae29d95736e7d1e299973dd35cd1dfffd8f6ad3605bc0ce
 docker/validator-dynamic/run.sh: 14: docker/validator-dynamic/run.sh: Syntax error: Bad for loop variable

Expected Behavior
Docker container running the validator node image started
System information
last version, testnet branch
rustc 1.42.0 (b8cedc004 2020-03-09)
Ubuntu 18.04

I replaced #!/bin/sh with #!/bin/bash to solve this problem.
Seems that for ((node=1; node<$nodes; node++)); is bash specific and does not work with Bourne shell

oh interesting...
can you submit a PR and add me as a reviewer?

@davidiw seems that I i cannot add reviewers
https://github.com/libra/libra/pull/3754


Cleanedup default example

This PR is just to demonstrate an issue I found

This Draft PR is intended to highlight this issue -> https://github.com/apollographql/graphql-tools/issues/1121


feat(Logging): Add logs with user and request information

Added ERROR, WARN, INFO und DEBUG level logs
INFO - Covers configuration information, request start/finish and most important events regarding Export/Import, creation and updating of business partners including BPNs
DEBUG - Some more details during important events
More in-depth error logging when we tackle exception handling
Logging user name and request id comes in handy when using tools like Kibana later but can be very verbose on local. So that's why you can disable adding the username and request id to each log in the application.properties.

https://www.veracode.com/security/java/cwe-117
The input must be logged before it is logged.


hexo-theme-Annie

Good evening!
I want to show my theme in the site. Of course, it can work normally. (Preview: hexo-theme-Annie)
Thank you very much.

Have you changed the indents? There are more than 2000+ lines changed.

I‘m sorry for that due to negligence the detalil of the document source/_data/themes.yml.
I will solve it as soon as possible.

Thanks for the quick fix. Closed as fixed in #754


Refactor fmt.Errorf %v to % w

What this PR does / why we need it:
Changed %v to %w in fmt.errorf.
reason : https://pkg.go.dev/fmt#Errorf
Which issue(s) this PR fixes:
Fixes #
Does this PR introduce a user-facing change?:
NONE

@seipan Thanks for the contribution, could you sign to your commit using git commit -s.
You can easily sign off your submitted commit using git commit --amend --signoff then force push to the same branch to update this PR

@khanhtc1202
Thank you very much for your warm acceptance of my first OSS contribution.

We are happy to hear that :smile:

Thank you so much!
We always welcome your PRs 👍


In the Beginning

About the PR

Added two new weapons, A Light Shining in Darkness(very weak gun for chaplain), and 2nd Testamendment(funi admeme gun)
Why / Balance

I liked Goonstation's chaplain design and I feel chap should play like a weird mix of med and sec
Media

[X ] I have added screenshots/videos to this PR showcasing its changes ingame, or this PR does not require an ingame showcase
https://github.com/Verbalase/Goob-Station/assets/131112412/e550059f-5c40-43cb-90c1-45e75f7da2c1
I had to cut the video really short because of file size limit, but the first gun takes a full mag to kill someone(it's damage is horrid)

Changelog

add: Added divine weaponry!

how do you get the light as chaplain
(sick btw)

how do you get the light as chaplain
(sick btw)

Chaplain spawns with it


it says i cant oppen error cant find module

@oppaidaisuki1211 try and do a discord uninstall and then press START key + R to open RUN and type %appdata% and enter. In that folder it opens delete all discord related stuff. Reinstall discord, login and install BD. Your issue should be fixed after this.


alpine linux not supported

Hi,
i tried this provider and could it be that alpine linux is not supported ?
the LUNs are created and mounted via iSCSI. So that's fine.
But when I try to use this pvc in a pod it is stuck in pending
reason
  Warning  FailedMount             6m10s (x20 over 37m)  kubelet                  MountVolume.MountDevice failed for volume "pvc-0b37a8dc-16fb-4b92-a8a6-4c45d66e3363" : rpc error: code = Internal desc = rpc error: code = Internal desc = Can't find device path [/dev/disk/by-path/ip-<IP_ADDRESS>:3260-iscsi-iqn.2000-01.com.synology:media1.pvc-0b37a8dc-16fb-4b92-a8a6-4c45d66e3363-lun-1]: file does not exist

from what I see is that the csi plugin tries to access the disk by /dev/disk/by-path which is created by udev and there is no systemd/udev in alpine linux.
is that true ? could we update the docs somewhere that systemd/udev is mandatory ?

sorry, wanted to open this issue on SynologyOpenSource/synology-csi and not here.

No problem. On the subject matter, I believe that Alpine Linux in the container should not be a problem. However, on the host that's a different story.


Fix Specials for some DynastyScans series

Some series have their specials labelled in a certain way which previously made cum overwrite regular chapters with the specials.
This changes the regex to have named groups, and a group to catch whether something is a chapter or a special is introduced.
A unit test which checks one series that previously exhibited this issue is added, in which the chapter names and the filenames are checked to be unique.
Somewhat relates to #35, but does not fix the original batoto-specific issue.

Current coverage is 97.76% (diff: 100%)

Merging #36 into master will increase coverage by <.01%

@@             master        #36   diff @@
==========================================
  Files            16         16
  Lines          1471       1474     +3
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
+ Hits           1438       1441     +3
  Misses           33         33
  Partials          0          0

Powered by Codecov. Last update 65a4b8b...a197b99


[SPARK-21925] Update trigger interval documentation in docs with behavior change in Spark 2.2

Forgot to update docs with behavior change.

Test build #81422 has started for PR 19138 at commit fb3a35e.

Test build #81422 has finished for PR 19138 at commit fb3a35e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81422/
Test PASSed.

LGTM. Thanks @brkyvz. Merging to master and 2.2


Add support for TRADFRI Control Outlet

This PR is the result of the effort of adding support in https://github.com/Koenkk/zigbee2mqtt/issues/365
I've only been able to test this with one outlet, so it would be great if someone else can validate the functionality added in this PR.

Thanks!


Include F Sharp for fun and profit

I's an epub version of the site, the author himself links to it. And it is awesome

Thanks! Merged as 97fb81cec8f81577813a432b89a54ae3106667ab , next time please rebase on top of master, your commit was based on a very old state of this repo (February). 👍


Update constants.py [BSC support]

Pancakeswap v2 factory and router contracts added.

Thanks!

@ErikBjare
Fixed that issue
https://github.com/uniswap-python/uniswap-python/issues/126
so it can be closed.


Support reactive expressions in arguments and options

  Software version info

Python              :  3.11.5 (main, Sep 11 2023, 08:31:25) [Clang 14.0.6 ]
Operating system    :  macOS-14.0-arm64-arm-64bit
Panel comms         :  default
holoviews           :  1.18.3
bokeh               :  3.3.4
colorcet            :  3.0.1
dask                :  2023.6.0
datashader          :  0.16.0
geoviews            :  1.11.0
hvplot              :  0.9.2
IPython             :  8.15.0
jupyterlab          :  4.0.11
matplotlib          :  3.8.0
notebook            :  7.0.6
numba               :  0.58.0
numpy               :  1.24.3
pandas              :  2.1.1
panel               :  1.3.1
param               :  2.0.1
pillow              :  10.0.1
pyarrow             :  11.0.0
pyviz_comms         :  2.3.0
scipy               :  1.11.3
spatialpandas       :  0.4.9
xarray              :  2023.6.0

I would expect reactive expressions (pn.rx(), param.rx()) to be treated the same as Panel Widgets and Param Parameters (i.e., support for all Reactive References), but only widgets and parameters seem to be accepted:
import hvplot.pandas, pandas as pd, panel as pn
df = pd.DataFrame([6,2,9], [1,2,3])
e = pn.widgets.IntSlider(value=2, start=1, end=10, name="e")
pn.Column(e,df.hvplot.line(line_width=e))

pn.Column(e,df.hvplot.line(line_width=e.param.value))

pn.Column(e,df.hvplot.line(line_width=e.rx()))

AbbreviatedException: TypeError: __str__ returned non-string (type rx)

For completeness, note that .apply.opts() can also be used successfully here:
import hvplot.pandas, pandas as pd, panel as pn
df = pd.DataFrame([6,2,9], [1,2,3])
e = pn.widgets.IntSlider(value=2, start=1, end=10, name="e")
pn.Column(e, df.hvplot.line().apply.opts(line_width=e))

Taking the initial example, we can see the object returned is not a HoloViews element but a Panel object:

I think we weren't very happy with non-HoloViews objects returned by .hvplot() calls (there might be an issue on that topic). I also have some vague memory of us talking about deprecating some of that functionality, since it can be done with .interactive():

It seems to me this issue is part of the greater issue of figuring out what to do with .interactive() now param.rx exists that I put on hvPlot roadmap's for this year. Cc'ing @philippjfr since he may have some ideas 🙃

Taking the initial example, we can see the object returned is not a HoloViews element but a Panel object:

This was an exceptionally bad decision on my part and should be undone. I'm not sure how to undo it cleanly though.

Returning a Panel objects; eventually we do want to accepts reactive references and then apply them with .apply.opts().

Ok, noted.

I'm not sure how to undo it cleanly though.

Perhaps with starting to emit a deprecation warning for the current behavior, and add a parameter (maybe global) for users to opt in the new behavior.


chore: 🔧 switched to yarn

Just to be consistent with CRA. Not that npm is lacking something at the moment

Pull Request Test Coverage Report for Build 364727427

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 95.925%

Totals

Change from base Build 362312257:
0.0%

Covered Lines:
245

Relevant Lines:
253

💛  - Coveralls


fix(@angular-devkit/build-angular): remove pure_getters

Closes #14316

main-es2015.js increased by 183 bytes.

Will rxjs 6.5.3 be the minimum supported version for v9?

@vikerman, I am not sure about that.
//cc @IgorMinar


Firefox not supported

Hey, looks like the transcripts don't load properly on Firefox.

Unable to reproduce, can you provide more details?

left = firefox, right = chrome-based browser

This is the transcript the user sent with the bug
https://tickety.top/transcripts?id=24873z4z6j7x

We are still running on the older release before you made the last change with react components

still unable to reproduce:

still unable to reproduce:

Hm maybe it was some extension then or something with the user settings


Makesuppfolder

I have moved all the images and data of which I am aware out of individual folders and into two new folders within the new Supplementary_Data folder. If you can't spell supplementary I'm sorry, please use tab complete ;)

Can I be a horribly annoying person and ask that we rename the directory to Supplementary_data without the second capital? That way it matches the format of the other folders (i.e. 08_Outputting_data)...

done!


feature request: add additional launch.json variable to refer to the compiled version of a file

problem
to begin a debugging session where a typescript file is not my entrypoint, but is instead an argument to my entrypoint, i must find a way to get the /path/to/the/compiled/JS file into my launch config.  this could be reduced in complexity if such a pseudo-variable were simply exposed.
it may sound a little meta--see the example below
desired state

add a new launch variable to point to the built version of the file

example
i like to use ava test runner.  to debug ava against a file in javascript, one uses a configuration as such:
    {
      ...
      "program": "${workspaceRoot}/node_modules/ava/profile.js",
      "args": [
        "${file}"
      ],
    }

this works fine when ${file} is JS.  in my case, it's TS.  thus, i need ${file} to point to my built TS file.
proposal

add a var ${file:out}/${file:built}/${file:compiled}

if an output file cannot be found, the current file is perhaps at least tried

    {
      ...
      "program": "${workspaceRoot}/node_modules/ava/profile.js",
      "args": [
        "${file:out}"
      ],
    }

now, the ava entrypoint will find my built src file, and sourcemaps will all work and i'll have a more natural debug workflow.

VS Code does not no anything about "building" or "build results". So we cannot introduce a new variable for that.
I suggest that you create an extension that just implements "commands" with the specific behaviour you are looking for. You can then use these commands from within a launch configuration via the ${command:my_cool_command}syntax.


Feat: Floating button to scroll to top

Describe the feature
I wish to add a floating button which on clicked would scroll back to the top.
Add ScreenShots

Record

[X] I agree to follow this project's Code of Conduct
[X] I'm a GSSOC'24 contributor
[X] I want to work on this issue

@VanshKing30 please assign me this issue under gssoc'24

@Asymtode712 Assigning this issue to you


I can't find any relevant program for streaming in firebase with esp32

Hello @mobizt please provide an example of firebase stream in esp32 to read data and control GPIO pins

Did you check this?

yes, but it gives API key invalid error with err code 400

Spend your time to read all the provided document carefully and try with all examples.
In all examples, there are detailed comments that guide and describe the purpose of each line of code which you should read it all.
For your code you should debug by yourself first.
If you try it enough and can't find the solution, you should provide the simple and clean code that you're already tried as well as the debug message that you've printed via serial monitor.

This is the one of authentication example which gives you all details e.g. how you can take all required authentication credentials.

If you don't know about the Firebase Authentication which use the time to learn, you can try the legacy method for authenticate as this example.


Path to monkeyrunner is incorrect

Hello,
your codelab states:

${ANDROID_HOME}/tools/monkeyrunner ./run_perf_tests.py ./ <INSERT_ID>

However, that is incorrect, as the monkeyrunner executable is located in ${ANDROID_HOME}/tools/bin/monkeyrunner

Thanks @IgorGanapolsky for posting this!
Initially I followed the Step 7 instructions and used:
${ANDROID_HOME}/tools/monkeyrunner ./run_perf_tests.py ./ <INSERT_ID>
which resulted in the following error:
bash: /Users/hlupico/Library/Android/sdk/tools/monkeyrunner: No such file or directory
Using ${ANDROID_HOME}/tools/bin/monkeyrunner instead resolved my issue.

Also important to note that folks with this issue will also have to update Line 69 of RunLocalPerfTestsTask to be:
def monkeyPath = Paths.get(sdkDir, "tools/bin", "monkeyrunner" + monkeyExt).toAbsolutePath().toString()


Remove optimizely JS

Fixes #1310
:bomb: Do not merge yet. :bomb: - will land after the tag

r+ \o/


Pytouch version sync with Cuda

The 'accelerate' lib is missing from the requirements.txt
The Pytouch version isn't synced with the Cude version - I had to install the following:
The default version was cu121, which didn't work and failed (RuntimeError: The NVIDIA driver on your system is too old) in this cell:

model_path = "model_artifacts"

pipe = StableDiffusionPipeline.from_pretrained(
    model_path, torch_dtype=torch.float16
).to("cuda")

g_cuda = None

import torch
print("PyTorch Version:", torch.__version__)

PyTorch Version: 2.1.2+cu121
after running the following commands and kernel restart it completed:
!pip uninstall torch -y
!pip cache purge
!pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

Kernel restart
import torch
print("PyTorch Version:", torch.__version__)

PyTorch Version: 2.1.2+cu118

@nadavw thank you for the information. Can you please let me know which notebook you referred to? Thanks.

notebooks/community/vertex_endpoints/torchserve/dreambooth_stablediffusion.ipynb

@telpirion can you please help take a look at this? Thanks.

@gericdong I can't help, I'm afraid. Sorry. (This is also a community notebook, so the threshold for fixes is higher.)
It looks like there's a missing dependency (accelerate) and torch needs to be upgraded to a higher version. Should be a quick fix. @nadavw has provided a lot of the required changes in their detailed write up.

@katiemn can you help take a look at this? Thanks.

We won't be able to support this notebook. Closed as not planned.


Update Carousel.svelte

export divClass so that the user can change the look/size of the carousel

Closes #
📑 Description

Status

[x] Not Completed
[ ] Completed

✅ Checks

[ ] My pull request adheres to the code style of this project
[ ] My code requires changes to the documentation
[ ] I have updated the documentation as required
[ ] I have checked the page with https://validator.unl.edu/
[ ] All the tests have passed
[ ] My pull request is based on the latest commit (not the npm version).

ℹ Additional Information

You can overwrite those settings by class property. There's no need to export that divClass.
<Carousel {images} duration={3000} class="flex rounded-none" />


Adding seed_extra_containers to support redfish and wireguard

Needed to resolve conversations in:
https://github.com/stackhpc/smslab-kayobe-config/pull/150#pullrequestreview-1107090341

Yes, tested on sms-lab

Tested it?

Yes, tested on sms-lab


Failure to resolve dependencies in Dart 2.12.0

The latest stable Dart release breaks the build for all of our applications that rely on angular_components due to the dependency on observable. It seems observable was patched the other week but no release has been made as yet.
angular: v6.0.1
angular_components: v1.0.2
Resolving dependencies... (1.4s)
The current Dart SDK version is 2.12.0.

Because no versions of observable match >0.22.2 <0.23.0 and angular_components >=0.13.0 depends on observable ^0.22.2, angular_components >=0.13.0 requires observable 0.22.2.
And because observable 0.22.2 depends on dart_internal ^0.1.1, angular_components >=0.13.0 requires dart_internal ^0.1.1.
So, because dart_internal ^0.1.1 requires SDK version >=2.0.0-dev.12.0 <2.12.0 and cog depends on angular_components ^1.0.2, version solving failed.

Add the following line to pubspec.yaml
dependency_overrides:
dart_internal: ^0.2.0

This is now fixed with 1.0.3


Input Icon - design & build internal component

Description / User Story
We need an internal inputIcon component for the TextInput. The component already exists in design here.
The current idea would be to implement a very dumb input icon component, which only brings its own colors and is clickable.

Acceptance Criteria

[ ] The Input Icon has been defined
[ ] An Input Icon component exists under internal components in code
[ ] The Input Icon is used by the same components that use it in design

[ ] Select?
[ ] Text Input?

[ ] Documentation of changed components has been updated

Background information

Related to #489

Moved out of 1.0 again, as we will just do a quicker fix with #1014 .
This ticket also was updated and now includes a timebox of 3-4 weeks.


Use dbt sources for staging area

The framework currently has macros that make direct calls to source tables. This setup doesn't take advantage of dbt sources, which facilitates the definition of where the source tables are, allows for source freshness tests and creates special objects for them in documentation.
Proposed changes
To dbt_project.yml

Adding a stitch_source variable

Create a sources.yml file within the schema repository of each source
version: 2

sources:
  - name: mailchimp
    database: stitch_database
    schema: mailchimp
    tables:
      - name: list_members
      - name: reports_email_activity
      - name: lists
      - name: campaigns

Changes to filter_etl_table macros

Only have 3 parameters that are passed: filter_stitch_table(source_name, source_table, unique_column)
Use dbt's source macro in the from statement: {{ source(source_name, source_table) }}

Changes to staging files

Call the filter_etl_table macros with the correct parameters: {{ filter_stitch_table(var('stitch_source'),var('stitch_clients_table'),'id') }}

┆Issue is synchronized with this Jira Task by Unito

Hi Olivier, this has now been done and is present in the most recent version of the repo

Hi Olivier, this has now been done and is present in the most recent version of the repo


Tag

What
Use this issue to discuss the NHS App tag component.

Related
NHS design system guidance
NHS design system discussion
GOV.UK Design System guidance
GOV.UK Design System discussion
Examples on the NHS App

Examples

Journeys found on

Request repeat prescription
Find NHS services near you
Prescriptions
Upcoming and past appointments

Notes
The GOV.UK Design System team have recently made some tag design changes to improve the accessibility and readability of the tag text.

Prescription and medicines use tag to tell status of an prescription order

As part of their work on a new prescription journey the team ran some usability testing with people who had access needs but did not use assistive technology. The testing included the use of status tags to indicate when prescriptions were ready to collect, part ready to collect or still being prepared.

Across the 5 sessions there were a couple of key findings:

participants were not sure why the status tags were purple or blue, as these colours had no clear meaning to the participants
participants expected the use of traffic light style colours for statuses
participants did not always seem completely clear what information within the card link the status applied too (ie an individual medicine vs the overall prescription)
participants were unlikely to click into the detailed view of the prescription, this may have been due to the very information rich nature of the card links (with the status tags playing a key role)

Quotes
‘The choice of blue and purple does seem a little random… preparing, is that the first step? Is there something else that it might say?’
‘I don’t understand why it’s green, blue and then purple, why isn’t it just the same colour?’
Reflection

Using purple/blue status tags for different but similar statuses may simply be introducing more complexity for users without much obvious need/benefit.
When using status tags it's important to think about exactly what information the status relates to and make sure the connection between those things is visually clear.

As part of our work in N&M we are looking at inbox management features. We have recently done some user testing (moderated and unmoderated) on flagging, along with sending out a survey, to understand how users might mark and more easily find messages they deem to be important.
As a result, we have settled on Flagging as the inbox management feature to meet this user need, for which we are likely going to use an orange tag for. Our thinking for using the orange tag is:

This colour is commonly used in other experiences where users can flag a message, such as on email services
It is a colour that stands out on the page, which is key for users who are scrolling through their inbox looking for something important to catch their eye

We did some initial testing with a blue tag but users thought this was too subtle. A few users suggested using green or red for the 'flagged' tag, but for semantic reasons we will be avoiding doing this.
The flag tag will be applied to a message via a link text button at the bottom of the message which acts somewhat as a toggle. Clicking 'Flag message' will apply the orange 'Flagged' tag to the top of the message which will also be visible on the message preview. Once a message is flagged, the link text button label changes to 'Unflag message'.

Interestingly, this might be the first case of a user-applied tag?

To be consistent throughout the NHS App we need an aligned approach to tag colours and what the colour represents.
Attached are the tags used by:
GP Services
Blue - By phone
Purple - In person

Appointments:
Red - action needed from user
Grey - cancelled
Yellow - pending change
Hospital docs & questionnaires:
Red - action needed from user
Yellow - in progress
Green - action complete (document = read, questionnaires = complete)

Prescriptions
Group level:

Item level:

Please add any other screenshots if others are being used - thank you

Rationale & guidance migrated from the NHS App Figma file


Wallet Multi button disconnect error

Describe the bug
A clear and concise description of what the bug is.
the wallet multi button isnt working in the react or material ui after connecting

Are you using Phantom? Do you see the button change to "Disconnecting..." but not disconnect? This might be a dupe of https://github.com/solana-labs/wallet-adapter/issues/755

I can confirm - I am not even using the wallet multi button, but rather calling disconnect() directly from WalletContextState.
Everything works as expected for Backpack / Solflare. Phantom doesn't disconnect successfully.


maven project compile args not passed to editor javac

https://github.com/apache/incubator-netbeans/blob/master/java/java.source.base/src/org/netbeans/modules/java/source/parsing/JavacParser.java
validateCompilerOptions removes any unsupported parameters.


fix(#1368): set color of the TokenClassification entities

fix #1368
This PR changes the number of color entities from 50 to infinity

Also, with these changes, maybe we could consider decreasing the number of available colors for entity tags?
cc @leiyre @Amelie-V @dvsrepo

Also, with these changes, maybe we could consider decreasing the number of available colors for entity tags?
cc @leiyre @Amelie-V @dvsrepo

Yes, we can experiment with more contrasting colors by limiting the number or we can also pass a list of custom colors.

After deploy changes the webapp  is not accessible and I see the following error:


add "supported by INNOQ" in footer

and, at the same time, try a centered footer, as in datamesh-architecture.com
take the graphical logo from the datamesh site.

Done for this site, please have a look if you like it like this. The size of the supported-by-innnoq image is responsive, being smaller on smaller screens.
Shall I also do this on other sites?

perfect.


feat: update topBar's searchBar's style

Changes description

Update the style of the searchBar (in the topBar) to fit Figma style
Context

Align Android widget style with Figma style
Checklist

[x] Make sure you are requesting to pull a topic/feature/bugfix branch. Please, don't request directly from your main!
[x] Check commits & PR names matches our requested structure. It must follow the https://www.conventionalcommits.org pattern.
[x] Check your code additions will fail neither code linting checks.
[x] I have reviewed the submitted code.
[x] I have tested on a phone device/emulator.
[x] I have tested on a tablet device/emulator.
[x] I have tested on a large screen device/emulator.
[x] If it includes design changes, please ask for a review with a core team designer.

Does this introduce a breaking change?

No

Screenshots
Phone

Tablet

Large screen

Other information

##Before

You can download the apk here :
sample-debug.apk :
https://app.bitrise.io/app/62ac2962b2dd627a/build/37f6b776-d38a-48f3-af8f-d172a502a2c6/artifact/ecf6e32488204c26/p/3ac9cdc370698a98896cc329afe8795e

You can download the apk here :
sample-debug.apk :
https://app.bitrise.io/app/62ac2962b2dd627a/build/b395af5d-dda4-4be0-ba27-58a5eb1da8f5/artifact/1bd73054afd5953a/p/7abee9c4a46fe85cb52cea2ad7c66a22

You can download the apk here :
sample-debug.apk :
https://app.bitrise.io/app/62ac2962b2dd627a/build/0089d69b-f5e0-49df-974d-0063a2be8e39/artifact/8775acd98931041d/p/c7c80b3df6fb66026abe06649eb0d7e7


Does comcat gave you money to named your code as their brand name ?

Does comcat gave you money to named your code as their brand name ?
Do you afraid they may Sue you.
I like the name.

No, as stated a few times before by the original creator, the name was mostly an afterthought, but it is pretty appropriate when considering it satirically. Comcast surely wouldn't be very happy, but from the standpoint of being satirical, I think there's no real harm Comcast the company can do to this project other than make the creator change the name. If that is even possible from a legal standpoint.


Image quantization

The current implementation of _continuous, $x/k$ maps 0 to 0 and k to 1 which
feels desirable. However, I beleive _discretize should be the inverse of
_continuous which results in the current implementation of _discretize being
$ceil(x*k - 0.5)$ which has "bins" of differing widths at the extremes (half
the width of the internal bins). This does not feel right. The TODO notes
describe a counter-proposal in which _discretize maps to constant-width bins
but the inverse means that _continuous no longer maps 0 and k to 0 and 1
respectivley. This also feels wrong.

@dtorop This has been bothering me for awhile. Any thoughts? I had trouble finding the "right way" to do this because I didn't know which keywords to search for.


wnd_title unsafe

Use wstring, not a pointer.

window_title data type changed to wstring from const wchar_t* in v_0.1.1.
I close this issue.


add nodeMonitorGracePeriod to kubeControllerManager shoot spec

/area control-plane
/kind enhancement
What this PR does / why we need it:
Currently gardener sets --node-monitor-grace-period hardcoded to 120s. There are situations, where the connection between a  shoot and the seeds is less stable (e.g. during maintenance work at the firewall or at the internet uplink between shoot and seed, or slow BGP failovers from one uplink to another witch take more than 2min to complete, etc.
In these situations, the kube-controller-manager node-lifecycle-controller marks all three nodes as NotReady and cuts of all shoot pods from all services as soon as the connections comes back (basically the situation described in https://github.com/gardener/machine-controller-manager/issues/354#issuecomment-553340093)
With this PR we all allow setting higher values for --node-monitor-grace-period in the shoot spec for certain shoots with known uplink/infrastructure issues.
Which issue(s) this PR fixes:
Relates to https://github.com/gardener/machine-controller-manager/issues/354
Special notes for your reviewer:
The implementation is analog to podEvictionTimeout
Release note:
It is now possible to set custom values for kube-controller-manager `--node-monitor-grace-period` via `.spec.kubernetes.kubeControllerManager.nodeMonitorGracePeriod` (should not be less than 2m).

Thank you @mwennrich for your contribution. Before I can start building your PR, a member of the organization must set the required label(s) {'reviewed/ok-to-test'}. Once started, you can check the build status in the PR checks section below.

Thanks!
Just some change requests to fix the protobuf generation.

Thanks, fixed now.

Generally, I am just wondering whether the DWD needs some adaptation to honor this configuration, @amshuman-kr any thoughts?

Shoot validation could check for nodeMonitorGracePeriod must be larger than 120s if that is the minimum required value for DWD.

@mwennrich you have to run make generate now.


Need to specify you only support restricted DAG - single parents only

Hi, your project looks nice, I needed to get method signatures so I ended up modifying it to pass in a dictionary eg:
signatures = {"notify_payment" : notify_payment.si("1")}

and get tasks like:
    def _get_task(name):
        task_name = task_id(name)
        return self.signatures.get(task_name, signature(task_name))

However, I ended up not being able to use it because my nodes can have multiple parents. Maybe you could put a note in there about that, or get that to work. I think you'll need to define the JSON as child : dependency instead. Good luck!

Hi Dave! Thanks for opening this issue, this is a feature I decided not to implement because I wanted to start really small and I was not sure this would be even necessary. My idea was to allow to refer to other nodes as well as to the Celery tasks itself. For example:
    {
        'task0': {
            'task01': {
                'task011': { },
            },
            'task02': {
                'task01': { },
                'tasks021': { }
            }
        }
    }

In this case, task01 has two "parents". However, it seems a little bit unclear to me if putting tasks inside task02.task01 means these tasks will execute at the same time as the tasks defined in task01 or if this should be considered an overwrite of tasks01. I guess the later as this allows more flexibility.


browers remote alias in .testcaferc.json is not working

What is the Current behavior?
{
  "browsers": "remote",
  "src": "index.js",
  "screenshotPath": "screenshots",
  "screenshotPathPattern": "${DATE}_${TIME}/test-${TEST_INDEX}/${USERAGENT}/${FILE_INDEX}.png",
  "speed": 0.5
}

I have the above settings in .testcaferc.json , when i run testcafe in command line,
the spinner never stops and no remote link appears. like this:
$ testcafe
Using locally installed version of TestCafe.
/

What is the Expected behavior?
works like testcafe remote
Your Environment details:

testcafe version: 1.1.2
node.js version: v10.15.1
command-line arguments:  testcafe
platform and version:  Windows 10

I was able to reproduce the issue.

I discussed the issue with the team and we decided not to include the fix in our nearest plans, since it can lead to some significant changes. We'll consider implementing this feature in the future, however, I cannot give you any estimates on it.


Issue changing mesh file format from a .vtk to .mdpa

Hi all,
Issue Description
I am trying to change a mesh file format from a  .vtk  to  .mdpa   but I am getting an error when I use meshio library.
Code
import sys
import meshio
# Changing file format from a vtk to mdpa
mesh=meshio.read("extrudedmesh.vtk")
meshio.mdpa.write("test.mdpa", mesh)

Output error
meshio.mdpa.write("test.mdpa", mesh)
File "/home/uchasco/.local/lib/python3.8/site-packages/meshio/mdpa/_mdpa.py", line 533, in write
for key, data in a.items():
AttributeError: 'list' object has no attribute 'items'

Question
I would like to fix this issue in order to advance with my research, do you have any idea what the reason can be?. Likewise, I attach a zip file where the original mesh ( extrudedmesh.vtk)  and the python code (changingfiletype.py) are included.
Fromvtktomdpa.zip
Thank you in advance,
Uxue

Fixed now. Note that MDPA cannot deal with mesh data yet, so this is just dropped.


Running nginx-vts-exporter and got error:

./nginx-vts-exporter &
[1] 14394
[root@office-monitoring nginx-vts-exporter-0.6.linux-amd64]# 2017/09/05 08:58:48 Starting nginx_vts_exporter (version=0.6, branch=HEAD, revision=e48bd6899d8864b11b9f71f852b049ec3ad2ae26)
2017/09/05 08:58:48 Build context (go=go1.8.3, user=root@3b0fc1d0acb6, date=20170808-07:09:08)
2017/09/05 08:58:48 Starting Server at : :9913
2017/09/05 08:58:48 Metrics endpoint: /metrics
2017/09/05 08:58:48 Metrics namespace: nginx
2017/09/05 08:58:48 Scraping information from : http://localhost/status
2017/09/05 08:58:51 json.Unmarshal failed invalid character '<' looking for beginning of value
2017/09/05 08:59:06 json.Unmarshal failed invalid character '<' looking for beginning of value
2017/09/05 08:59:21 json.Unmarshal failed invalid character '<' looking for beginning of value
2017/09/05 08:59:36 json.Unmarshal failed invalid character '<' looking for beginning of value
2017/09/05 08:59:51 json.Unmarshal failed invalid character '<' looking for beginning of value
2017/09/05 09:00:06 json.Unmarshal failed invalid character '<' looking for beginning of value
2017/09/05 09:00:21 json.Unmarshal failed invalid character '<' looking for beginning of value
2017/09/05 09:00:36 json.Unmarshal failed invalid character '<' looking for beginning of value
Do you konw how to resolving the issue as above?
Thank you!

You should make sure http://localhost/status is the url where vts-module exposes its metrics.
I think this is the point you should take into account.

Yes, I can access http://localhost/status from my browser.

And this url should return json format metrics.

Hi hnlq715,
Thanks for your help, I tried change nginx.conf and change the code "vhost_traffic_status_display_format html;" to "vhost_traffic_status_display_format json;" and now it's ok:
./nginx-vts-exporter &
[1] 4941
2017/09/08 08:31:56 Starting nginx_vts_exporter (version=0.6, branch=HEAD, revision=e48bd6899d8864b11b9f71f852b049ec3ad2ae26)
2017/09/08 08:31:56 Build context (go=go1.8.3, user=root@3b0fc1d0acb6, date=20170808-07:09:08)
2017/09/08 08:31:56 Starting Server at : :9913
2017/09/08 08:31:56 Metrics endpoint: /metrics
2017/09/08 08:31:56 Metrics namespace: nginx
2017/09/08 08:31:56 Scraping information from : http://localhost/status
But I have another questions, "nginx-vts-exporter" process will terminated automatically about in a day, do you konw how can I setup log and to find the root cause?

Glad to here this 😃
The log of nginx-vts-exporter would direcetly print into stdout, and you can use nohup ./nginx-vts-exporter & to capture log into nohuo.out file.

I created log file for nginx-vts-exporter, but when the process terminated, no any other information:
tailf /var/log/nginx-vts-exporter.log
2017/09/11 14:58:39 Starting nginx_vts_exporter (version=0.6, branch=HEAD, revision=e48bd6899d8864b11b9f71f852b049ec3ad2ae26)
2017/09/11 14:58:39 Build context (go=go1.8.3, user=root@3b0fc1d0acb6, date=20170808-07:09:08)
2017/09/11 14:58:39 Starting Server at : :9913
2017/09/11 14:58:39 Metrics endpoint: /metrics
2017/09/11 14:58:39 Metrics namespace: nginx
2017/09/11 14:58:39 Scraping information from : http://localhost/status
I don't know why the process terminted automatically, please advise.

Maybe the process exits when your ssh terminal quits? Please try nohup ./nginx-vts-exporter & to launch this program

Yes, you are right, I used  "nohup" and the process is running ok now.
Thanks you very much!

Nice :-)

Scraping information from : http://localhost/status
How to replace it to more correct path like http://localhost/status/format/json ? I made
export NGINX_STATUS=http://localhost/status/format/json but unsuccessfully.

Scraping information from : http://localhost/status
How to replace it to more correct path like http://localhost/status/format/json ? I made
export NGINX_STATUS=http://localhost/status/format/json but unsuccessfully.

try http://localhost/status?json


Worker autotuning

Is your feature request related to a problem? Please describe.
Temporal worker should be able to autotune themselves based on some set of rules
Additional context
https://docs.temporal.io/dev-guide/worker-performance

I do not believe this is a Go-specific request. This is a general feature for SDKs.

This issue is to track the Go Specific implementation


home screen detail view CodeCardAction should support other code card options

Is your feature request related to a problem? Please describe.
want to make code cards for the advanced streams, many of which are multi part, and also include links to the completed games.
Describe the solution you'd like
other actions is the obj used to support multiple buttons on a homescreen card. It would be nice to expand this to use a larger subset of CodeCard's options; e.g. it would be nice to be able to do something like this:
{
    "name": "Game Idea Generator",
    "description": "Build a game idea generator to help decide on your next game",
    "youTubeId": "48uOkTLnpDk",
    "otherActions": [
        {
            "label": "Video 2"
            "youtubeId": "asdjfklasdjfl"
        }
        {
            "label": "Completed Project",
            "url": "https://makecode.com/_KXqTM06c1H6v",
            "editor": "blocks",
            "cardType": "example"
        }
    ]
}

and have it show a card with a buttons to go to the first video, second video, and one to open up the linked project, and the given labels for the projects on the buttons

Fixed


Allow sorting of claim related functions

The Issue
Per discussion in https://github.com/lbryio/lbry/pull/1216#issuecomment-392835700,
we should enable the ability to pass a sort parameter to claim related functions in order to control the sorting method.
The original issue, which was addressed to provide a consistent sort by default for claim_list/claim_list_mine, can be found here: https://github.com/lbryio/lbry/issues/1098
file_list recently went through a similar enhancement, the PR can be found here: https://github.com/lbryio/lbry/pull/1174
Claim related functions:
https://lbryio.github.io/lbry/cli/#claim_list_by_channel
https://lbryio.github.io/lbry/cli/#channel_list
https://lbryio.github.io/lbry/cli/#claim_list
https://lbryio.github.io/lbry/cli/#claim_list_mine
System Configuration

LBRY Daemon version:
LBRY App version:
LBRY Installation ID:
Operating system:

Anything Else

Screenshots

i think @eukreign did this in 0.38

No, but we can do it as part of https://github.com/lbryio/lbry/issues/1853


Allow both string and name type

This PR allows the old syntax, and the syntax I suggested in #3 .
@mbardelmeijer can you tell me what you think about this ? I am open to any modification :)

Oh wow, very cool! I'll take a look!

Oh wow, very cool! I'll take a look!

Can you add a test for the name & string variants with a component that consists of 2 parts as well. For instance: user-create, and one for the nested variant suggested in #3, something like table.user-search?
Then we can ensure that behavior that we want in the end is checked by tests 👍
Other than that, looking very good. Glad that we can merge this into the 1.x release cycle!

Can you add a test for the name & string variants with a component that consists of 2 parts as well. For instance: user-create, and one for the nested variant suggested in #3, something like table.user-search?
Then we can ensure that behavior that we want in the end is checked by tests 👍
Other than that, looking very good. Glad that we can merge this into the 1.x release cycle!

And here we go !
Those tests cover the 2 points to asked, just need a readme update (incoming)

And here we go !
Those tests cover the 2 points to asked, just need a readme update (incoming)

Awesome, thank you so much! I'll tag 1.2.0 in a bit.

Awesome, thank you so much! I'll tag 1.2.0 in a bit.


Revisit scf.for bufferization.

A few minor adjustments to scf.for bufferization.

When tracking number of uses, specifically look for destructive
update patterns to treat than as "non real" uses since these uses
are due to tensor abstraction limitations.
Refactor the walk to store and also stop the walk at the
subtensor_insert that is a destructive update.
During scf.for conversion, replace the operands of the scf.yield
with the values for the arg that represent the initArg of the
loop. This effectively marks the (result, initArg, regionArg) set as
a no-op that is canonicalized later.
Remove the now unecessary loop manipulation added previously to make
the loop dead.

@asaadaldien looked at some of the things that I was surprised that just worked out of the box. Most changes are just refactoring, except for the walk to get the result tensor. That needs to handle subtensor_insert, and this also helps remove some copies.
I would suggest ignoring the second commit for the review. That is just a clean up past the integrate commit which accounts for reshape op now being split. It is not relevant to this PR really, but need it to land the PR.