Implement Iptables based Proxy

This is an implementation of https://github.com/GoogleCloudPlatform/kubernetes/issues/3760.
It implements an iptables-restore wrapper, a dummy 'loadbalancer' that serves as a config.EndpointsConfigHandler, and a purely iptables based version of Proxier, as well as a method for determining whether to use the new ProxierIptables or the old Proxier based on the iptables version. kube-proxy now autoselects between the two and uses the new ProxierIptables + DummyLoadBalancerIptables pair when possible.
TODO:

[x] Write new Proxier and LoadBalancer (ProxierIptables and DummyLoadBalancerIptables)
[x] Update kube-proxy to select between old Proxier and ProxierIptables
[x] Basic iptables rule generation
[x] Faster rule generation (switched to using bytes.Buffer[s] for string concatenation)
[x] Sticky Sessions
[x] Delete old rule chains
[x] Testing. We need more tests and this needs to be more thoroughly run through e2e when done. (mostly done now, just need more verification on e2e and the iptables rules)
[x] Select iptables minimum version for switching over from Proxier to ProxierIptables (I've selected a version now, but we may want to change it, this probably needs some discussion)
[x] Make rules fully deterministic (namely make the chain generation use hashing instead of a counter).
[x] Parse iptables-save output to extract existing chains and rules
[x] Use iptables-save output to preserve counters

Further TODO:

[ ] Use iptables-save output to minimize excess rule rewriting.
[ ] Handle the edge case with pod->self from https://github.com/BenTheElder/kubernetes/issues/3

This has a long ways to go before it should be merged, but I wanted to get an issue/PR up tracking the state of it since otherwise I've been pretty quietly working on this.
I'd also like to get some eyes on it just to verify that it's headed in the right direction.
Vagrant/e2e haven't been playing very nice [bugging out during validation mostly, occasionally the vbox kext has even crashed my laptop(!)] so testing this has been very slow.
/cc @thockin

I'm not sure what happened to the commit history here, looks like I messed up a rebase or something.
I'll try to clean it up when this is ready.

I've pushed some smaller PR's working on breaking down some of the things that this will eventually rely on. I'll clean it and the commit history up and try to finish debugging/vetting the iptables rules once https://github.com/GoogleCloudPlatform/kubernetes/pull/9563 lands.
After that I'll be on track to move on to writing new tests hopefully.

@BenTheElder Thanks for your comments on #7245.
2 things worth drawing your attention to:
Firstly, in Calico, instead of attaching the container veth to a local bridge, we leave the host end in the root namespace and use Linux routing instead of bridging.  We don't assign an IP address to the veth in the root namespace (only the container namespace).  In the current userspace proxy, we find that REDIRECT rules don't work correctly to redirect traffic to local ports.  I don't think this is an issue in the iptables proxy, since you'll be forwarding directly to the service instance pod, but it's something to keep in mind.
Secondly, it is highly desirable for the traffic that arrives at the service instance pod to have the correct source IP (and source port, although in practice port is less important than IP) from the pod that is connecting to the service.  (Rather than, say, an IP associated with the pod host.)  This is because Calico uses IP tables rules to enforce network policy.  Kubernetes doesn't support defining network policy, but will be doing so very soon with the introduction of namespaces, so it's important we get the proxy working in concert with those efforts.
Hopefully that makes sense!  I'm happy to discuss further if you want clarification or detail.

@spikecurtis Thanks.
I currently am only using REDIRECT in a certain case of nodePorts in the same was as the userspace proxy but otherwise DNAT, you can see the rule generation in syncProxyRules here: https://github.com/BenTheElder/kubernetes/blob/iptables_proxy/pkg/proxy/proxieriptables.go#L247
I notice that in #7245 that nodePorts were not implemented in proxier when you implemented those changes, you may want to rebase from upstream and look into that.
I believe the source IP should be preserved correctly as I'm only using dnat otherwise; no MASQUERADE etc, but that is one of the goals of this implementation so any error in that respect should be fixed and working before this lands.
Other than that, I'm currently adding/maintaining the rules for connecting PREROUTING and OUTPUT to the proxy the same way as the userspace version; but the rest of the rules are synced with iptables-restore periodically, flushing the existing Kubernetes chains and restoring freshly generated rules/chains for atomicity. Hopefully that shouldn't be a problem but it does mean trying to add rules to Kubernetes' chains could be problematic. It doesn't look like calico does this though, looking at #7245.

How easy will be to automate the removal of the new iptables rules, if kube-proxy is terminated and k8s is uninstalled from a node?
see https://github.com/mesosphere/kubernetes-mesos/issues/353

@jdef Well all of the chains are known, we have 4 chains that always exist (KUBE-PORTALS-CONTAINER, KUBE-PORTALS-HOST, KUBE-NODEPORT-CONTAINER, KUBE-NODEPORT-HOST) and then a number of KUBE-SERVICE- prefixed chains generated in this version. I track all of the chains internally so that the KUBE-SERVICE-* chains are cleaned up and then regenerated each update (as well as the other 4); but should kube-proxy be terminated we could parse iptables-save for :<CHAIN_NAME> and :KUBE-SERVICE* and flush then delete each of those chains.

The history should be cleaned up now. I've just finished running a full e2e before pushing this and i see 13 failures (output: https://gist.github.com/BenTheElder/f8d7f70fa5bd45f9ad06#file-e2e-snippet-L5130) however as noted in https://github.com/GoogleCloudPlatform/kubernetes/issues/9681 these failures occur on master currently and seem to be related to e2e + vagrant.

That last commit also fixes an odd bug: while running e2e i saw instances where the port allocator was returning 0 which appears to be default behavior for one without a configured port-range. Iptables of course won't let you use 0 for your ports so I have a fallback routine that attempts to get a port by opening then a socket with port 0 and getting the actual port then closing it and returning the port.

Pretty typical rules on a minion from e2e currently look like:
[vagrant@kubernetes-minion-2 ~]$ sudo iptables-save
# Generated by iptables-save v1.4.21 on Wed Jun 17 05:57:33 2015
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [2:120]
:POSTROUTING ACCEPT [2:120]
:DOCKER - [0:0]
:KUBE-NODEPORT-CONTAINER - [0:0]
:KUBE-NODEPORT-HOST - [0:0]
:KUBE-PORTALS-CONTAINER - [0:0]
:KUBE-PORTALS-HOST - [0:0]
:KUBE-SERVICE-0 - [0:0]
:KUBE-SERVICE-0_0 - [0:0]
:KUBE-SERVICE-1 - [0:0]
:KUBE-SERVICE-1_0 - [0:0]
:KUBE-SERVICE-2 - [0:0]
:KUBE-SERVICE-2_0 - [0:0]
-A PREROUTING -m comment --comment "handle ClusterIPs; NOTE: this must be before the NodePort rules" -j KUBE-PORTALS-CONTAINER
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A PREROUTING -m addrtype --dst-type LOCAL -m comment --comment "handle service NodePorts; NOTE: this must be the last rule in the chain" -j KUBE-NODEPORT-CONTAINER
-A OUTPUT -m comment --comment "handle ClusterIPs; NOTE: this must be before the NodePort rules" -j KUBE-PORTALS-HOST
-A OUTPUT ! -d <IP_ADDRESS>/8 -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m addrtype --dst-type LOCAL -m comment --comment "handle service NodePorts; NOTE: this must be the last rule in the chain" -j KUBE-NODEPORT-HOST
-A POSTROUTING -s <IP_ADDRESS>/24 ! -o cbr0 -j MASQUERADE
-A KUBE-PORTALS-CONTAINER -d <IP_ADDRESS>/32 -p tcp -m comment --comment "portal for default/kube-dns:dns-tcp" -m state --state NEW -m tcp --dport 53 -j KUBE-SERVICE-0
-A KUBE-PORTALS-CONTAINER -d <IP_ADDRESS>/32 -p tcp -m comment --comment "portal for default/kubernetes:" -m state --state NEW -m tcp --dport 443 -j KUBE-SERVICE-1
-A KUBE-PORTALS-CONTAINER -d <IP_ADDRESS>/32 -p udp -m comment --comment "portal for default/kube-dns:dns" -m state --state NEW -m udp --dport 53 -j KUBE-SERVICE-2
-A KUBE-PORTALS-HOST -d <IP_ADDRESS>/32 -p tcp -m comment --comment "portal for default/kube-dns:dns-tcp" -m state --state NEW -m tcp --dport 53 -j KUBE-SERVICE-0
-A KUBE-PORTALS-HOST -d <IP_ADDRESS>/32 -p tcp -m comment --comment "portal for default/kubernetes:" -m state --state NEW -m tcp --dport 443 -j KUBE-SERVICE-1
-A KUBE-PORTALS-HOST -d <IP_ADDRESS>/32 -p udp -m comment --comment "portal for default/kube-dns:dns" -m state --state NEW -m udp --dport 53 -j KUBE-SERVICE-2
-A KUBE-SERVICE-0 -j KUBE-SERVICE-0_0
-A KUBE-SERVICE-0_0 -p tcp -m recent --set --name KUBE-SERVICE-0_0 --mask <IP_ADDRESS> --rsource -j DNAT --to-destination <IP_ADDRESS>:53
-A KUBE-SERVICE-1 -j KUBE-SERVICE-1_0
-A KUBE-SERVICE-1_0 -p tcp -m recent --set --name KUBE-SERVICE-1_0 --mask <IP_ADDRESS> --rsource -j DNAT --to-destination <IP_ADDRESS>:6443
-A KUBE-SERVICE-2 -j KUBE-SERVICE-2_0
-A KUBE-SERVICE-2_0 -p udp -m recent --set --name KUBE-SERVICE-2_0 --mask <IP_ADDRESS> --rsource -j DNAT --to-destination <IP_ADDRESS>:53
COMMIT

As far as I can tell the service and endpoint monitoring should be working 100%, the chain tracking works, iptables.Restore(...) works,  logging is fine (but we might want to change some of the log levels), the only thing left is just ensuring that the rules generated in syncProxyRules() (in proxieriptables.go) are functioning as needed, and then writing some more tests to verify all of the components.
I'll be working on those tests today, but i'd appreciate any help reviewing the rule generation.
cc @lxpollitt

@BenTheElder The person on my team I would most like to look at this is @Symmetric, but he is on vacation.  He's back in the office Tuesday next week (23rd).  He should be able to look at it then.  CC'ing @fasaxc in case he gets a chance to look at this later today too.

Thanks!
On Jun 18, 2015 6:11 PM, "Alex Pollitt"<EMAIL_ADDRESS>wrote:

@BenTheElder https://github.com/BenTheElder The person on my team I
would most like to look at this is @Symmetric
https://github.com/Symmetric, but he is on vacation. He's back in the
office Tuesday next week (23rd). He should be able to look at it then.
CC'ing @fasaxc https://github.com/fasaxc in case he gets a chance to
look at this later today too.
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/9210#issuecomment-113302574
.

I'd be happy to help with testing this with my application if that would be useful.

@statik that would be. I'm currently testing in a vagrant test cluster, on which a number of tests are currently always failing even on master so it's been difficult to finish validating it.

Rebased to current master.

@BenTheElder I've just started giving this another look, I'll apply our patches on top of the latest code here and check that the iptables rules look correct. Should (hopefully) have something tomorrow, I'll let you know how I get on.

Thanks!
On Thu, Jun 25, 2015, 20:51 Paul Tiplady<EMAIL_ADDRESS>wrote:

@BenTheElder https://github.com/BenTheElder I've just started giving
this another look, I'll apply our patches on top of the latest code here
and check that the iptables rules look correct. Should (hopefully) have
something tomorrow, I'll let you know how I get on.
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/9210#issuecomment-115448113
.

Hi @BenTheElder , I gave your PR a spin, and I'm seeing some unexpected rules; it looks like  services (e.g. kube-dns) are now getting the old REDIRECT rules, instead of DNAT:
-A KUBE-PORTALS-CONTAINER -d <IP_ADDRESS>/32 -p udp -m comment --comment "default/kube-dns:dns" -m udp --dport 53 -j REDIRECT --to-ports 60906
-A KUBE-PORTALS-CONTAINER -d <IP_ADDRESS>/32 -p tcp -m comment --comment "default/kube-dns:dns-tcp" -m tcp --dport 53 -j REDIRECT --to-ports 42567
-A KUBE-PORTALS-CONTAINER -d <IP_ADDRESS>/32 -p tcp -m comment --comment "default/kubernetes:" -m tcp --dport 443 -j REDIRECT --to-ports 60461
-A KUBE-PORTALS-HOST -d <IP_ADDRESS>/32 -p udp -m comment --comment "default/kube-dns:dns" -m udp --dport 53 -j DNAT --to-destination <IP_ADDRESS>:60906
```

I'm on iptables 1.4.21. The ShouldUseProxierIptables logic looks incorrect:

    return (major >= IPTABLES_MIN_MAJOR) && (minor > IPTABLES_MIN_MINOR) && (patch > IPTABLES_MIN_PATCH)

This will return false for 1.4.21, I think you want >= on the minor version. I checked the previous commit and the DNAT rules are getting created as expected.

I'll test a bit more with my changes merged into your HEAD^ (without the ShouldUseProxierIptables) and let you know how that goes.

Hi @Symmetric, thanks. Yes that should be minor >=, that's a typo, it should be >= for all three, I'll patch that, and the redirect rules are only be generated for NodePorts bound to the Zero IP.
The current rules are based on comments in the existing Proxier iptablesContainerPortalArgs discussing DNAT not working on the Zero IP.
Looking at #7245 NodePort's don't appear to be implemented in that Proxier version yet. What would you expect those rules to look like?

Hmm, if we allow any traffic to go through the kube-proxy then we lose the true source IP for packets arriving at the service pod, so our source-filter ACLs will be broken in that case. I may be able to work around this case by mandating the --proxyIP flag, so it's not the end of the world. Failing that I can just mandate that Calico users don't use nodePort if they want do use ACLs.
However, I'm wondering if there is another way around this; it seems a shame to have the kube-proxy userspace process still kicking around just for the single case of handling nodePort services. In the old proxier.go, the comment said
	// Unfortunately, I don't know of any way to listen on some (N > 1)
	// interfaces but not ALL interfaces, short of doing it manually, and
	// this is simpler than that.

I'm wondering if some variant of the "doing it manually" option is now worth implementing, compared to the complexity cost of keeping kube-proxy around. @thockin may know off the top of his head if it would be feasible.

I know we can make DNAT to localhost work on recent kernels (>= 3.6) by setting net.ipv4.conf.all.route_localnet=1 which should solve:
	//   * DNAT to <IP_ADDRESS> = Packets just disappear - this seems to be a
	//     well-known limitation of iptables.

But I'm not sure about the other issues outlined (including the one you mentioned):
https://github.com/BenTheElder/kubernetes/blob/iptables_proxy/pkg/proxy/proxier.go#L694
	// This is tricky.
	//
	// If the proxy is bound (see Proxier.listenIP) to <IP_ADDRESS> ("any
	// interface") we want to use REDIRECT, which sends traffic to the
	// "primary address of the incoming interface" which means the container
	// bridge, if there is one.  When the response comes, it comes from that
	// same interface, so the NAT matches and the response packet is
	// correct.  This matters for UDP, since there is no per-connection port
	// number.
	//
	// The alternative would be to use DNAT, except that it doesn't work
	// (empirically):
	//   * DNAT to <IP_ADDRESS> = Packets just disappear - this seems to be a
	//     well-known limitation of iptables.
	//   * DNAT to eth0's IP = Response packets come from the bridge, which
	//     breaks the NAT, and makes things like DNS not accept them.  If
	//     this could be resolved, it would simplify all of this code.
	//
	// If the proxy is bound to a specific IP, then we have to use DNAT to
	// that IP.  Unlike the previous case, this works because the proxy is
	// ONLY listening on that IP, not the bridge.
	//
	// Why would anyone bind to an address that is not inclusive of
	// localhost?  Apparently some cloud environments have their public IP
	// exposed as a real network interface AND do not have firewalling.  We
	// don't want to expose everything out to the world.
	//
	// Unfortunately, I don't know of any way to listen on some (N > 1)
	// interfaces but not ALL interfaces, short of doing it manually, and
	// this is simpler than that.
	//
	// If the proxy is bound to localhost only, all of this is broken.  Not
	// allowed.

I finally managed to do some testing of this tonight, I built and deployed this branch into GCE. I was hopeful that the software running in my pod would be able to see the source IP of the system that was connecting to the load balancer. Instead, I still see a <IP_ADDRESS> address as the origin. I am wondering whether this is an artifact of the way the GCE load balancers work or is something not working as expected in the new proxier? When I am running code in AWS (separate from kubernetes) that passes traffic through an ELB, my code is able to get the correct source IP of the traffic that is coming in to the load balancer.

hmm, just found this, so maybe the GCE load balancer isn't going to pass on the client info the way the AWS one does http://aws.amazon.com/about-aws/whats-new/2013/07/30/elastic-load-balancing-now-supports-proxy-protocol/

@statik Ideally expected behavior is that we preserve the source IP, this should be doing that as we use DNAT for pretty much everything.

I was mistaken that things were working differently in AWS :) I don't think there is any problem with this branch.

FWIW on my early integration branch I hacked the kube-proxy to just use DNAT to the host's IP instead of REDIRECT, and it worked. (this diff deletes that function from the PR https://github.com/GoogleCloudPlatform/kubernetes/commit/91ba5713539dc365436e893a0118503fbd86f051#diff-9cb51239c40a339d14e794fe66b04bf7L511 ).
I suspect that it might be some dependency on what IPs are on the system; in the Vagrant setup the kube-proxy is listening on <IP_ADDRESS>, which happens to be the lowest numbered IP. I think this bears further testing, but I could believe that when the kube-proxy is sending the return packet to the local pod, Linux may just pick an arbitrary IP on the node to use as the source, which may or may not correspond to the IP that the REDIRECT rule hit by the outgoing packet. If instead of REDIRECT, we DNAT to the lowest IP on the machine, that might "just work" (tm).
@BenTheElder I've got some other work to do before I can spend more time on this, what's your timeline looking like? I'll probably be able to dig more in the second half of next week; will that still be useful?

Excellent! Thank you for testing this :)
On Jul 1, 2015 10:08 AM, "Elliot Murphy"<EMAIL_ADDRESS>wrote:

I was mistaken that things were working differently in AWS :) I don't
think there is any problem with this branch.
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/9210#issuecomment-117687559
.

@Symmetric, hmm, well I'll look more into the DNAT/REDIRECT today; and yes that will most likely still be useful. I'm waiting on @thockin to review and this is not going to be merged until post 1.0 is my understanding so until this has been checked off I will just be doing what I can to ensure that it does.
I'll be keeping this up to date with master primarily at the moment.

Thanks again to both of you @statik @Symmetric. I'm very happy to have some feedback on this.

Rebased to current master and built locally, shippable should be a flake as seen in https://github.com/GoogleCloudPlatform/kubernetes/issues/10420 (?)

@Symmetric I'm still not sure about the DNAT/REDIRECT, clearly in some cases we can just use DNAT (as we do already) but I don't think we always can per the comments.
According to git blame those were written by thockin, so maybe he can elaborate.
@thockin PTAL -- I can write a more detailed design overview if needed but the tl;dr is:

This uses a new Proxier and 'LoadBalancer' to track Services and Endpoints and synchronizes iptables rules via iptables-restore mimicing functionality from the existing implementations as much as possible. We switch between the implementations in kube-proxy based on a method that checks the iptables version for a minimum version.

I have two smaller PRs out and one through that contain part of the changes here as spin offs in an attempt to make the changes less monolithic; but ultimately this is just a big change with most of the new lines being in the new 'Proxier' and 'LoadBalancer' as well as the tests.
Realistically it would probably (?) be best to keep the changes together here since they only really make sense together.

From earlier comments this seems to be functioning well when tested; however e2e is currently somewhat flakey for me locally both on this and on master, mainly due to performance issues under vagrant.
I'll be running them on some more powerful hardware again today with hopefully better luck (there has also been some work on removing irrelevant tests from vagrant).
I'll also be pushing another rebase to master to keep this from falling behind; What is the timeline looking like for v1.0 since this seems to be post 1.0?

Nice to finally see all green on shippable again.

@MikaelCluseau looks like the googlebot is complaining about your commits, can you check on that please?

Rebased to latest master again.

Oh, it seems I need to sign something... I'll check how to do that.

Thanks. https://github.com/GoogleCloudPlatform/kubernetes/blob/master/CONTRIBUTING.md#contributor-license-agreements should have the details on the CLA(s).

I followed the login link, and submitted my CLA.

@googlebot should be good now.

Oh, yes, it's another e-mail... @googlebot confirm

@googlebot are you happy now? :)

This is barred by the issue at https://github.com/BenTheElder/kubernetes/issues/3 currently.

@thockin @erictune could someone help us placate the googlebot? It's adding a lot of noise to the PR, there are now 8 duplicate posts about the CLA.

Thanks!

... And it's back. @erictune googlebot put it back :/

I don't think it is possible for me to shut googlebot up as long as there are multiple authors.  If Michael Cluseau is okay with it, I think that you should just squash his commits into yours. That should make googlebot happy.

@MikaelCluseau would this (squashing the pr commits) be acceptable? I will probably need to squash the commits anyhow prior to merging later on given that I plan to add more for https://github.com/BenTheElder/kubernetes/issues/4.

@BenTheElder do as required, I did everything I could on my side, it seems that adding a secondary email address to my account after having signed locks me in a state were the CLA app still tells me I already signed, but doesn't include the secondary email address I added following the link on the same page, and doesn't allow me to "unsign/sign" either. There must be a missing job to correlate changes in Google accounts' emails to CLAs.
Anyway, I can either edit these small commits to change my email address in them, or let you squash, as is easier for you.

I'm not really familiar with the CLA system but It's normal to squash
commits before merge so I'll probably just do that regardless.
On Thu, Jul 16, 2015 at 4:18 PM, MikaelCluseau<EMAIL_ADDRESS>wrote:

@BenTheElder https://github.com/BenTheElder do as required, I did
everything I could on my side, it seems that adding a secondary email
address to my account after having signed locks me in a state were the CLA
app still tells me I already signed, but doesn't include the secondary
email address I added following the link on the same page, and doesn't
allow me to "unsign/sign" either. There must be a missing job to correlate
changes in Google accounts' emails to CLAs.
Anyway, I can either edit these small commits to change my email address
in them, or let you squash, as is easier for you.
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/9210#issuecomment-122077560
.

Note that https://github.com/BenTheElder/kubernetes/issues/4 is now the blocking issue for this.
https://github.com/BenTheElder/kubernetes/issues/3 is mostly taken care of, we just need to decide how to take care of the sysctl settings now required when working with flannel.
I plan to open an issue related to that, probably when this starts to go under review.

sysctl settings now required when working with flannel

I don't think it's linked to flannel. It will hit anyone upgrading to a 3.18+ kernel without taking care of modprobe'ing br_netfilter,  as can be seen here:

Since Linux kernel 3.18-rc1, you have to modprobe br_netfilter to enable bridge-netfilter.

Ah. Well we'll probably need a separate (but linked) issue here to discuss
how to handle documenting that, etc.
On Thu, Jul 16, 2015 at 5:46 PM, MikaelCluseau<EMAIL_ADDRESS>wrote:

sysctl settings now required when working with flannel
I don't think it's linked to flannel. It will hit anyone upgrading to a
3.18+ kernel without taking care of modprobe'ing br_netfilter, as can be
seen here http://ebtables.netfilter.org/documentation/bridge-nf.html:
Since Linux kernel 3.18-rc1, you have to modprobe br_netfilter to enable
bridge-netfilter.
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/9210#issuecomment-122112823
.

Hey @BenTheElder, been thinking about the last use of the kube-proxy, and I think I've come up with a set of rules that will allow us to remove it.
I don't think the DNAT/REDIRECT issue applies here. Instead, if we do both DNAT and SNAT for packets that hit a node on the nodePort, then I think we're fine.
In more detail:
If we have traffic from an external load balancer hitting the node:
LBIP => nodeIP:nodePort
and full-NAT that to:
nodeIP => podIP:svcPort
then we don't hit the REDIRECT/DNAT issue, since that was only a problem when DNAT'ing to nodeIP. In this scheme we'll always be DNAT'ing to a pod IP, which never a local IP (but will always be routable by the node).
The corner case is when the source of the packet is another pod, i.e.
podAIP => nodeIP:nodePort
Which is NAT'ed to
nodeIP => podBIP:svcPort
This case will be handled fine by these rules, though we'll obscure the source IP, so it's not great for Calico. I'd prefer to ban this flow entirely and explicitly drop packets pod=>nodePort, since I think there's not a great reason for them to avoid using the service IP (and it will make all firewalling schemes simpler later on), that's a broader discussion to be had with @thockin et. al.
LMK if you think this will work, or if you need any more detail on the proposal.

Hi @Symmetric, I've been reading and re-reading your post, and I'm not quite sure I understand what you are suggesting.
For the full-NAT, you want to do full-NAT on the host for packets destined to the node port depending on the source ip? And otherwise keep the rules as is?
As far as obscuring the source IP goes, I think that's fine if it's necessary, but I don't really know that I should be making those decisions... Perhaps we should also move the discussion back over to the original issue at https://github.com/GoogleCloudPlatform/kubernetes/issues/3760 as well, as the discussion here hasn't really brought in the k8s team so far, and we've had other possible rule-rewrite discussions e.g. here: https://github.com/BenTheElder/kubernetes/issues/3

This has been rebased to current master as of https://github.com/GoogleCloudPlatform/kubernetes/commit/1367076d2e4e3b4e581923edf0c7bc69c004b70a.

So as I understand it, the only reason you have the kube-proxy still hanging around (except for the iptables-compatability mode) is to handle nodePort traffic (shout if that's incorrect and I've missed something). The rules I've proposed are (I think) what you'd need to handle that case purely in iptables. The effect is basically the same as the old kube-proxy.
So I believe these rules would simplify your code significantly (i.e. no need to spin up the kube-proxy in both cases), while covering this case in the same way. Happy to raise in the main issue if you prefer though.

Sorry -- a bit more clarification is in order.
What I'm proposing is that for the only remaining case where you're still forwarding to the kube-proxy, i.e. if the dest-port == nodePort, instead do a SNAT (probably MASQUERADE, so that the packet looks like it's originating on the node), and then DNAT using the same rules that we're already using for the service's virtual IPs.
For all other cases (i.e. where the dest IP is the service VIP), we just use the existing DNAT rules, so this should ONLY affect the nodePort case.

@Symmetric, ok I understand what you mean now.
Well while that sounds like it would work fine and possibly simplify things quite a bit, doesn't that mean that we totally lose source-ip preservation as far as the actual services are concerned?

In the case that we're currently using the kube-proxy, we are already losing the source IP of the external LB. As long as this change only applies in that case, then we're not worse off.
For Calico, if you want to police the sourceIP fully then you would have to forbid nodePort. Not ideal, but workable.
The issue that I'm concerned about with just DNATing nodePort traffic is this:
If the traffic flows:
LB -> node1:nodePort
And the service pod is on node2, then the full flow will be:
LB -> node1:nodePort -> node2 -> pod:svcPort
The srcIP will still be the LB, so the response will go
pod -> node2 -> LB
Since node2 can route directly to the LB.
Now we lose the opportunity to un-DNAT to restore the correct source IP for the return packet (that can only happen on node1).
This is a corner of DC networking that I'm not intimately familiar with; it may be that it's possible to have the LB correlate the response packet from node2, even though it is not in the LB's TCP session with node1. The discussion on #10921 seems to suggest that something like this would work. But I don't know of any way to do that. @thockin?
If we can make this reverse path work, then I think there's no reason to SNAT, and we should just use the normal DNAT rules (no kube-proxy) for the nodePort case too.

That seems quite reasonable; I'm going to see about finishing up some stuff I've been working on locally (iptables-save checking etc) while we hopefully get some input from other developers/users, and revisit rule changes once I finish or we receive some more input.

Status Update:
I will rewrite this into smaller commits for review in the morning, and then probably go ahead and add the masquerade self rule.
Remaining from https://github.com/BenTheElder/kubernetes/issues/3 we still need to do the sysctl for the nf-bridge module kubernetes-wide, the masquerade self rule, and hairpin mode on bridges.
From https://github.com/BenTheElder/kubernetes/issues/4 I have solved deterministic chain names / rule generation and in theory the counters should be preserved now, but more intelligent usage of the iptables-saveoutput could still be achieved for possible performance gains; this should not hold back review but I still plan to work on it later.

The history here has been broken into separate commits roughly for each component added and should probably be squashed after review. In retrospect the commit adding ProxierIptables in particular is still pretty big but is a single file, I can break if up further if necessary.

Noting ahead of time that some of the other commits probably need some renaming as well. DummyLoadBalancerIptables in particular is one of the earliest things I wrote and is a bit unwieldy. Perhaps EndpointsTracker or something more a concise.

Not completely sure that MASQUERADE rule is ideal, will look back into after i fix the DummyLoadBalancerIptables rename / cleanup.

From @thockin  on an old soon to be squashed commit, talking about syncProxyRules():
I feel like this is more complex that it needs to be. If we set up the chains right it should be a simple case of asserting what we want and then cleaning up after.
Something like:
initIptables() {
    EnsureChain(TableNAT, "KUBE-SERVICES")
    EnsureRule(TableNAT, ChainOutput, "-j", "KUBE-SERVICES")
}

syncIptables() {
    chains := []chain{"KUBE-SERVICES"}
    rules := []rule{}
    for each service we should have {
        for each port on service {
            svcportchain := "KUBE-SERVICE-" + portname
            chains = append(chains, svcportchain)
            rules = append(rules, rule{
                chain: "KUBE-SERVICES",
                args: []string {
                    "-d", serviceIP+"/32",
                    "-m", "state", "--state", "NEW",
                    "-p", "tcp", "-m", "tcp",
                    "--dport", servicePortStr,
                    "-j", svcportchain
                }
            })
            for each endpoint {
                svcportendpointchain := make chain name
                add a chain for each svc+port+endpoint
                if affinity is on {
                    add the -m recent -rcheck rule to jump from svcportchain to svcportendpointchain
                }
            }
            for each endpoint
                add a --mode random --probability rule to jump from svcportchain to svcportendpointchain
            }
        }
    }
    write a tempfile with table name, all chains, then all rules, then COMMIT
    iptables-restore the tempfile --noflush

    // now clean up leftovers
    chains = iptables.ListChains(TableNAT)  // run iptables -t nat -S
    for each chain {
        if !chain.HasPrefix("KUBE-SERVICE-") {
            return
        }
        svcname = parse chain
        if svcname not in known list {
            add chain to list of stale chains
        }
     }
     for each stale chain {
          flush chain
      }
      for each stale chain {
           delete chain
       }
}

The layout this makes is :
OUTPUT & PREROUTING jump to KUBE-SERVICES
KUBE-SERVICES checks for accesses to portals and jumps to per-service chains
per-service chains apply balancing and jump to per-endpoint chains
The iptables-restore atomically (I think) recreates the whole space for chains we mean to keep
The final loops clean up the debris

My reply to above comment, also soon to be squashed away:
Discussed in IRC:

will still need to handle counters, but want to keep parsing of state in util/iptables package.
still need hashing or similar to handle chain length limits

Otherwise seems like a very clean simplification, will implement after some more discussion.

@BenTheElder You around?  I am ready for next stage review when you are.

@thockin I am, working on cleaning it up some.

When I'm done we won't need a separate endpoints tracker.

Huh what happened there.

@googlebot all good, I authored them.

oh nope, the commit history is fudged somehow, will fix.

Fixed, but i lost most of the cleanup. Still, this is much smaller than the original PR, and has the cli flag-gate. I'll get on cleaning it up some more like we discussed.

Let me know when ready for next round of review

@thockin, some more cleanup work is in, still planning to do a lot more. Also not sure about where the merge conflicts are coming from, will check into that.

Rebased again. Merge conflicts fixed.

@thockin, ready for some review, could use more work but it's much smaller and I've moved the parsing out of the proxier and fixed the logging and comments.

@thockin, for the semver stuff, should we still be checking for a minimum iptables version with ShouldUseProxierIptables? If we're just not going to support the userspace proxy in the future then we can remove all of that, semver stuff included.
If not i'll update the semver stuff per the comments.
Stale comments, mis-copies, moving protocol to portal, the chain parsing, etc have all been addressed and are ready for the next push.

Semver stuff still needs doing, I think I have addressed all other comments so far.

Also need to unit test the iptables-save chain lines parsing.

Going home now.  Back later.  If you push again, squash and I will look again.

Ok, will do :)
On Mon, Aug 10, 2015 at 9:35 PM, Tim Hockin<EMAIL_ADDRESS>wrote:

Going home now. Back later. If you push again, squash and I will look
again.
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/9210#issuecomment-129664893
.

ahhh i did it again. rebased too much. fixing.

history should be fixed.

Shippable says:
k8s.io/kubernetes/pkg/proxy/iptables
_output/local/go/src/k8s.io/kubernetes/pkg/proxy/iptables/proxier_test.go:20:2: cannot find package "k8s.io/pkg/util/iptables" in any of:
/root/.gvm/gos/go1.4/src/k8s.io/pkg/util/iptables (from $GOROOT)
/root/workspace/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/pkg/util/iptables (from $GOPATH)
/root/workspace/src/k8s.io/kubernetes/Godeps/_workspace/src/k8s.io/pkg/util/iptables
FAIL	k8s.io/kubernetes/pkg/proxy/iptables [setup failed]

Missing kubernetes after k8.sio in an import. thanks, saves me running them
again locally. (still will but can push a fix first).
On Mon, Aug 10, 2015 at 11:33 PM, Tim Hockin<EMAIL_ADDRESS>wrote:

Shippable says:
k8s.io/kubernetes/pkg/proxy/iptables
_output/local/go/src/
k8s.io/kubernetes/pkg/proxy/iptables/proxier_test.go:20:2: cannot find
package "k8s.io/pkg/util/iptables" in any of:
/root/.gvm/gos/go1.4/src/k8s.io/pkg/util/iptables (from $GOROOT)
/root/workspace/src/
k8s.io/kubernetes/_output/local/go/src/k8s.io/pkg/util/iptables (from
$GOPATH)
/root/workspace/src/
k8s.io/kubernetes/Godeps/_workspace/src/k8s.io/pkg/util/iptables
FAIL k8s.io/kubernetes/pkg/proxy/iptables [setup failed]
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/9210#issuecomment-129691219
.

There are more typos in proxier_test.go, fixing those now.

Overall looking pretty nice!  The substance of my comments has dropped far enough that, once we get the larger issues nailed, it's done.
On startup (NewProxier) you should tear down the old-style rules.  Just flush the chains and remove them, and remove the jump rules from OUTPUT and PREROUTING.
Oh.  e2e.  at least one test in test/e2e/service.go is no longer applicable - you should update that test case to cover old and new forms, with a TODO to remove old eventually.
Does this need any further e2e testing?  Can you think of anything more we could do to try to break this new model?

Most of the other comments should be handled in a commit soon.
Tearing down the old rules I'll get to shortly, if not tonight, tomorrow.
e2e, well the nodePort stuff won't work for now and route to self. Not sure
what else we need to do off the top of my head. I'll think about it
tomorrow, getting late here, have to go to the dentist in the morning
unfortunately.
On Tue, Aug 11, 2015 at 12:43 AM, Tim Hockin<EMAIL_ADDRESS>wrote:

Overall looking pretty nice! The substance of my comments has dropped far
enough that, once we get the larger issues nailed, it's done.
On startup (NewProxier) you should tear down the old-style rules. Just
flush the chains and remove them, and remove the jump rules from OUTPUT and
PREROUTING.
Oh. e2e. at least one test in test/e2e/service.go is no longer applicable

you should update that test case to cover old and new forms, with a TODO
to remove old eventually.

Does this need any further e2e testing? Can you think of anything more we
could do to try to break this new model?
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/9210#issuecomment-129701734
.

Regarding e2e: the e2e test intentionally flushes iptables rules and uses the old-style names

Ah, yeah we'll have to update that.
I'll do a pass over the e2e tests tomorrow, about to push for the night,
head off to grab some sleep.
On Tue, Aug 11, 2015 at 1:18 AM, Tim Hockin<EMAIL_ADDRESS>wrote:

Regarding e2e: the e2e test intentionally flushes iptables rules and uses
the old-style names
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/9210#issuecomment-129708826
.

TODO:

Teardown old style rules in iptables NewProxier()
e2e
Pass over mutex (should be fine now in theory, but worth checking)

Later TODO:

route to self (masquerade, hairpin)
nodePorts
bridge-nf modprobe

When you have a new push, squash and I'll pass it over again - I think we're close

Ok, looking at e2e right now. I'll see about tearing down the old rules
next, then push that and squash.
On Aug 11, 2015 2:21 PM, "Tim Hockin"<EMAIL_ADDRESS>wrote:

When you have a new push, squash and I'll pass it over again - I think
we're close
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/9210#issuecomment-130000495
.

@thockin, tearing down old rules added, squashed.

All my comments are minor.  Fix these up and fix up the e2e to work in both modes and it's LGTM.  We have a lot more to do :)

Just need to fix up e2e, then i'll push and squash.

pushed. should have the last round of comments and the e2e iptables rules stuff removed.

From my logs:
I0811 20:32:20.398166   16604 server.go:115] Using iptables Proxier.
I0811 20:32:20.398172   16604 proxier.go:121] Tearing down userspace rules.
E0811 20:32:20.407393   16604 proxier.go:150] Error removing userspace chain: (error deleting chain "KUBE-PORTALS-CONTAINER": exit status 1: iptables: Directory not empty.
)
E0811 20:32:20.408512   16604 proxier.go:153] Error removing userspace chain: (error deleting chain "KUBE-PORTALS-HOST": exit status 1: iptables: Directory not empty.
)
E0811 20:32:20.414080   16604 proxier.go:165] Error removing userspace chain: (error deleting chain "KUBE-NODEPORT-HOST": exit status 1: iptables: Directory not empty.
)
E0811 20:32:20.414670   16604 proxier.go:168] Error removing userspace chain: (error deleting chain "KUBE-NODEPORT-CONTAINER": exit status 1: iptables: Directory not empty.
)

note the errors and the ) wrapping lines.  just get rid of the parens

Ok, next round of fixes inbound soon.

Pushed & squashed some more fixes. Hopefully should take care of all current comments.
Need to test it some more.

Need to run out for a few hours.  Test test test.  Start thinking about how
to benchmark this (focus on on latency).
On Tue, Aug 11, 2015 at 2:41 PM, Benjamin Elder<EMAIL_ADDRESS>wrote:

Pushed & squashed some more fixes. Hopefully should take care of all
current comments.
Need to test it some more.
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/9210#issuecomment-130085996
.

For future, it's useful not to squash so I can see deltas, and then squashe when I am happy with them :)

Need a rebase to make shippable happy

Ok, will wait next time :)
On Tue, Aug 11, 2015 at 8:51 PM, Tim Hockin<EMAIL_ADDRESS>wrote:

For future, it's useful not to squash so I can see deltas, and then
squashe when I am happy with them :)
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/9210#issuecomment-130125030
.

I am going to dinner.  Back later.  If you have a new push up I wil lLGTM it and we can start on next steps.  I'll patch it into my own cluster and play with it and try to break what you have done so far.
Next steps:

DNAT-to-self corner case (hairpin)
sysctl(s)
NodePorts
any other things needed for feature parity?
load test and measure latency (I want a packaged test and a graph!)

From the GCE e2e log:
• Failure in Spec Teardown (AfterEach) [146.308 seconds]
Kubectl client
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:634
  Simple pod [AfterEach]
  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:207
    should support exec
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:169

    Aug 11 18:33:00.706: Couldn't delete ns namespace e2e-tests-kubectl-5mvxl was not deleted within limit: timed out waiting for the condition, pods remaining: []

    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubectl.go:82```
I don't think that's from anything here, will see if there has been a patch upstream since I last rebased.

@thockin, As for the test I have some (simple) ideas, but, where would this be packaged ideally? e2e? Something new under test/?
I was thinking we can just run it iptables and userspace mode, and measure latency from a simple service (and the kube-proxy load), perhaps a hello world http server or a tcp echo service; but I'm not sure where to start yet.

I'll make a last pass, but LGTM at this point.  Nice work.  Thanks for hanging in for the ride.
As for test: it could be an e2e or a contrib tool.  Fire up a Service backed by a single Pod on node X.  Run a test pod on node Y.  Do 1 million (or 100K or something) hits of the service IP, build a histogram of latency.  Then we can decide what the magic test-failure-threshold is for e2e :)

Thanks :)
Squashing and pushing now; I'll start on the tool/test next.
On Wed, Aug 12, 2015 at 12:08 AM, Tim Hockin<EMAIL_ADDRESS>wrote:

I'll make a last pass, but LGTM at this point. Nice work. Thanks for
hanging in for the ride.
As for test: it could be an e2e or a contrib tool. Fire up a Service
backed by a single Pod on node X. Run a test pod on node Y. Do 1 million
(or 100K or something) hits of the service IP, build a histogram of
latency. Then we can decide what the magic test-failure-threshold is for
e2e :)
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/9210#issuecomment-130154261
.

shippable is unhappy - rebase again?

well i squashed, did a fresh fetch and nothing new to rebase onto.

Shippable is still angry, and I can't see why.
+++ [0812 04:17:18] Building go targets for linux/amd64:
cmd/genconversion
+++ [0812 04:17:19] Placing binaries
+++ [0812 04:17:21] Building go targets for linux/amd64:
cmd/genconversion
+++ [0812 04:17:22] Placing binaries
goimports not in path, run go get golang.org/x/tools/cmd/goimports
!!! Error in /root/workspace/src/k8s.io/kubernetes/hack/update-generated-conversions.sh:28
'"${KUBE_ROOT}/hack/after-build/update-generated-conversions.sh" "$@"' exited with status 1
Call stack:
1: /root/workspace/src/k8s.io/kubernetes/hack/update-generated-conversions.sh:28 main(...)
Exiting with status 1
!!! Error in /root/workspace/src/k8s.io/kubernetes/hack/after-build/verify-generated-conversions.sh:40
'"${KUBE_ROOT}/hack/update-generated-conversions.sh"' exited with status 1
Call stack:
1: /root/workspace/src/k8s.io/kubernetes/hack/after-build/verify-generated-conversions.sh:40 main(...)
Exiting with status 1
!!! Error in ./hack/verify-generated-conversions.sh:28
'"${KUBE_ROOT}/hack/after-build/verify-generated-conversions.sh" "$@"' exited with status 1
Call stack:
1: ./hack/verify-generated-conversions.sh:28 main(...)
Exiting with status 1

Huh. I'll look at what that script does.
On Aug 12, 2015 12:27 AM, "Tim Hockin"<EMAIL_ADDRESS>wrote:

Shippable is still angry, and I can't see why.
+++ [0812 04:17:18] Building go targets for linux/amd64:
cmd/genconversion
+++ [0812 04:17:19] Placing binaries
+++ [0812 04:17:21] Building go targets for linux/amd64:
cmd/genconversion
+++ [0812 04:17:22] Placing binaries
goimports not in path, run go get golang.org/x/tools/cmd/goimports
!!! Error in /root/workspace/src/k8s.io/kubernetes/hack/update-generated-conversions.sh:28
'"${KUBE_ROOT}/hack/after-build/update-generated-conversions.sh" "$@"' exited with status 1
Call stack:
1: /root/workspace/src/k8s.io/kubernetes/hack/update-generated-conversions.sh:28 main(...)
Exiting with status 1
!!! Error in /root/workspace/src/k8s.io/kubernetes/hack/after-build/verify-generated-conversions.sh:40
'"${KUBE_ROOT}/hack/update-generated-conversions.sh"' exited with status 1
Call stack:
1: /root/workspace/src/k8s.io/kubernetes/hack/after-build/verify-generated-conversions.sh:40 main(...)
Exiting with status 1
!!! Error in ./hack/verify-generated-conversions.sh:28
'"${KUBE_ROOT}/hack/after-build/verify-generated-conversions.sh" "$@"' exited with status 1
Call stack:
1: ./hack/verify-generated-conversions.sh:28 main(...)
Exiting with status 1
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/9210#issuecomment-130156096
.

So it's complaining about goimports not being in the path and then exiting in:
hack/update-generated-conversions.sh
if ! which goimports >/dev/null; then
	echo "goimports not in path, run go get golang.org/x/tools/cmd/goimports"
	exit 1
fi

Not sure why that would be showing up now. Digging some more....

If it isn't clear, I am running it and finding anything I dislike.  gonna run a few more tests, maybe pick nits ad then LGTM

Ok, watching here and pushing fixes for them all when you're done.
On Wed, Aug 12, 2015 at 1:00 AM, Tim Hockin<EMAIL_ADDRESS>wrote:

If it isn't clear, I am running it and finding anything I dislike. gonna
run a few more tests, maybe pick nits ad then LGTM
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/9210#issuecomment-130164893
.

I've updated for everything here so far but it will take time to check that it functions correctly.

Going to go ahead and push, so you can look it over, still testing locally.

a few last nits on the last commit.
I ran a pretty full battery of manual tests against this and it held up.  Something is up with load-balancer IPs, but we won't hold this up any longer

let me know when ready for LGTM

i've pushed the fixes, just need to squash when ready.

@thockin, let me know when to squash or if any last nits. otherwise, good to me.
shippable still seems unhappy but i don't think that's from the changes here.

LGTM
1 last nit.
I'm seeing failure to connect to load-balancer IPs.  We can debug it later.
@k8s-oncall would appreciate a prio merge tomorrow - this is behind a flag gate and GSoC ends soon.  No idea what's up with shippable.

fixing nit and squashing.

oh yeah, squash please

#12568 rebasing against that should fix shippable it looks like.

LGTM

Reviewing log...

I figured out the LB issue, it's a GCE thing, we have a fix in the works.
Next: graphs, nodeports, and hairpins.

@ArtfulCoder you'll have to patch your external IPs work onto this, too

Also the bridge-nf module sysctl/modprobe right?

yes - I think of what's left as the :advanced use cases" :)

@thockin, which do you want to tackle first? I'll open a PR in the morning.
Though I'm not sure how we want to handle nodePorts yet (obviously with some iptables rules, but...).
The other two "advanced uses" could just be some init scripts (salt-stack maybe?) to probe/load the module and set hairpin mode on the bridges or something, and the graphs can be from a  contrib script/tool.

Shippable is green again :green_heart:

@thockin if I may, the hairpin requirement is for running an Apache Kafka cluster without having to put the advertised hostname to <IP_ADDRESS> in /etc/hosts, which doesn't seems to be an advanced use case to me ;) It will probably surprise some users (here, literally generating more than 100MBps of "hard-to-understand-why" logs), and I think it should be fixed before integrating the iptables-proxy implementation in a release. Of course, I can understand this shouldn't, but I really don't know how many other use-cases it may impact. At least, I give this issue a +1 in the priority vote :)
In any case, of course, I'll test with the current setup (with the workaround for when hairpin is disabled).

I think by "advanced use cases" @thockin is mainly referring to things that
need to be handled beyond basic service-portal support. We intend to reach
full parity and test everything thoroughly before the new proxy is
eventually un-gated, but those can be in later change sets. Right now it's
behind an "unofficial" cli flag that defaults to forcing the userspace
proxy.
It's not quite intended for real use yet, and all of the "advanced use
cases" will be taken care of shortly.
I don't have anything to do with releases of course, but I don't believe it
will be integrated untill everything listed is complete.
Thanks for testing :)
On Aug 12, 2015 4:38 AM, "MikaelCluseau"<EMAIL_ADDRESS>wrote:

@thockin https://github.com/thockin if I may, the hairpin requirement
is for running an Apache Kafka cluster without having to put the advertised
hostname to <IP_ADDRESS> in /etc/hosts, which doesn't seems to be an advanced
use case to me ;) It will probably surprise some users (here, literally
generating more than 100MBps of "hard-to-understand-why" logs), and I think
it should be fixed before integrating the iptables-proxy implementation in
a release. Of course, I can understand this shouldn't, but I really don't
know how many other use-cases it may impact. At least, I give this issue a
+1 in the priority vote :)
In any case, of course, I'll test with the current setup (with the
workaround for when hairpin is disabled).
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/9210#issuecomment-130217669
.

Better safe than sorry ;-)

Don't get me wrong, we know how to solve it but we wanted to get the core
code in first :)
On Wed, Aug 12, 2015 at 1:38 AM, MikaelCluseau<EMAIL_ADDRESS>wrote:

@thockin https://github.com/thockin if I may, the hairpin requirement
is for running an Apache Kafka cluster without having to put the advertised
hostname to <IP_ADDRESS> in /etc/hosts, which doesn't seems to be an advanced
use case to me ;) It will probably surprise some users (here, literally
generating more than 100MBps of "hard-to-understand-why" logs), and I think
it should be fixed before integrating the iptables-proxy implementation in
a release. Of course, I can understand this shouldn't, but I really don't
know how many other use-cases it may impact. At least, I give this issue a
+1 in the priority vote :)
In any case, of course, I'll test with the current setup (with the
workaround for when hairpin is disabled).
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/9210#issuecomment-130217669
.

On 08/13/2015 02:19 AM, Tim Hockin wrote:

Don't get me wrong, we know how to solve it but we wanted to get the core
code in first :)

Totally fine, I just wanted to express my view about this "secondary
list item" should some time constraint arise :)

:fireworks: :smile:

W00t!!

I'm looking at NodePorts.  I want you to focus on the test - I want to show a graph of how this is better :)

even if the test is largely manual, just needs to be reproducible.

Sounds good. :)
On Wed, Aug 12, 2015 at 12:42 PM, Tim Hockin<EMAIL_ADDRESS>wrote:

I'm looking at NodePorts. I want you to focus on the test - I want to show
a graph of how this is better :)
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/9210#issuecomment-130365502
.


LogPython: Failed to init Stable Diffusion Img2Image pipeline. Exception was This modeling file requires the following packages that were not found in your environment: accelerate. Run pip install accelerate

unable to initialize the models for some reason! would love any type of guidance on this front!

Try pulling the latest code from main and then click the Update dependencies button in the plugin window and let it restart the editor when it's complete. Let me know if that solves the issue.

@neonvoid Just wanted to confim if updating the dependencies fixed your issue?


Some Polymer components don't work as expected

Hi, I'm testing Polymer components integration.
I tried exactly the sample in official docs (the paper-drawer-panel) with the only difference that I added responsive-width attribute, and removed every binding (straight html code), as follows:
<link rel="import" href="/bower_components/paper-drawer-panel/paper-drawer-panel.html">
<link rel="import" href="/bower_components/paper-toolbar/paper-toolbar.html">
<link rel="import" href="/bower_components/paper-menu/paper-menu.html">
<link rel="import" href="/bower_components/paper-item/paper-item.html">
<link rel="import" href="/bower_components/paper-scroll-header-panel/paper-scroll-header-panel.html">
<link rel="import" href="/bower_components/paper-icon-button/paper-icon-button.html">
<link rel="import" href="/bower_components/iron-icons/iron-icons.html">

<template>
  <paper-drawer-panel responsive-width="768px">

      <paper-toolbar class="paper-header">
        Menu
      </paper-toolbar>
      <paper-menu>
        <paper-item>Dashboard</paper-item>
        <paper-item>Documents</paper-item>
      </paper-menu>

      <paper-toolbar class="paper-header">
        <paper-icon-button icon="menu" tabindex="1" paper-drawer-toggle></paper-icon-button>

Test App

        <iron-icon icon="autorenew"></iron-icon>
      </paper-toolbar>

  </paper-drawer-panel>
</template>

It works as expected in desktop browsers, but it fails in IOS (9.x, in simulator or real device).
When it loads on iPad portrait (768px wide) the sidebar is not collapsed, but if I try in a desktop browser it works when the windows is under 768px wide.
Plain Polymer implementation works well, as in this plunker: https://embed.plnkr.co/ECsaxV/
Has anyone tested these integration in a complex app?
Failing at this simple scenario makes the whole "Aurelia + Polymer" story unsuitable for production use..

I just ran it in an iOS 9.2 iPad Retina simulator, and the sidebar was collapsed. Which iPad version were you using?

Hi
I found the culprit (maybe)..
I was following the article on Polymer in the official Aurelia docs, and, instead of starting from a skeleton on github, I integrated polymer in my existing app.
Following what is reported on the docs, I created a template with the imports at the top of the file, and in index.html I included only the polymer.html import and the polyfill script. You can see the view template in my original post.
Everything seemed to work correctly, but I had a lot of weird behaviours.
So I started from scratch, and followed the official polymer advice to collect all imports in a single html import (typically an elements.html file) and include it in the index.html . This seems to be the only reliable way to vulcanize the imports, so one should follow this approach anyway...
Doing so fixed everything!
It definitely needs more investigation, because if it's confirmed that including the imports in a view template is not reliable, the official docs should be changed accordingly...
Cheers, P.

Glad you found something! What was the rest of your setup like? There are a couple subtle ordering issues that might also be involved

It was an aurelia app created from scratch (not based on a skeleton), with the latest beta packages installed with JSPM.
In the index.html file I included the webcomponents_lite script and the polymer html import, and I boostrapped Aurelia exactly as depicted in the Polymer doc page.
Using bower, I installed the following:
{
  "name": "web-client-au",
  "private": true,
  "dependencies": {
    "webcomponentsjs": "^0.7.20",
    "polymer": "^1.2.4",
    "paper-elements": "^1.0.7",
    "iron-elements": "^1.0.8"
  }
}

The main and only Aurelia view contained what you see in my first post, and the view model was empty.
Maybe the problem was some missing iron dependencies, used by the paper-drawer-panel when you specify responsive behaviour via responsive-width property...

Huh, I can't seem to find anything different between the two. I think paper-drawer-panel has a responsive-width by default, but there could still be any missing dependencies. When you say the view model was empty, do you mean the file was empty or the class?


Feature Request

I feel this tool could be greatly improved by having different tabs for HSL, RGB and Hex. This would allow for quick switching between definition types as well as allow for the option to use RGB sliders or Hex sliders. It would also be nice to be able to use the keyboard to move the sliders.

See https://github.com/thomaslindstrom/color-picker/issues/3

This will be included in the upcoming v2.0.0.


Fix selection bugs

add additive/subtractive selection when the user holds down a modified key (cmd/ctrl/shift)
fix various bugs related to tap events in cyJsRenderer.
The renderNetwork function was accessing state like networkView from outside the hook.  Those values would not be up to date and the render function would be rendering the wrong values.  Updated the renderNetwork hooks dependencies and changed the hook to call renderNetwork with them.

This fixes bugs such as:

clicking the network after loading would render the wrong visibility values for nodes/edges
clicking the subnetwork edge causes all the edges to disappear

I am missing knowledge/context about the selection feature in general, and in relation to the subnetwork viewer.  Should we also allow multiple selection for the subnetwork viewer?  As of right now the user can only click a single node in the subnetwork view.


Sanitizing hardwares entry when creating a workflow

In this commit a "Note" has been in place with template and workflow which defines what you can provide as a key for hardware indexing

@parauliya Please ensure any PR on this repo has a preview link added to the description.
Adding here for now: changset-preview


Add parallell views and deactivated views

Hey!
I'm using Ionicframework (which uses angular-ui-router) to build an app. Native apps usually have parallell views (see the Spotify app, for example, where Search, Browse, Discover etc are parallell) and each of these parallell views have a history.
For example, if you go into Spotify's Search view, do a search and go on to the Artist view and then back to Search view again the search results are still there. Similarly if you do a search in the Search view then go to the Discover view and back to the Search view again, the search results are still there.
Would be great to be able to use named ui-views to create parallells (suggesting to use | (pipe) prefix):

When a search-view is active, all other ui-views on the same level get's a "display:none" (or any other way of hiding, whatever is fastest).
The state could be defined as:
$stateProvider
  .state( "|search", { url: "/search", templateUrl: "search.html", controller: "searchCtrl" } );
  .state( "|discover", { url: "/discover", templateUrl: "discover.html", controller: "discoverCtrl" } );

And for ancestors (suggesting to use > separator to indicate that parent views are preserved):
.state( "|search>artist", { url: "/search/artist/:id", templateUrl: "artist.html", controller: "artistCtrl" } );
.state( "|search>album", { url: "/search/album/:id", templateUrl: "album.html", controller: "albumCtrl" } );

If you search and select an artist, the artist.html will appended as a child to the "|search" ui-view and the search.html content will be applied with "display:none" to be hidden. When going back, the artist.html will be removed from the DOM and the search.html will be shown again.
Does it sound reasonable? Feedback appreciated! Maybe the state name and the name of the ui-view shouldn't be connected as suggested?

https://gist.github.com/eddiemonge/f6a58169c2846731a1eb


Support callbacks/tool use in JSON generation

Is your feature request related to a problem? Please describe.
When generating a complex schema, sometimes I want to interleave generation, templating, and tool use. One way to support this would be via callbacks attached to a particular JSONpath.
Describe the solution you'd like
class Book(BaseModel):
    title: str
    isbn: str

class Author(BaseModel):
    name: str
    books: List[Book]

def lookup_isbn(generation_state):
  isbn = _lookup_isbn_by_author_and_title(author=generation_state.current_object.parent.name, title=generation_state.current_object.title)
  return isbn

lm += prompt
callbacks = {r"author.books[\d].isbn": lookup_isbn}
lm += guidance.json(schema=schema, callbacks=callbacks)

Describe alternatives you've considered
Another way would be to support temporarily interrupting generation somehow, but I think this reduces down to something very similar to the callback approach. A less clean approach (IMO) would be inheritance-based, pushing the current JSON library methods under a class and allowing the user to inherit from them, e.g. overriding _gen_json_object and hooking in that way.
Additional context
See also discussion on #876 and #889 (some kind of lazy approach is likely necessary here)

It's an interesting use case. It wouldn't work with our server-side things right now, since it involves the server running arbitrary code. I do know that @Harsha-Nori has been thinking about possible ways around this limitation.

This is a very interesting idea.
One issue that comes up here also comes up with a much simpler case that we don't (yet :crossed_fingers: ) support: adding a "pattern" field to string types. If the pattern allows for (or worse yet, demands) strings with unescaped quotes, for example, then the generated json won't be valid. We could always post-process the generation to escape the string (which would have to happen in a "stateful" context), I'd be concerned that we're knocking the LLM off-distribution during the generation of that string. I foresee a similar problem with any callback that is generative/a guidance function.
@riedgar-ms if we can at least generate a stateful grammar, then it's on us to make sure what can run server-side runs server-side and what has to run client-side runs client side.

Interesting point about pattern elements -- I had not considered this in my templating implementation, and I agree I think the in-distribution-ness/perplexity likely suffers. I naively implemented the post-processing approach and would hope that perplexity recovers for tokens generated after the post processed string. Fortunately I think problematic characters are pretty rare for my use case, but definitely not so rare as to be ignorable. Will need to look at it.


Extend JaxSimModelReferences to support applying forces to frames

We already have JaxSimModelReferences.apply_link_forces to apply forces to the links. Now that frame support is finalized (#147), it would be great adding a new JaxSimModelReferences.apply_frame_force method that allows to apply a 6D force to a given frame. In this way, the logic can directly detect the parent link of the frame and automatically express the 6D frame force to the frame of the parent link.

Given a frame force ${} _F F$ expressed in inertial representation, the corresponding force on the parent link will be given by:
{} _L F = {} _L X ^F {} _F F

in which:
\begin{align}
{} _L X ^F &= \text{Adj}{} _L H ^F \\
{} _L H ^F &= ({} ^L H _F) ^\top = ({} ^L H _W {}^W H _F) ^\top
\end{align}

This logic can be parallelized for every considered frame.

I want to extend a bit the math since I'm reviewing #209 and there's something that might be not 100% correct.
Let's consider the following two frames:

A link frame $L$.
A generic frame $F$ rigidly attached to link $L$ and having a fixed pose defined by ${}^L \mathbf{H}_F$.

Let's now consider a 6D force $\mathbf{f}$ applied to the origin of frame $F$. The aim of the desired helper is to express such force to the link frame $L$. Since our internal buffers store velocites and forces in inertial-fixed representation, we need to also convert the frame force to inertial-fixed.
To be as generic as possible, let's consider an input 6D force expressed in frame $Y \in \left[ W, F, F[W] \right]$ that describes inertial-fixed, body-fixed, and mixed representations.
What we need to do is the following:
{}_L \mathbf{f} = {}_L \mathbf{X}^Y {}_Y \mathbf{f}

where ${}_L \mathbf{X}^Y = {}^Y \mathbf{X}_L^\top = {}^L \mathbf{X}_Y^{-\top} $.

Thanks a lot @diegoferigo! I've updated my comment accordingly


Add known issue for client-side encryption

@jaschrep-msft Thanks for the feedback - edits are in place.

@seanmcc-msft If this looks good, could you please approve? thx


LogStream throws EOFException if the stream contains less than 8 bytes

Description
When trying to read the output of a command launched through DockerClient.execStart(), an EOFException is thrown if the output of the command is less than 8 characters using v6.0.0:shaded
How to reproduce
The following unit test demonstrates the problem.
@Test
public void readStdoutOfCommandWithOutputLessThan8Chars() throws DockerCertificateException, DockerException, InterruptedException, IOException {
  DockerClient client = createDockerClient();
  String simpleEcho = client.execCreate("container_id", new String[] { "echo" ,"Hello" }, DockerClient.ExecCreateParam.attachStdout(true), DockerClient.ExecCreateParam.attachStderr(true)).id();

  LogStream logStream = client.execStart(simpleEcho, DockerClient.ExecStartParameter.TTY);
  ByteArrayOutputStream stdOut = new ByteArrayOutputStream();
  ByteArrayOutputStream stdErr = new ByteArrayOutputStream();

  logStream.attach(stdOut, stdErr);
  System.out.println("======== OUTPUT =======");
  System.out.println(stdOut);
}

Produces the following trace:
java.lang.RuntimeException: java.io.EOFException
    at com.google.common.base.Throwables.propagate(Throwables.java:160)
    at com.spotify.docker.client.LogStream.computeNext(LogStream.java:61)
    at com.spotify.docker.client.LogStream.computeNext(LogStream.java:35)
    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
    at com.spotify.docker.client.LogStream.attach(LogStream.java:142)
    at Test.readStdoutOfCommandWithOutputLessThan8Chars(Test.java:61)
    // JUnit noise omitted
Caused by: java.io.EOFException
    at com.spotify.docker.client.LogReader.nextMessage(LogReader.java:59)
    at com.spotify.docker.client.LogStream.computeNext(LogStream.java:59)
    ... 31 more

Whereas the following test
@Test
public void readStdoutOfCommandWithOutputMoreThan8Chars() throws DockerCertificateException, DockerException, InterruptedException, IOException {
  DockerClient client = createDockerClient();
  String simpleEcho = client.execCreate("container_id", new String[] { "echo" ,"HelloWorld" }, DockerClient.ExecCreateParam.attachStdout(true), DockerClient.ExecCreateParam.attachStderr(true)).id();

  LogStream logStream = client.execStart(simpleEcho, DockerClient.ExecStartParameter.TTY);
  ByteArrayOutputStream stdOut = new ByteArrayOutputStream();
  ByteArrayOutputStream stdErr = new ByteArrayOutputStream();

  logStream.attach(stdOut, stdErr);
  System.out.println("======== OUTPUT =======");
  System.out.println(stdOut);
}

Prints out the expected output:
//Misc logs
======== OUTPUT =======
HelloWorld

What do you expect
Command output should be read.
What happened instead
EOFException was thrown
Software:

docker version:  1.12.2
Spotify's docker-client version: v6.0.0:shaded

Also reproduced on v6.1.0:shaded.

thanks for the bug report @bayanr. I was able to reproduce this on master with docker v.1.12.1.

@bayanr can you add details about how the container that you are exec-ing the echo command inside is created? Is the tty setting set to true in the createContainer(..) call?

I think that what might be happening is that, when attaching to a container with a tty, the output from the attach Remote API call the output stream is the raw data from the process PTY.
com.spotify.docker.client.LogReader is always expecting the second type of output from this API call, for non-tty containers:

When the TTY is disabled, then the stream is multiplexed to separate stdout and stderr.
The format is a Header and a Payload (frame).
...

Hi @mattnworb sorry for the late response. This is not a setting I am modifying, so whatever the default is I guess. Here is a unit test which is launching a container with a configuration similar to what I have as well as performing the exec.
@Test
public void readStdoutOfCommandWithOutputLessThan8Chars() throws DockerCertificateException, DockerException, InterruptedException, IOException {
  DockerClient client = createDockerClient();

  ContainerConfig config = ContainerConfig.builder()
    .image("httpd")
    .env("SETTING_1=VALUE_1")
    .build();

  ContainerCreation container = client.createContainer(config);
  client.startContainer(container.id());
  Thread.sleep(2000);
  String simpleEcho = client.execCreate(container.id(), new String[] { "echo" ,"Hello" }, DockerClient.ExecCreateParam.attachStdout(true), DockerClient.ExecCreateParam.attachStderr(true)).id();

  LogStream logStream = client.execStart(simpleEcho, DockerClient.ExecStartParameter.TTY);
  ByteArrayOutputStream stdOut = new ByteArrayOutputStream();
  ByteArrayOutputStream stdErr = new ByteArrayOutputStream();

  logStream.attach(stdOut, stdErr);
  System.out.println("======== OUTPUT =======");
  System.out.println(stdOut);
}

I tried explicitly setting TTY to true and the exception still happens.

Apparently the multiplexed stream never happens. Setting TTY to both true and false yields the same exception which is quite weird.

Can you try just doing either stdout or stderr but not both?

I tried with stdout only, same exception is thrown.

As far as I can tell the stream returned by execStart is multiplexed when you do not set the detach or tty parameters.
This test passes successfully:
  @Test
  public void readStdoutOfCommandWithOutputLessThan8Chars()
      throws DockerCertificateException, DockerException, InterruptedException, IOException {

    DockerClient client = DefaultDockerClient.fromEnv().build();

    final ContainerConfig containerConfig = ContainerConfig.builder()
        .image("spotify/alpine:latest")
        .cmd("sh", "-c", "while true; do date; sleep 1; done")
        .build();

    final ContainerCreation creation = client.createContainer(containerConfig);

    final String containerId = creation.id();
    client.startContainer(containerId);

    log.info("started container {}", containerId);

    assertTrue("Container " + containerId + " is not running",
        client.inspectContainer(containerId).state().running());

    final ExecCreation execCreation = client.execCreate(
        containerId,
        new String[]{"echo", "Hello"},
        ExecCreateParam.attachStdout(),
        ExecCreateParam.attachStderr());
    log.info("exec created: {}", execCreation);

    // ** I removed the DockerClient.ExecStartParameter.TTY param here ** /
    LogStream logStream = client.execStart(execCreation.id());

    ByteArrayOutputStream stdOut = new ByteArrayOutputStream();
    ByteArrayOutputStream stdErr = new ByteArrayOutputStream();

    logStream.attach(stdOut, stdErr);
    assertThat(stdOut.toString(), is("Hello\n"));
  }

It does seem like a bug though that the LogStream instance returned by execStart or attachContainer is unaware that there are some cases where the stream will not be multiplexed. This could definitely be improved.

This is definitely a bug at https://github.com/spotify/docker-client/blob/e385b577a617734e4fd87307822924b855a2d46a/src/main/java/com/spotify/docker/client/LogReader.java#L59
As mentioned before, It seems the logic for determining whether the stream is a TTY-enabled or not depends on reading the first 8 bytes, but if there's fewer, then we're clearly in the TTY-enabled case because the stream definitely doesn't have the correct form (it's a raw stream). Instead of throwing the exception (when n != 8), we should go into the code path that handles raw streams (in the else).
This is probably code that I didn't touch when I added support for both types of streams (spotify/docker-client@824af85) and went undetected (likely because most raw streams that truly contain TTY formatting codes end up being larger than 8 bytes)

thanks to @rgrunber for addressing this in #551


Clarify the difference between https://github.com/terraform-providers/terraform-provider-mailgun

I found now, that https://www.terraform.io/docs/providers/mailgun/ is using the V3 API as well. Does it mean it no longer make sense to use this provider? Would be great to have the difference highlighted somehow.

I've tried migrating to the mentioned provider and it seem they use API v3 now and support region argument for resources, so I was able to even migrate existing resources to it.

Sounds like time to archive this provider then! I left a followup on https://github.com/terraform-providers/terraform-provider-mailgun/issues/17#issuecomment-680441956

Sounds reasonable to me, thanks!


FirebaseCppApp-6_7_0.bundle cannot be opened because the developer cannot be verified.

Please fill in the following fields:
Unity editor version: 2019.2.12f1
Firebase Unity SDK version: Latest
Firebase plugins in use (Auth, Database, etc.): database
Additional SDKs you are using (Facebook, AdMob, etc.):  none
Platform you are using the Unity editor on (Mac, Windows, or Linux):  Mac OSX Catalina 10.15.1
Platform you are targeting (iOS, Android, and/or desktop):  ALL
Scripting Runtime (Mono, and/or IL2CPP):  IL2CPP
Please describe the issue here:
(Please list the full steps to reproduce the issue. Include device logs, Unity logs, and stack traces if available.)

Install firebase SDK
Follow quickstart example
Run project

The issue is that when I push player I get the following popups from the security of Catalina.

FirebaseCppApp-6_7_0.bundle” cannot be opened because the developer cannot be verified. Are you sure you want to open it?
By opening this app, you will be overriding system security which can expose your computer and personal information to malware that may harm your Mac or compromise your privacy.

Please answer the following, if applicable:
Have you been able to reproduce this issue with just the Firebase Unity quickstarts (this GitHub project)?  YES
What's the issue repro rate? (eg 100%, 1/5 etc)
100%

Also have this issue, after upgrading to Catalina!

same here: https://drive.google.com/file/d/1_RRjEvfrAI9fYk10IXofKPr1AT7bSA6u/view?usp=sharing

Just go into security settings on your mac, and you will see firebase blocked because it has an untrusted developer. Allow it there and you are good to go. I solved it that way👍🏻

Yup, it just works!
On Tue, 19 Nov 2019, 13:20 NikolicSlobodan<EMAIL_ADDRESS>wrote:

Just go into security settings on your mac, and you will see firebase
blocked because it has an untrusted developer. Allow it there and you are
good to go. I solved it that way👍🏻
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/firebase/quickstart-unity/issues/515?email_source=notifications&email_token=AA473QTFTVWETLSH6SF5NY3QUPDY5A5CNFSM4JOG6DXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEN27ZA#issuecomment-555462628,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA473QR3C2EALIHYGVGBPGLQUPDY5ANCNFSM4JOG6DXA
.

Thanks for bringing this to our attention, and thanks for adding that way to solve it.  We will look into what is causing this issue.

I think what @SlobodanNikolic said is the right solution.  Unfortunately this is how Catalina works.


Split up Event and Event processing

The Event model is currently used for two purposes: Duplicating the Event object from the Stripe database, as well as tracking webhook processing requests.
I would like to propose splitting up the two into the following:

Event
WebhookRequest

The Event model will be a 1:1 of the Event Stripe object. The WebhookRequest model however will be created whenever a webhook pings us.
That means that if a webhook fails once and is retried by Stripe, we create two WebhookRequest objects.
This also means we may be able to consolidate both the concept of a webhook request and a webhook processing exception: A WebhookRequest can have an internal status, which gets updated as we attempt processing the webhook (and is set to failed, with an exception/traceback if one was caught during processing).
Thoughts?

I forgot to specify the use case: I'm looking into this because I want to reliably enumerate certain event types, such as charge.failed. This may mean syncing previous events which are missing from the db (webhooks won't help us!).
This also greatly improves our ability to debug webhook issues. The code has a TODO to "catch more exceptions" which is very reasonable.

I just implemented this model in dj-paypal: https://github.com/dj-stripe/dj-paypal/commit/f00a6be780d64f1f4bd4286a48ff4e94458e2324

This landed a while ago


fix(ui-markdown-editor): add 0 to element before last in anchor and focus - i134

Issue #134
Changes

fix(toggle block of toolbar helpers getting node and path from node descendants ):add 0 to the element before last in anchor and focus-#134

Flags
Related Issues

https://github.com/accordproject/web-components/issues/134

@hohooio823 just a future reference for AP's commit guide


First commit implementing generated code.

First commit implementing generated code. Duplicate fields were removed and the AddConfigurationSettingAsync, AddConfigurationSetting, DeleteConfigurationSettingAsync, and DeleteConfigurationSetting methods were changed to integrate the generated code.
Contributing to the Azure SDK
Please see our CONTRIBUTING.md if you are not familiar with contributing to this repository or have questions.
For specific information about pull request etiquette and best practices, see this section.

API change check
APIView has identified API level changes in this PR and created following API reviews.
Azure.Data.AppConfiguration


Typo in https://github.com/open-component-model/ocm/tree/main/cmds/helminstaller#configuring-subcharts

What happened:
Text reads "are set ny preceeding"
What you expected to happen:
Text should read "are set by preceeding
How to reproduce it (as minimally and precisely as possible):
Look here https://github.com/open-component-model/ocm/tree/main/cmds/helminstaller#configuring-subcharts
Anything else we need to know:
Environment:

Fixed with PR #374


Show a failed activity log when attaching one queue via SAS

Storage Explorer Version: 1.13.0
Build:  20200504.7
Branch: master
Platform/OS: Windows 10/  Linux Ubuntu 16.04/ macOS Catalina
Architecture: ia32/x64
Regression From:  Previous release(1.13.0)
Steps to reproduce:

Expand one storage account -> Queues.
Select one queue -> Attach it via SAS.
Check the result.

Expect Experience:

Succeed to attach the queue.
Show a successful log message.

Actual Experience:

Show a failed activity log.
The attached queue displays after refreshing manually.

There's already a fix that's been merged for this.

@craxal
Verified this issue on build 20200509.1.

Show a successful activity log when attaching queues.
The attached queue doesn't auto display.

@v-xianya Yes, as I mentioned earlier, there's already a fix that's been merged. The current state of this bug is "Verifying", meaning we still need to check for things on our end. Packages have not yet been updated, so you're not going to see the fix yet.


feature: document jest-snapshot

Publishing jest-snapshot as a standalone module was a great idea, and one that has already proven usable outside of Jest through for instance the chai-jest-snapshot plugin for Chai.
That plugin is unfortunately dependent on Chai, and I'd like to see a standalone version that is not tied that closely to the assertion library. bahmutov/snap-shot was perfect in that regard, but I could not get it to produce the nice diffs and the same snapshot format that Jest uses which is why I landed on the chai-jest-snapshot plugin (which is simply wrapping jest-snapshot).
In that regard the documentation is a bit lacking, in fact non-existent. Would it be possible to create some kind of README documenting the official API? I could of course just try to fiddle with the code in chai-jest-snapshot, but I would not be sure what are public methods and not.
Jest 0.20.3

It's unlikely that we'll be able to write documentation for this module at this point, I'm sorry. However, anything that we export from the code, you are free to use. If we end up changing the API, you'll notice when we release the next major release and we put it into our release notes normally :)

Fair enough :)


Microsoft Surface Pro

Firefox preset for Surface Pro :

Screenshot from Surface Pro 3 :

Looking verey good as well but suppose we can leave this one open for someone with a SP1 or SP2?

:+1: Sounds good to me.


add method to Location to reconstruct URL

The Location object represents a path: https://github.com/rackt/react-router/blob/master/modules/Location.js
It would be nice to have a way to reconstruct the full URL, just a string with the query and hash attached.
I would use this to store the current URL in my app state. This way, I can still snapshot my entire app state, and load it later. Part of the loading process would be to start react-router with the URL from the snapshot.

Actually, looks like I can access the path property of the history object:
React.render(
  Router(
    { history: history,
      onUpdate: function() { console.log(this.history.path); }},
    Route({ path: "/", component: App },
          Route({ path: "item", component: Item }))
  ),
  document.querySelector('#mount')
);

If I go from / to /item?foo=bar, it logs /item?foo=bar which is exactly what I want. If this is guaranteed to work in all environments, this is good.


[v3] how can I manual trigger mathjax render

I wanna trigger mathjax render when my tui-editor change event is triggered

See the documentation, as well as the interactive examples, in particular this one.


Hyperkit segfaults if format=qcow is passed to it without qcow-config

Hyperkit segfaults if a qcow disk is used, but the qcow-config option is not explicitly passed to it.
Here's an example:
$ FLAGS="-A -s 0:0,hostbridge -s 31,lpc -l com1,stdio -m 4G -s 2:0,virtio-net"

This segfaults:
# com.docker.hyperkit $FLAGS -s "4,virtio-blk,file:///tmp/test.qcow2,format=qcow" -f "kexec,vmlinuz,initramfs.img,earlyprintk=serial console=ttyS0"

While this doesn't:
# com.docker.hyperkit $FLAGS -s "4,virtio-blk,file:///tmp/test.qcow2,format=qcow,qcow-config=" -f "kexec,vmlinuz,initramfs.img,earlyprintk=serial console=ttyS0"

I'm not sure if qcow-config is intended to be a mandatory option, but if it is, hyperkit should probably throw an error instead of segfaulting.

I think it was supposed to be an optional thing rather than mandatory, @djs55 is that correct?

It was supposed to be optional-- I'll take a look.
On 9 Dec 2016 9:50 am, "Ian Campbell"<EMAIL_ADDRESS>wrote:

I think it was supposed to be an optional thing rather than mandatory,
@djs55 https://github.com/djs55 is that correct?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/docker/hyperkit/issues/88#issuecomment-265974671, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAMHuntnKSazNBoX4NgEtL83TmrrIqFSks5rGSRVgaJpZM4LIpbt
.


Mongo Data-Sync

There were some errors noticed during puppet runs.

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate declaration: File[/etc/unattended-reboot/no-reboot/govuk_env_sync] is already declared in file /usr/share/puppet/production/current/modules/govuk_env_sync/manifests/task.pp:72; cannot redeclare at /usr/share/puppet/production/current/modules/govuk_env_sync/manifests/task.pp:72 on node postgresql-primary-1.backend.staging.publishing.service.gov.uk
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run```

- This change is intended to correct the duplicate declaration.

Note: Original change https://github.com/alphagov/govuk-puppet/pull/7927

https://trello.com/c/8wzAOsrV/1314-adapt-jenkins-jobs-to-run-mongo-backup-jobs

Solo: @suthagarht

There should be a spec for the new class. That will help pick up things like the variables being  used outside of their scope.


No such file or directory

Hello Guys
I can't build/compile tdlight
os: CentOS 8
first of all i can`t found "./compile-all.sh" and i try to use "compile-natives-package.sh"
but without success you could see my output
# ./compile-natives-package.sh
Generating maven project...
Current directory: /opt/3/tdlight-java-natives-master
Operating system: linux
Architecture: amd64
Td implementation: tdlight
Deleting old data...
Creating missing folders...
Generating pom.xml...
Copying source files...
Done.
Generating td tools...
Current directory: /opt/3/tdlight-java-natives-master
Operating system: linux
Architecture: amd64
Td implementation: tdlight
CPU cores count: -- -j1
CMake extra arguments: ''
Deleting old data...
Copying implementation files...
Patching implementation files...
Configuring CMake...
-- The CXX compiler identification is GNU 8.3.1
-- The C compiler identification is GNU 8.3.1
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Could NOT find ccache (this is NOT an error)
-- Found OpenSSL: /usr/lib64/libcrypto.so (found version "1.1.1g")
-- Found OpenSSL: /usr/include /usr/lib64/libssl.so;/usr/lib64/libcrypto.so
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE
-- Performing Test HAVE_STD14
-- Performing Test HAVE_STD14 - Success
-- Performing Test HAVE_CXX_FLAG_WALL
-- Performing Test HAVE_CXX_FLAG_WALL - Success
-- Performing Test HAVE_CXX_FLAG_WEXTRA
-- Performing Test HAVE_CXX_FLAG_WEXTRA - Success
-- Performing Test HAVE_CXX_FLAG_WIMPLICIT_FALLTHROUGH_2
-- Performing Test HAVE_CXX_FLAG_WIMPLICIT_FALLTHROUGH_2 - Success
-- Performing Test HAVE_CXX_FLAG_WPOINTER_ARITH
-- Performing Test HAVE_CXX_FLAG_WPOINTER_ARITH - Success
-- Performing Test HAVE_CXX_FLAG_WCAST_QUAL
-- Performing Test HAVE_CXX_FLAG_WCAST_QUAL - Success
-- Performing Test HAVE_CXX_FLAG_WSIGN_COMPARE
-- Performing Test HAVE_CXX_FLAG_WSIGN_COMPARE - Success
-- Performing Test HAVE_CXX_FLAG_WDUPLICATED_BRANCHES
-- Performing Test HAVE_CXX_FLAG_WDUPLICATED_BRANCHES - Success
-- Performing Test HAVE_CXX_FLAG_WDUPLICATED_COND
-- Performing Test HAVE_CXX_FLAG_WDUPLICATED_COND - Success
-- Performing Test HAVE_CXX_FLAG_WALLOC_ZERO
-- Performing Test HAVE_CXX_FLAG_WALLOC_ZERO - Success
-- Performing Test HAVE_CXX_FLAG_WLOGICAL_OP
-- Performing Test HAVE_CXX_FLAG_WLOGICAL_OP - Success
-- Performing Test HAVE_CXX_FLAG_WTAUTOLOGICAL_COMPARE
-- Performing Test HAVE_CXX_FLAG_WTAUTOLOGICAL_COMPARE - Success
-- Performing Test HAVE_CXX_FLAG_WVLA
-- Performing Test HAVE_CXX_FLAG_WVLA - Success
-- Performing Test HAVE_CXX_FLAG_WNON_VIRTUAL_DTOR
-- Performing Test HAVE_CXX_FLAG_WNON_VIRTUAL_DTOR - Success
-- Performing Test HAVE_CXX_FLAG_WUNUSED_PARAMETER
-- Performing Test HAVE_CXX_FLAG_WUNUSED_PARAMETER - Success
-- Performing Test HAVE_CXX_FLAG_WCONVERSION
-- Performing Test HAVE_CXX_FLAG_WCONVERSION - Success
-- Performing Test HAVE_CXX_FLAG_WSIGN_CONVERSION
-- Performing Test HAVE_CXX_FLAG_WSIGN_CONVERSION - Success
-- Performing Test HAVE_CXX_FLAG_WCXX14_COMPAT_PEDANTIC
-- Performing Test HAVE_CXX_FLAG_WCXX14_COMPAT_PEDANTIC - Failed
-- Performing Test HAVE_CXX_FLAG_WDEPRECATED
-- Performing Test HAVE_CXX_FLAG_WDEPRECATED - Success
-- Performing Test HAVE_CXX_FLAG_WUNUSED_COMMAND_LINE_ARGUMENT
-- Performing Test HAVE_CXX_FLAG_WUNUSED_COMMAND_LINE_ARGUMENT - Failed
-- Performing Test HAVE_CXX_FLAG_QUNUSED_ARGUMENTS
-- Performing Test HAVE_CXX_FLAG_QUNUSED_ARGUMENTS - Failed
-- Performing Test HAVE_CXX_FLAG_WODR
-- Performing Test HAVE_CXX_FLAG_WODR - Success
-- Performing Test HAVE_CXX_FLAG_FLTO_ODR_TYPE_MERGING
-- Performing Test HAVE_CXX_FLAG_FLTO_ODR_TYPE_MERGING - Success
-- Performing Test HAVE_CXX_FLAG_WMAYBE_UNINITIALIZED
-- Performing Test HAVE_CXX_FLAG_WMAYBE_UNINITIALIZED - Success
-- Found ZLIB: /usr/lib64/libz.so (found version "1.2.11")
-- Found ZLIB: /usr/include /usr/lib64/libz.so
-- Found JNI: /usr/java/jdk-11.0.11/lib/libjawt.so
-- Found JNI: /usr/java/jdk-11.0.11/include;/usr/java/jdk-11.0.11/include/linux;/usr/java/jdk-11.0.11/include /usr/java/jdk-11.0.11/lib/libjawt.so;/usr/java/jdk-11.0.11/lib/server/libjvm.so
-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY
-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY - Success
-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY
-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY - Success
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR - Success
-- Could NOT find Readline (missing: READLINE_INCLUDE_DIR READLINE_LIBRARY)
-- Could NOT find Readline (this is NOT an error)
-- Configuring done
-- Generating done
-- Build files have been written to: /opt/3/tdlight-java-natives-master/generated/td_tools
Generating cross compilation tools...


Change values in settings.py to use a seperate/dynamic/secure source

Example:
When talking (in the readme) about adding a real database back end, the current content suggests embedding the credentials in the settings file - this should never happen.  Instead, the settings file should pull those values from either a secrets/config object or the environment within the container.

This may have been addressed while resolving another issue.


Sees Temp of Room and sends commands with no action to mini split

Describe The Bug:

Reeds Temp of the Room and HomeBridge show Power on command and set cool but the Unit does not respond at all.
To Reproduce:

Homebridge Accessories or homekit
Click Room
Room Tstat
Pick cool or heat
chose temp set point
Expected behavior:

Mini split unit from off mode should
turn on
set cool/Heat

set temp set point
Logs:
Show the Homebridge logs here, remove any sensitive information.

[4/20/2023, 9:33:31 PM] [homebridge-cielo] Sending power on
[4/20/2023, 9:33:41 PM] [homebridge-cielo] Setting mode to cool
[4/20/2023, 9:38:44 PM] [Homebridge UI] Changes to config.json saved.
[4/20/2023, 9:38:48 PM] [homebridge-cielo] Restarting child bridge...
[4/20/2023, 9:38:48 PM] Got SIGTERM, shutting down child bridge process...
[4/20/2023, 9:38:53 PM] [homebridge-cielo] Child bridge process ended
[4/20/2023, 9:38:53 PM] [homebridge-cielo] Process Ended. Code: 143, Signal: null
[4/20/2023, 9:39:00 PM] [homebridge-cielo] Restarting Process...
[4/20/2023, 9:39:01 PM] [homebridge-cielo] Launched child bridge with PID 1274
[4/20/2023, 9:39:01 PM] Registering platform 'homebridge-cielo.cielo'
[4/20/2023, 9:39:01 PM] [homebridge-cielo] Loaded homebridge-cielo v1.1.2 child bridge successfully
[4/20/2023, 9:39:01 PM] Loaded 1 cached accessories from cachedAccessories.0E9E6427F249.
[4/20/2023, 9:39:01 PM] [homebridge-cielo] Loading accessory from cache: Game Room
[4/20/2023, 9:39:01 PM] Homebridge v1.6.0 (HAP v0.11.0) (homebridge-cielo) is running on port 47595.
[4/20/2023, 9:39:03 PM] [homebridge-cielo] Restoring existing accessory from cache: Game Room
[4/20/2023, 9:40:59 PM] [homebridge-cielo] Sending power on
[4/20/2023, 9:41:09 PM] [homebridge-cielo] Setting mode to cool
[4/20/2023, 9:52:01 PM] [homebridge-cielo] Communication Error: Error: Connection Closed.
at WebSocket. (/homebridge/node_modules/homebridge-cielo/node_modules/node-smartcielo-ws/Cielo.js:158:27)
at WebSocket.emit (node:events:513:28)
at WebSocket.emitClose (/homebridge/node_modules/homebridge-cielo/node_modules/ws/lib/websocket.js:246:10)
at TLSSocket.socketOnClose (/homebridge/node_modules/homebridge-cielo/node_modules/ws/lib/websocket.js:1148:15)
at TLSSocket.emit (node:events:525:35)
at node:net:320:12
at TCP.done (node:_tls_wrap:588:7)
[4/20/2023, 9:52:01 PM] [homebridge-cielo] Reconnecting in 30 seconds...
[4/20/2023, 9:52:33 PM] [homebridge-cielo] Restoring existing accessory from cache: Game RoomPlugin Config:
Show your Homebridge config.json here, remove any sensitive information.

{
"username": "my login",
"password": "mypassword",
"ip": "my home pub static ip",
"macAddresses": [
"E8DB8486780A"
],
"_bridge": {
"username": "0E:9E:64:27:F2:49",
"port": 47595
},
"platform": "cielo"
}
Screenshots:

Environment:
Local Network No Vlans

Plugin Version: homebridge-cielo v1.1.2
Homebridge Version:  [v1.6.0]
Node.js Version:
NPM Version:
Operating System:  Docker

this is what I get if i manually turn it on from the cielowigle home app this feed back from the unit.
[4/20/2023, 10:41:10 PM] [homebridge-cielo] Sending power on
[4/20/2023, 10:41:20 PM] [homebridge-cielo] Setting mode to cool
[4/20/2023, 10:41:20 PM] [homebridge-cielo] Sending power on
[4/20/2023, 10:41:29 PM] [homebridge-cielo] Setting temperature to 69 °F / 20.6 °C
[4/20/2023, 10:41:30 PM] [homebridge-cielo] Setting mode to cool
[4/20/2023, 10:41:32 PM] [homebridge-cielo] Setting temperature to 70 °F / 21.1 °C
[4/20/2023, 10:41:38 PM] [homebridge-cielo] Sending power on
[4/20/2023, 10:41:42 PM] [homebridge-cielo] Setting temperature to 73 °F / 22.8 °C
[4/20/2023, 10:41:44 PM] [homebridge-cielo] Setting temperature to 71 °F / 21.7 °C
[4/20/2023, 10:41:48 PM] [homebridge-cielo] Setting mode to heat
[4/20/2023, 10:47:50 PM] [homebridge-cielo] This plugin generated a warning from the characteristic 'Current Heating Cooling State': characteristic was supplied illegal value: number 3 exceeded maximum of 2. See https://homebridge.io/w/JtMGR for more info.
[4/20/2023, 10:48:11 PM] [homebridge-cielo] Setting temperature to 71 °F / 21.7 °C
[4/20/2023, 10:48:56 PM] [homebridge-cielo] This plugin generated a warning from the characteristic 'Current Heating Cooling State': characteristic was supplied illegal value: number 3 exceeded maximum of 2. See https://homebridge.io/w/JtMGR for more info.
[4/20/2023, 10:48:58 PM] [homebridge-cielo] This plugin generated a warning from the characteristic 'Current Heating Cooling State': characteristic was supplied illegal value: number 3 exceeded maximum of 2. See https://homebridge.io/w/JtMGR for more info.