Use etcd to allocate PortalIPs instead of in-mem

Addresses #473
Basic design:

The service registry asks an allocator for an IP (same as today)
That allocator is backed by an in-memory state (represented as a bit array of allocated IPs in a range), which is in turn backed by etcd
On an allocation request, the local state is reserved, then we try an atomic CAS on etcd to set the allocation
If the CAS fails, we reload the state from etcd and try the allocation again
If there is no state in Etcd all service calls fail with an error "Allocation is disabled"
A controller loop is responsible for booting the state in etcd on creation, and then periodically recalculating the allocation map based on what is actually calculated from services.

Each allocator in the master has a lock and we use optimistic concurrency to guard changes to the global state (each master can only make progress by performing the CAS) so each master will contend during high service creation load.
The repair controller logs any errors - it is the responsibility of the administrator to see those issues and correct them via delete/recreate of a service.
A subsequent optimization will be to move non-requested portalIP allocation to a controller loop (distinct from the Repair controller).
The new allocator should be IPv6 safe.

@thockin if you want to see WIP.

Taking off WIP, this is ready for review.

@a-robinson said he could take a look at this today.

I'll need to run e2e before this merges (after all review is taken care of) but I've had green runs.

First two commits are in separate pulls, only the last two are part of this change.

Why did you choose to not do async IP allocation in a controller instead? That would reduce it to the master election case.

Since we support explicit PortalIP allocation via direct set of spec.portalIP, we needed to have the direct case working (acquire cluster state, allocate, CAS).  We can in a subsequent follow up easily split so that services that don't request a portalIP bypass the inline allocation, and then we can spawn a second controller in controller-manager that can do high performance allocation (single writer, low contention because direct service IP allocation is rare).
----- Original Message -----

Why did you choose to not do async IP allocation in a controller instead?
That would reduce it to the master election case.

Reply to this email directly or view it on GitHub:
https://github.com/GoogleCloudPlatform/kubernetes/pull/7704#issuecomment-98845767

I'd also prefer to see this as/in a controller, but taking a look now.

It's a nice next step, but not strictly required to improve the state we have now.  That other change is more disruptive to clients.
----- Original Message -----

I'd also prefer to see this as/in a controller, but taking a look now.

Reply to this email directly or view it on GitHub:
https://github.com/GoogleCloudPlatform/kubernetes/pull/7704#issuecomment-98855388

I understand that if we made PortalIP allocation async. that it would create another instance of cluster-wide validation (#2457), similar to ReplicationController overlap detection or user-specified node assignment, but we'll need to address that eventually, anyway. And, as you pointed out, manual PortalIP assignment is rare.

On May 4, 2015, at 7:16 PM, Brian Grant<EMAIL_ADDRESS>wrote:
I understand that if we made PortalIP allocation async. that it would create another instance of cluster-wide validation (#2457), similar to ReplicationController overlap detection or user-specified node assignment, but we'll need to address that eventually, anyway. And, as you pointed out, manual PortalIP assignment is rare.

Right - that's extra work that would build on everything here (no extra primitives).  I just don't want to introduce that in the same pull and destabilize everything.
The subsequent pull would be a controller loop that iterates over services and calls into an instance of the etcd allocator.  And that's it.

—
Reply to this email directly or view it on GitHub.

We could relax portalIP in general and make it a request, with a status.portalIP filled by the allocator.  That's a more dramatic change, but does lose the ability to rely on the portalIP not changing.
I'm mostly looking to get to HA master in the smallest number of steps.

On May 4, 2015, at 7:16 PM, Brian Grant<EMAIL_ADDRESS>wrote:
I understand that if we made PortalIP allocation async. that it would create another instance of cluster-wide validation (#2457), similar to ReplicationController overlap detection or user-specified node assignment, but we'll need to address that eventually, anyway. And, as you pointed out, manual PortalIP assignment is rare.
—
Reply to this email directly or view it on GitHub.

The etcd prefix logic seems fine, but IMO this design is not where we should be headed long term. I think the component handling IP assignment actually ought to be a separate process from apiserver. I think it should run like scheduler, filling in PortalIPs for new services. It should be level-triggered, and self-healing such that an admin doesn't need to intervene if some bad data does get into the system. (If two services end up with the same portalIP it should modify one of them.)
Yes, this would change the API when you make a new service, but I'd argue that we should do that sooner rather than later, when it will be harder.
The Repair thingy bothers me a little bit because only one apiserver should run it, but we don't want master election for apiserver.
This does look like an improvement over the current state of affairs, so I won't block it. (although I hope that #7273 merges first! ;) )

On May 4, 2015, at 7:46 PM, Daniel Smith<EMAIL_ADDRESS>wrote:
The etcd prefix logic seems fine, but IMO this design is not where we should be headed long term. I think the component handling IP assignment actually ought to be a separate process from apiserver. I think it should run like scheduler, filling in PortalIPs for new services. It should be level-triggered, and self-healing such that an admin doesn't need to intervene if some bad data does get into the system. (If two services end up with the same portalIP it should modify one of them.)
Yes, this would change the API when you make a new service, but I'd argue that we should do that sooner rather than later, when it will be harder.

It's a big change.  If we want to do it now, we probably need to weigh whether we would hold 1.0 for it.  I'm a bit antsy because we're not going to rebase after the next few weeks, so if we break this after that we'll ship a version of services to folks that will be incompatible with upstream Kube for a while.
On the other hand, if we break this now it probably breaks a fair number of client tools and apps.  You're right, that's not going to get easier.

The Repair thingy bothers me a little bit because only one apiserver should run it, but we don't want master election for apiserver.

It's level based in that it can be run concurrently.  It makes no decisions, merely reports conditions an admin must fix.

This does look like an improvement over the current state of affairs, so I won't block it. (although I hope that #7273 merges first! ;) )
—
Reply to this email directly or view it on GitHub.

On May 4, 2015, at 7:46 PM, Daniel Smith<EMAIL_ADDRESS>wrote:
The etcd prefix logic seems fine, but IMO this design is not where we should be headed long term. I think the component handling IP assignment actually ought to be a separate process from apiserver. I think it should run like scheduler, filling in PortalIPs for new services. It should be level-triggered, and self-healing such that an admin doesn't need to intervene if some bad data does get into the system. (If two services end up with the same portalIP it should modify one of them.)

One risk here is that this change breaks the premise of portal IP a bit - you can't hardcode it, because it could always get broken.  Maybe we can get away with it only changing in extremely rare cases... But again, a big change in our mental model.
Also, because we overlap user request and system assigned at spec.portaIIP we can't distinguish which service to best change, and we can't reject user input.  Which makes setting spec.portalIP as a user pretty worthless (so it should not be encouraged).

Yes, this would change the API when you make a new service, but I'd argue that we should do that sooner rather than later, when it will be harder.
The Repair thingy bothers me a little bit because only one apiserver should run it, but we don't want master election for apiserver.
This does look like an improvement over the current state of affairs, so I won't block it. (although I hope that #7273 merges first! ;) )
—
Reply to this email directly or view it on GitHub.

On Mon, May 4, 2015 at 5:01 PM, Clayton Coleman<EMAIL_ADDRESS>wrote:

On May 4, 2015, at 7:46 PM, Daniel Smith<EMAIL_ADDRESS>wrote:
The etcd prefix logic seems fine, but IMO this design is not where we
should be headed long term. I think the component handling IP assignment
actually ought to be a separate process from apiserver. I think it should
run like scheduler, filling in PortalIPs for new services. It should be
level-triggered, and self-healing such that an admin doesn't need to
intervene if some bad data does get into the system. (If two services end
up with the same portalIP it should modify one of them.)

One risk here is that this change breaks the premise of portal IP a bit -
you can't hardcode it, because it could always get broken. Maybe we can get
away with it only changing in extremely rare cases... But again, a big
change in our mental model.
Also, because we overlap user request and system assigned at spec.portaIIP
we can't distinguish which service to best change, and we can't reject user
input. Which makes setting spec.portalIP as a user pretty worthless (so it
should not be encouraged).

Yeah I think that this is a good example of why we should never have fields
that are sometimes set by the system and sometimes set by humans. I think
you earlier mentioned having the system set an IP field in status; I could
get behind that. Then it would be easy for the ip-controller to figure out
which of two services it would be best to change the IP for.

On Mon, May 4, 2015 at 4:57 PM, Clayton Coleman<EMAIL_ADDRESS>wrote:

On May 4, 2015, at 7:46 PM, Daniel Smith<EMAIL_ADDRESS>wrote:
The etcd prefix logic seems fine, but IMO this design is not where we
should be headed long term. I think the component handling IP assignment
actually ought to be a separate process from apiserver. I think it should
run like scheduler, filling in PortalIPs for new services. It should be
level-triggered, and self-healing such that an admin doesn't need to
intervene if some bad data does get into the system. (If two services end
up with the same portalIP it should modify one of them.)
Yes, this would change the API when you make a new service, but I'd
argue that we should do that sooner rather than later, when it will be
harder.

It's a big change. If we want to do it now, we probably need to weigh
whether we would hold 1.0 for it. I'm a bit antsy because we're not going
to rebase after the next few weeks, so if we break this after that we'll
ship a version of services to folks that will be incompatible with upstream
Kube for a while.

Yeah, I don't want to hold 1.0 for it.

—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/7704#issuecomment-98887064
.

I assume ServiceIPAllocation needsto be an API object in order for us to store it in etcd? It feels pretty weird to have to register something in our public API definitions (all of v1beta1/2/3) when it's purely private state.

Technically it's part of our storage model, but yes, etcd_helper depends heavily on the conversions to work.  I also realized that we don't support resourceversion on listmeta, which is why I left this as objectmeta.  I think the issue on refactoring runtime to separate our the different concerns covers that problem, but I would have preferred to have less overhead related to serializing this object.  Creating generic frameworks is a rathole, as usual.

On May 4, 2015, at 8:26 PM, Alex Robinson<EMAIL_ADDRESS>wrote:
I assume ServiceIPAllocation needsto be an API object in order for us to store it in etcd? It feels pretty weird to have to register something in our public API definitions (all of v1beta1/2/3) when it's purely private state.
—
Reply to this email directly or view it on GitHub.

Ok, so to summarize:
Three big decisions need to be made:

Do we continue to have strong guarantees on the immutable uniqueness of portal IPs, or do we relax them?
Do we continue to allow users to strong request a portal IP
Do we need to introduce "eventual portaIIP" behavior before 1.0 to prevent people from assuming that a spec.portalIP is set?

If we don't relax 2 we still need the inline allocator, otherwise users really get no guarantee.  If we relax 1 that means we depend on DNS more and service IP less, which means it's more important for clients to know what DNS names are available for a particular service.
1/2 are fundamental to our service objects, 3 is smaller.

I have not read this PR (on phone) but here's my take as de facto owner of
this "area".
Having a sync portal IP fill-in is a net negative on the UX unless we
change the semantic of that field.
I assert: If a user specified that field, we MUST NOT give them anything
but that IP. It succeeds as-is or fails. Today we fail the POST. That is
nice and simple. The only behavior I think is acceptable is to mark a
Service as Pending or Failed, with an event or something saying that the
portal IP is busy. I still think this is a net worse UX. Especially when
you repeat this pattern for service host ports (in progress).
Now, the advent of " internal LB" (imagine using an haproxy pool instead of
kube-proxy) might change things, but I would just as soon cross that bridge
when we come to it.
I will read the PR later today.
On May 4, 2015 6:58 PM, "Clayton Coleman"<EMAIL_ADDRESS>wrote:

Ok, so to summarize:
Three big decisions need to be made:

Do we continue to have strong guarantees on the immutable
uniqueness of portal IPs, or do we relax them?
Do we continue to allow users to strong request a portal IP
Do we need to introduce "eventual portaIIP" behavior before 1.0 to
prevent people from assuming that a spec.portalIP is set?

If we don't relax 2 we still need the inline allocator, otherwise users
really get no guarantee. If we relax 1 that means we depend on DNS more and
service IP less, which means it's more important for clients to know what
DNS names are available for a particular service.
1/2 are fundamental to our service objects, 3 is smaller.
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/7704#issuecomment-98903974
.

The spec is the desired state, whether chosen by the user or by a system component or extension.
We shouldn't change spec non-empty fields. Policy is documented here:
https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/api-conventions.md#late-initialization
I agree with @thockin-cc that services with portal IPs that were already otherwise taken would need to be in some failed state. If this could only happen to services with user-selected IPs, I'm not overly concerned about async failure being more painful than synchronous failure.
Are users likely to choose service host ports in the same range as auto-allocated ports?

On May 4, 2015, at 11:23 PM, Brian Grant<EMAIL_ADDRESS>wrote:
The spec is the desired state, whether chosen by the user or by a system component or extension.
We shouldn't change spec non-empty fields. Policy is documented here:
https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/api-conventions.md#late-initialization
I agree with @thockin-cc that services with portal IPs that were already otherwise taken would need to be in some failed state. If this could only happen to services with user-selected IPs, I'm not overly concerned about async failure being more painful than synchronous failure.

What's the use case that underlies async user selected portalIPs?
For sync it's "I have a bunch of other hard coded config that means if the service doesn't exist it's pointless to continue", ie treat portalIP as immutable and special.
For async it's "pretty please let me have a vanity IP, but I'll still have to code around not getting it"?  Ie portalIP shouldn't be known up front?
I think I agree with Tim that the value of the sync requested IP is certainty (barring an awesome use case for async).  If we can't give certainty, let's just nuke the feature.

Are users likely to choose service host ports in the same range as auto-allocated ports?

  We should either let them hardcode it or force them not to hardcode it, but not both.

—
Reply to this email directly or view it on GitHub.

On Mon, May 4, 2015 at 8:23 PM, Brian Grant<EMAIL_ADDRESS>wrote:

The spec is the desired state, whether chosen by the user or by a system component or extension.
We shouldn't change spec non-empty fields. Policy is documented here:
https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/api-conventions.md#late-initialization
I agree with @thockin-cc that services with portal IPs that were already otherwise taken would need to be in some failed state. If this could only happen to services with user-selected IPs, I'm not overly concerned about async failure being more painful than synchronous failure.

Think of it this way.  Starting a fixed IP service (for example, we
have to do this for DNS) changes from:

POST Service
Receive response

to

POST Service
Receive success response
GET Service
If status != active and status != failed then goto 3

Puke.  What is the principal deciding what fields should be handled
async?  I'm not seeing it.

Are users likely to choose service host ports in the same range as auto-allocated ports?

The proposal is a single service host port range.  If a user asks for
a port outside this range, fail.  If the user asks for a port inside
this range, use the same logic as Portal IP.  Admittedly this is going
to be rare, but it follows that it should be possible.

POST Service
Receive success response
GET Service
If status != active and status != failed then goto 3

If you really want to optimize the UX of this, I can imagine the POST handler doing steps 3 & 4... This is gross but it should be done by tools, not humans, so I'm not sure we should optimize for it. Additionally, that's already the pattern you have to follow to get a pod into the system.

Puke.  What is the principal deciding what fields should be handled async?  I'm not seeing it.

Cluster-wide constraints are checked/enforced by some specialized component. We should do the same thing we do for scheduling pods, whatever that may be. It sounds like a better error handling path would make this pill easier to swallow? Arguably that could be used for pod scheduling, too...

What drives the async approach, such as we do with scheduling? Latency, pluggability, and the need to analyze cluster-wide state (at least theoretically). All of those issues apply to scheduling. Do they apply here? Maybe not, though I could envision delegating IP allocation to a separate network control plane.
Another motivation for async would be to make creation work the same way (or at least look the same) as post-creation self-healing or auto-tuning. Both pods and RCs have this property.
What else might require synchronous allocation/mutation? Maybe service host ports. Probably master election. So at least portal IPs wouldn't be the only case.
I'm not really opposed to this approach, esp. since the work has been done, but it does move away from the goal of making apiserver a dumb, CRUDy server, and would work differently than all other current allocation/scheduling/reconciliation functionality, differently than all other objects, and differently than services with external load balancers.
As for the UX of async instantiation, we have to solve that general problem for our other objects, anyway.

----- Original Message -----

What drives the async approach, such as we do with scheduling? Latency,
pluggability, and the need to analyze cluster-wide state (at least
theoretically). All of those issues apply to scheduling. Do they apply here?
Maybe not, though I could envision delegating IP allocation to a separate
network control plane.

I think that falls into the "external IP" camp.  Service PortalIP I have been thinking of and distinct and separate - it's an "internal IP" that is unique per cluster.    The original goal was to have the internal stable IP - the ability to pick it, and the ability for it to be allocated up front, both contribute to the usability of the cluster when viewed through that lens.  If you relax those guarantees, then it becomes much more of a layer on top.  Do we want people to have and use a stable internal IP?  Or was it a mistake to give it a central place in the scheme (and pretend that it's stable and a pet)?

Another motivation for async would be to make creation work the same way (or
at least look the same) as post-creation self-healing or auto-tuning. Both
pods and RCs have this property.
What else might require synchronous allocation/mutation? Maybe service host
ports. Probably master election. So at least portal IPs wouldn't be the only
case.
I'm not really opposed to this approach, esp. since the work has been done,
but it does move away from the goal of making apiserver a dumb, CRUDy
server, and would work differently than all other current
allocation/scheduling/reconciliation functionality, differently than all
other objects, and differently than services with external load balancers.
As for the UX of async instantiation, we have to solve that general problem
for our other objects, anyway.

Reply to this email directly or view it on GitHub:
https://github.com/GoogleCloudPlatform/kubernetes/pull/7704#issuecomment-98970441

I view services (and pods) as core, not "layered on top". These require infrastructure orchestration and/or node agents, not just cluster-level control-plane components interacting just with the apiserver.
That said, the networking implementation needs to be pluggable.
Anyway, I'm fine with this approach. It's an implementation detail we can change later, if necessary.

Thanks for making the changes - the non-etcd parts look reasonable enough to me, I'll wait on @lavalamp to LGTM those and give @thockin a chance to take a look.

The big difference I see between portal IP allocation and ~everything else
async (load balancers, scheduling, replication, nodes, ...) is that those
things have natural control loops that fix reality when it doesn't match
intent.  Portal IPs pretty much have no manifestation in reality - there's
nothing to fix and no control loop.
On Tue, May 5, 2015 at 9:50 AM, Brian Grant<EMAIL_ADDRESS>wrote:

I view services (and pods) as core, not "layered on top". These require
infrastructure orchestration and/or node agents, not just cluster-level
control-plane components interacting just with the apiserver.
That said, the networking implementation needs to be pluggable.
Anyway, I'm fine with this approach. It's an implementation detail we can
change later, if necessary.
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/7704#issuecomment-99138379
.

This is quite the change...
My biggest question is whether we can do this without the API being involved?  The apiserver replicas all have a handle to an (abstract) etcd.  why not carve out some space there and use it directly, rather than push it though the API?

We'd have to implement all the resource versioning logic ourselves (atomic update).  Since that's known and proven I'd like to continue to use it.  It's fairly easy later to drop down to a lower level of abstraction if we implement the necessary primitives.  I just didn't want to create new code that has new bugs - instead using proven known bugs.
----- Original Message -----

This is quite the change...
My biggest question is whether we can do this without the API being involved?
The apiserver replicas all have a handle to an (abstract) etcd.  why not
carve out some space there and use it directly, rather than push it though
the API?

Reply to this email directly or view it on GitHub:
https://github.com/GoogleCloudPlatform/kubernetes/pull/7704#issuecomment-99344143

No new API is actually exposed by this change, correct? It's just using the API machinery to create versioned objects in etcd, which I think is reasonable, though it would be nice for that functionality to be more cleanly factored eventually.
It would be useful to write up a brief description of the design is some relevant design doc (services.md?).

Write up added, all comments addressed.  I generalized the object in the api machinery to be a "RangeAllocation" so we can use it for ports or UIDs or SELinux labels in the future.

No new APIs, correct.

All green.... :)

LGTM. Last call for comments from @lavalamp or @thockin before I merge midday tomorrow.

There are a few unaddressed comments still (mostly requests for comments).
I will finish this re-review in the AM
On Wed, May 6, 2015 at 11:12 PM, Alex Robinson<EMAIL_ADDRESS>wrote:

LGTM. Last call for comments from @lavalamp https://github.com/lavalamp
or @thockin https://github.com/thockin before I merge midday tomorrow.
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/7704#issuecomment-99734219
.

I humbly submit my comments for your judgement
----- Original Message -----

There are a few unaddressed comments still (mostly requests for comments).
I will finish this re-review in the AM
On Wed, May 6, 2015 at 11:12 PM, Alex Robinson<EMAIL_ADDRESS>wrote:

LGTM. Last call for comments from @lavalamp https://github.com/lavalamp
or @thockin https://github.com/thockin before I merge midday tomorrow.
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/7704#issuecomment-99734219
.

Reply to this email directly or view it on GitHub:
https://github.com/GoogleCloudPlatform/kubernetes/pull/7704#issuecomment-99757867

I'm definitely not going to have time to get through this so I'll defer to Tim.

I think those are the last questions.

Comments addressed.  The range behavior is now that the first and last address is the CIDR are omitted - given that a CIDR is a linear sequence in the bit space with all zeros or all 1s at the edges (matching the bit mask) I'm fairly certain that is accurate, but if not I blame my lack of formal network training.

LGTM - just a couple of minor points on the last commit.

Now with a diagram, if the diagram is wrong or not clear let me know so we can encode it.
----- Original Message -----

LGTM - just a couple of minor points on the last commit.

Reply to this email directly or view it on GitHub:
https://github.com/GoogleCloudPlatform/kubernetes/pull/7704#issuecomment-100101757

LGTM, One doc nit.

LGTM - squash and let's roll

Waiting for green.
----- Original Message -----

LGTM - squash and let's roll

Reply to this email directly or view it on GitHub:
https://github.com/GoogleCloudPlatform/kubernetes/pull/7704#issuecomment-100305114

Shippable is failing with a python out of range error on a hook.  can you manually run?  I know this hook just changed to python this week.
./hack/verify-boilerplate.sh
Traceback (most recent call last):
File "./hack/../hooks/boilerplate.py", line 87, in
sys.exit(main())
File "./hack/../hooks/boilerplate.py", line 83, in main
if not file_passes(filename, extention, ref, p):
File "./hack/../hooks/boilerplate.py", line 38, in file_passes
while data[0] != "/*\n":
IndexError: list index out of range

Fails locally, but same way.
----- Original Message -----

Shippable is failing with a python out of range error on a hook.  can you
manually run?  I know this hook just changed to python this week.
./hack/verify-boilerplate.sh
Traceback (most recent call last):
File "./hack/../hooks/boilerplate.py", line 87, in
sys.exit(main())
File "./hack/../hooks/boilerplate.py", line 83, in main
if not file_passes(filename, extention, ref, p):
File "./hack/../hooks/boilerplate.py", line 38, in file_passes
while data[0] != "/*\n":
IndexError: list index out of range

Reply to this email directly or view it on GitHub:
https://github.com/GoogleCloudPlatform/kubernetes/pull/7704#issuecomment-100336211

Pester eparis?   He wrote it

Went green, can merge on monday

Just looked at this and was wondering if you were going to do uids this
way. This clarifies that piece of the puzzle for me significantly.
On Fri, May 8, 2015 at 11:41 PM Clayton Coleman<EMAIL_ADDRESS>wrote:

Went green, can merge on monday
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/7704#issuecomment-100420808
.

Yes

On May 9, 2015, at 12:31 AM, Paul Morie<EMAIL_ADDRESS>wrote:
Just looked at this and was wondering if you were going to do uids this
way. This clarifies that piece of the puzzle for me significantly.
On Fri, May 8, 2015 at 11:41 PM Clayton Coleman<EMAIL_ADDRESS>wrote:

Went green, can merge on monday
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/7704#issuecomment-100420808
.

—
Reply to this email directly or view it on GitHub.

Green retests.  Merging to beat the rush today.

Added a test flake, will debug in https://github.com/GoogleCloudPlatform/kubernetes/issues/8060

Our continuous integration end to end tests have been failing consistently since this merge (details below).  I'm going to confirm that it's this merge, and if so, roll back.
Identified problems
Networking should function for intra-pod communication
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/networking.go:263
Timed out. Cleaning up. Details:
{
"Hostname": "nettest-0",
"Sent": null,
"Received": null,
"Errors": null,
"Log": [
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again."
],
"StillContactingPeers": true
}
Monitoring verify monitoring pods and all cluster nodes are available on influxdb using heapster.
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/monitoring.go:51
monitoring using heapster and influxdb test failed
Services should provide RW and RO services
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/service.go:193
unexpected error listing services using ro service: an error on the server has prevented the request from succeeding

If it is my change, I wonder if it's something different in the config options i missed for gce.  The core controllers have to start for endpoints to be read, it's possible I broke one of the paths.

On May 11, 2015, at 5:49 PM, Quinton Hoole<EMAIL_ADDRESS>wrote:
Our continuous integration end to end tests have been failing consistently since this merge (details below). I'm going to confirm that it's this merge, and if so, roll back.
Identified problems
Networking should function for intra-pod communication
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/networking.go:263
Timed out. Cleaning up. Details:
{
"Hostname": "nettest-0",
"Sent": null,
"Received": null,
"Errors": null,
"Log": [
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again."
],
"StillContactingPeers": true
}
Monitoring verify monitoring pods and all cluster nodes are available on influxdb using heapster.
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/monitoring.go:51
monitoring using heapster and influxdb test failed
Services should provide RW and RO services
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/service.go:193
unexpected error listing services using ro service: an error on the server has prevented the request from succeeding
—
Reply to this email directly or view it on GitHub.

Those error messages indicate that the kubernetes-ro service is not working.

The symptoms there look like the core controller not running.  Possible in one of my final review changes I dropped part of the endpoint sync loop.

On May 11, 2015, at 5:49 PM, Quinton Hoole<EMAIL_ADDRESS>wrote:
Our continuous integration end to end tests have been failing consistently since this merge (details below). I'm going to confirm that it's this merge, and if so, roll back.
Identified problems
Networking should function for intra-pod communication
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/networking.go:263
Timed out. Cleaning up. Details:
{
"Hostname": "nettest-0",
"Sent": null,
"Received": null,
"Errors": null,
"Log": [
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again.",
"Unable to read the endpoints for e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6/nettest: Get http://<IP_ADDRESS>:80/api/v1beta1/endpoints/nettest?namespace=e2e-tests-nettest-5b39782d-6afa-437c-83bd-8aacc59e7bb6: read tcp <IP_ADDRESS>:80: connection reset by peer; will try again."
],
"StillContactingPeers": true
}
Monitoring verify monitoring pods and all cluster nodes are available on influxdb using heapster.
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/monitoring.go:51
monitoring using heapster and influxdb test failed
Services should provide RW and RO services
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/service.go:193
unexpected error listing services using ro service: an error on the server has prevented the request from succeeding
—
Reply to this email directly or view it on GitHub.

Which is likely the core controllers loop.  If there are no other candidates go ahead and revert my change while i track it down.

On May 11, 2015, at 5:58 PM, Daniel Smith<EMAIL_ADDRESS>wrote:
Those error messages indicate that the kubernetes-ro service is not working.
—
Reply to this email directly or view it on GitHub.

Local up cluster is definitely triggering the core sync loop. Hrm. Digging more into vagrant...
----- Original Message -----

Which is likely the core controllers loop.  If there are no other candidates
go ahead and revert my change while i track it down.

On May 11, 2015, at 5:58 PM, Daniel Smith<EMAIL_ADDRESS>wrote:
Those error messages indicate that the kubernetes-ro service is not
working.
—
Reply to this email directly or view it on GitHub.

Hrm... I may have flipped the order of the service registration accidentally (so ro <IP_ADDRESS> sometimes and sometimes <IP_ADDRESS>).
Were there e2e tests hardcoding <IP_ADDRESS>?
----- Original Message -----

Those error messages indicate that the kubernetes-ro service is not working.

Reply to this email directly or view it on GitHub:
https://github.com/GoogleCloudPlatform/kubernetes/pull/7704#issuecomment-101060507

I don't see any.
----- Original Message -----

Hrm... I may have flipped the order of the service registration accidentally
(so ro <IP_ADDRESS> sometimes and sometimes <IP_ADDRESS>).
Were there e2e tests hardcoding <IP_ADDRESS>?
----- Original Message -----

Those error messages indicate that the kubernetes-ro service is not
working.

Reply to this email directly or view it on GitHub:
https://github.com/GoogleCloudPlatform/kubernetes/pull/7704#issuecomment-101060507

If the core controllers fail to startup the cluster will not have a kubernetes or kubernetes-ro service (unless created externally or old data).  So the absence of kubernetes[-ro] on the cluster would be indicative that the master didn't start properly.
----- Original Message -----

I don't see any.
----- Original Message -----

Hrm... I may have flipped the order of the service registration
accidentally
(so ro <IP_ADDRESS> sometimes and sometimes <IP_ADDRESS>).
Were there e2e tests hardcoding <IP_ADDRESS>?
----- Original Message -----

Those error messages indicate that the kubernetes-ro service is not
working.

Reply to this email directly or view it on GitHub:
https://github.com/GoogleCloudPlatform/kubernetes/pull/7704#issuecomment-101060507

No-- it got an IP address, which means the part of the master that creates
the service ran. But the ip address is not connected up, which means the
endpoints were not updated or kube-proxy is broken.
On Mon, May 11, 2015 at 3:52 PM, Clayton Coleman<EMAIL_ADDRESS>wrote:

If the core controllers fail to startup the cluster will not have a
kubernetes or kubernetes-ro service (unless created externally or old
data). So the absence of kubernetes[-ro] on the cluster would be indicative
that the master didn't start properly.
----- Original Message -----

I don't see any.
----- Original Message -----

Hrm... I may have flipped the order of the service registration
accidentally
(so ro <IP_ADDRESS> sometimes and sometimes <IP_ADDRESS>).
Were there e2e tests hardcoding <IP_ADDRESS>?
----- Original Message -----

Those error messages indicate that the kubernetes-ro service is not
working.

Reply to this email directly or view it on GitHub:

https://github.com/GoogleCloudPlatform/kubernetes/pull/7704#issuecomment-101060507

—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/7704#issuecomment-101069468
.

Definitely works locally.  So I don't think it's a flaw in the controller loop or the core proxy code.  It could be a data race thing, but we should be retrying every 10 seconds to create the service.  We also run the initial state setup on apiserver start in a predictable order so that they are guaranteed to be consistent.  Although the ro and the regular being flopped is suspicious.
----- Original Message -----

No-- it got an IP address, which means the part of the master that creates
the service ran. But the ip address is not connected up, which means the
endpoints were not updated or kube-proxy is broken.
On Mon, May 11, 2015 at 3:52 PM, Clayton Coleman<EMAIL_ADDRESS>wrote:

If the core controllers fail to startup the cluster will not have a
kubernetes or kubernetes-ro service (unless created externally or old
data). So the absence of kubernetes[-ro] on the cluster would be indicative
that the master didn't start properly.
----- Original Message -----

I don't see any.
----- Original Message -----

Hrm... I may have flipped the order of the service registration
accidentally
(so ro <IP_ADDRESS> sometimes and sometimes <IP_ADDRESS>).
Were there e2e tests hardcoding <IP_ADDRESS>?
----- Original Message -----

Those error messages indicate that the kubernetes-ro service is not
working.

Reply to this email directly or view it on GitHub:

https://github.com/GoogleCloudPlatform/kubernetes/pull/7704#issuecomment-101060507

—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/7704#issuecomment-101069468
.

Reply to this email directly or view it on GitHub:
https://github.com/GoogleCloudPlatform/kubernetes/pull/7704#issuecomment-101069958

It may be the public read only service port.
----- Original Message -----

Definitely works locally.  So I don't think it's a flaw in the controller
loop or the core proxy code.  It could be a data race thing, but we should
be retrying every 10 seconds to create the service.  We also run the initial
state setup on apiserver start in a predictable order so that they are
guaranteed to be consistent.  Although the ro and the regular being flopped
is suspicious.
----- Original Message -----

No-- it got an IP address, which means the part of the master that creates
the service ran. But the ip address is not connected up, which means the
endpoints were not updated or kube-proxy is broken.
On Mon, May 11, 2015 at 3:52 PM, Clayton Coleman<EMAIL_ADDRESS>wrote:

If the core controllers fail to startup the cluster will not have a
kubernetes or kubernetes-ro service (unless created externally or old
data). So the absence of kubernetes[-ro] on the cluster would be
indicative
that the master didn't start properly.
----- Original Message -----

I don't see any.
----- Original Message -----

Hrm... I may have flipped the order of the service registration
accidentally
(so ro <IP_ADDRESS> sometimes and sometimes <IP_ADDRESS>).
Were there e2e tests hardcoding <IP_ADDRESS>?
----- Original Message -----

Those error messages indicate that the kubernetes-ro service is not
working.

Reply to this email directly or view it on GitHub:

https://github.com/GoogleCloudPlatform/kubernetes/pull/7704#issuecomment-101060507

—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/7704#issuecomment-101069468
.

Reply to this email directly or view it on GitHub:
https://github.com/GoogleCloudPlatform/kubernetes/pull/7704#issuecomment-101069958

Found the culprit https://github.com/GoogleCloudPlatform/kubernetes/pull/8085
Cut and paste error.
----- Original Message -----

Definitely works locally.  So I don't think it's a flaw in the controller
loop or the core proxy code.  It could be a data race thing, but we should
be retrying every 10 seconds to create the service.  We also run the initial
state setup on apiserver start in a predictable order so that they are
guaranteed to be consistent.  Although the ro and the regular being flopped
is suspicious.
----- Original Message -----

No-- it got an IP address, which means the part of the master that creates
the service ran. But the ip address is not connected up, which means the
endpoints were not updated or kube-proxy is broken.
On Mon, May 11, 2015 at 3:52 PM, Clayton Coleman<EMAIL_ADDRESS>wrote:

If the core controllers fail to startup the cluster will not have a
kubernetes or kubernetes-ro service (unless created externally or old
data). So the absence of kubernetes[-ro] on the cluster would be
indicative
that the master didn't start properly.
----- Original Message -----

I don't see any.
----- Original Message -----

Hrm... I may have flipped the order of the service registration
accidentally
(so ro <IP_ADDRESS> sometimes and sometimes <IP_ADDRESS>).
Were there e2e tests hardcoding <IP_ADDRESS>?
----- Original Message -----

Those error messages indicate that the kubernetes-ro service is not
working.

Reply to this email directly or view it on GitHub:

https://github.com/GoogleCloudPlatform/kubernetes/pull/7704#issuecomment-101060507

—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/7704#issuecomment-101069468
.

Reply to this email directly or view it on GitHub:
https://github.com/GoogleCloudPlatform/kubernetes/pull/7704#issuecomment-101069958


[Windows] Implement Windows host configuration

Create Transarent HNS Network in agent initialization.
Use netroute to configure routing rules when new Node join the
cluster on Windows.
Extract OS specific functions under pkg/agent to seperate files.
Move iptables configuraitons to prepare host networking stage,
and Windows creates HNS Network at the same stage.

/test-all

/test-all

/test-all

/test-e2e

/test-all

/test-all

/test-all

/test-all

/test-all

/test-all

/test-all

/test-all

/test-e2e

/test-e2e

/test-e2e

/test-all

/test-all

/test-all

/test-conformance

/test-all

/test-conformance

@wenyingd @tnqn : what is the plan for this PR - we want to merge Quan's PR first before this one?

Yes, I will wait for Quan's PR merged first, and then update this one.

@tnqn @jianjuns I have rebased "windows" branch with latest master code, and this PR is changed to consume new route interfaces. Please help review.

/test-all

/test-all


Add Dock Dodger

An old app (left from 2007) that still works with 10.11.
It removes Dock icon from the apps, as advertised. Only link
to the app is on MacUpdates.

This app is discontinued and not supported by the vendor anymore and the original homepage of the app is not available: http://foggynoggin.com/dockdodger/
Even though the app is working, we shouldn't add this to the main homebrew-cask repository. However, you can always add it to homebrew-unofficial if you still need it.


Improve Precision Xtra error handling and increase timeout

What it says on the tin. For more details, see https://trello.com/c/NRFZHzbo

 [uploader] PXtra times out if there are many BG records on the meter


Update registering-calling-bot.md

Updating the notice, so that people know that they can use this API to record so long as they set the correct API.

Docs Build status updates of commit f23fe0f:
:warning: Validation status: warnings

File
Status
Preview URL
Details

msteams-platform/bots/calls-and-meetings/registering-calling-bot.md
:warning:Warning
View
Details

msteams-platform/bots/calls-and-meetings/registering-calling-bot.md

Line 52, Column 1: [Warning-hard-coded-locale] Link 'https://docs.microsoft.com/en-us/graph/api/call-updaterecordingstatus' contains locale code 'en-us'. For localizability, remove 'en-us' from links to most Microsoft sites.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit eac132b:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

msteams-platform/bots/calls-and-meetings/registering-calling-bot.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit eac132b:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

msteams-platform/bots/calls-and-meetings/registering-calling-bot.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel


Support behavior property in Clover IIIF Viewer

Description
Add Clover viewer support of behavior property to support a "2-up" or "side-by-side" book view on Image works. This could be a change to use the iiif commons thumbnail panel.
Use case 2 here:
https://iiif.io/api/cookbook/recipe/0009-book-1/
UNC Examples:
https://dcr.lib.unc.edu/record/287eb48a-c9f8-4c48-a7fa-b4210c0c6b01
It's possible this would be collaborative work (within Samvera!)
Image component
https://github.com/samvera-labs/clover-iiif/blob/main/src/components/Image/index.tsx#L27
Painting component to render the Image viewer
https://github.com/samvera-labs/clover-iiif/blob/main/src/components/Viewer/Painting/Painting.tsx#L152-L159
Thumbnails rendered in Media component:
https://github.com/samvera-labs/clover-iiif/blob/main/src/components/Viewer/Media/Media.tsx#L114-L122
Done Looks Like

[x] Update @iiif/vault to @iiif/helpers
[ ] Clover supports behavor property -- at least paging and individuals
[ ] We can apply behavior paged to a Manifest and see items as 2-up
[ ] Paged items render correctly in thumbnails
[ ] Annotations and Content Search still work as expected

See: https://samvera-labs.github.io/clover-iiif/docs/viewer/demo?iiif-content=https://studio.esmero.io/do/e5758f97-34be-4fa7-af0f-2e76ccb3c401/metadata/iiifmanifest/default.jsonld

[ ] Content State still works as expected

Steps to Test/Verify

[ ] Does this require developer to verify?
[ ] one
[ ] two

Ah, nope. I think GitHub moved it because I approved closed that related PR for the first punchlist item (Vault). Moving it back to open.


Sync logs even if panic occurred while bootstrapping

Codecov Report

Merging #69 (7f4a587) into master (c801d68) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##            master       #69   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files            1         1
  Lines           53        70   +17
=========================================
+ Hits            53        70   +17

Impacted Files
Coverage Δ

boot.go
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c801d68...7f4a587. Read the comment docs.


Action generator doesn't respect Ruby file naming

How to reproduce this:
% lotus new bookshelf && cd bookshelf && bundle
% bundle exec lotus generate action Web dashboard#index

      insert  apps/Web/config/routes.rb
      create  spec/Web/controllers/dashboard/index_spec.rb
      create  apps/Web/controllers/dashboard/index.rb
      create  apps/Web/views/dashboard/index.rb
      create  apps/Web/templates/dashboard/index.html.erb
      create  spec/Web/views/dashboard/index_spec.rb

Another case is true for controller and action names:
% bundle exec lotus generate action web AncientBooks#ToggleVisibility
      insert  apps/web/config/routes.rb
      create  spec/web/controllers/AncientBooks/ToggleVisibility_spec.rb
      create  apps/web/controllers/AncientBooks/ToggleVisibility.rb
      create  apps/web/views/AncientBooks/ToggleVisibility.rb
      create  apps/web/templates/AncientBooks/ToggleVisibility.html.erb
      create  spec/web/views/AncientBooks/ToggleVisibility_spec.rb

This last case also generates wrong routes:
# apps/web/config/routes.rb
get '/AncientBooks', to: 'AncientBooks#ToggleVisibility'

@nguyenngoc2505 do you want to take care of this too?

Yes, I got it.

@nguyenngoc2505 What's the ETA of this? We're pushing to release v0.3.2 in a couple of hours.

@jodosha I'm doing.

@jodosha @joneslee85 I've made new pull-req to do fixed this. Please review this. Thanks!
https://github.com/lotus/lotus/pull/232

@nguyenngoc2505 Merged as it is. :clap: /cc @joneslee85


MudSelect with MultiSelection=true doesn't fire bind:after event correctly.

Bug type
Component
Component name
MudSelect
What happened?
I have a control like this:
   <MudSelect T=" int? " Label="Month" @bind-Value="searchService.Month" @bind-Value:after="FilterChanged"
                                MultiSelection="true" Clearable="true" Variant="UIConstants.MudVariant">
            @foreach( var choice in monthChoices )
            {
                <MudSelectItem T=" int? "<EMAIL_ADDRESS>                    @choice.Key
                </MudSelectItem>
            }
        </MudSelect>

and a method:
    private Task FilterChanged()
    {
        Console.WriteLine("The filter changed");
        return Task.CompletedTask;
    }

If I change a value in the MudSelect, the FilterChange does not fire. If I modify the MudSelect to set MultiSelection=false the FilterChanged event fires as expected.
Expected behavior
I would expect the :after event to correctly fire when the multiple-selection value updates.
Reproduction link
Not possible, because TryMudblazor doesn't seem to support :after
Reproduction steps
Something like this:
@using System.Linq

<MudSelect T="int?" @bind-Value="theValue"
                    @bind-Value:after="FilterChanged"
                    MultiSelection="true" Clearable="true" >
            @foreach( var choice in new [] { 1, 2, 3, 4, 5 } )
            {
                <MudSelectItem T=" int? " Value="@choice">
                    @choice
                </MudSelectItem>
            }
</MudSelect>

@code {

    private int? theValue;

    private Task FilterChanged()
    {
        Console.WriteLine("The filter changed");
        return Task.CompletedTask;
    }
}

Relevant log output
No response
Version (bug)
6.1.9
Version (working)
No response
What browsers are you seeing the problem on?
Chrome
On what operating system are you experiencing the issue?
Windows, Mac OSX
Pull Request

[ ] I would like to do a Pull Request

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Could you try same with MudSelectExtended? It should work on extended component, https://www.nuget.org/packages/CodeBeam.MudExtensions

Could you try same with MudSelectExtended? It should work on extended component, https://www.nuget.org/packages/CodeBeam.MudExtensions

I haven't tried. I'd really just like the MudBlazor component to work; adding in extended components is just another dependency to manage, and bulks up the WASM size too.

Understand, that's why we keep the package size as less as possible.


The bot can get rate limited during a storm (and probably in large rooms)

t2bot.io was half-broken where it was receiving pings but not able to send any events out. When it eventually did get through a 9 hour backlog and shoved that out to the servers, the bots promptly spammed pongs back out. The two bots I have that aren't t2bot.io both got rate limited.
Possible mitigations:

Don't reply to pings that have a very old origin_server_ts (5 min?)
Raise the interval even more to prevent large backlogs

Shouldn't happen anymore now that the bot only sends an event every 5 minutes


Serve the GraphiQL UI

Although currently "the server is GraphiQL enabled", to use it appears to require standing up GraphiQL separately (perhaps via the electron app that is linked) and then pointing at this server.
Feature request:
Optionally (controlled by a switch) serve the GraphiQL UI from json-graphql-server.

The GraphiQL interface is served by this package. Just open the url (http://localhost:3000) in a web browser to see it.
If not, which version are you using? What do you see in the browser when you go to that URL?

I confirm that the GraphiQL interface is served.


Simplify code and add tags in the markup

In this modifications add the opcions the add tags in the markup. In my case ir very util

Coverage decreased (-1.4%) to 96.457% when pulling 094b2efcea7017fc99f086c4f4edc0566742b2e7 on Ymil:dev into 7b2d8b357f8603a92887819cf5c57c2eaec0117a on starboarder2001:develop.


Remove right column from versions tab of simulator pages

Example:
https://biosimulators.org/simulators/tellurium/latest
The right column (pull command, license, website) for a specific version might be confused with the meta data for the individual versions. Lets remove the column to eliminate this potential source of confusion.
Instead

Add a "Copy to clipboard" button to the "Image" column of the versions table

Fixed in dev-simulators


Code sample ListLogger.cs BeginScope gives CS8633 nullability mismatch warning in net8

[Enter feedback here]
I am trying to resolve CS8633 nullability mismatch warning in net8 for our implementation of ListLogger
I have created a new project following the documentation.
I am getting warnings for NullScope.cs and ListLogger.cs
Do you have and updated document for net8?
	TestProject1	C:\Users\mparsons\source\Test\ListLoggerTest\TestProject1\NullScopeBase.cs	6	Active
Warning	S1186	Add a nested comment explaining why this method is empty, throw a 'NotSupportedException' or complete the implementation.	TestProject1	C:\Users\mparsons\source\Test\ListLoggerTest\TestProject1\UnitTest1.cs	6	Active
Warning	S3881	Fix this implementation of 'IDisposable' to conform to the dispose pattern.	TestProject1	C:\Users\mparsons\source\Test\ListLoggerTest\TestProject1\NullScope.cs	5	Active
Warning	CS8633	Nullability in constraints for type parameter 'TState' of method 'ListLogger.BeginScope(TState)' doesn't match the constraints for type parameter 'TState' of interface method 'ILogger.BeginScope(TState)'. Consider using an explicit interface implementation instead.	TestProject1	C:\Users\mparsons\source\Test\ListLoggerTest\TestProject1\ListLogger.cs	9	Active
Warning	CS8767	Nullability of reference types in type of parameter 'exception' of 'void ListLogger.Log(LogLevel logLevel, EventId eventId, TState state, Exception exception, Func<TState, Exception, string> formatter)' doesn't match implicitly implemented member 'void ILogger.Log(LogLevel logLevel, EventId eventId, TState state, Exception? exception, Func<TState, Exception?, string> formatter)' (possibly because of nullability attributes).	TestProject1

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 369aefa6-74da-3001-f835-15abdfffd5d6
Version Independent ID: 2cc2a2db-d059-3252-7a1a-4261b7d3c37c
Content: Develop Azure Functions using Visual Studio
Content Source: articles/azure-functions/functions-develop-vs.md
Service: azure-functions
GitHub Login: @ggailey777
Microsoft Alias: glenga

@MarkParsonsOS
Thanks for your feedback! We will investigate and update as appropriate.

@MarkParsonsOS It looks like the samples on this doc target the .net 6. To accomplish the same in .net 8, could you make a new post on the MS Q&A channel instead?
Since we didn't determine any faulty steps/changes to this doc upon reviewing your feedback, we will now proceed to close this thread for now. If there are further questions regarding this matter, please reopen it and we will gladly continue the discussion.


Not all topics are synced

Hello,
I have quite a peculiar problem where only specific topics are synced. First, I will explain my unique setup (which is quite similar to the one in PR #115).
I am on a machine A with a robot arm B. Both machines have their respective ROS masters. I cannot access the machine on robot B due to it being a proprietary robot and therefore, must launch multimaster nodes through a terminal in machine A with robot B's ROS_MASTER_URI.
I use two launch files, one for my machine A and another for robot B. Both are launched from machine A though.
Machine A launch file:
<node name="master_discovery" pkg="fkie_master_discovery" type="master_discovery" />
<node name="master_sync" pkg="fkie_master_sync" type="master_sync" output="screen">
        <rosparam param="sync_topics"> ['/s1/joint_states', '/robot/joint_states'] </rosparam>
</node>

Robot B launch file (called on Machine A with export ROS_MASTER_URI=http://robot.local:11311)
I use a different rcp_port similar as to what was mentioned in PR #115.
I also relay the topic /robot/joint_states to the topic /s1/joint_states as I would like to eventually connect a second robot with the same namespace and I cannot change the namespace myself.
<node name="master_discovery" pkg="fkie_master_discovery" type="master_discovery">
        <rosparam param="rpc_port"> 11613 </rosparam>
</node>
<node pkg="topic_tools" type="relay" name="s1_relay" args="/robot/joint_states /s1/joint_states"/>

The terminal output of machine A is
started roslaunch server http://QuantumMopeSCI:43563/

SUMMARY
========

PARAMETERS
 * /master_sync/sync_topics: ['/s1/joint_state...
 * /rosdistro: melodic
 * /rosversion: 1.14.13

NODES
  /
    master_discovery (fkie_master_discovery/master_discovery)
    master_sync (fkie_master_sync/master_sync)

ROS_MASTER_URI=http://<IP_ADDRESS>:11311

process[master_discovery-1]: started with pid [22226]
process[master_sync-2]: started with pid [22227]
[INFO] [1659399736.178650]: ignore_hosts: []
[INFO] [1659399736.185075]: sync_hosts: []
[INFO] [1659399736.192174]: sync_topics_on_demand: False
[INFO] [1659399736.197407]: resync_on_reconnect: True
[INFO] [1659399736.200915]: resync_on_reconnect_timeout: 0
[INFO] [1659399736.203511]: listen for updates on /master_discovery/changes
[INFO] [1659399753.291191]: [robot.local] ignore_nodes: ['/node_manager', '/param_sync', '/master_sync', '/rosout', '/node_manager_daemon', '/zeroconf', '/master_discovery']
[INFO] [1659399753.299708]: [robot.local] sync_nodes: []
[INFO] [1659399753.306485]: [robot.local] ignore_topics: ['/master_sync/*', '/master_discovery/*', '/rosout', '/rosout_agg', '/zeroconf/*']
[INFO] [1659399753.314488]: [robot.local] sync_topics: ['/robot/joint_states', '/s1/joint_states']
[INFO] [1659399753.321949]: [robot.local] ignore_services: ['/*get_loggers', '/master_sync/*', '/master_discovery/*', '/zeroconf/*', '/*set_logger_level', '/node_manager_daemon/*']
[INFO] [1659399753.330428]: [robot.local] sync_services: []
[INFO] [1659399753.335239]: [robot.local] ignore_type: ['fkie_multimaster_msgs/SyncTopicInfo', 'fkie_multimaster_msgs/MasterState', 'fkie_multimaster_msgs/SyncServiceInfo', 'fkie_multimaster_msgs/SyncMasterInfo', 'bond/Status']
[INFO] [1659399753.340280]: [robot.local] ignore_publishers: []
[INFO] [1659399753.343617]: [robot.local] ignore_subscribers: []
[WARN] [1659399753.432426]: Resolved host of ROS_MASTER_URI robot.local=<IP_ADDRESS> and origin discovered IP=<IP_ADDRESS> are different. Fix your network settings and restart master_discovery!
[INFO] [1659399753.865409]: SyncThread[robot.local] Requesting remote state from 'http://<IP_ADDRESS>:11613'
[INFO] [1659399753.873816]: SyncThread[robot.local] Applying remote state...
[INFO] [1659399753.886515]: SyncThread[robot.local] Requesting remote md5sums 'http://<IP_ADDRESS>:11613'
[INFO] [1659399753.959848]: SyncThread[robot.local] remote state applied.
[INFO] [1659399755.662658]: SyncThread[robot.local] Requesting remote state from 'http://<IP_ADDRESS>:11613'
[INFO] [1659399755.703939]: SyncThread[robot.local] Applying remote state...
[INFO] [1659399755.731761]: SyncThread[robot.local] Requesting remote md5sums 'http://<IP_ADDRESS>:11613'
[INFO] [1659399755.738407]: SyncThread[robot.local] remote state applied.
[INFO] [1659399766.217683]: ROS masters obtained from '/master_discovery/list_masters': ['quantummopesci', 'robot.local']
[INFO] [1659399796.235996]: ROS masters obtained from '/master_discovery/list_masters': ['quantummopesci', 'robot.local']
[INFO] [1659399826.251965]: ROS masters obtained from '/master_discovery/list_masters': ['quantummopesci', 'robot.local']

The terminal output for robot B on my machine A is
started roslaunch server http://<IP_ADDRESS>:42433/

SUMMARY
========

PARAMETERS
 * /master_discovery/rpc_port: 11613
 * /rosdistro: melodic
 * /rosversion: 1.14.13

NODES
  /
    master_discovery (fkie_master_discovery/master_discovery)
    s1_relay (topic_tools/relay)

ROS_MASTER_URI=http://robot.local:11311

process[master_discovery-1]: started with pid [22306]
process[s1_relay-2]: started with pid [22307]

Now, here comes the problem. Essentially, I can see both /s1/joint_states and /robot/joint_states topics in my rostopic list for robot B but my rostopic list for machine A shows the following where only /robot/joint_states is synced.
/diagnostics
/master_discovery/changes
/master_discovery/linkstats
/robot/joint_states
/rosout
/rosout_agg

The only distinguishing factor I can deduct is that /s1/joint_states is a topic that I create with a node from machine A with robot B's ROS_MASTER_URI while /robot/joint_states is published on robot B's hardware.
In fact, I can sync any topic that the robot itself publishes on bootup while all topics I manually publish do not sync.
I wouldn't think this would cause any issues but maybe there is something wrong with my setup.
Do you have any insights as to what might be causing this problem?
Thanks so much.

Hi,
since the monitoring of the ROS master by the master_discovery causes a lot of network load, a "remote" start of the master_discovery was never intended and tested.
In such a case I would start the whole system with the ROS_MASTER_URI of robot B.
If you still want to use your configuration, I'll have to look for the bug. However, it may then take some time, since I can not estimate the effort now.

Unfortunately, I will still have to stick to this configuration.
Essentially, I would like to eventually sync two robots (B and C).
Both these robots cannot be accessed through their onboard ROS and have identical topic names.
Because of this, the only workaround for dual arm collaboration would be to relay all desired topic names to unique names and then sync it to my local ROS on computer A.
The only thing stopping me from achieving a working setup is the relayed topic name not being synced properly while any topic published by the robot itself is (which is not useful since we have topic name clashes between the two robots).
I appreciate any help towards discovering the bug. I understand it will probably be somewhat difficult.
In the meantime, I may be able to circumvent this by using a rather messy setup with two additional intermediary ROS masters. D and E, meant to relay the topics.
robot B --> ROS D (relays B's topics to unique names) --> ROS A <-- ROS E (relays C's topics to unique names) <-- robot C
This will obviously accrue alot of unwanted bandwidth but I'm not sure how else to solve the problem. Please let me know if you any insights into this.
Thank you!

Hi,
I have now looked at the problem more closely and there is indeed a way to parameterize the master_sync so that it should work without patches. At least it worked in my replayed scenario ^^
I would also suggest reducing the checking of changes on the remote host by the master_discovery.
<node name="master_discovery" pkg="fkie_master_discovery" type="master_discovery" >
        <rosparam param="rosmaster_hz">0.1</rosparam>
</node>
<node name="master_sync" pkg="fkie_master_sync" type="master_sync" output="screen">
        <rosparam param="check_host">False </rosparam>
        <rosparam param="sync_topics"> ['/s1/joint_states'] </rosparam>
</node>

Sorry, the function of this parameter has slipped my mind!

Unfortunately, the check_host param did not solve the issue.
I am happy to say that I was able to solve the issue by toggling a different param, sync_remote_nodes!
In fact, the description of this param matches perfectly the problem I was having and is set to false as default:

~sync_remote_nodes (boolean, default: False)
The nodes which are running not at the same host as the ROS master are not synchronized by default. Use sync_remote_nodes to sync these nodes also.

Setting this to true solved my issue.
Before I close this issue though, I would like to ask another question about syncing two robots B and C (with identical topic names) to a control station A. Although syncing works perfectly fine for a single robot, the moment I sync a second robot, one of the two robots essentially crashes and requires a reboot. From the previous issues I've read, I'm assuming this is because the topic names overlap and even if I set ignore_hosts or sync_topics, the overlapping published topics of B are still viewable to C through A. I have no way of changing the namespace of topics with these two robots. Is there any way I can sync B and C to control station A while B and C are completely unseen by eachother so as to prevent crashing?
Thanks so much for your help again.


fix: upgrade to version 3.1.60

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To complete the pull request process, please assign macox
You can assign the PR to them by writing /assign @macox in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@jenkins-x-bot-test: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

tls
976b9c7468e5feeb602b7ec72f5970c75a6880f6
link
/test tls

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the jenkins-x/lighthouse repository. I understand the commands that are listed here.


Revert "Fix onnx export for quantized MPT models 2 (#1647)"

This reverts commit e8ff0f6bd5ed4fcb8d281f30462462b0125b4ddf.
These commit breaks our current YOLOv8 flows (After #1680 ); reverting them for now, a fix will be put in later for MPT models

@eldarkurtic noting here for you, let's talk through potential general solution that we can use here

I just verified that reverting the earlier commit (https://github.com/neuralmagic/sparseml/commit/e8ff0f6bd5ed4fcb8d281f30462462b0125b4ddf) works.


RFE: support custom user data

I want to create a metal instance with custom user data so that i can reserve cpu cores and disable CPU P state, so that i can use the reserved cores to run VMs.
I have tested the following user data and it works on the Equninix console.
runcmd:
  - cp /boot/grub2/grub.cfg /boot/grub2/grub.cfg.bak
  - sed -i '/^[[:space:]]*linux / s/$/ isolcpus=1-7,9-15 intel_pstate=disable/' /boot/grub2/grub.cfg
  - reboot

Since the runner has already set the user data here, it could be possible to check if there is a custom user data provided and run it after runner configuration is finished.
cc @vprashar2929

custom_script option was included in v0.4.0


Unable to create new container using bc insider

HI,
trying to create new container using bc insider.
OS Windows Server 2016 - 64 bit
Below is the error
Initialization of container bcinsider failed
At C:\Program Files\WindowsPowerShell\Modules\navcontainerhelper\<IP_ADDRESS>\ContainerHandling\Wait-NavContainerReady.ps1:44 char:17
+ ...             throw "Initialization of container $containerName failed" ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (Initialization ...cinsider failed:String) [], RuntimeException
    + FullyQualifiedErrorId : Initialization of container bcinsider failed

below are the details.

NavContainerHelper is version <IP_ADDRESS>
NavContainerHelper is running as administrator
Host is Microsoft Windows Server 2016 Standard - ltsc2016
Docker Client Version is 18.09.1
Docker Server Version is 18.09.1
Using image bcinsider.azurecr.io/bcsandbox-master:us-ltsc2016
Creating Container bcinsider
Version: 16.0.11104.0-us
Style: sandbox
Platform: 16.0.11059.0
Generic Tag: <IP_ADDRESS>
Container OS Version: 10.0.14393.3443 (ltsc2016)
Host OS Version: 10.0.14393.3443 (ltsc2016)
Using locale en-US
Using process isolation
Disabling the standard eventlog dump to container log every 2 seconds (use -dumpEventLog to enable)
Using license file C:\myFolder\mylicense.flf
Files in C:\ProgramData\NavContainerHelper\Extensions\bcinsider\my:

AdditionalOutput.ps1
license.flf
MainLoop.ps1
SetupVariables.ps1
updatehosts.ps1
Creating container bcinsider from image bcinsider.azurecr.io/bcsandbox-master:us-ltsc2016
73854aa201767396beaf45d861ab56dfafbd529454ee07ad59c00fa1879ecfeb
Waiting for container bcinsider to be ready
Initializing...
Setting host.containerhelper.internal to <IP_ADDRESS> in container hosts file
Starting Container
Hostname is bcinsider
PublicDnsName is bcinsider
Using NavUserPassword Authentication
Starting Local SQL Server
Starting Internet Information Server
Creating Self Signed Certificate
Self Signed Certificate Thumbprint C6F3F6BE92D861CA278357F8D0826DCD4C19B1BB
Modifying Service Tier Config File with Instance Specific Settings
Starting Service Tier
Registering event sources
Creating DotNetCore Web Server Instance
The data is invalid. (Exception from HRESULT: 0x8007000D)
at Set-NavSiteAuthenticationSettings, C:\Program Files\Microsoft Dynamics NAV\160\Web Client\Modules\NAVWebClientManagement\NAVWebClientManagement.psm1: line 654
at New-NavWebSite, C:\Program Files\Microsoft Dynamics NAV\160\Web Client\Modules\NAVWebClientManagement\NAVWebClientManagement.psm1: line 631
at New-NAVWebServerInstance, C:\Program Files\Microsoft Dynamics NAV\160\Web Client\Modules\NAVWebClientManagement\NAVWebClientManagement.psm1: line 501
at , C:\Run\SetupWebClient.ps1: line 26
at , C:\Run\navstart.ps1: line 157
at , C:\Run\start.ps1: line 121
at , : line 1Error
Initializing...
Setting host.containerhelper.internal to <IP_ADDRESS> in container hosts file
Starting Container
Hostname is bcinsider
PublicDnsName is bcinsider
Using NavUserPassword Authentication
Starting Local SQL Server
Starting Internet Information Server
Creating Self Signed Certificate
Self Signed Certificate Thumbprint C6F3F6BE92D861CA278357F8D0826DCD4C19B1BB
Modifying Service Tier Config File with Instance Specific Settings
Starting Service Tier
Registering event sources
Creating DotNetCore Web Server Instance
The data is invalid. (Exception from HRESULT: 0x8007000D)
at Set-NavSiteAuthenticationSettings, C:\Program Files\Microsoft Dynamics NAV\160\Web Client\Modules\NAVWebClientManagement\NAVWebClientManagement.psm1: line 654
at New-NavWebSite, C:\Program Files\Microsoft Dynamics NAV\160\Web Client\Modules\NAVWebClientManagement\NAVWebClientManagement.psm1: line 631
at New-NAVWebServerInstance, C:\Program Files\Microsoft Dynamics NAV\160\Web Client\Modules\NAVWebClientManagement\NAVWebClientManagement.psm1: line 501
at , C:\Run\SetupWebClient.ps1: line 26
at , C:\Run\navstart.ps1: line 157
at , C:\Run\start.ps1: line 121
at , : line 1

> Initialization of container bcinsider failed
> At C:\Program Files\WindowsPowerShell\Modules\navcontainerhelper\<IP_ADDRESS>\ContainerHandling\Wait-NavContainerReady.ps1:44 char:17
> + ...             throw "Initialization of container $containerName failed" ...
> +                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>     + CategoryInfo          : OperationStopped: (Initialization ...cinsider failed:String) [], RuntimeException
>     + FullyQualifiedErrorId : Initialization of container bcinsider failed

Did you retry the operation to see whether it happens the same place?

yes, tried in multiple docker machines

What PowerShell script are you using (without the insider password:-))?

i am having the same issue

PS C:\Users\vmadmin> $imageName = "bcinsider.azurecr.io/bcsandbox-master:us"
$containerName = "BC"
$auth = "UserPassword"
$credential = New-Object pscredential 'xxxxxxxxxxxxxx', (ConvertTo-SecureString -String 'xxxxxxxxxxxx' -AsPlainText -Force)
$licenseFile = "C:\Users\vmadmin\Desktop\License_051018.flf"
New-navcontainer -accept_eula                 -imageName $imageName
-containerName $containerName                 -auth $auth
-credential $credential                 -updateHosts
-isolation hyperv ` -licenseFile $licenseFile -includeAL

Will try to repro

New-BCContainer -accept_eula                 -imageName bcinsider.azurecr.io/bcsandbox-master:us
-containerName bcinsider                 -auth NavUserPassword
-licenseFile 'C:\myFolder\mylicense.flf'                 -updateHosts
-alwayspull

The same issue for us - CZ insider image.
Jiří

Yep, I can repro this - looks like latest insider builds are broken (which can happen, but shouldn't)
Will investigate

do you have any previous build version that is working in the meantime?

I can see that my pipeline was able to create a container with 16.0.11044.0, so you can use:
bcinsider.azurecr.io/bcsandbox-master:16.0.11044.0-us-ltsc2019 (or ltsc2016 if you are running server 2016)
OR
I found out that the reason for this is, that URL rewrite and ASPNETCORE wasn't installed correctly on the base image - I don't know why, but that seems to be the reason.
This should be fixed tomorrow (with new builds) or you can add
-usegenericImage 'mcr.microsoft.com/dynamicsnav:generic-ltsc2019'
if you are running Windows Server 2019.
-usegenericImage 'mcr.microsoft.com/dynamicsnav:generic-ltsc2016'
if you are running Windows Server 2016.
and
-useBestContainerOS
if you are running Windows 10 1903 or 1909.
That should also work (or at least - it works here).

thanks freddy it worked :)

This was caused by a Windows Update to the Docker Build Servers. Read the full story here: https://freddysblog.com/2020/02/14/hyperv-isolation-to-the-rescue/  as you might yourself be hit by that security update.


Errors: Audio and low quality output video

What went wrong
If you use the auto-editor to output a xml file to put it into DaVinci Resolve software, the stuttering sound is very annoying. I didn't get this error before. Now I don't understand why. For the most part I export to DaVinci Resolve for further editing.
Previously, using the command: auto-editor D:\video\example.mp4 -exr gave a good result (no error) but now doing so the sound is corrupted and the video speed is naturally faster without Use another command.
If you use auto-editor to output mp4 files without any command, the video quality will be low. The original video is more than 300Mb but the new video is only 80Mb. Blurry image is not clear. I tried the video codec command and the video bitrate still doesn't work.
Paste the commands you've tried
auto-editor D:\video\example.mp4 -exr

and also
auto-editor D:\video\example.mp4

Describe what went wrong
Low quality video or problem with audio sound in DaVinci Resolve
What's your platform? (if applicable)
Python Version: 3.9.6 64-bit
Platform: Windows 10
Config File path: C:\Users\KyThuat88\AppData\Local\Programs\Python\Python39\Lib\site-packages\auto_editor\config.txt
FFmpeg path: C:\Users\KyThuat88\AppData\Local\Programs\Python\Python39\Lib\site-packages\auto_editor\ffmpeg\Windows\ffmpeg.exe
FFmpeg version: 4.3
Auto-Editor version 21w46b

Tried both methods. Can't confirm. Closing.


Make the next toast hiding the current one

Hello,
great plugin! I'm having some (noob) trouble with multiple toasts though (on Android).
I would like the next toast to erase / replace the previous one if it's still being shown. I tried using window.plugins.toast.hide(); before showing the new toast but it hides the previous AND next one (basicaly nothing appears)
Am I missing something?

Can you show the code which hides and shows the toast?

Sure, thanks for the fast answer!
(I'm also using this controller to play an audio file first)
$scope.play = function(src,message) {
var media = new Media(src, function onSuccess() {
// release the media resource once finished playing
media.release();
},
function onError(e){
console.log("error playing sound: " + JSON.stringify(e));
});
media.play();
// TOAST
window.plugins.toast.hide();
$cordovaToast.show(message, 'long', 'bottom').then(function(success) {
console.log("The toast was shown");
}, function (error) {
console.log("The toast was not shown due to " + error);
});
};

I'm guessing you need to put that showLongBottom('txt') call in the successcallback of the hide function, since those Cordova functions are asynchronous:
..hide(function() {
  // ..show here
});

Sorry didn't managed to make it work.
Is that what you meant?
$cordovaToast.hide().then(function(success) {
          $cordovaToast.show(message, 'long', 'bottom');
        });

I don't think ngCordova supports the hide function, so try this (didn't test this code, just typed it here):
window.plugins.toast.hide(function() {
  // you're now in the succes callback handler of the 'hide' function
  $cordovaToast.show(message, 'long', 'bottom');
});


Update main.yml

Selinux blocks httpd connections to database
Description of PR

Type of change

Feature Pull Request
Bugfix Pull Request
Docs Pull Request
Fixes an issue

Thanks! 👍


indent_size not honoured in Visual Studio Code Insiders

I am running the Visual Studio Code Insiders build. My .editorconfig specifies indent_size = 2, but this does not appear to be applied in the IDE. This works in the stable build.

Did you install the editor config exension in the insider build? The insider and stable build are full separate install that do not share extensions.

Oh yeah, thanks!
On Wed, 8 Jun 2016, 12:18 Erich Gamma<EMAIL_ADDRESS>wrote:

Did you install the editor config exension in the insider build? The
insider and stable build are full separate install that do not share
extensions.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/vscode-editorconfig/issues/26#issuecomment-224560294,
or mute the thread
https://github.com/notifications/unsubscribe/AA4QCWI-GAFHJEZXDzoZwxcuAUz542Qwks5qJqUQgaJpZM4Iwz9e
.


Why not using vgo to manage the third party library dependencies?

Why not using vgo to manage the third party library dependencies?

LGTM


Combined ready and data arrays in seg_queue

This combines the ready block and the data block in seq queue. Combining these has some good cache properties -

Loads of the ready variable in pull will likely read the data into the cache, and same with loads of the data in push.
Writes to the data and ready variable may well be combined.
Elements are farther apart in the cache, so writes from pushers won't invalidate reads from pullers as much.

In my basic (and somewhat variable) tests, this saw performance gains of:

mpmc: 5-15% increase
spmc: 0-5% increase
spsc: 15-35% increase

Also, this version is generally more consistent performance-wise probably due to less cache thrashing.

Sorry for the slow review -- this looks good! I'm happy to merge after a rebase.

Just rebased!

Lovely, and nice perf wins. Thanks again for your help with the library!


Upload RTOS artifacts to separate bucket

Adding logic to upload RTOS artifacts to separate production bucket
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/cherrypick release-0.20

/lgtm
/approve


[SPARK-37320][K8S][TESTS] Delete py_container_checks.zip after the test in DepsTestsSuite finishes

What changes were proposed in this pull request?

This PR fixes an issue that py_container_checks.zip still remains in resource-managers/kubernetes/integration-tests/tests/ even after the test Launcher python client dependencies using a zip file in DepsTestsSuite finishes.
Why are the changes needed?

To keep the repository clean.
Does this PR introduce any user-facing change?

No.
How was this patch tested?

Confirmed that the zip file will be removed after the test finishes with the following command using MiniKube.
PVC_TESTS_HOST_PATH=/path PVC_TESTS_VM_PATH=/path build/mvn -Dspark.kubernetes.test.namespace=default -Pkubernetes -Pkubernetes-integration-tests -pl resource-managers/kubernetes/integration-tests integration-test

Test build #145205 has started for PR 34588 at commit 7352e20.

Test build #145205 has finished for PR 34588 at commit 7352e20.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145205/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49674/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49674/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49674/

Test build #145208 has started for PR 34588 at commit d326b62.

Test build #145208 has finished for PR 34588 at commit d326b62.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145208/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49677/

Merged to master/3.2/3.1.

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49677/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49677/


fix leak in planner

We found excessive memory usage when using the library on scale with huge requests. The following fix cleans the planner visitor fields so that they do not grow excessively..

added tag v1.67.3


Table view pagination

Long lists take a long time to render, so we need to optimise these views by using pagination. This is part of the react-md data table API.
https://react-md.mlaursen.com/components/data-tables

See #104


Sidecar: no supported bucket was configured, uploads will be disabled

Using latest releases of Prometheus-Operator and Thanos Sidecar (v0.4.0
What happened - Launched Prom Op with a Thanos Sidecar and receive the following error in the sidecar:  caller=sidecar.go:123 msg="no supported bucket was configured, uploads will be disabled"
What you expected to happen. The sidecar would connect to the S3 compatible Object Storage
How to reproduce it (as minimally and precisely as possible):
Have the following in the Prometheus Operator values yaml file:
    thanos:
      baseImage: improbable/thanos
      version: v0.4.0
      peers: thanos-peers.prod.cluster:10900
      objectStorageConfig:
        key: bucket.yaml
        name: thanos-objstore-config

I have a secret name thanos-objstore-config in the same namespace with a key bucket.yaml and value:
type: S3
config:
  endpoint: "minioingress.rook-minio.<IP_ADDRESS>.xip.io"
  bucket: "thanos"
  access_key: "MINIO_Access_Key"
  secret_key: "MINIO_Secret_Key"
  insecure: true
  signature_version2: false

The actual Thanos Store container has no problems with the bucket config used to access this same bucket.

I am pretty sure this is something with how operator configures Thanos. Can you double check what the actual Sidecar and Prometheus YAML looks like when it is generated and running on Kubernetes?

That sounds right - the yaml doesn't have any reference to the argument.
args:
        - sidecar
        - --prometheus.url=http://<IP_ADDRESS>:9090/
        - --tsdb.path=/prometheus
        - --cluster.address=[$(POD_IP)]:10900
        - --grpc-address=[$(POD_IP)]:10901
        - --cluster.peers=thanos-peers.prod.cluster:10900

Hm, maybe then just not deployed yet?

It's all definitely been deployed - the whole operator itself is working except for the bucket config.  It's really strange.

Hi, what is the status of this? Did you managet to get the sidecar uploading the blocks?
From the discussion it sounds more related to the prometheus operator than actual thanos issue.

I gave up on Thanos. Far too many moving pieces for the size of our environment. Especially when a core piece doesn’t work. I switched to remote writing our Prometheus instances to a VictoriaMetrics server and couldn’t be happier.

Ok, sorry to hear that but whatever works for you is fine.
So I'm closing this fo now.

If anyone else stumbles upon this issue like I did - I believe kube-prometheus-stack changed the format of values for thanos-sidecar's objectStore config:

I have not tested it thoroughly yet, but at least the generated configuration looks right and the "no supported bucket was configured" message disappeared from logs

If anyone else stumbles upon this issue like I did - I believe kube-prometheus-stack changed the format of values for thanos-sidecar's objectStore config:  I have not tested it thoroughly yet, but at least the generated configuration looks right and the "no supported bucket was configured" message disappeared from logs

This helped, thanks!

If anyone else stumbles upon this issue like I did - I believe kube-prometheus-stack changed the format of values for thanos-sidecar's objectStore config:  I have not tested it thoroughly yet, but at least the generated configuration looks right and the "no supported bucket was configured" message disappeared from logs

Works thanks!  <3

If anybody is looking when this happened:
https://github.com/prometheus-community/helm-charts/pull/3918