[SPARK-20434][YARN][CORE] Move kerberos delegation token code from yarn to core

What changes were proposed in this pull request?
Move kerberos delegation token code from yarn to core, so that other schedulers (such as Mesos), may use it.
Summary:

Move registered ServiceCredentialProvider classes from yarn to core
Deprecates yarn.security.ServiceCredentialProvider
Move ConfigurableCredentialManager from yarn to core
Add HadoopAccessManager trait, which parameterizes HadoopFSCredentialProvider with scheduler-specific functionality, such as that in YARNHadoopAccessManager
Add YARNConfigurableCredentialManager, which loads the now deprecated yarn.security.ServiceCredentialProvider services

How was this patch tested?
unit tests

Test build #76044 has started for PR 17723 at commit ad4e33b.

Test build #76044 has finished for PR 17723 at commit ad4e33b.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76044/
Test FAILed.

Test build #76045 has started for PR 17723 at commit e15f1ab.

Test build #76045 has finished for PR 17723 at commit e15f1ab.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76045/
Test FAILed.

Test build #76046 has started for PR 17723 at commit a546aab.

Test build #76046 has finished for PR 17723 at commit a546aab.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76046/
Test FAILed.

Test build #76047 has started for PR 17723 at commit d6d21d1.

Test build #76047 has finished for PR 17723 at commit d6d21d1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76047/
Test PASSed.

Merged build finished. Test PASSed.

cc @vanzin @jerryshao @skonto
BTW @vanzin, I decided to parameterize HadoopFSCredentialProvider with a new HadoopAccessManager object, for which YARN provides a custom YARNHadoopAccessManager.  I did this instead of conditioning on SparkHadoopUtil.get.isYarnMode, since I prefer functional parameterization over global values.

Hi @vanzin.  Any thoughts on this one?

Really busy with all sorts of things, will get to it eventually.

Thanks, do you have an approx. ETA?  Knowing that will help me not ping you too much.

EOW, maybe.

For example, use of UGI for spark security in core.

UGI, as far as I understand, is being used mostly as a credential store and wrapper around the JAAS / GSSAPI code. Replicating that code would be very hard and error prone.
If at some point in the future someone decides they want to support some different paradigm, I think that would be the time to think about abstracting that stuff. Right now I don't really see why using UGI outside of YARN is bad (aside from assumptions that the Hadoop API makes and Michael has run into already).

I was actually surprised the mesos is.

It's not that Mesos is compatible. It's that as far as Spark is concerned, UGI is mostly a container for credentials, and that's how it's being used here.

@vanzin While UGI does provide credential store and implementations for GSSAPI, it also does a bunch of other things - which are leveraged by hadoop ipc, hadoop common, etc.
Leveraging ugi from a non hadoop env is always a potential risk - due to evolution of ugi to support hadoop usecases, expectations of hadoop api from ugi, and interaction between hadoop and non hadoop deployments (for example - a non hadoop based resource manager application interacting with a hadoop based service).
As an implementation detail, mesos could be depending on UGI - but introducing this into core security should merit careful consideration. If possible, we should abstract things out so that use of UGI is an implementation detail of yarn and mesos (which, if I understand correctly, is the argument you made for current use of UGI by @mgummelt ) - while keeping core open to other implementations.

@mridulm if you're talking about ServiceCredentialProvider.obtainCredentials, which is a public interface that exposes UGI (Credentials in this case), then sure, it's probably worth to tweak it so that the UGI dependency is hidden. Perhaps even credentialsRequired could be hidden behind obtainCredentials to remove another Hadoop API from this interface.
I just don't think it's worth it, at this point in time, to spend time worrying about the underlying implementation using UGI or not.

There are multiple aspects of concern here :

Dependency on yarn api from core.
Exposure of yarn api in core interfaces.
Introduction of yarn specific implementation details in core.
Backward compatibility for existing exposed interfaces.

Based on your comments @vanzin, for (1) core is transitively already depending on yarn - now we will be explicitly depending on it.
(2) is a blocker - we cannot have yarn specific api being exposed in core interfaces.
(3) is an issue for future evolution of the project - we could either fix it now, or incur tech debt and revisit it in future if it becomes a problem.
We also have to ensure that changes made to accommodate above should not cause, as much as possible, changes to existing implementations (4) - which are based on yarn resource manager api.

No YARN API is being exposed in core. There are hadoop security APIs being exposed in the current patch (Credentials). It's different.
Michael's patch is not exposing YARN's way of using UGI and propagating tokens. I can buy not exposing Credentials in the API, but there's nothing in the functionality being added to core that is YARN-specific. If anything, it's specific to handling security in Hadoop-based services, which is exactly what the goals are.
YARN just happens to use the same APIs to achieve what it does (credential propagation, token renewal, automatic token cancellation, etc). But because YARN uses those APIs, it does not mean they are YARN APIs.

@vanzin Since hadoop 1 is no longer maintained, I was referring to hadoop security as yarn security - you are right, they are not the same : though in what I described, replacing one with another, should not change the concerns I hope ?
UGI and associated infra in hadoop security is tied to its ecosystem. hadoop-security is specific to how hadoop manages security (use of kerberos, delegation tokens, how they are renewed, implicit renewals, etc) - it is not a standalone security library. It also includes implementation hooks for hadoop ipc, shared state for hadoop services, implementation idioms, etc.
Depending on hadoop api as a library (codec, encryption, etc) is different from depending on it to define how spark manages security. Having said that, we do depend on hadoop specific api's exposed in spark - but they are for exposing corresponding hadoop functionality and is also usually historical in nature (use in SparkContext, RDD, etc for example).
In yarn resource manager, it makes logical sense to use hadoop security - since spark becomes a yarn service. In mesos or other non hadoop based schedulers, I am not sure it does (other than implementation simplicity in case they follow same security paradigm as hadoop).
Depending on hadoop security in core for spark security should be evaluated on its merits and we should be cognizant of the implications of doing so - implementation simplicity should not be the only driving factor.

In yarn resource manager, it makes logical sense to use hadoop security - since spark becomes a yarn service

No. Spark is a YARN client. Not a service.

In mesos or other non hadoop based schedulers, I am not sure it does

The part you're missing is that it's not YARN that's mandating the use of UGI / delegation tokens for security. It's HDFS, Hive and HBase, all of which are perfectly valid services to use in a Mesos cluster (or any other cluster manager for that matter). So any solution for this problem that claims to support those services needs, at some point, to use UGI APIs.
It's one thing to avoid exposing UGI APIs through Spark APIs. That might be a good thing to do. But you just cannot avoid the use of those APIs if you want to support those services that are based on the Hadoop security APIs.

Depending on hadoop security in core for spark security should be evaluated on its merits

The main merit is that you cannot connect to secured Hadoop services without using those APIs. Unless you know something I don't know.
UGI is also already used in core for some security-related features. So it's not like it's being "added".

Hey @vanzin @mridulm.  Sorry for joining the party a bit late.  I just read through the discussion.
It seems the first point of contention is the distinction between Hadoop and YARN.  This PR relies on Hadoop libraries in core, but it shouldn't rely on yarn.  If it is, that's a mistake and I should fix that.
Then the discussion becomes whether we should rely on Hadoop in core.  It looks like @mridulm acknowledges we're already using Hadoop in core, so I hope we agree that this PR doesn't create a new problem, but that it does increase the coupling.
I agree that, ideally, all Hadoop-specific code would be factored out into a separate hadoop module.  But, as @vanzin points out, doing so would be a massive undertaking.  Spark identity and access control is based on Hadoop security (UserGroupInformation), and Hadoop filesystems are exposed through HadoopRDD.  We'd have to, at minimum, create an entirely new Spark access control interface for which UGI is just one provider.
And also as @vanzin points out, ultimately, there's no way to get around the requirement of using Hadoop security libraries such as UGI if our goal is to access Hadoop services.  Hadoop services require Hadoop delegation tokens, rather than some more broadly applicable security standard.  And I hope we agree we don't want to duplicate the UGI client code in both the yarn and mesos module (sharing that client code was the whole motivation of this PR).
So the only alternative I see would be to create a separate hadoop module, place all this code there, and create new interfaces that Hadoop-specific code would implement.  One obstacle to that is the massive amount of work.  The other is that I'm not a huge fan of creating interfaces when we only have on implementation, since you often end up with the wrong interface, so you have to rewrite it anyway.
So my proposal is that we acknowledge that splitting out all Hadoop code from core and placing it behind a common interface should be done at some point, but we wait to do it until we at least have some other security provider that would implement that interface.
BTW, I'll definitely go back and ensure that no Hadoop interfaces are publicly exposed in core.

There is a distinction between what spark core implements and exposes - and what the actual SPI implementations depend on. My point is that spark core should not depend on hadoop-security; since we could need to support other models.
The actual implementation which authenticates/authorizes for specific hadoop services (hdfs, hive, hbase, etc) could be leveraging UGI + kerberos principal/keytab + corresponding service token api, etc. That is an implementation detail of the credential provider - and is decoupled from spark core itself. This PR is bundling the SPI impl's also as part of core, but that is a packaging detail.
The lifecycle of a credential provider, acquisition of credentials (tokens, DT, etc), distribution and eventual application to executors should not depend on hadoop-security from spark core's point of view: specific implementations could be doing ugi.getCurrentUser.addCredentials(creds), just as some other impls might be doing something else.
In my earlier comments, I am not sure if this distinction came across.

It would be easier if instead you said what particular API that is being proposed here you have issues with. Is it the storing of credentials in UGI? Or what?
If that's what you're saying. I don't see why it's a problem to store credentials, even non-Hadoop ones, in UGI. It's just a container. Just like HDFS looks at UGI and only fetches the credentials it needs, some app blah that wants to hook up to Spark would do the same and look for its credentials there. I don't see a ton of benefits in abstracting that away into yet another security abstraction layer.
And if for some reason that doesn't work, well, how do we know? Unless you have a concrete example, it's hard to develop an interface to satisfy an unknown use case.

@vanzin Looking at it from view of specific API usage, while relevant for analyzing implementation, is not the whole picture. Is hadoop-security model sufficient for spark ? Will spark need to support others as well - and if it does, will our design allow for it ? [1].
A specific implementation of what we expose will be dependent on hadoop-security (existing credential providers).
In order to make progress here, let me highlight a few areas where hadoop-security is used explicitly or implicitly in current code.

Pre-requisites - principal/keytab : they are specific to use of kerberos in hadoop-security.
Environment in which credential provider's are invoked.

Expected to be within ugi returned via ugi.loginUserFromKeytabAndReturnUGI
Current credential providers assume this as impl detail (for example hbase - perhaps others too).

Use of o.a.h.security.Credential in our interface.

The api is specific to hadoop-security - and assumes a model of secret keys and TokenIdentifier.
The latter might be non trivial to adapt for non hadoop-security implementations.

Application of acquired tokens at executors/driver.

For hadoop-security, this becomes ugi.getCurrentUser.addCredentials(Credentials)
For others, it could be something specific to its implementation.

I am assuming token distribution should not be an issue - though we might have multiple implementations for it.
We are also assuming a model of periodic renewal of credentials (based on expiry) - are there other models we would need to support ?
[1] Having said this, I am not suggesting that we must not use hadoop-security as basis for introduction of this feature into spark core : if we find the model suitable for spark's needs (present and future), that is great. I did not see an analysis being done here on that regard - If I missed it (likely), please point me to relevant discussion.

Hi @mgummelt, hopefully I have clarified some of my thinking above. Responding to specific points below.

It seems the first point of contention is the distinction between Hadoop and YARN. This PR relies on Hadoop libraries in core, but it shouldn't rely on yarn. If it is, that's a mistake and I should fix that.

My query was not regarding the actual credential provider implementations themselves (which will require their dependencies), but whether spark core needs to depend on the api.
Put another way, suppose we moved credential provider implementations into a separate module - will spark core still need to depend on this or not ?
@vanzin's made the point that since we depend on hadoop-client, which depends on hadoop-security - this does not matter anymore :-)

Then the discussion becomes whether we should rely on Hadoop in core. It looks like @mridulm acknowledges we're already using Hadoop in core, so I hope we agree that this PR doesn't create a new problem, but that it does increase the coupling.

Hopefully I covered this in my earlier comments.
I was not revisiting use of hadoop in core (that is pervasive in spark), but whether hadoop-security model is sufficient for what we are attempting.

And also as @vanzin points out, ultimately, there's no way to get around the requirement of using Hadoop security libraries such as UGI if our goal is to access Hadoop services. Hadoop services require Hadoop delegation tokens, rather than some more broadly applicable security standard. And I hope we agree we don't want to duplicate the UGI client code in both the yarn and mesos module (sharing that client code was the whole motivation of this PR).

Definitely agree on not duplicating code !
Paraphrasing what you mentioned earlier and elaborating, I am trying to understand if:

We can assume hadoop-security is sufficient for our usecases.

In this case, we leverage existing implementations as-is (more or less) - and can expose hadoop-security in our interface definitions (traits, execution environment, how we use the credentials).

hadoop-security becomes one supported model (and currently only one). For example, model definition could be:

Defining pre-requisites (principal/keytab, external credential update, etc).
environment setup (UGI. loginUserFromKeytabAndReturnUGI.doAs, etc currently used).
Application of acquired credentials (UGI.getCurrentUser.addCredentials) at executors and driver.
credential provider's declaring which model they are for.

Perhaps some other solution (synthesis of the above ? new ?)

So the only alternative I see would be to create a separate hadoop module, place all this code there, and create new interfaces that Hadoop-specific code would implement. One obstacle to that is the massive amount of work. The other is that I'm not a huge fan of creating interfaces when we only have on implementation, since you often end up with the wrong interface, so you have to rewrite it anyway.

I share your concerns here ! Creating incorrect interface we get stuck with, a single implementation causing our interfaces to be very specialized, potentially over-designing.
I am trying to understand if what we have is sufficient or do we need to look more closely at our dependency on hadoop-security.
I am not very familiar with mesos or kubernetes, but a cursory search indicated they have other forms of authentication ? If yes can you comment if, with the current model, will mesos be able to evolve to support others ?
Since I do not have sufficient compelling examples to give, I am unable to convince @vanzin :-)

So my proposal is that we acknowledge that decoupling all Hadoop code into a hadoop module and placing UGI behind a common interface should be done at some point, but we wait to do it until we at least have some other security provider that would implement that interface.

Since we are exposing credential providers as an api from core, we will need to support it.
If it is possible to support other models in future without breaking our exposed interfaces - that is something which would be an excellent way forward too.

@mridulm the main argument for just dealing with Hadoop security is that it's been sufficient since the inception of Spark. I have never seen anyone ask for integration with any other type of system. Most of the non-Hadoop systems supported by Spark don't need this kind of functionality (e.g. S3 just uses secrets stored in configuration or a credential store - both of which are handled by the Hadoop libraries btw, not Spark).
That being said, I think this whole discussion is distracting from the main goal of the change, which is to expose Hadoop-based security in Mesos-based applications. Would it make you more comfortable if the new API were kept private[spark]? It would limit extensibility in the case of Mesos (would be restricted to built-in providers), but would free us from this discussion and allow progress to happen.
Really, personally, unless someone can given me a concrete example, I don't see a pressing need to have a lengthy discussion about non-Hadoop-based security frameworks.

My point is that spark core should not depend on hadoop-security;

@mridulm You keep mentioning hadoop-security as if it's a library.  It's not.  UserGroupInformation and Credentials, for example, are are security classes in hadoop-commons, which core already depends on.  So this coupling already exists.  Is your concern that we're increasing this coupling?

@vanzin:

@mridulm the main argument for just dealing with Hadoop security is that it's been sufficient since the inception of Spark. I have never seen anyone ask for integration with any other type of system.

Support for long running applications (which require token renewal, etc) was added much later in spark - in 1.4 IIRC : https://issues.apache.org/jira/browse/SPARK-5342

Would it make you more comfortable if the new API were kept private[spark]? It would limit extensibility in the case of Mesos (would be restricted to built-in providers), but would free us from this discussion and allow progress to happen.

If we are not exposing an api for spark core, while maintaining backward compatibility - I am fine with the change. (Please see [1] below too)
Either we move implementations from yarn to core - or a new module which yarn and mesos depends on (if that helps).
@mgummelt:

@mridulm You keep mentioning hadoop-security as if it's a library. It's not. UserGroupInformation and Credentials, for example, are are security classes in hadoop-commons, which core already depends on. So this coupling already exists. Is your concern that we're increasing this coupling?

When I refer to hadoop-security, I do not mean a maven package - but use of org.apache.hadoop.security for handling authentication/authorization. hadoop-common contains a collection of libraries & utilities, bundled together for convenience; security package implements classes in context of security design of hadoop.
[1] If we want to base security in spark on hadoop-security, and think it is sufficient for our current & anticipated needs -  we should be explicit about the (design) dependency.
We should solicit opinion about it in dev@ and proceed based on feedback from from the list (this PR discussion might not be followed by many interested parties).
As @vanzin mentioned above, there have not integration requests for other systems - so perhaps it is sufficient for our needs and I might be being overly paranoid (due to my past experiences with api design).

Support for long running applications (which require token renewal, etc) was added much later in spark

That's different and not what this change is about. Support for Hadoop security (i.e. delegation tokens) has existed at least since Spark 1.0 (I'm hazy before that since that's when I started playing with Spark). And it doesn't change the fact that it's the only custom security framework that people have ever tried to use with Spark, as far as I'm aware.
Hadoop security is different and puts a lot of burden on clients to do things; there are good reasons for that, but it means that it's not as simple as just providing a password. I wish there was a library that made all this easier (wouldn't it be great if there was a single service to contact and ask for "delegation token for service X", like the Kerberos TGS?), but that's not the case.
I can think of 3 other types of systems that Spark supports (directly or through extensions):

Those with no security. e.g. Kudu (as far as I know that's still in the roadmap), Kafka 0.8, etc
Those that are happy with just a simple secret stashed somewhere. e.g. S3, JDBC drivers, etc. Even though I've never seen it, I also count cert-based authentication here, since I'm pretty sure you can achieve that with existing features in Spark.
Systems that implemented Kerberos-based auth but not Hadoop delegation tokens. (Looking at you, Kafka 0.10.) That means it's really hard to use those services in a distributed environment where security is enabled.

None of those require code in Spark to handle things specially (the third would, but then you'd run into the good reasons why delegation tokens exist in the first place, so they really should start using them instead).

If we are not exposing an api for spark core, while maintaining backward compatibility

I guess I'm a little less queasy than you are about exposing an unstable API. That's what the "Unstable" annotation means to me. It's an API that is still being designed, and exposing it serves as both a way to let people write extensions that fit the model and also collect feedback about things that don't fit well.
I have issues with the things you're suggesting for a few different reasons.

keeping the API private means people won't extend it, so we won't get feedback if there's any.

moving the API to a separate module is a distinction without a difference. It will still be a public Spark API, and still should follow the rules of backwards compatibility. It would just increase coupling since it's very unlikely that core wouldn't call into that module (since many people have asked for Hadoop auth support in standalone too - I have some issues with the security model there but it's a separate discussion).

trying to work on an abstract interface to rule them all is a wild goose chase. Unless you can point me to the contrary, we don't have an example of what a different system would look like, so whatever abstract interface we end up with will still be heavily modeled after the Hadoop system. Not exposing Hadoop types is not a great gain if the whole mechanism still works like Hadoop security. (It makes Spark's handling of backwards compatibility easier, probably, but here the model is more important than the types exposed in the API.)

Yes, exposing an unstable interface risks more work in the future. We may have to change it, and then we have to decide whether to write code to keep compatibility, or break all users. It's a risk, but at the same time, we have a few years of only needing to support this model, so it doesn't seem like it's that big of a risk to me.
In the case that this elusive other system shows up in the future, we'll probably have a lot more issues; we could try to merge both APIs or just handle it with a completely separate one. In either case, there will be work. So I'm really not seeing the benefit of going out of our way to mitigate future work. It's better to do that work when we have a better idea of what it even looks like.

@vanzin Thanks for the details and examples, that definitely clarifies things better for me !
I agree with you that opening it up, even if as unstable api, would be better than keeping it private[spark] (hence why I started down this tangent to begin with :-) ).
My main concern, as I mentioned before, has been whether we would need to support auth models different from hadoop security or not. If I understand from @mgummelt, mesos does not require it - but I am unsure about kubernetes (perusing their documentation left me unsure) or any others.
As I mentioned before, would be good to atleast solicit opinions on the dev@ list on this.

If I understand from @mgummelt, mesos does not require it

Mesos has its own authn/authz system, but it has nothing to do with this PR: https://github.com/apache/spark/blob/master/resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala#L82

I also have to admit that even after all this discussion, I'm still unclear what the concern is.  Hadoop security classes, such as UGI are already exposed to users.  For example, through --proxy-user.
Also, even if ServiceCredentialProvider, which depends on hadoop, is a public interface, that doesn't mean we can't support non-hadoop security.

but I am unsure about kubernetes (perusing their documentation left me unsure) or any others

That's kinda orthogonal. This PR isn't Mesos-specific. Mesos, AFAICT, does not support Kerberos (or so I've gathered from Michael's comments on JIRA and here). But that has no bearing on whether applications running on Mesos can talk to Hadoop services that use kerberos. So, this is purely about handling Hadoop authentication for Hadoop services.
The authentication used to launch the application on Mesos has nothing to do with the auth used to talk to HDFS or other services. The same applies if you replace Mesos with any other cluster manager.

Do we need to generalize ServiceCredentialProvider to support non-hadoop delegation tokens?

I don't see a need for that.

@vanzip wrote:

So, this is purely about handling Hadoop authentication for Hadoop services.

This was my point - we should not introduce system specific api's into spark core infrastructure api's/spi's : unless
a) we have explicitly based our support on it, or
b) generalized it sufficiently that we can support others, or
c) keep it an impl detail in core (but exposed in yarn for backward compatibility).
IMO (a) or (b) require a dev@ discussion.
Until now, this (hadoop security) support was restricted to yarn in spark (with a couple of minor other uses iirc).
@mgummelt:

The only thing the new ServiceCredentialProvider interface enforces is that the credentials must be added to a Credentials object, which is a hadoop class.

The spi makes assumptions about the environment within which the credential provider is invoked, how  the tokens are updated at driver/executor as well in addition to use of Credentials - and these are driven by hadoop security DT design.

Do we need to generalize ServiceCredentialProvider to support non-hadoop delegation tokens?

IMO that depends on what the answer to the design choice above is.
If (a) or (c) - then no.
If (b), then yes.

This was my point - we should not introduce system specific api's into spark core infrastructure api's/spi's

Sorry, I still have no idea what your point is. How do you suggest we support Hadoop security without having code to do it? How can a Spark application running on Mesos talk to secure HDFS without this code?
Please explain that, because so far, I'm really confused about exactly what you think this change is about.
Adding support for Hadoop security does not preclude adding support for other systems.

a) we have explicitly based our support on it

What does this mean?

To ask a more direct question:
The only public interface being added in this change is ServiceCredentialProvider. It's an interface that service-specific libraries (e.g. a Solr connector, or a Kudu connector) would extend, and user applications would never touch. It's also an existing interface, which was in the YARN module but, really, has functionality that is independent of YARN.
So the question is: why is it a bad thing to expose this interface more widely? It serves a specific purpose (provide a way for services that use Hadoop-style security to integrate with Spark applications). It doesn't dictate that all Spark code must use Hadoop delegation tokens, it just allows Spark applications to better integrate with secure Hadoop services.
And, on top of that, I don't see a point in abstracting it further. It would just be abstraction for the sake of abstraction. If a different class of secure services requires something like that at some point, we can evaluate then what to do. But right now, that class of services does not exist.

The only public interface being added in this change is ServiceCredentialProvider. It's an interface that service-specific libraries (e.g. a Solr connector, or a Kudu connector) would extend, and user applications would never touch. It's also an existing interface, which was in the YARN module but, really, has functionality that is independent of YARN.

As I detailed above, the exposed spi contract is not just the interface, but also how it is to be used by spark.
It includes the environment obtainCredentials is invoked within, use of Credentials as container for tokens and how Credentials are leveraged in driver/executor.
This, in entirety, defines the public contract - not just the methods in the interface : and this is dependent on how hadoop-security authenticates/authorizes services.

And, on top of that, I don't see a point in abstracting it further. It would just be abstraction for the sake of abstraction.

It would ensure that spark is not tied to hadoop security, but can continue to leverage it for managing long running applications/services.
Once interfaces are exposed, and have been exposed for a few releases, it is exceedingly difficult for users if we break/modify them - irrespective of the stability annotation's we add.
A straw man pseudocode proposal which elaborates is:
@Unstable
// rename
trait BaseServiceTokenProvider[C, T] {

  // Must match a supported ServiceTokenManager.serviceType
  def serviceType: String

  def serviceName: String

  def credentialsRequired(conf: C): Boolean

  def obtainCredentials(conf: C, tokenContainer: T): Option[Long]
}

// Existing ServiceCredentialProvider becomes
@Unstable
trait ServiceCredentialProvider extends BaseServiceTokenProvider[Configuration, Credentials] {
  override def serviceType: String = ServiceTokenManager.HADOOP_SECURITY_SERVICE_TYPE
}

// Modify ConfigurableCredentialManager to segregate by serviceType of ServiceTokenProvider
// which will be queried by ServiceTokenManager. If unknown serviceType, then log error.

@Unstable
trait ServiceTokenManager {

  val HADOOP_SECURITY_SERVICE_TYPE = "hadoop-security"

  def serviceType: String

  // Return false if validation fails. Invoked in driver.
  def configure(conf: org.apache.spark.SparkConf): Boolean

  // Invoked in driver. If required, application of tokens to driver to be done as part of
  // acquireTokens
  def acquireTokens(manager: ConfigurableCredentialManager): Array[Byte]

  // Invoked in executor
  def applyTokens(data: Array[Byte])
}

// Existing implementation will be encapsulated within this class.
private[spark] class YarnServiceTokenManager extends ServiceTokenManager {

  override def serviceType: String = HADOOP_SECURITY_SERVICE_TYPE

  // In driver
  override def configure(conf: SparkConf): Boolean = {
    // save principal, keytab; create freshHadoopConf
    // configure
    true
  }

  override def acquireTokens(manager: ConfigurableCredentialManager): Array[Byte] = {
    val ugi = UGI.loginUserFromKeytabAndReturnUGI(principal, keytab)
    val tempCreds = ugi.getCredentials
    ugi.doAs {
      manager.getCredentialProviders(serviceType).flatMap { provider =>
        // existing code in ConfigurableCredentialManager.obtainCredentials
      }.foldLeft(Long.MaxValue)(math.min)
    }

    // apply it to driver
    UGI.currentUser.addCredentials(tempCreds)

    // serialize tempCreds to Array[Byte]
    serialize(tempCreds)
  }

  override def applyTokens(data: Array[Byte]): Unit = {
    // deserialize data into Credentials; apply it to UGI.currentUser
  }
}

// Spark core itself becomes decoupled from hadoop-security with the above.
// It also allows us to add other models if applicable.
// Changes to driver to leverage above would essentially be:

// a) load ServiceTokenManager's via ServiceLoader.

// b) create CredentialManager - which loads BaseServiceTokenProvider via service loader.
// b.1) Reject any token provider which has a service type not loaded in (a)
// c) configure (a)

managers.foreach(_.configure(sparkConf))

// token acquisition
val acquiredTokenMap = managers.map(manager => (manager.serviceType, manager.acquireTokens(credentialManager)))

// serialize and send acquiredTokenMap to executors.

// At executors, application is:

acquiredTokenMap.map {
  case (managerType, tokenData) => serviceTokenManagers(managerType).applyTokens(tokenData)
}

This ensures that future requirements are reasonably captured without any implementation specific changes required to the interface or behavior.

Based on the PR title Move kerberos delegation token code from yarn to core, it sounds like it moves the whole delegation token logics into core. I found it does not move the logics for pushing and propagating delegation tokens. Does that mean Mesos plans to use a different solution for delegation token management?

As I detailed above, the exposed spi contract is not just the interface, but also how it is to be used by spark.

There's very little lee way on that front. Those semantics are mostly already defined by Hadoop security; as I said, you can try to create some abstraction on top of that, but it would be abstraction for the sake of abstraction: the Hadoop implementation would basically be exactly this code, and we don't have an example of any other types of services to see if the abstraction actually makes sense for them.

It would ensure that spark is not tied to hadoop security

There's nothing tying Spark to Hadoop security here. There's code that allows Spark to use Hadoop services with security enabled, but that does not preclude other types of services from being used. I've given many examples of that already.

Once interfaces are exposed, and have been exposed for a few releases, it is exceedingly difficult for users if we break/modify them

Then we make an effort to not break them. Adding an abstraction that does not abstract anything useful does not help with that. If anything, it makes things worse.

// Modify ConfigurableCredentialManager to segregate by serviceType of ServiceTokenProvider

See, you're already making assumptions that these other systems use credentials in a certain way.

override def acquireTokens(manager: ConfigurableCredentialManager): Array[Byte] = {

Now you're exposing ConfigurableCredentialManager as a public API, which it was never meant to be.

// Spark core itself becomes decoupled from hadoop-security with the above.

It doesn't. You just created an interface that mimics the Hadoop security interfaces, and if you change the semantics of those, you'll end up breaking Hadoop security.

This ensures that future requirements are reasonably captured

No, you just created an interface that is Hadoop security without Hadoop types. That does not take into account any other service type.
So, again, abstraction for the sake of abstraction. It is much simpler if instead of trying to do this, you leave them as separate concerns. The Hadoop-based security interface exposes Hadoop semantics. If later there's a different type of service, it will probably need to expose a different public interface for services to provide their credentials.
Then, whether internally to Spark those two can be handled similarly (so that code can be shared) is a different question that can only be answered when those services are known.

// Invoked in executor
def applyTokens(data: Array[Byte])

This is maybe the only good idea I see in your proposal, but at the same time, it's already internally handled by Spark with the current code and doesn't need to be exposed to the credential provider implementations.
I really don't understand what the abstraction is buying anybody at this point. If you don't know what other types of services you want to support, by definition you can't create an abstraction. So, as I pointed out, your abstraction looks exactly like the only type of service it does support - Hadoop. Which makes it kinda pointless as an abstraction.

@gatorsmile

Does that mean Mesos plans to use a different solution for delegation token management?

There's still Mesos code to be added for this; if you trace back through the JIRA links, Michael submitted a PR earlier that had both this change + the Mesos code, I just asked him to separate the two.
At the end, both YARN and Mesos code will call into these APIs to create delegation tokens; there might be some extra code needed in Mesos, since YARN already has built-in support for managing the initial token set (distribution and renewal). But creating and distributing new delegation tokens after the TTL of the initial set should be the same for both.

@gatorsmile Yes.  This PR originally contained the Mesos propagation code as well, but I split it out to simplify the review process (can't you see how simple it's become!) :)
The old PR is here: https://github.com/apache/spark/pull/17665

I see. Will try to catch the whole history. Thanks! @mgummelt @vanzin
Also cc @jerryshao  I just read his design doc. It sounds like he also expect moving ServiceCredentialProvider and ConfigurableCredentialManager  from Yarn to core for supporting his proposed Credentials push command.

@vanzin @mridulm
I think we've reached the crux of the disagreement.  @mridulm thinks we should generalize ServiceCredentialProvider.  @vanzin and I think this would do more harm than good since we don't have another delegation token provider to generalize against.
You two might know the PR protocol better than I do.  How can we make progress?  Does @mridulm get a -1 (veto) on this PR?  If so, I'm willing to just write the generalization in order to make progress, unless of course @vanzin is a -1 on that.  Should I open this up to dev@?  Maybe add a deadline for comments and see where we are after that?  What if we're still split?
To complicate matters even further, I'll be on vacation next week.  Feel free to continue discussions, but I'll be absent until May 15.

Technically anyone can -1 a change, AFAIK. In practice I see that done very rarely.
To clarify my position, I see it as counter-productive to try to create an abstraction now. We don't know what this abstraction would look like, so as Mridul's example proves, the abstraction is basically just Hadoop-security-without-Hadoop-types. We don't know if that's going to be useful to anyone else, which makes its standing as an abstraction questionable.
Exposing Hadoop security directly at this point has a single drawback I can see: potentially exposing Hadoop types in the API. That's already done in a bunch of parts of core, so I don't see that as a strong argument. And hiding those from the API doesn't really hide them from the implementations; the Hadoop credential providers will still deal with UGI and Credentials internally, and since everybody is running under the same class loader, they'll be the same classes. So there's still tight coupling, just not explicit in the API.
Also, exposing Hadoop security this way does not preclude adding an abstraction (like the one Mridul sketched out) later. Assuming that there exists a class of services that has a very similar requirement for how to create and distribute credentials, you just add the abstraction at that point, and have an implementation of the new interface that wraps the old Hadoop interface. It's pretty easy to do, and avoids having to figure out what this hypothetical interface would look like right now.

I didn't read through the super long debate here, but I have a strong preference to not expose Hadoop APIs directly. I'm seeing more and more deployments out there that do not use Hadoop (e.g. connect directly to cloud storage, connect to some on-premise object store, connect to Redis, connect to some netapp appliance, connect directly to a message queue or just run Spark on a laptop).
Hadoop APIs were designed for a different world pre Spark. Serialization is painful (Configuration?) to deal with, API breaking changes are painful to deal with, size of the dependencies are painful to deal with (especially considering the single node use cases in which ideally we'd just want a super trimmed down jar).
As you can see (although most of you that have chimed in here don't know much about the new components), the newer components (Spark SQL) does not expose Hadoop APIs.

I didn't read through the super long debate here, but I have a strong preference to not expose Hadoop APIs directly. I'm seeing more and more deployments out there that do not use Hadoop (e.g. connect directly to cloud storage, connect to some on-premise object store, connect to Redis, connect to some netapp appliance, connect directly to a message queue or just run Spark on a laptop).

Are you saying exposing the Hadoop API in a public interface (like ServiceCredentialProvider) would make it more difficult to integrate with non-Hadoop data stores?  How so?

I'm saying avoid exposing Hadoop APIs. Wrap them around something if possible.

@vanzin I agree with you, but it looks like the best way to make progress is to generalize the interface.  Are you OK with this?

I'm saying avoid exposing Hadoop APIs. Wrap them around something if possible.

If that's all you'd like to see, it's not hard. But at the same time, it doesn't solve a whole lot of problems; as I mentioned, both the implementation of the provider and the Spark code calling into the provider have to use Hadoop APIs, so they still have to agree at some point.
To try to give a more concrete example: let's say that we hide Hadoop types, and now some credential provider uses a UGI API that's only available in Hadoop 2.9, and someone tries to run it on Spark built Hadoop 2.6. Even though Spark hasn't exposed the Hadoop API, the dependency leaks, and things will still not work.

it looks like the best way to make progress is to generalize the interface. Are you OK with this?

If masking the Hadoop types in the API make people happy, sure, do that. I don't see much benefit, since the implementation will still be pretty coupled to Hadoop one way or another, but it's not a big change from the current change anyway.
I just don't want this to turn into "let's solve the problem of how to support all the security frameworks out there", when we have no idea of what other security frameworks we even might want to support.

OK, I'll submit an updated version once I get back from vacation around May 15th.

@vanzin:

// Modify ConfigurableCredentialManager to segregate by serviceType of ServiceTokenProvider

See, you're already making assumptions that these other systems use credentials in a certain way.

serviceType defines which ServiceTokenManager to use for configuring/managing the BaseServiceTokenProvider loaded - there is nothing specific about how credentials are used.
Probably should have been more detailed in my proposal, apologies for the confusion - detailing it below:
We can do it two ways :
a) I was initially envisioning ConfigurableCredentialManager will essentially be doing only service loader lookup for BaseServiceTokenProvider and partitioning the loaded BaseServiceTokenProvider's by serviceType.
All other existing methods (iirc token acquisition ?) will move into YarnServiceTokenManager.
The only reason to do this would be for each ServiceTokenManager to not need to do service load.
b) Eliminate ConfigurableCredentialManager entirely, and have each ServiceTokenProvider load their specific sub-interface of BaseServiceTokenProvider; which gives better type safety.
And perhaps also eliminate the confusion above ?

override def acquireTokens(manager: ConfigurableCredentialManager): Array[Byte] = {

Now you're exposing ConfigurableCredentialManager as a public API, which it was never meant to be.

See above if we dont want to expose ConfigurableCredentialManager.
As I mentioned above, other than as a container for BaseServiceTokenProvider, ConfigurableCredentialManager does not really do anything else now.
The token acquisition is an implementation detail of the specific ServiceTokenProvider and will move into that.

This ensures that future requirements are reasonably captured

No, you just created an interface that is Hadoop security without Hadoop types. That does not take into account any other service type.

The life cycle is fairly generic IMO, and actually matches general SPI patterns - not sure why you think it is specific to hadoop-security, probably I am missing something ?
It currently does :

configure/initialize
invoke
apply

Perhaps you refer to the specificity that configure and invoke happens in driver and application in executor ?

I really don't understand what the abstraction is buying anybody at this point. If you don't know what other types of services you want to support, by definition you can't create an abstraction.

The abstraction defines a general lifecycle of how spark expects ServiceTokenManager to conform to - without tying it to any specific implementation.
If the concern is that it requires more iteration/evolution, I am fine with that.
Bottomline is, unless there is consensus that we want to adopt hadoop security as the model for authenticating/authorizing not just existing - we should not be dependent on it.

How can we make progress? Does @mridulm get a -1 (veto) on this PR?

Veto is an extreme measure, and not my intention :-)
It is not like I dont understand where @vanzin is coming from, and I agree with him for the most part - and he probably understands hadoop security way better than I do.
As @vanzin mentioned, I am just more queasy about interface change, even if marked Unstable :-)

To complicate matters even further, I'll be on vacation next week. Feel free to continue discussions, but I'll be absent until May 15.

Ah, looks like it is not just me who will be leaving for vacation soon !

b) Eliminate ConfigurableCredentialManager entirely, and have each ServiceTokenProvider load their specific sub-interface of BaseServiceTokenProvider; which gives better type safety.

This may solve that particular problem, but why? What are you gaining from it? What if you never have to implement any different credential provider, because there's no need to?
That's my main problem with your suggestion. You're trying to create something generic without any specific need for it other than wanting to create something generic. You're assuming that at some time in the future this generality will pay off, but you don't know.
On the other hand, there's a lot of simplification that can be achieved by kicking the "let's create something generic" can down the road.

The life cycle is fairly generic IMO,

But it's just a small piece of the puzzle. For example, for Hadoop token providers, obtainCredentials has to be called with a very specific environment set up depending on when it's called. For example, when creating new delegation tokens to work around the 7-day TTL, a new UGI has to be created by logging in with the user provided keytab, or no new delegation tokens will be created (because the currently active UGI still has valid tokens).
So you've abstracted some aspects of how the mechanics of shipping tokens back and forth works, but you haven't abstracted enough that it's possible to write this feature without baking in logic about how Hadoop security works.
What I'm trying to say is that it's a futile goal to try to create this generic interface now. You're going to end up with either of the following:

an overly generic interface, and make Hadoop token providers (which are the very thing we want to support!) way more complicated to code, since all the semantics that are now in the internal Spark code would be pushed to the providers. I really don't like this, because to be useful the interface would become more complicated, so that all things related to Hadoop are exposed in some way (e.g. "here's the kerberos keytab so that you can create your own UGI so that you can create new delegation tokens").

a generic interface with a single Hadoop-specific implementation, which exposes its own interface so that Hadoop token providers can avoid having to re-implement all of the Hadoop-specific semantics. I assume this is what you mean with your (b) suggestion I quoted above.

I really don't like the first option, and I think it's premature to invest time in the second option.
So instead, what I'm proposing is to just expose the Hadoop-specific one. If it helps, change names to make it really clear that it's really specific to how Hadoop security works. Mask Hadoop names if desired, so that you have wrappers, and the API Spark exposes is not dependent on Hadoop (even though it is at runtime).
And then, when we have a second use case for something like this, then think of how you can merge the two with a generic interface, if that would be even possible. You will have a ton more context about what's needed at that time, instead of trying to guess it up front.

@vanzin

But it's just a small piece of the puzzle. For example, for Hadoop token providers, obtainCredentials has to be called with a very specific environment set up depending on when it's called.

On the contrary, this is handled if you read the proposal - ServiceTokenManager.acquireTokens will do the necessary setup : before acquiring tokens from BaseServiceTokenProvider's which match its service type.
To elaborate:

configure - grab principal/keytab/configure itself based on the SparkConf provided.
acquireTokens - setup environment specific to the model.

Execute BaseServiceTokenProvider.obtainCredentials
In case of hadoop security, obtainCredentials will be invoked within ugi setup; in case of other models, it could be something else - that is an implementation detail which must not leak out.
serialize result to Array[Byte] and return.

applyTokens - at the executor's - for application of the obtained bytes.

Each ServiceTokenManager would have different deserialized object representing tokens, and different semantics on how tokens are applied to the executor environment.

The whole point of generalizing is that existing functionality should continue to work, without needing to depend on specifics of hadoop-security.
Given @rxin's comment above, I am actually more convinced that taking a hadoop-security only route is detrimental to future integration with other systems - I just did not have examples to elaborate my point.

On the contrary, this is handled if you read the proposal - ServiceTokenManager.acquireTokens will do the necessary setup

Yes, that's approach (b) which I said still assumes that the interface you created is enough for other security systems. Which is a big if, since we don't know any of them.

Given rxin's comment above, I am actually more convinced that taking a hadoop-security only route is detrimental

Reynold mentioned that a lot of users don't need Hadoop security, which is true. But he didn't mention any other security system that actually needs anything like this, so that's not an argument for a super generic interface.
Which is what I've been saying all along. You don't know of any other system that needs anything like this, so you don't know if the interface you're creating will be useful to them or cover all their needs. If there the least bit of difference in how this hypothetical system works, your abstraction goes down the drain.
Instead, we have actual use cases for Hadoop security. They don't benefit from the abstraction you're trying to create at all, since internally they still need to deal with UGI and all the rest of the Hadoop security API. So why force the abstraction right now?
I'm not saying that abstraction are bad. I'm just saying that at this point they're completely unnecessary and avoidable, and I haven't seen any use case for them here. Give me a single use case and I'll agree with you, but I haven't seen one.

BTW, if @mgummelt is ok with spending time working on this super generic interface, then it's up to him. I find it unnecessary and a waste of resources at this point in time, but it's his time. :-)
At best it will prove useful, most probably it will forever live with a single Hadoop-specific implementation, and at worst we'll have to change it to accommodate a system we did not predict, which is basically the same risk we have with just exposing the Hadoop one in the first place.

I'm working on this now, and am definitely willing to execute the plan we've agreed on, but the more I think about it, the more I think it makes sense to make ServiceCredentialProvider private and not pluggable.
The benefit is that we don't have to design a possibly incorrect delegation token interface until we actually have another implementation to design against.
The cost is that Mesos users would not be able to provide their own delegation token implementations.  However, YARN users still would, since the yarn.ServiceCredentialProvider would still exist.  I would remove the deprecation.
@vanzin @mridulm Does this work for both of you?  We briefly considered this at the beginning of this PR conversation, and I think both of you suggested it's acceptable.
Out of curiousity, what was the benefit of making delegation token provision pluggable in the first place?  What other Hadoop delegation token providers other than HDFS, HBase, and Hive are there?  What was the be

@mgummelt We have in house delegation provider for HiveServer2, multi HBase cluster. I think this is useful in Hadoop world. So better to keep it.

@jerryshao Are those providers different than the Hive and HBase providers already in the Spark codebase?
Regardless, with what I'm proposing, the yarn.ServiceCredentialProvider would remain, so you would still retain the ability to plugin those providers for the YARN scheduler to use.
It's just the Mesos scheduler (or any other scheduler that uses core) that wouldn't support pluggable providers.  I'm not worried about this, because I haven't seen any demand from Mesos users for any provider other than HDFS, Hive, and HBase.  We can expose it once, if ever, this demand arises, but by keeping it private for now, we gain the option to redesign the interface if a new provider type emerges, which is one of the things @vanzin and I were worried about.  And we still wouldn't be exposing Hadoop types publicly, which is what rxin and @mridulm were worried about.

I'm fine with that if you're ok with Mesos being restricted to the built-in providers.

Test build #77064 has started for PR 17723 at commit 38adaae.

I have:

Made ServiceCredentialProvider private
Made ConfigurableCredentialManager load a hardcoded list of (HDFS, Hive, HBase) providers, rather than service loading
Renamed ServiceCredentialProvider to HadoopDelegationTokenProvider, a more specific name
Addressed all style comments

Ready for review.

Test build #77065 has started for PR 17723 at commit 92ac3f0.

Test build #77210 has started for PR 17723 at commit cd58b6c.

Test build #77210 has finished for PR 17723 at commit cd58b6c.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77210/
Test FAILed.

Merged build finished. Test FAILed.

Test build #77212 has started for PR 17723 at commit bf758e6.

Test build #77212 has finished for PR 17723 at commit bf758e6.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77212/
Test FAILed.

Merged build finished. Test FAILed.

Test build #77232 has started for PR 17723 at commit e820b09.

Test build #77232 has finished for PR 17723 at commit e820b09.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77232/
Test FAILed.

retest this please

Test build #77259 has finished for PR 17723 at commit e820b09.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77259/
Test PASSed.

@gatorsmile All comments addressed.  Ready for re-review.

@gatorsmile @vanzin @mridulm Any thoughts here?

I'm super busy with things as usual. But I hope to be able to take a look at this sometime this week.
I'm also sort of waiting for Mridul to chime in since he's the one that's super concerned about the API here...

Perfect, thanks.
@mridulm Any thoughts?  He might still be out on vacation...

Hi @mgummelt, I am quite tied up and will be back from vacation July mid - which is when I will have bandwidth for reviews (which is, IMO, too long to wait for this PR !).
I see that Wenchen Fan is already taking a look along with @vanzin, I will try to follow the discussions, but wont be able to participate unfortunately. Thanks for the great work !

Thanks @mridulm.  We'll move forward without you on this one.  Have a great vacation.

@vanzin All comments addressed.  Ready for re-review (after tests pass).

Test build #77683 has started for PR 17723 at commit 376dba0.

Test build #77683 has finished for PR 17723 at commit 376dba0.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77683/
Test FAILed.

Merged build finished. Test FAILed.

Test build #77686 has started for PR 17723 at commit 0ffe8f0.

Test build #77686 has finished for PR 17723 at commit 0ffe8f0.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77686/
Test FAILed.

Test build #77688 has started for PR 17723 at commit 7796e14.

Test build #77688 has finished for PR 17723 at commit 7796e14.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77688/
Test FAILed.

Test build #77761 has started for PR 17723 at commit 1479c60.

Test build #77761 has finished for PR 17723 at commit 1479c60.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77761/
Test PASSed.

@vanzin tests are passing.  ready for re-review.

Test build #77779 has started for PR 17723 at commit 4d57f7b.

Test build #77779 has finished for PR 17723 at commit 4d57f7b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77779/
Test PASSed.

You still haven't updated the PR description. It still references things that you have removed from the PR.
I tested the latest version with our kerberos tests and so far it looks good.

Thanks.  I updated the description.

Test build #77820 has started for PR 17723 at commit 7e2f90d.

Test build #77822 has started for PR 17723 at commit 563b80a.

Test build #77820 has finished for PR 17723 at commit 7e2f90d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77820/
Test PASSed.

Test build #77822 has finished for PR 17723 at commit 563b80a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77822/
Test PASSed.

Thanks! Will review it tomorrow.

Test build #77941 has started for PR 17723 at commit c684d88.

@gatorsmile I've expanded the description, and included a before/after class hierarchy.

Test build #77941 has finished for PR 17723 at commit c684d88.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77941/
Test FAILed.

retest this please

Test build #77946 has started for PR 17723 at commit c684d88.

Test build #77946 has finished for PR 17723 at commit c684d88.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77946/
Test FAILed.

retest this please

Test build #77953 has started for PR 17723 at commit c684d88.

Test build #77953 has finished for PR 17723 at commit c684d88.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77953/
Test FAILed.

@gatorsmile All comments addressed.  Ready for re-review.  Unit tests are failing on master, so I'm waiting for those to be resolved to get a passing build.

retest this please

Test build #78062 has started for PR 17723 at commit c684d88.

LGTM except one comment.
cc @vanzin @mridulm @cloud-fan @rxin

Test build #78062 has finished for PR 17723 at commit c684d88.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78062/
Test PASSed.

Merged build finished. Test PASSed.

Test build #78072 has started for PR 17723 at commit c4149dd.

Test build #78072 has finished for PR 17723 at commit c4149dd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78072/
Test PASSed.

Merged build finished. Test PASSed.

LGTM, merging to master.

w00t
Thanks everyone.


Property:bed

Definition
The full name of the lithostratigraphic bed from which the cataloged item was collected

Dimension

Existing property
dwc:bed

Existing class
Geological Context

Existing property identifier
https://dwc.tdwg.org/terms/#dwc:bed

Format
list

Required
No

Repeatable
No

Constraints

Examples
Harlem coal

Notes

Does a vocabulary exist?


Fix inconsistent state between WAL and saved Snapshot

Type of change

Bug fix

Description

It is a bug on etcd.
When an orderer has not participated in the consensus for some time and crashes during the process of writing the latest snapshot to the file, a panic error occurs after the restart.
This bug has been fixed in etcd, but not in fabric.
This PR fixes it into the fabric.

@Param-S @jiangyaoguo what is your opinion on this?

What's the progress for this PR?

What's the progress for this PR?

@Param-S and @tock-ibm are looking at it, please standby :)

Able to recreate the issue by terminating the orderer process just after saving walsnap entries and before saving snap file.
func (rs *RaftStorage) saveSnap(snap raftpb.Snapshot) error {

if err := rs.wal.SaveSnapshot(walsnap); err != nil {
return errors.Errorf("failed to save snapshot to WAL: %s", err)
}

terminate the process here

if err := rs.snap.SaveSnap(snap); err != nil {
return errors.Errorf("failed to save snapshot to disk: %s", err)
}

2022-08-18 04:04:19.431 PDT 031a PANI [orderer.consensus.etcdraft] loadState -> 5 state.commit 15 is out of range [0, 3] channel=test-system-channel-name node=5
> [unrecovered-panic] runtime.fatalpanic() /usr/local/go/src/runtime/panic.go:1065 (hits goroutine(1):1 total:1) (PC: 0x441b00)
Warning: debugging optimized function
	runtime.curg._panic.arg: interface {}(string) "5 state.commit 15 is out of range [0, 3]"

I will continue to go through the PR changes more and update here.

What's the progress?

I could not spend time on this last week. I will work on this next couple of days & confirm.

What's the progress for this PR?

What's the progress for this PR?

I'm sorry, but we're just too busy to review it.
We will get there eventually but unfortunately I cannot commit to a deadline.

@Mergifyio rebase

Trying to get checks re-triggered... let me try to Close and Re-open.

What's the progress for this PR? @Param-S

@zghh @Param-S
This one has been stale for some time, what is the status of it?
A few questions that may help to clarify the severity:
Why was this opened against release-2.2 instead of main branch? Is the problem resolved on main branch and release-2.5 already given the upgrade of etcdraft in those branches?
What is the overall impact after the problem occurs? The Description says a "panic error occurs after the restart". Will the panic occur after every subsequent restart? Is there any resolution of the problem, or the orderer node must be abandoned and a new orderer node created to replace it?
Has this problem been observed in practice?


_pickle.unpicklingerror: invalid load key, '<'.

I run the source code it gave, but i met the problem below.
How can i fix?

You need to download TRBA by yourself, dont use the code below.
Then you can run demo.py in Colab.


Create a Sponsors component for the fifth theme

As per provided designs, Create a Sponsors component for the fifth theme

Reference for the design: https://www.figma.com/file/0ISIz6XklEToDd4KXfIWRB/SCoRe-Lab-Theme-designs?node-id=3%3A4
Cc: @charithccmc, @dinithminura

I'd like to work on this issue next @Abhijay007.

@Abhijay007 I'm done with open PRs on other assigned issues. Can I start working on this next ? ( if it's ready to be implemented)

I would like to work on this issue. So can you assign this issue to me?


Use Parquet instead of CSV for better performance

Refactor the code so all the I/O related operations are encapsulated in data_engine
Convert CSV to Parquet with 2X saving in file size...Amazing
Create a utility function for ppl who've already downloaded historical data in CSV format, and they can easily use the utility function to convert all CSV to Parquet ease.

Fixed in e8808cc0f8ae971042eeb44cb6dc7a80cad4bbde

A quick comparision between CSV and Parquet

CSV
Parquet

Row-based storage format.
A hybrid of Row-based and column-based storage formats.

It consumes a lot of space as no default compression option is available. For example, a 1TB file will occupy the same space when stored on Amazon S3 or any other cloud.
Compresses data while storing, thus consuming less space. A 1 TB file stored in Parquet format will take up only 130GB of space.

Query run time is slow because of the row-based search. For each column, every row of data has to be retrieved.
Query time is about 34 times faster because of the column-based storage and presence of metadata.

More data has to be scanned per query.
About 99% less data is scanned for the execution of the query, thus optimizing performance.

Most storage devices charge based on the storage space, so CSV format means the high storage cost.
Less storage cost as data is stored in compressed, encoded format.

File schema has to be either inferred (leading to errors) or supplied (tedious).
File schema is stored in the metadata.

The format is suitable for simple data types.
Parquet is suitable even for complex types like nested schemas, arrays, dictionaries.

Credit: https://geekflare.com/parquet-csv-data-storage/


is empty include allowed?

Both the 1.0 and 1.1 versions of the specification state:

The value of the include parameter MUST be a comma-separated (U+002C COMMA, “,”)
list of relationship paths. A relationship path is a dot-separated (U+002E FULL-STOP, “.”)
list of relationship names.

It is unclear whether an empty value for include represents an empty list of relationship paths or a single (invalid) relationship path.

This seems to be unclear from the specification indeed. I think it should be considered as supported because otherwise a client would not be able to request to not include any resource.
The specification allows a server to include related resources by default:

An endpoint MAY return resources related to the primary data by default.

It allows a client to tell the server, which resources should be included:

An endpoint MAY also support an include request parameter to allow the client to customize which related resources should be returned.
[...]
If an endpoint supports the include parameter and a client supplies it, the server MUST NOT include unrequested resource objects in the included section of the compound document.

Using an empty include query parameter a client can tell the server to not include any related resources. Even if the server would do so by default.
For Sparse Fieldsets the meaning of an empty list is explicitly specified:

Sparse Fieldsets
[...]
The value of any fields[TYPE] parameter MUST be a comma-separated (U+002C COMMA, “,”) list that refers to the name(s) of the fields to be returned. An empty value indicates that no fields should be returned.

I think such a clarification should be added for include query parameter as well. At least if my interpretation of the specification is correct.

This seems to be unclear from the specification indeed. I think it should be considered as supported because otherwise a client would not be able to request to not include any resource.
The specification allows a server to include related resources by default:

An endpoint MAY return resources related to the primary data by default.

It allows a client to tell the server, which resources should be included:

An endpoint MAY also support an include request parameter to allow the client to customize which related resources should be returned.
[...]
If an endpoint supports the include parameter and a client supplies it, the server MUST NOT include unrequested resource objects in the included section of the compound document.

Using an empty include query parameter a client can tell the server to not include any related resources. Even if the server would do so by default.
For Sparse Fieldsets the meaning of an empty list is explicitly specified:

Sparse Fieldsets
[...]
The value of any fields[TYPE] parameter MUST be a comma-separated (U+002C COMMA, “,”) list that refers to the name(s) of the fields to be returned. An empty value indicates that no fields should be returned.

I think such a clarification should be added for include query parameter as well. At least if my interpretation of the specification is correct.


Question: How to reset my chart if the config parameters change at runtime

Hello
I have the realtime config setup and working fine.
I want to add functionality that a user could select the refresh interval of the chart. When this happens, I update the config in the chart object, but the chart is not reset.
So I am updating the duration, refresh and ttl in the following code but still the original duration and refresh rates are being used.
        this.chart.options.scales.xAxes.realtime = {             duration: 1000 * 60 * 60,             refresh: 1000 * 10,             ttl: 1000 * 60 * 60,             onRefresh: () => this.onRefresh(),             delay: 100         }
What should I do to get this to work?
Thank you!

Your code has two problems (assuming it uses chartjs-plugin-streaming 1.8 and Chart.js 2.8).

xAxes option is an array, so it should be xAxes[0]
The change won't be effective immediately without calling chart.update()

So, the following code should work as expected.
this.chart.options.scales.xAxes[0].realtime = {
  duration: 1000 * 60 * 60,
  refresh: 1000 * 10,
  ttl: 1000 * 60 * 60,
  onRefresh: () => this.onRefresh(),
  delay: 100
};
this.chart.update({duration: 0})


Update test and README about tsx compatibility

tsx has been compatible with other loaders since around the end of 3.x and the start of 4.x when the aforementioned issue https://github.com/privatenumber/tsx/issues/264 was resolved.
Update the test to use a newer version of tsx which does not exhibit the bug and the README to say tsx is compatible.

the failing test is likely due to the node upstream bug mentioned here https://github.com/iambumblehead/esmock/issues/304
thanks for the PR will merge and publish later today

I tried running the tsx tests different times since that issue was closed, but this is the first time I have seen them pass. There were other tsx issues that prevented tsx from working, even after url-params-issue was resolved.


Scaffold core element generates: "Failed to execute 'querySelector' on 'Element': '#packages/core_elements/src/core-scaffold/menu' is not a valid selector."

This issue was originally filed by<EMAIL_ADDRESS>
What steps will reproduce the problem?

 What do you see instead?
I would like to see something like:
http://www.polymer-project.org/samples/layout-elements/scaffold-app.html
But I see:
"Failed to execute 'querySelector' on 'Element': '#packages/core_elements/src/core-scaffold/menu' is not a valid selector."
FYI:
When I use icon in HTML file directly:
<core-icon-button id="menuButton" icon="menu"></core-icon-button>
It works.

Attachment:
project-without-packages.zip (2.91 KB)

This issue has been moved to dart-lang/polymer-dart#227.


within-docker time zone doesn't match local time

To reproduce:
From CLI:
>>> from utils import sp_utils
>>> sp_utils.convert_to_local_time(sp_utils.get_past_time())
'10/25/24 01:12AM'

From Docker container:
# python
Python 3.11.10 (main, Oct 17 2024, 18:57:41) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
# python
Python 3.11.10 (main, Oct 17 2024, 18:57:41) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from utils import sp_utils
>>> sp_utils.convert_to_local_time(sp_utils.get_past_time())
'10/25/24 08:11AM'

This is now addressed with an optional offset in the format_timestamp helper function.


"Failed to encode"

Describe the issue
I can not install a particular mod
To Reproduce

Go to https://store.gx.me/mods/ot1jje/akko-lavender-purple/
Click on 'Install with GXM'
It failed to encode

Expected behavior
The mod installs on my browser
Screenshots

System Information

OS: Windows 10
Browser: Firefox
Version: 124.0.1

sorry for the slow response time, i had seen the e-mail for the issue right as i was about to pass out (i was very tired)

seems like the server is forbidding the file from being downloaded, i'll look into this further rn

i don't think this is something i can fix on my end? i even tried opening it in the actual Opera GX browser and the requests still failed. it's probably misconfiguration on Opera's side, so i'm completely out of luck here 😓
nevertheless, i did a quick lookup and found out that someone's already extracted the files from the mod you wanted. you can download the files from this repo and create a local mod as a workaround.
sorry that this isn't really "solved", but i think i'm powerless for this one 😭

Thanks so much for checking this out and getting back to me!
Do you happen to have a guide on how to make a local mod as a workaround? i'm a noob

Thanks so much for checking this out and getting back to me!
Do you happen to have a guide on how to make a local mod as a workaround? i'm a noob

dont worry i got you, i'll explain it from the beginning
so first off you're gonna want to go to the place to download all the sounds: https://github.com/FireXDragon007/Akko-Lavender-Purple-Mechvibes-customs-
once you're there you're gonna wanna hit the green "Code" button, then the "Download zip" button

once that's downloaded, go ahead and open it. you should see a single folder in there called "Akko-Lavender-Purple-Mechvibes-customs-".
drag that somewhere else on your computer. for my example i'll do a folder on my desktop.

once that's done, go back to Firefox and open the settings menu for GXM.
scroll aaaalll the way down...

...'til you see the "Mods" button.

give that a click and you'll end up in the Mod Manager.
at the bottom of this page you should see the "Create" button. scroll down if you have to!

that'll open up the mod creation screen.
go ahead and name the mod. once you've done that, go down to the "Sound effects" section and click "Choose files".

find that "Akko-Lavender-Purple-Mechvibes-customs-" folder from the previous step. you'll see a bunch of .wav files in there.
just select everything in there (either by dragging the mouse over everything, or CTRL + A) and hit "Open" on the bottom right.

you should see all the files appear in the sound effects section.
all of them will have a dropdown next to them saying "- select an option -". for each one, click on the dropdown and set the sound type appropriately

hit the big red "Create" button, and you should see your mod appear in the list!

finally, re-open the GXM settings, scroll aaalll the way down to the Keyboard Sounds section, and pick your mod from the dropdown.

you should be all set after that! ^^


Removing (commercial) and (commercially)

Per suggestion by @chaals, removed (commercial) and (commercially), except for (commercial) counterparty or counterparties which were replace by transacting counterparties.

@humbitious ... could you please approve and merge?

@humbitious ... could you please approve and merge?

@kthomas @humbitious ... please approve PR. standards WG meeting on 3/9 approves changes


The most recent version of Seaside no longer uses/supports GRPackge class

The Grease test GRPackageTest >> #testPackages has started failing in internal tests and it seems that the Seaside extensions expected by that test are no longer provided by the most recent version of Seaside.
GRPackageTest >> #testPackages is the only test failure that we've seen ... and it fails when loaded against extent0.seaside.dbf (for GemStone <IP_ADDRESS>) in internal tests ...
GsDevKit_stones/smalltalkCI tests are running fine, so the changes that I make will likely only affect the internal tests ...

I think this should be fixed now...


Errors when Unfocusing Mono Editor Window

Godot version
v4.2.rc2.mono.official [1ba920fad]
System information
Windows 10 Enterprise - OpenGL API 3.3.0 NVIDIA 391.35 - Compatibility - Using Device: NVIDIA - GeForce GT 520MX
Issue description
If a project contains a C# script, it prints out the following error every time the editor window is unfocused.
  Parent node is busy setting up children, `add_child()` failed. Consider using `add_child.call_deferred(child)` instead.
  scene/main/window.cpp:1642 - Condition "!is_inside_tree()" is true.

Steps to reproduce

Create a new project with Godot Mono (only tested with Compatibility, I can't run Mobile and Forward+)
Create a new node and attach a C# script onto it
Unfocus the editor window (by opening another app, Alt+Tab, or even just running the scene)

Minimal reproduction project
unfocus error.zip

Tested on a different System, error doesn't happen.
Godot v4.2.rc2.mono - Windows 10.0.19044 - GLES3 (Compatibility) - NVIDIA GeForce GTX 1660 SUPER (NVIDIA; <IP_ADDRESS>17) - Intel(R) Core(TM) i5-4460  CPU @ 3.20GHz (4 Threads)

Couldn't reproduce in my machine either (windows 11, v4.3.dev.mono.custom_build [d76c1d0e5])


dartfmt doesn't support --enable-experiment=non-nullable

bin mit$ ./dartfmt --enable-experiment=non-nullable bin/main.dart

Could not find an option named "enable-experiment".

Issue moved to dart-lang/dart_style #915 via ZenHub


Doesn't work for me...

Sorry, still doesn't work for me. I'm following the instructions very very carefully. I'm getting this:
root@ghost-caucasusandmercury:~# cd /var/www/ghost
root@ghost-caucasusandmercury:/var/www/ghost# ghost-purge-images display --content-key=69891a65807dc1931c98406ded --admin-key=5cefed645c7944048554b2d5:b50084fd0cdda8fa7cb93fb2f17c2537cd294aa1f540b3f307ca6dbedc48da8b
🔑 Required parameter CONTENT_KEY is missing!
🔑 Required parameter ADMIN_KEY is missing!
❓ Need a key?

On your Ghost admin > Integrations > Create custom integration
Copy Content API & Admin API Keys

On a side note, I'm having Ghost-CLI version: 1.9.9 as I'm unable to update it for some reason.
Any idea?

💡 The issue was related about the hyphen - in the param names. This is just fixed in v2.1.1
@Caucasus-and-Mercury please update to the last version running npm install -g ghost-purge-images and try again, thanks for your patience!


fixed compilation errors in generated code

Thanks for this @rocketnik . Yeah, this would fix a compile error in Swift 4 which I'm guessing you're using. A proper Swift 4 update will be coming, but this is a good start and is still Swift 3 compatible.
Can you please run swift test and then commit the modified generated fixtures

Hello @yonaskolb, I made a fix to the code generator (qualifying a name), where I did get a compile error in the generated code before. The tests are failing. I can not really follow them. Could you please have a look at them?

 If they are different to what is stored the test fails. To fix this just run swift test or run tests in Xcode on SwiftGenKit which will generate the files which you can then commit. It's a bit of a funny process :)

Hell @yonaskolb, the tests are fixed now. Please consider merging the PR.


fix: windows下gf run命令行使用args参数导致无法运行

https://github.com/gogf/gf/issues/2383

Codecov Report
Base: 79.61% // Head: 79.66% // Increases project coverage by +0.04% :tada:

Coverage data is based on head (2fc8f7f) compared to base (39fede6).
Patch has no changes to coverable lines.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2414      +/-   ##
==========================================
+ Coverage   79.61%   79.66%   +0.04%
==========================================
  Files         598      596       -2
  Lines       49682    49562     -120
==========================================
- Hits        39554    39483      -71
+ Misses       8131     8089      -42
+ Partials     1997     1990       -7

Flag
Coverage Δ

go-1.15-386
79.64% <ø> (+<0.01%)
:arrow_up:

go-1.15-amd64
?

go-1.16-386
79.61% <ø> (-0.02%)
:arrow_down:

go-1.16-amd64
79.65% <ø> (-0.01%)
:arrow_down:

go-1.17-386
79.66% <ø> (+<0.01%)
:arrow_up:

go-1.17-amd64
?

go-1.18-386
?

go-1.18-amd64
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

os/gfsnotify/gfsnotify_watcher_loop.go
78.15% <0.00%> (-7.57%)
:arrow_down:

container/gmap/gmap_hash_str_any_map.go
97.00% <0.00%> (-1.00%)
:arrow_down:

contrib/config/kubecm/kubecm.go

contrib/config/kubecm/kubecm_kube.go

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Error won't allow mod to run

So when I last git cloned this mod, I got this error "Unable to checkout '2296304e62412b32e77bebe5843d5cce3df73ce2' in submodule path 'irc' And I decided to open a issue and see if it's fixable.

df229cabe6b3c4ef0b1de784603f1f8b4dfe305c


llvmPackages.libcxx: unset _LIBCPP_USE_AVAILABILITY_APPLE

Should fix a common regression from #126411
Can go to staging or staging-next depending on how fast this can be reviewed, or whether we have time for a quick darwin stdenv rebuild.
cc #148396

(I generally know almost nothing about Apple-specific stuff.)

(I generally know almost nothing about Apple-specific stuff.)

I don't think many of us feel like they do :) In this instance I was pinging you because: you are watching over the staging-next regressions. Also I need help deciding where this change should go this or next staging iteration, and you probably have a better idea about timeline and amount of acceptable breakages for this.

I'm not sure how bad these issues are.  Hydra isn't very busy now, but the current staging-next has been running for relatively long time (over two weeks) and full x86_64-darwin rebuild would surely add at least several days.

(am attempting to build the packages which have had their tweaks removed...)

Result of nixpkgs-review pr 151151 run on x86_64-darwin 1

1 package built:

nix

Not entirely sure how it's related but I get:
builder for '/nix/store/sfjsywkwvyfhc4dbvmm3fwllajl1l7c3-automake-1.16.5.drv' failed with exit code 2; last 10 log lines:
    GEN      doc/aclocal.1
    GEN      lib/Automake/Config.pm
    GEN      runtest
    GEN      t/ax/cc-no-c-o
    GEN      t/ax/shell-no-trail-bslash
    GEN      t/ax/test-defs.sh
    GEN      doc/automake-1.16.1
  help2man: can't get `--help' info from automake-1.16
  Try `--no-discard-stderr' if option outputs to stderr
  make: *** [Makefile:3743: doc/automake-1.16.1] Error 127

on macos 10.15

@risicle I think saw this error on x86_64-linux recently. Although I don't remember what that was about, could be that I just worked it around by enabling sandboxing.

Let's pull the trigger on this. The issue affects compilation with never standards, so users will definitely be affected if we don't fix it now.


Remove Periodic Behavior

Linting
It Should lint
Contents
I've added a set of functions to preparation and test_preparation that revolve around removing periodic behavior from an image after a fourier transform has been applied to the data.
Reviewer
Anyone may review

@ojha-aditya Ive added you as a reviewer for this request please do so as soon as possible.

@ojha-aditya
Friendly reminder that you've been assigned to review this

@ojha-aditya
I fixed line 171.
Line 64 in test_preparation.py is someone else's code. Can't do much about that.
Branch should be synced

Branch is still not synced but I updated that myself as it is late and people might need the code. You can still fix the pylint error even if it is someone else's code because everyone will get a pylint fail message otherwise.
@kylemasc917 the rest looks good to merge.

Okay I removed the line pylint error myself as it is too close to the deadline. @kylemasc917 looks good to merge.

@ojha-aditya I fixed line 171. Line 64 in test_preparation.py is someone else's code. Can't do much about that. Branch should be synced

ok merging


Remove ddoc references to 'second-generation std.random'

While it would definitely be nice to deliver on this idea, better not to mention it until we have an actual implementation lined up and approved.
Requested in forum discussion<EMAIL_ADDRESS>

LGTM, thanks!


Uboot bootstate

Updating bootstate env support for uboot
Adding potential eMMC tinker board support
#214

Updated to address comments.


Ensure distribution contains all groovy modules

Expose more modules as groovy-all used to contain them all as part of the build script classpath.

I'm just mentioning it because I remember we had the conversation in the past: instead of having Gradle always including "all" Groovy modules (which is not quite true), we could also very well have the build authors explicitly ask for them. It's probably too late for 7.0, but we could consider something like:
dependencies {
   implementation localGroovy()
   implementation localGroovy("json")
}

Good point! I think it would be great to go that way to also avoid those modules in the distribution. But to pull this off, we should actually introduce deprecation warnings if you depend on any module that would require a new localGroovy dependency


AccessToken is null after app restart

Description
fbsdk stopped working after the upgrade from RN 0.39 to RN 0.40. AccessToken returns null after app restart, even tho everything worked right before the upgrade of the RN itself.
react-native-fbsdk header have been fixed from <RCTThing.h> to <React/RCTThing.h>
Seems like the problem only occurs on the iOS platform. Android works fine.
As said Keychain and AppDelegate has been setup and was untouched, only RN was updated, when the problem started to occur :(
Reproduction
Upgrade from 0.39 to 0.40
Solution
No solution :/
Additional Information

React Native version: 0.40
Platform: iOS only
Operating System: MacOS

Closing this issue because it has been inactive for a while. If you think it should still be opened let us know why.


xarm7

Thank you for sharing such great work. Could you please introduce how to use the code for xArm7？ My robot is UR5

from robot.utils import create_transform, transform_to_vec
no create_transform, transform_to_vec?

Hi! Thank you for your interest in our work. We just pushed an update including those functions; it seems an outdated file was pushed previously. Thank you for catching this!
To run policies on a UR5, you would first need to mount our gripper onto it, which would need to include a XL430 Dynamixel motor for actuation as well as an iPhone. We don’t have access to a UR5, so we haven’t made a mount for it, but you should be able to use the xArm 7 mount as a reference and create a new mount.
In terms of using the xArm code, you’d just need to set the following configs in robot-server start_server.yaml defaults: “controller: xarm” and “camera: iphone”. Also, the xArm ip would need to be set in network.yaml. Then, the documentation for running policies should largely follow once the gripper Dynamixel is also connected (the robot server would need to be run in a conda/mamba environment containing the arm's controller package).
To run the policies on the xArm, we add in a transform from the iPhone position to end-effector, then convert the policy's relative action from iPhone coordinate frame to xArm coordinate frame, and finally use the robot’s IK to execute the relative action. The same steps should be able to be performed on a UR5.
We also no longer have this xArm/gripper setup in our lab, but please let me know if you face any issues and I’d be glad to set it back up again to help debug if needed.

Thank you for your patient response. I would like to ask how I can determine the positions of the end effector and the phone. The coordinate system of my robotic arm's end effector is as follows:

How can I make the model execute the task all at once instead of manually performing it step by step?

Great question! Using the robot-server and imitation-in-homes code from the gpt_retry branch, rather than main, would be better for having the task perform all at once, as the UI doesn't currently support this functionality. Running run.py on gpt_retry branch does not provide the UI to run policies, and you can instead easily run them through the command line. I've included some commands for running below:

h + ↵: home the robot
r + ↵: change home height of the robot
↵: takes 1 step of the policy
[integer n] + ↵: takes n steps of the policy
e + ↵: auto-runs evaluation using gpt retrying (if the use_gpt config is set and API key is provided)
q + ↵: quits the process

For example, 10 + ↵ would execute 10 steps of the policy consecutively, rather than needing to manually step through. e + ↵ will use a fixed time horizon set by the TIME_HORIZON variable, and run the policy until the time limit is reached, this way you also do not need to perform it step-by-step.
The reason we often run step-by-step is that our primary robot, the Hello Robot Stretch, does not having a very good blocking controller. Its low-level controllers also do not move linearly from current to desired pose, so we aren't able to simply run our policies at a higher frequency, and must wait until the current action is complete before performing the next. The methods I mentioned above ([integer n] + ↵ and e + ↵) use time.sleep() in order to achieve this.


Remove starting the application logic from config to bootstrap folder

Currently we have app.php file for starting the application in cofig folder, but we should use that file for configuring the application like we do in database.php config file. All the logic for starting the application should instead be in app.php file in bootstrap folder which then calls all of the config files.

With commit fca8141 we now have the logic of starting the application in bootstrap/app.php file instead of in config/app.php. app.php file in config folder now behaves just like database.php which sets the data of the application.
Also with this commit we now call and start the Horizon router directly from app.php in boostrap file instead of starting it in Application class. I find this better as it is more transparent and you know exactly what happens when you start your application.

Code of the app.php file.

<?php
// * Start the application
use Abyss\Core\Application;
use Abyss\Outsider\Outsider;
use Abyss\Horizon\Horizon;

// * Start the application
Application::start(dirname(__DIR__));

// * Import configs
$app_config      = require_once __DIR__ . '/../config/app.php';
$outsider_config = require_once __DIR__ . '/../config/database.php';

// * Configure app
Application::configure($app_config);

// * Connect to the database
Outsider::connect($outsider_config);

// * Start the router
Horizon::start();


Rounding error @ interdrone

Describe the bug
Lost money during roulette.
To Reproduce

Go to Rio lobby at 4:30pm
Put 5 dollars on red
Wait
If you won, go back to step 2
You will observe that you just lost your money

Expected behavior
Become rich and famous
Actual behavior
Remain poor
Additional context
Happens @ interdrone

Fyi @JonasVautherin

This is a feature.

@Stifael we probably need to talk about "Waking up in Vegas", but for now this is a "Not Going to Fix"


Testing 1.1.1~ynh2

fixes prices configuration
PR Status

[x] Code finished and ready to be reviewed/tested
[x] The fix/enhancement were manually tested (if applicable)

!testme

!testme