DRILL-7871 StoragePluginStore instances for different users

DRILL-7871: StoragePluginStore instances for different users
Description
This feature allows to have the separate plugin set per user. Profiles are restricted per user too.
Documentation
To enable the feature:
drill.exec.security.user.auth.separate_workspace
See doc fr more info
Testing
Unit tests. Manual testing of Drill WebUI and JDBC connection via SqlLine. Details in doc

@vdiravka, looks like the test you have added is failing in CI, could you please take a look?
Could you please share some functional-level doc with the description use cases and behavior, so it will be easier to do the code review?

This one is difficult. We are reviewing code under time pressure when the design itself raises many questions. Let's start with the design. First, it is too short, it leaves many issues unspecified. So, I have to infer them from reviewing the code. As noted before, the storage plugin registry is complex; so having to reverse-engineer a design at the code level makes the process slower than it might otherwise be.
This PR adds two separate features: a new options level and changes to the plugin registry. Since these are independent concepts, they should appear as separate PRs: one for the options, another for plugins. That will reduce the cognitive load on us poor reviewers.
So, the design of options:

But it appears different users have the common Storage Plugins and System Options.

There is no possibility for Drillbit configuring individually for every user and persistent between sessions (Note: SessionOptions are removed after closing session)

The above is by design. System options are just that: options for the system. They are meant to be set by the administrator to control the Drill cluster as a whole. Drill also has Session options: these are per user session. If we want per-user option settings, then we need a new level: User options.
Options form a hierarchy. At present that hierarchy, from most to least specific, is:

Per-query options or session options
System options
Options configured in the Drill config system.

It sounds like you want to add a new layer: user options:

Per-query options or

Session option
Per-user option

System options...

Suppose we have per-user system options: I set my queue length to 10 items, you set your queue length to 100 items. But, the queue is a global resource: which is used? This is just one example of how intractable "per-user system options" is as a concept. So, we need actual user options.
One obvious choice is simply to persist the session options. I login and change foo to 10. Drill simply persists that change so that next time I log in, foo is 10. Easy, but is that the right answer? Drill has a serious flaw that the behavior of any query depends on options: JSON modes, all-text mode and so on. That was done as a quick and dirty short-term solution to ship the product, but is still with us and still causing issues. If I set an option to "all text mode", and run my query, I get a result. If I share that query with you, but you don't set the all-text option, the query will produce different results (or, probably, fail.)
So, to "fix" options, we need to push those options into the query (with hints, say), or we need options associated with a query, as having per-user options won't solve the "options associated with a query" problem.
So, we probably do not want to just persist session options as they are used to control specific queries. Instead, we'd want to add a new system: SET USER OPTION ... so that the hierarchy is as above.
The design is silent on many important questions:

What is the new hierarchy?
How does the (not an) administrator set truly-global options if all users can set their own?
How does the user set, review and clear the user options?
Suppose a user is logged in twice. In session 1 they change foo to 10. Do we expect session 2 (on a different Drillbit) to see that change? This is a very difficult concurrency question. Of course they do. But, when? In the middle of query planning? At the start of the next SQL statement? After a time period? Only at the next login? If concurrently, how will one Drillbit track changes made to the user options by another Drillbit? How can the use a "force resync" to keep options synchronized?
Can the admin see the per-user options? If so, how? SHOW OPTIONS FOR USER 'bob'? Remember, the admin is tasked with policing the system, they must be able to see all options.
How do we solve the current ambiguity of true system options (the query queuing options mentioned above) vs. user/session options? At present, the solution is ad-hoc and we just trust that most options are not changed. If a naive user can change any option, we've got issues. Do we need a permissions table to say which users can change which options? (And, categorize true system options as read-only at the user and session levels.)
You will need USER version of all of the option commands.
You will need complete tests that demonstrate that the hierarchy works as expected.

The good news is that, in terms of implementation, Drill's option system is already hierarchical, so adding a new layer should be very easy.
All of this is complex. It should be done as its own PR along with a detailed design.

Let's now move onto the primary topic of this PR: plugins. Again, this is a complex topic, especially if we think about how the solution would work at scale. I'm afraid that, since the design is so sparse and leaves out many details, that we're perhaps going down the wrong path. Let's take each topic in turn so we can see what's what.
First, what is the structure we want? To answer this we need to separate the code (Java libraries, which I'll call a "connector", following Presto) and configs (the JSON things that users see, what we normally mean when we say "plugin".)
Drill will only ever work with a fixed set of connectors: those available on the class path at runtime. AFAIK, there has never been interest in adding connectors at runtime (no "dynamic connectors") and, for security reasons, I'm not sure that doing so would even be a good idea.
Connectors are presently used by any number of configs. And, every query creates an instance of the objects in the connector (the readers, the planner objects, etc.) So, the structure we have is:

Config (a JSON object in the persistent store)
Connector (a library)

Plugin (an instance of the plugin class from a connector, along with its config.)

Readers, planner objects, created per-query from a plugin

The plugin registry binds connectors to configs to produce plugins (where by "plugin" I mean an instance of the plugin class initialized with a config.) This is some mighty complex code!  I'll omit the details for now.
So, what does this mean? For one, it means we only need one copy of the connector. The "new" plugin registry is designed to handle this idea. My "class path plugin" and your "class path plugin" use the same plugin library. I'll assume that the answer above, that each user has their own copy of the code, represents a misunderstanding because of Drill's horribly ambiguous names in this area.
Next, the plugin registry holds two kinds of "plugins". First are those configured in the UI and that can be resolved in queries. Familiar things like cp, dfs and so on. Second are ad-hoc plugins: those created on-the-fly based on table properties in a query, or plugin properties passed along in a query definition. This is done so that queries work even if, right after planning completes, someone deletes or changes the stored config: the next query sees the new config, the executing query uses the config stored in the query. (Else, disaster would result when some Drillbits see one config, others another.)
There is a system plugin that offers the system tables. This one is meant to be shared by all users. It has no config options, but if it did, they would be set at the system level by the admin. It makes no sense for each user to have their own copy. (It might make sense to disallow certain system tables for some users, but that is a different question.)
Now we have the "regular" configured plugins: a JSON config and an associated plugin instance. The goal here is to define a new level so that individual users have these items.
First, I'll question if the requirements are correct. Do we really want plugins associated with each user? If you and I both use tale "foo", do I have to ask you to send me your JSON so I can create a new copy? What if you change something? What if we have 100s of users all with copies? Making copies is a "demo only" feature, it does not scale in production. Remember: reuse by copy and paste is for amateurs who will throw away their solution, not for professionals who want to minimize total costs.
Let's think of a use case. A firm has five departments with 10 people per department. Some people in one department can see some data in another. For example, the VP of marketing is allowed to see Sales and Dev data. The CFO can see everything. Oh, and the employees change regularly.
This is not a new idea. How do other tools handle this? Oracle? SQLServer? They operate at the level of schemas (databases) and tables. So, think of a plugin (config in storage, config + connector at runtime) as a shared, named object.
So, what we want are groups of plugins. "Marketing", "Sales", "Ops". Within each are the configs for that set of tables. Users can be given permissions on the whole group, or on individual plugins. Administrators or DBAs (those who understand production systems) set up the configs. Users use those to which they are given permission. Or, if we want to be cheap, we have a huge global name space of plugins, and require all names to be globally unique, but you can only see those to which you have been given permission. (How will this work with 100 plugins and 1000 users? That results in 10,000 distinct permissions to maintain.)
Users might want to create their own, individual configs not meant to be shared. Fine. Have a "user" group that holds configs for just that user. If I want to promote my new "foo" config, I ask an Admin to add it to the "Dev" group so you can see it.
If the above is roughly in the ball park, we'd want to work out the details in a detailed design. There are likely a dozen "gotchas" that the above omitted. Who removes configs for users no longer with the organization? What connectors is a user allowed to create configs for? How can the admin see all configs to police users or track down odd behavior "some user did something silly that puts huge load on an external system"?
Think of security: who decides who gets permission on what? Is it done manually? With what UI? How does that work in an organization of 10 people? 100? 1000? Since we are dealing with permissions based on roles (role-based security), can/should we tie onto the corporate LDAP or similar system? What is the API for that?
Configs should hold security keys, but those should not be in JSON, stored in the persistent store. (Drill has always done this wrong for systems other than MFS; another reason Drill fails in production.) How are configs connected with a "vault" to get keys, passwords, tickets and the like at runtime? On S3, the use of temporary tokens is the right way to do permissions, no one gives away S3 keys any more. So, how does a plugin obtain and refresh its S3 token?
These are hard topics. Thanks for taking on this big challenge! Suggestion: pick a small subset to start with, but ensure that subset allows us to add the pieces needed when the solution moves into production.

Hi @paul-rogers,
I started design for system options from adding the new additional layer. But this approach is much more complex and we should cover all the cases you described.
The design for the feature proposed by me is other: we just don't use global system options at all (in case separate_workspace is enabled). Every user has it's own system options. It gives a great functionality, which drill required for a long time, I think. We used session options for this purpose, but it can't be serialized. I also abandon the idea to Store Session options for the same reason, it has other concept and couples with a Query.
About plugins the same thing, I tried to keep things as simple as it is possible (it corresponds to your last sentence).

I agree we can factor out system plugins and use that shared plugins for all users. It is minor improvement, it allows to keep less things in memory. This is not a blocker for this PR. I will create a ticket and proceed working on it after finishing this PR.
Also I agree with it is nice to have Group/Organization level of plugins and System Options. I noted about it in Design document. But it is much more complex improvement, requires introducing a lot of new rules and grants. I thought to consider working on it after finishing on this PR. Because now separate plugins and options can provide all things needed for users, but setting up is not so comfortable as it can be with Group/Organization level of these instances. And there are more similar instances are placed in memory now (separate instances for Groups/Organizations will improve this too)
About concurrency and distributed systems issues, they are all the same as it is now. Since event now one user can run query and other can edit the plugin config. Now the picture is even better, since two user can't sue common plugins registry or system options. Whey their own ones. They don't impact each other.

About dividing the PR I also thought about it initially, because leverage of System Options is other than plugins, so a lot of things for options is other. But options and plugins are combined with a common idea - to have the separate workspace for users: plugin configs, system options and to see their own query profiles only. And the approach for implementation is very similar - to replace the global instances with their own user-session ones.
So I can here to divide the PR in two parts or just in two commits for easier review. But I think we can also go with one PR and commit.
So let me know do you like the above approach for this feature with a two (or more) further improvements or possibly you want to bring mail discussion on this topic with other devs?

Hi @vdiravka,
Thanks for the note, you make good points. Let's see where we can compromise. Let's start with options.
It makes no sense for users to have their own system options (my favorite example is the length of the global, shared query queue.) So, SYSTEM options can't be per-user. This means that the SYSTEM level has to remain as it is. We can cut corners: still allow users to change system options, we just ignore those values. (For example, IIRC, the query queue, the value is taken from the SYSTEM options; we ignore SESSION values.) That is, we just trust that user ignorance will prevent them from breaking things; they won't know about the system options.
If we have any form of user-property persistence, then we have to worry about race conditions. I don't see how we can ignore that issue unless we are strict and we say that user options are read once: when the user starts a session. If I change all-text-mode in session 1, it will not affect session 2. This seems reasonable for the "toy" use cases where Drill is now used.
That is, when a session starts, we set the session options to the saved per-user values from the persistent store. A nice result is that, at runtime, there is no difference between user and session options. This fact should save some work.
The user has to be able to set the user options. We don't need a new option level. Simply add a new statement: ALTER USER SET <name>=<value>. This would do two things:

Write the given key/value pair into the persistent store for the user.
Change the session option in the current session to that new value.

Again, we let the user change anything they want; we rely on the fact that users will only care about one or two options, and only when someone suggests a change.
The above is simple, but limited. The limits are no worse than today. It seems we could extend the feature in the future without breaking anything. (For example, broadcast user option changes to all sessions. Apply security to system options. Etc.)
Then, we simply ignore the issue that the options users care about are associated with queries (all the Parquet, JSON, and other options that affect reader behavior.) We simply decide to be surprised when the user encounters those issues. User: "Gee, I set the all-text-mode option and my queries run file. But, when Bob runs them, they fail...". Our answer: "have Bob change the same option" This may break Bob's own queries, so we have Bob change the option back. This is a horrible user experience, but is the same as what Drill offers today, so we're not breaking anything.
Will this work?

Next, let's discuss plugins. Let's start with three goals:

System plugins work for everyone (we can add security later).
By default, plugins are available to everyone. That is, the Drill out-of-the-box plugins are available so that, say, cp works and people can try examples from the documentation.
There is an optional way to create per-user plugin (configs).

What is the simplest way to do this? Maybe by changing the plugin resolution rules. When the planner looks for a plugin, it tries two names: $$<user>$$<plugin> and <plugin>. We use the resulting name in the query plan. So, if I have (unwisely) created my own cp plugin, when I ask for cp, that's what I get:  $$paul$$cp. If I do not have my own dfs, then when I ask for it, I get the (global) dfs. (The $$<user>$$ is something I just made up; it can be anything as long as the marker is not likely to ever appear in a real plugin name. Maybe ..<user>.. or \<user>\ would be better since they are not legal SQL.)
When a user stores a plugin, and the per-user config feature is enabled, then the UI appends the user prefix. The UI filters out all plugins owned by other users. When the admin works with plugins, there is no prefix. This allows the admin to see, and change, both cp and $$paul$$cp. If I leave the organization, the admin can remove all the $$paul$$... plugins.
I suspect you'll find that, with this solution, no change is needed in the plugin registry. Plans will resolve to the correct reader configs. The reader configs in the query plan will just work (as they would in your design.)
How does this help sharing? Simply ask the admin to change the name of the plugin from $$paul$$whizbang to just whizbang. Now, everyone can see it and my queries work without change. Super simple. We have only two security levels (everyone or single user), but that is far better than reuse by copy/paste. The design could be extended to the role-based one without breaking anything.
The next step is store the per-user plugins in separate persistent stores. This is the "cleaner" design, but now you have to create the means for an admin to see, change, and remove the user-level plugins. You also have to extend the plugin registry, which is, as we noted, a complex beast. Any change requires extensive unit tests.
By the way, when I revised the plugin registry a while back, I fixed dozens of bugs. Most were simply because I added tests and found all kinds of broken stuff. Concurrency was broken, upgrades were broken, expiring of old plugins was broken. and so on. I still have scars. So, I'm trying to save you from similar pain.
Will something like this work and avoid the hard work we discussed earlier?

One more thought: it is not clear from the PR or design document why you need these changes, so I'm having to guess. I may be guessing wrong. Can you spell out the use case?
For example, I could imagine that you need the persistent user options because you want to use the REST API, and that API does not have a persistent session, so options don't persist. As it turns out, someone kindly recently added the ability to pass along options on each query request. This is, in fact, how Impala works to avoid the need for session state. So, you can achieve the same result by keeping options on the client.
Or, you can achieve the same result by adding a session concept to the REST API. Log in and get a session token back. Send that token on subsequent requests to identify the user and to key to a cached session object. The session ends after a timeout or on an explicit logout call.
If the per-user plugins are to allow multi-tenant operation, then the problem is considerably more complex as much of Drill is not designed for that. (You'd want to secure query profiles, police resource usage, etc.) If it is to allow different users in the same organization to define their own plugins, then you have the sharing issue we discussed.
In short, if you can explain your goals, we can perhaps help you find a workable solution that achieves the goals. Without knowing the goals, then I'm free to make up my own understanding of what you want...

Motivation:

The goal is to allow multi-tenant operations. So it doesn't matter what API user uses: REST, JDBC ot ODBC. This is true that Drill initially wasn't designed for that, but we ca add it.

Options:

The user has to be able to set the user options. We don't need a new option level. Simply add a new statement: ALTER USER > SET =. This would do two things:
Write the given key/value pair into the persistent store for the user.
Change the session option in the current session to that new value.

This is also questionable, because it breaks SessionOptionManager extends InMemoryOptionManager:
This is an OptionManager that holds options in memory rather than in a persistent store. Options stored in
SessionOptionManager, QueryOptionManager, and FragmentOptionManager are held in memory
(see options) whereas the SystemOptionManager stores options in a persistent store.

Most likely all easy ways here are some sort of hacks. So let's do that in correct manner: adding the new UserOptionManager as a layer between SystemOptionManager and SessionOptionManager.

The next step is store the per-user plugins in separate persistent stores. This is the "cleaner" design, but now you have to
create the means for an admin to see, change, and remove the user-level plugins. You also have to extend the plugin registry, > which is, as we noted, a complex beast. Any change requires extensive unit tests.

Different PS can be chosen per Drill doc. This is independent from Plugins. We have HBasePersistentStore, InMemoryStore, LocalPersistentStore, MongoPersistentStore, NoWriteLocalStore, ZookeeperPersistentStore implementations for PS.
If you mean different instances of the samePS, that is not needed. PSRegistry is just a code, which allows to access actual PS. We can manage access via the common PSRegistry. Or we can reconsider it in future.

Plugins:

About plugins it makes sense to add Global plugins and to use it in case User plugin is absent. It's like a plugins fallback.
And system plugins can be used only in scope of Global plugins. I don't think user name in schema is a nice and elegant solution. I think plugins fallback is simpler and more convenient approach.

Conclusion:

So I am going to add this functionality to current implementation and separate out user options part of this PR into the new one.

@vdiravka, thanks for the explanation. Let's be careful not to confuse existing implementation with new requirements. Yes, there is a comment that says that the system options are persistent and session options are not. Be careful to read this correctly. At the time the comment was written, system options were the only persistent options. System options, by definition, have to be persistent. The comment did not forbid session options from being persistent; it simply observes that, at that time, for the reasons we discussed, session options were not persistent. Nor does the comment say that, if you add persistent user-level options, that they have to be system options because only system options are persistent: you are free to add persistence to session options.
The gist of your requirement is to let users persist some of their session options. We discussed that this may not work as one might think: we have to think through the issues. The point was that concurrency of user options will be a mess if they work like system options: if changes to user/session options are written to persistent storage immediately. This will screw up queries as I persist "all-text-mode" in one session, which breaks a query in another. Point is, user options must have semantics different from system options.
Before we worry about code details, we have to get the semantics right. To repeat:

We need a concurrency guarantee. One choice is "options are propagated to all active sessions with unknown latency, just like system options." Another, more dependable, is "session options are initialized on session startup, after which they do not reflect changes made in other sessions." Whatever it is, the definition has to be stable and explainable.
Some options apply to queries: we've been using "all-text-mode" as an example. Such options will be set frequently. Since these options are per-query, we cannot set the option across all user sessions immediately, as that will break queries. Instead, the user has to tell us when they want to persist that setting. Otherwise the setting is per-session (that is, per-query.)

Is it a bad thing that the behavior of queries depends on session options? Certainly. That was a poor design choice. But, it is how Drill works, and we have to design new features to reflect this choice.
The proposal I outlined is the simplest way to achieve with concurrency semantics we could actually support. Feel free to suggest a better design; but let's discuss that design before we reduce it to code. Either way, the design has to reflect our requirements and constraints, not just follow existing code.
I strongly recommend splitting this PR into two: one for options, another for plugins. Else, discussion becomes overly complex.

@vdiravka, on the plugins side, let's please do list our requirements and design goals. Else, I worry that the comments I make get lost, and I will this have to repeat them over and over.
Requirements as I extract them from your description:

Provide a bootstrap option to enable the new semantics. The default is off, meaning that Drill behavior will not change by default.
Retain global plugin configurations. (They allow multiple users to share the same configuration without copy/paste.)
Add per-user plugin configurations.
Users can add and change only their own per-user configurations. (No user can "share" a config with another user: either the config must be copied to a global config by the admin, or the second user must make their own copy.)
If a user configuration happens to have the same name as a global configuration, the per-user configuration takes precedence. (If the admin make a global version of a now-shared config, the user must delete his own copy in order to use the now-shared version.)
No non-admin user can see or change the configuration for another user. (That is, if two people work in the same group, and want to share a config, they must do so via copy/paste, or by asking the admin to make the plugin global to all users.)
No non-admin user can change a global configuration.
The admin user can see, add, modify or delete any plugin configuration, both global and per-user. (Without this, the system is not maintainable as the admin won't be able to manage per-user configurations.)

Now, one can disagree with these requirements. For example, in a multi-tenant environment, we do not one tenant to see the plugins for another tenant. But, we might want all users in "Tenant A" to see a shared set of configs. Your description appears to assume that "Tenant" = "User": one user per tenant. However, a typical model is for each human in a tenant to have their own user account. Please spell out the desired semantics.
Aside: Clarifying the desired tenant model may help with the options discussion. I've been equating "user" with "person". If your tenant model equates "user" with "tenant", so you are tying to define per-tenant defaults, then please clearly say so in the design. If you want per-tenant default, I would suggest a somewhat different design for options.
Finally, if the goal is a true multi-tenant model, in which tenants are distinct business entities (rather than different departments within a single entity), then we must also ensure each tenant has access to only their own query profiles. Will there be another PR for this? Is there a roll-up Jira ticket for all the issues involved in true multi-tenant operation?
Also, the Drill UI assumes a single organization. If the UI is to be exposed to multi-tenant users (so they can monitor queries, see query profiles, etc.), then the UI must change. Tenants should not be able to see details, or change the state of, Drillbits. Session options should reflect tenant values. Probably other changes. Will there be a design or PR for this?
Further, each tenant must have guarantees on resources. That is, Tenant A should not be able to run a huge query that denies resources sold to Tenant B. This is a very hard problem. If you don't solve the hard problem, the options and plugins are somewhat moot. The option and plugin features are handy, but do not, by themselves, give you multi-tenant support in Drill.

@vdiravka, let's think about your larger goal.  If the goal is a true multi-tenant model, in which tenants are distinct business entities (rather than different departments within a single entity), then we must also ensure each tenant has access to only their own query profiles. Will there be another PR for this? Is there a roll-up Jira ticket for all the issues involved in true multi-tenant operation?
Also, the Drill UI assumes a single organization. If the UI is to be exposed to multi-tenant users (so they can monitor queries, see query profiles, etc.), then the UI must change. Tenants should not be able to see details, or change the state of, Drillbits. Session options should reflect tenant values. Probably other changes. Will there be a design or PR for this?
Further, each tenant must have guarantees on resources. That is, Tenant A should not be able to run a huge query that denies resources sold to Tenant B. This is a very hard problem. If you don't solve the hard problem, the options and plugins are somewhat moot. The option and plugin features are handy, but do not, by themselves, give you multi-tenant support in Drill.
System tables could "leak" information between tenants. Should system tables be disabled? What changes would be needed to ensure that no per-tenant information "leaks" to another tenant?

@vdiravka, I had an opportunity to get a bit more background info for this project. I am going out of my way to try to facilitate this PR; normally we'd require that the PR author provide this information so that us reviews can simply review the code, and not have to reverse engineer requirements and design.
Sounds like the requirements are for a very specific light-weight multi-tenant model: one that allows tenants to set options, create storage plugin configs, and run queries, but not access any other part of Drill. Tenants are to be trusted to not make mistakes. Specifically:

A tenant has a set of "system" options (maybe call them tenant options) available when that tenant creates a Drill session.
A tenant can define a set of storage plugin configs which are visible to only that tenant. Perhaps call these tenant plugin configs.
A tenant can run queries that use the tenant options and tenant plugin configs.

This use case is limited compared to the normal multi-tenant requirements. The following appear to be restrictions for this project:

A tenant does not have access to the Drill Web Console or the Drill REST API and thus does not have access to query profiles.
A tenant does not have access to Zookeeper or the Drill native API. Queries sent by the tenant must go through an intermediate software layer provided by the service provider.
A tenant does not have access to Drill logs to diagnose failed queries.

The above restrictions say that the feature is not useful for open source Drill users who use the Drill-provided UI and APIs. This makes the feature of very limited appeal to the Drill community. So, one of our challenges is to design the feature in a way that users of the "out-of-the-box" Drill can benefit.
Additional restrictions for this one use case:

A tenant cannot start, stop or restart Drillbits, nor can they change startup properties.
A tenant cannot upload a UDF nor can a tenant provide custom connectors (storage plugin classes). (Note that DRILL-7916 is working at cross-purposes to this PR.)
Tenants are trusted to not change system-wide performance-related options (queueing, resource allocation, etc.) The resulting behavior, if those options are changed, is undefined and must be dealt with by the service provider if they occur.
No provision for the Drill admin to view or modify tenant options or plugin configs. If such behavior is desired, a service provider must write tools that work with Drill's persistent storage.
Tenants are trusted to not consume excess resources, so no resource isolation between tenants. Tenant A might try to sort a trillion rows, which might deny resources to other tenants.
Tenants cannot (?) create views or a metadata store.
Parquet metadata caching is either unsupported (?) or must be written to the tenant's S3 bucket; Drill provides no storage for the metadata.

The above limit the solution, but leave the door open to eventually providing more general multi-tenant support.
A final question is the relation between tenant and user. This PR assumes that they are identical: that "fred" is either a normal Drill user in "normal mode", or a tenant in "tenant mode." That is, each tenant has a single Drill user (which works in this use case because of the intermediate software layer.) This explains why this PR is labeled as "instances for different users", the the discussion has revealed the goal to be "instances for different tenants."
Since the "tenant = user" model applies to only this one use case, it again is not a general enough feature to add to the Apache Drill code. Instead, Apache Drill must provide a generally useful solution.
Prior notes have explained how a "per-user" model should work (it requires sharing between users). Specifically a, "config per user" solution must recognize that users work on a team, allow sharing, and permit admin abilities. Similarly, an "options per user" solution must persist only a subset of options (those which are neither system nor per-query options), and must solve the synchronization issue.
Notes have also explained that the customary definition of "tenant" is an organization with multiple users. A true multi-tenant solution must allow multiple users per tenant, and provide a path toward full tenant isolation later.
The challenge here is to find a design that balances the very specific, ad-hoc, unusual needs of this one use case, with something that can evolve to become of general use to the Drill community.

@vdiravka, here I will suggest the broad outline of a design which meets the very limited goals of this PR while also providing a path forward for a generally useful set of Apache Drill features.
We introduce one new concept, the tenant. Key concepts:

A tenant is a name space for options and plugin configs. The tenant level is optional; by default, no tenants exist.
Each user has an optional associated tenant ID. A user with no tenant ID works at the system level. (Example: the Drill admin, or a user who sets up demo configs shared by all users.) A user with an associated tenant ID gives the tenant name space to use. There may be 0, 1 or many users associated with a tenant.
A persistent tenant options name space sits between the system options and the session options in the option hierarchy. Users "inherit" options from the system layer, unless overridden at the tenant layer, unless overridden at the session layer.
Queries and plugin config-related API calls resolve plugin configs within the tenant plugin config name space. On lookups, if the config is not found in the tenant name space, repeat the search in the global name space. (Allows system-wide configs for such things as demo data.)

Now we can decide how much of the above to implement in this PR, leaving the rest to future PRs. Given your goals, it would seem the minimum is:

Provide an API or other means to define tenants and to associate tenants with users.
Provide documentation for how to inspect or remove per-tenant name spaces (e.g. if the tenant cancels service or is moved to another cluster.)
Tenant membership is enforced for the native and REST query APIs.
Tenant users can see the Drill Console, but features that are not "tenant-ready" should be disabled. That is, a tenant user might only be able to run a query, and see their options, but not see query profiles, etc.

SQL semantics:

The ALTER SYSTEM command affects the tenant option store given by the user's tenant ID, if any. System (e.g. server-wide) options are set using the same command issued by a user with no tenant ID.
System tables are not available to users associated with a tenant ID. (Later, this can be modified to filter system table contents to include only information for that tenant.)

An possible code design might be:

Add a tenant registry which can be as simple as the known, valid tenant IDs. This "registry" can simply be an API, with a default implementation that does nothing.
Add a field to the user to hold the tenant ID. When resolving a user name, obtain the tenant ID along with the admin/normal-user access level.
Unfortunately, Drill does not have a User object, we assume the only information for a user is the user name. Define a new User object which holds the user ID and tenant ID. (Ideally, it would also hold the security level, etc.)
We know that the system must determine if the user is an admin. Extend that mechanism to also provide the tenant ID. Modify that code to provide a new method which returns the User object. A storage plugin manager client can either use an existing User object, or obtain a new one by calling to this new User factory method.
Introduce the tenant option manager which acts like the system option manager, and persists to a location that includes the tenant ID in the name. Persistence is to a location which includes the tenant ID in the name.
When a session starts, if the user has a tenant ID, add the tenant option manager to the session option manager stack. This one change should ensure that the option solution "just works" everywhere else.
Modify the storage plugin manager to require a User object for every config CRUD method. The User says which tenant store to read from or write to.
The storage plugin registry currently uses keys to locate configs. This key is extended with the tenant ID. Modify config resolution to use the tenant ID. If the config is not found in the tenant store, or the tenant ID is not set for the user, repeat the search with the default "system" config store. When modifying configs, write to the persistent store given by the tenant ID.
Modify all clients of the storage plugin manager to pass along the User object, creating one if needed.
System tables do not have an associated config. For system tables (only), enforce the rule above: if a plugin lookup resolves to the system plugin, and the user has an associated tenant ID, act as if the plugin is not found.

This proposed design accomplishes two goals. First, it is limited in scope to only the limited needs of the PR author. Second, it is constructed in a way that others can extend the solution toward a full multi-tenant solution in the future without breaking backward compatibility.
Note that this design is a multi-tenant solution, not a per-user solution. I would suggest changing the title of the PR to match the intent to (partially) solve the multi-tenant problem.
The good news is that this is more-or-less in line with some of the code in this PR.
@vdiravka, please review these suggestions, and the need to offer a solution that is of general interest to the Drill community. Feel free to use the above, or to modify them, to propose a revised set of requirements and designs. Once we're on the same page at the design level, attention can return to review of the code itself.

Converting to draft while there is still discussion about how we approach the design.


Linked to new location for Northwind download.

Summary
At the request of my manager, we have added the Northwind and Pubs scripts to the SQL Server Samples repo. I have replaced the previous link in this article with a link to the new location, with simplified and up-to-date instructions in the Readme there.

Checks have failed; closing and reopening to begin new build.


can not support Type in denpency and interface Type

for example , a struct like below :
type VO struct{
code: string
msg : string
data: interface {}
}
I can not reference it in response annotation
also ,can not reference a type not in current file :
VoMap defined in other package
c.JSON(http.StatusOK, VoMap{"code": 200, "org": org})

@zzq392342084  swag doesn't support interface{} type yet.  I want to support it in further version(pull request welcome 😄 ). For parsing efficiency, swag now only search in project root folder which contains main.go file.

ok,thanks!


fix getAuth() in auth.js

Fix Error: source-google-docs: Cannot read property 'generateAuthUrl' of undefined

Fix #31

Close #31

Published in v1.14.2


Welcome system does not work

I made the entry to my server where the bot is and it does not generate the welcome image or the greeting although no error is registered in the console of the server where the bot is hosted

an issue with the package, not sure when it'll be fixed, but its not my issue so i can't help


DDS hover event doesn't get triggered after entering a directory

What system are you running Yazi on?
macOS
What terminal are you running Yazi in?
kitty 0.35.1
Did you try the latest code to see if this problem got fixed?
Tried, but the problem still
yazi --debug output

Yazi
    Version: 0.2.5 (0f84717 2024-06-20)
    Debug  : false
    OS     : macos-aarch64 (unix)

Ya
    Version: 0.2.5

Emulator
    Emulator.via_env: ("xterm-kitty", "")
    Emulator.via_csi: Ok(Kitty)
    Emulator.detect : Kitty

Adapter
    Adapter.matches: Kitty

Desktop
    XDG_SESSION_TYPE: None
    WAYLAND_DISPLAY : None
    DISPLAY         : None

SSH
    shared.in_ssh_connection: false

WSL
    /proc/sys/fs/binfmt_misc/WSLInterop: false

Variables
    SHELL              : Some("/opt/homebrew/bin/zsh")
    EDITOR             : Some("nvim")
    ZELLIJ_SESSION_NAME: None
    YAZI_FILE_ONE      : None
    YAZI_CONFIG_HOME   : None

Text Opener
    default: Some(Opener { run: "${EDITOR:=vi} \"$@\"", block: true, orphan: false, desc: "$EDITOR", for_: None, spread: true })
    block  : Some(Opener { run: "${EDITOR:=vi} \"$@\"", block: true, orphan: false, desc: "$EDITOR", for_: None, spread: true })

tmux
    TMUX   : false
    Version: No such file or directory (os error 2)

Dependencies
    file             : 5.41
    ueberzugpp       : No such file or directory (os error 2)
    ffmpegthumbnailer: No such file or directory (os error 2)
    magick           : 7.1.1-33
    fzf              : 0.53.0
    fd               : 10.1.0
    rg               : 14.1.0
    chafa            : 1.14.0
    zoxide           : 0.9.4
    unar             : No such file or directory (os error 2)
    jq               : No such file or directory (os error 2)

--------------------------------------------------
When reporting a bug, please also upload the `yazi.log` log file - only upload the most recent content by time.
You can find it in the "/Users/sam/.local/state/yazi" directory.

Describe the bug
the hover event doesn't get triggered after entering a directory
Expected Behavior
The hover event should get triggered after entering a directory. Because external programs might use this as an indication to find out what the currently "hovered" item is
To Reproduce
Run yazi --local-events=hover and observe the events
Configuration
No response
Anything else?
https://github.com/sxyazi/yazi/assets/86842365/98944a26-6dc8-4d55-ac8b-0f80b093b153

Fixed in https://github.com/sxyazi/yazi/pull/1187, thanks to @Xerxes-2 for the PR! 🎉


Boostrap LSP

This PR implements a very simple LSP that responds to HoverRequests.  As part of the request, it parses the code in the file from the hover request was made from and does type inference.  It responds with a generic "Hello world!" string.  In the future we'll determine the type for the symbol at the given cursor location, but we need #405 before that can happen.
One thing that took me a while to figure out what was that all of the println!()s that I had scattered around were interfering with the stdio communication between the LSP server and the client.  This PR changes all of them to be eprintln!()s so that their output goes to stderr instead.

Codecov Report

Merging #406 (f290ccd) into main (349bd2c) will decrease coverage by 0.57%.
The diff coverage is 19.68%.

:exclamation: Current head f290ccd differs from pull request most recent head 4c35a9a. Consider uploading reports for the commit 4c35a9a to get more accurate results

@@            Coverage Diff             @@
##             main     #406      +/-   ##
==========================================
- Coverage   92.16%   91.58%   -0.58%
==========================================
  Files          51       52       +1
  Lines       14865    14968     +103
==========================================
+ Hits        13700    13709       +9
- Misses       1165     1259      +94

Impacted Files
Coverage Δ

crates/crochet_lsp/src/main.rs
1.03% <1.03%> (ø)

crates/crochet_infer/src/expand_type.rs
80.50% <50.00%> (ø)

crates/crochet_infer/src/lib.rs
95.92% <66.66%> (ø)

crates/crochet_dts/src/parse_dts.rs
89.73% <69.23%> (+0.21%)
:arrow_up:

crates/crochet_infer/src/infer_expr.rs
97.11% <100.00%> (+0.01%)
:arrow_up:

crates/crochet_infer/src/unify.rs
91.52% <100.00%> (ø)

crates/crochet_infer/src/unify_mut.rs
100.00% <100.00%> (ø)

crates/crochet_infer/src/util.rs
79.61% <100.00%> (ø)

crates/crochet_parser/src/lib.rs
94.33% <100.00%> (ø)

... and 2 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Try again using debian:8 for older glibc

Building on debian:10 results in a binary that requires glibc 2.27 which is not available on some LTS server Linuxes like CentOS 7. This commit attempts to make a compatible jffi.so using debian:8, which ships with glibc 2.19. This is not as old as 2.17 but may remove enough dependencies on newer glibc to be sufficient.
See #138
Original version of this PR did not fully update the builds to use debian:8.

Also confirmed working in #138. Ship it!


[SPARK-21479][SQL] Outer join filter pushdown in null supplying table when condition is on one of the joined columns

What changes were proposed in this pull request?
In left/right outer joins, constraints derived from the preserved side can be propagated to the null-supplying side (not the other way around). Meanwhile, we do not want to include the join condition in the constraints set since this is an outer join and otherwise is-not-null constraints would be inferred incorrectly.

Add method QueryPlanConstraints#additionalValidConstraintsForInference to accommodate this requirement from the Join operator.
Override QueryPlanConstraints#additionalValidConstraintsForInference in the Join operator.

How was this patch tested?

Added 3 new tests in InferFiltersFromConstraintsSuite.
Updated 1 test reference in ConstraintsSuitePropagationSuite.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1471/
Test PASSed.

Merged build finished. Test PASSed.

Test build #88185 has started for PR 20805 at commit deb4719.

Test build #88185 has finished for PR 20805 at commit deb4719.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88185/
Test FAILed.


Fix output of local state for idl list command

Hi there,
I'd like to push a fix for the idl list command upstream. It's really minor but fixes the output so that the column 'local' displays the state between the local service versions and the remote ones correctly.
Thanks,
Pascal

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Kitten timer stuck at 0

My kitten timer has gone down to 0 and the kitten has not aged

Fixed it


Just question about inject/provide

What problem does this feature solve?
Is there any reason that can not inject provided value in itself?
I see that initInjections(vm)  is being called before initProvide(vm)
https://github.com/vuejs/vue/blob/3cac5c766fc7b75cacf0a51dbfe10269eb3caf6a/src/core/instance/init.js#L56
so I see it's not simple
What does the proposed API look like?
it('hope this can work', () => {
  const vm = new Vue({
    provide: { foo: 1 }
  })

  const child = new Vue({
    parent: vm,
    provide: { foo: 2 },
    inject: ['foo'],
  })

  expect(child.foo).toBe(2)
})

for same usage both parent and child
import util from 'util'

const parent = new Vue({
  provide: { util },
  inject: ['util'],
  mounted(){
    this.util.sameCodeForUse()
  }
})

const child = new Vue({
  parent: parent,
  inject: ['util'],
  mounted(){
    this.util.sameCodeForUse()
  }
})

I'm using
computed: { util: () => util }
but It looks ugly

By having injected values resolved first, you can actually provide values based on them.

@rayrutjes Sorry I can't understand. please explain in detail?

Provided values should be able to rely on $data.
$data should be able to rely on injected values.
So the order must be:

inject
state
provide

@rayrutjes oh, I saw in source code.
so it means provide to it self should not happen, right? 😢

import util from 'util'

const parent = new Vue({
  provide: { util },
  computedLike: { util },
  mounted(){
    this.util.sameCodeForUse()
  }
})

const child = new Vue({
  parent: parent,
  inject: ['util'],
  mounted(){
    this.util.sameCodeForUse()
  }
})

If there is some option like computed but without function just inject property to this without watch will nice..


Implement incremental scans

cc @everplays - because you mentioned it.
There's two tiers to this:

Implement query parameters for the /search and /match APIs that allow a since= search. They would probably just filter on since > entity.last_changed. This gives you the ability to do incremental queries with the state managed externally.
Implement some sort of state in the API itself. In this scenario, a client would submit a /match-style entity and a callback mechanism (e.g. a webhook) and the API would then store this query, execute it at a set interval and notify the client whenever a new result is available. This could also be implemented a REST primitive, a la POST /alerts and GET /alerts?since=xxx.

I'm curious: which of these did you have in mind?

I think the first option would be less work and also more flexible. The second solution would require more infrastructure:

queue system (with a backend: redis, rabbitmq, sqs, ...)
storing match entries (probably just elasticsearch/opensearch)

About the first solution: do you think the client would be responsible for keeping track of since or we can expose a bit more info in /catalog?


[Feature] Add documentation to the wandb sweep page on gitbook-docs

Is your feature request related to a problem? Please describe.
I can't figure out how to use the wandb sweep cli tool to update a running sweep.
Describe the solution you'd like
I would like the documentation for wandb sweep to be filled out so that I can find out how to use the update function
Describe alternatives you've considered
I have tried to call the wandb sweep <sweep_id> --  but got an oblique error
<sweep_id> looks like xk2thesj (I changed a letter for privacy)
jsm81$ python3 -m wandb sweep <sweep_id> --update dim_latent_space=15
400 response executing GraphQL.
{"errors":[{"message":"name required for project query","path":["project"]}],"data":{"project":null}}
wandb: ERROR Error while calling W&B API: name required for project query (<Response [400]>)
Traceback (most recent call last):
  File "/home/home4/jsm81/.local/lib/python3.6/site-packages/wandb/apis/normalize.py", line 24, in wrapper
    return func(*args, **kwargs)
  File "/home/home4/jsm81/.local/lib/python3.6/site-packages/wandb/sdk/internal/internal_api.py", line 460, in sweep
    "specs": specs,
  File "/home/home4/jsm81/.local/lib/python3.6/site-packages/wandb/sdk/lib/retry.py", line 102, in __call__
    result = self._call_fn(*args, **kwargs)
  File "/home/home4/jsm81/.local/lib/python3.6/site-packages/wandb/sdk/internal/internal_api.py", line 133, in execute
    six.reraise(*sys.exc_info())
  File "/home/home4/jsm81/.local/lib/python3.6/site-packages/six.py", line 719, in reraise
    raise value
  File "/home/home4/jsm81/.local/lib/python3.6/site-packages/wandb/sdk/internal/internal_api.py", line 127, in execute
    return self.client.execute(*args, **kwargs)
  File "/home/home4/jsm81/.local/lib/python3.6/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 52, in execute
    result = self._get_result(document, *args, **kwargs)
  File "/home/home4/jsm81/.local/lib/python3.6/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 60, in _get_result
    return self.transport.execute(document, *args, **kwargs)
  File "/home/home4/jsm81/.local/lib/python3.6/site-packages/wandb/vendor/gql-0.2.0/gql/transport/requests.py", line 39, in execute
    request.raise_for_status()
  File "/usr/local/lib/python3.6/dist-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.wandb.ai/graphql

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/home4/jsm81/.local/lib/python3.6/site-packages/wandb/cli/cli.py", line 89, in wrapper
    return func(*args, **kwargs)
  File "/home/home4/jsm81/.local/lib/python3.6/site-packages/wandb/cli/cli.py", line 753, in sweep
    found = api.sweep(sweep_id, "{}", entity=entity, project=project)
  File "/home/home4/jsm81/.local/lib/python3.6/site-packages/wandb/apis/internal.py", line 97, in sweep
    return self.api.sweep(*args, **kwargs)
  File "/home/home4/jsm81/.local/lib/python3.6/site-packages/wandb/apis/normalize.py", line 26, in wrapper
    raise CommError(err.response, err)
wandb.errors.CommError: <Response [400]>

Error: <Response [400]>

Additional context
This is the web page i'm talking about:
https://gitbook-docs.wandb.ai/ref/cli/wandb-sweep

@JeremySMorgan we'll update the docs, but until then the easiest way to resolve this error is to provide the entire sweep path, i.e. "USERNAME/PROJECTNAME/SWEEPID"

@vanpelt @dannygoldstein Thanks for the quick follow up on this. Ya, it's not clear that the SWEEP_ID in CONFIG_YAML_OR_SWEEP_ID refers to USERNAME/PROJECTNAME/SWEEPID instead of xk2thesj (for example).
Side note - are you really supposed to pass your entire yaml file in as the argument to the wandb sweep? (Assuming that's what the CONFIG_YAML_ is in reference to?
Also - it looks like the docs haven't been updated https://gitbook-docs.wandb.ai/ref/cli/wandb-sweep

CONFIG_YAML_ refers to a file path that resolves to a YAML config file.  I thought @dannygoldstein updated the docs so it may just not be merged into master yet.

Sorry wasn’t merged yet, just merged now.

From: Chris Van Pelt @.>
Sent: Thursday, September 2, 2021 10:34:24 AM
To: wandb/client @.>
Cc: Danny Goldstein @.>; Mention @.>
Subject: Re: [wandb/client] [Feature] Add documentation to the wandb sweep page on gitbook-docs (#2617)
CONFIG_YAML_ refers to a file path that resolves to a YAML config file. I thought @dannygoldsteinhttps://github.com/dannygoldstein updated the docs so it may just not be merged into master yet.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/wandb/client/issues/2617#issuecomment-911909711, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAVEFYCYVMY3GFWWDO7CTEDT76YSBANCNFSM5DDPUEVQ.


python3Packages.flake8-builtins: init at 2.3.0

Description of changes
Add flake8-builtins to nixpkgs. It's a plugin to flake8.

Things done

Built on platform(s)

[ ] x86_64-linux
[x] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/) - tested behavior of flake8 when flake8-builtins is present with nix-shell -I nixpkgs=$(pwd) -p  'python39.withPackages(ps : with ps; [ flake8-builtins flake8])'
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 300097 run on x86_64-linux 1

4 packages built:

python311Packages.flake8-builtins
python311Packages.flake8-builtins.dist
python312Packages.flake8-builtins
python312Packages.flake8-builtins.dist

thanks for review
I realized tests weren't running, adding them now

Result of nixpkgs-review pr 300097 run on x86_64-linux 1

4 packages built:

python311Packages.flake8-builtins
python311Packages.flake8-builtins.dist
python312Packages.flake8-builtins
python312Packages.flake8-builtins.dist

Result of nixpkgs-review pr 300097 run on x86_64-darwin 1

4 packages built:

python311Packages.flake8-builtins
python311Packages.flake8-builtins.dist
python312Packages.flake8-builtins
python312Packages.flake8-builtins.dist


Test actions thoroughly

Our step by steps can now be in any of a number of different 'states', and accessed by a number of different 'actors', all of which affects the actions available on the step by step.
We've started a dedicated feature file to test that the correct actions apply at the correct times, but this PR fills in some of the gaps:

[x] Tests that there is the right quantity of Primary, Secondary and Tertiary actions
(won't silently fail if another action creeps in, for example)
[x] Test actions for when step by step is 2i approved
[x] Test actions for when step by step is scheduled
[x] Test actions for when step by step is published
[x] Test actions for when step by step is edited after publish

Trello: https://trello.com/c/Qle5c1QQ/43-write-tests-display-the-appropriate-buttons-depending-on-the-state-of-the-sbs-and-who-is-logged-in

Sorry to be a pain and don't let my comments stop this being merged as I can see this is really close and builds on existing work. More just adding this comment as food for thought (hopefully).
I think we're at risk here of unit testing the view through feature tests here. Which risks creating precedence/confusion in the purpose of feature tests - using classic testing paradigms this feels like inverting the test pyramid concept towards the test ice cream cone.
Where I think these stray from being feature tests is that we just check that a button is there, that itself is not a feature. It could be considered a feature that an approved step by step can be published for example, so there could be arguments that rather than testing presence of buttons these tests are scenarios of a feature - but it could become superfluous given they'll take you into existing feature journeys (I imagine the maths quickly adds up).
So I'm kind of wondering have we hit a point where we have a view that is sufficiently complex we should write a unit test for it (which would probably be more concise and much faster to run), if that's a smell that we have logic that should be tested elsewhere or if maybe we're double testing as a lot of these aspects are part of existing features?
Again sorry to be a pain, and interested in your thoughts - maybe I'm misinterpreting our approach.

@kevindew agreed - sounds like the view logic should ideally be in a presenter, and this test suite would be moved to a presenter unit test suite. 👍 Perhaps in a future PR?

@kevindew agreed - sounds like the view logic should ideally be in a presenter, and this test suite would be moved to a presenter unit test suite. 👍 Perhaps in a future PR?

Yup sounds good 👍


[Feature request] ColBERT V2

Hi @raphaelsty,
first of all, thanks a lot for your this project. I really appreciate its simplicity and effectiveness.
Question: do you have any plans to implement ColBERT V2?
Best wishes.

Hi @filippo82, I think it could be cool to add distillation loss of course. I plan to improve the loss function of the train module in the following weeks, there won't be any breaking change.
But before releasing a ColBERTV2 and its centroids algorithm I plan to release another method to accelerate ColBERT retriever, it will be really fast and still accurate, work in progress.

Hi @filippo82, I did release neural-cherche 1.1.0 which improve loss stability and brings better default parameters to models. Also I did release neural-tree in order to accelerate ColBERT. Feel free to open a PR if you are interested in Knowledge Distillation


Rewrite eloquent-sluggable adaptor and fake fields

I knew there must be a better way to handle JSON columns. This looks like a really good one.
We could use it to simplify the way JSON columns are used in:

fake fields
translatable fields usong eloquent-sluggable
fake translatable fields

test

Is that MySQL specific?

Doesn't look like it. It's called MySQLManager, but it uses Eloquent everywhere - so it should work for PosgreSQL and others too.

Please re-open if this is still relevant.

I think this is an important refactoring, but yes, it's not urgent so I agree to closing it.
/remind me to open this issue again in 2 months

Also:
/remind me to check if reminders work in 5 minutes
:-)


Custom enum/union parsing/serialization

Hi, I want to serialize a type such as:
type State =
    | None
    | OverflowedState
    | RestartedState

type Main =
    { State: State
    }

to have the following JSON:
{
    "state": "<none|overflowed_state|restarted_state>"
}

I tried using transforms for this, but they don't seem to get called for enum/union values. Is there any way to do it?

ping?

Any changes here?

I also have a thought about marking each Enum so it can be like this:
type Encryption =
    | [<JsonField("aes-128-gcm")>] AES // serializes to and deserialize from "aes-128-gcm"
    | [<JsonField("chacha20-poly1305")>] ChaCha20 // serializes to and deserialize from "chacha20-poly1305"

Can borrow scheme and some impl from https://github.com/eiriktsarpalis/TypeShape/blob/main/src/TypeShape/Applications/UnionContract.fs#L9


Better module support

We've done a lot of work to not fail a template bases on Modules.  However there are a lot of things we can do to build out better support.
Reading the registry to determine the parameters and resource types.  This allows us to validate naming conflicts, Ref/GetAtt relationships, if a module parameter is being used by a condition which means you can't ref/getatt that value, limits, etc.
Some of the big work to be done include:

We need to support the Registry schema for the rules describe above.
We need to download the registry schema from a users account/region. This would include us needing to have credentials to get these files.  Today cfn-lint doesn't require credentials and was completely designed to be offline validation. However with Registry Resource Types and Modules we need to think about starting being able to query an account inside cfn-lint.

We need to download the registry schema from a users account/region. This would include us needing to have credentials to get these files. Today cfn-lint doesn't require credentials and was completely designed to be offline validation. However with Registry Resource Types and Modules we need to think about starting being able to query an account inside cfn-lint.

Was thinking we could provide a script that could be run separately and allow the output to be passed in similar to https://github.com/aws-cloudformation/cfn-python-lint/pull/1732

However with Registry Resource Types and Modules we need to think about starting being able to query an account inside cfn-lint.

+1


add variable for horizon timeout

add variable for horizon timeout, with default value of 1800

This LGTM. IT won't run tests because it's a PR from a fork, but this is very simple. Just waiting for the master one to merge for "proper procedure".

Still LGTM


Add search to Study List

Define StudyListFilter
Add StudyListFilter to StudyList view
Add search boxes for short name and full name in StudyList view

looks good, thanks!


create Vue.js wrapper

In addition to the React, Polymer wrapper

see -> https://github.com/datavisyn/vue-lineup


Possible License

I love this minimal Hugo theme that you've created. Any plans to add a license on this repo?

Just going to bump this back up. I'd love to use this theme (and contribute!).


feat(bqls): Add bqls support

Adds support for BigQuery language server.
thanks @kitagry

Thanks. But we have some rules for new servers. Please see https://github.com/neovim/nvim-lspconfig/blob/master/CONTRIBUTING.md#adding-a-server-to-lspconfig
could you provide some more information about this server how popular it is

I would say not very popular (yet!) but maybe @kitagry has some more context (there is a plugin that adds the config dynamically).
I was hoping that adding support would increase the exposure of this LS, which appears to provide better support for BigQuery than the existing sqlls server.

It's a great server. But it's only 7 stars at the moment. And it doesn't have the same download volume as other editors like vscode. So I think we can wait a little longer to avoid our repo becoming bloated with some small and unpopular server configuration.


Add 3.7 features to python wrapper

Adds a number of new features to the python wrapper:

af_pad
meanvar
inverse square root
pinverse to python wrapper
NN convolve and gradient functions to wrapper
reduce by key
confidenceCC

Not sure, do we need it separately ?


Show if a product belongs to one of the lists on the summary tab

Show if a product belongs to one of the user's lists on the summary tab

I will work on this.


Lacking defaults

When deploying without Celery and RabbitMQ, the environment variables will still have to be set, as other roles (NGinx, Web) still depend on those variables.

Some conditionals would help.


single color channel

Hi! I've seem many other post mentioned batch size is essential here to keep the training stable. However I do only have one GPU which only allows me to use batch size under 8. I wonder is there any ways that I can reduce the number of RGB channel of the model to 1? The task / dataset I'm working on is relatively simple and can basically ignore the colors.
Thanks in advance!

Hi @petercmh01. I think reducing the number of image channels won't affect memory too much. I believe the vast bulk of memory consumption comes from the intermediate layers of the StyleGAN generator and the Spatial Transformer which aren't affected by image channel size.
There are a few things you can try to get good performance if you have 1 gpu. Probably the most surefire way is to use gradient accumulation (you can find an untested code snippet in Issue https://github.com/wpeebles/gangealing/issues/10 which uses 8x gradient accumulation). Using grad accumulation will increase your effective batch size. The downside is that training will be quite a bit slower, as for every one backward pass you'd need to multiple forward passes. For what it's worth, it may be that a batch size of 16 or 24 is enough, so you could try using only 2x or 3x rounds of accumulation.
Another thing you could try is reducing the --stn_channel_multiplier argument to e.g., 0.25. This argument controls the number of channels in intermediate conv layers of the Spatial Transformer. Using a smaller value should let you use a bigger batch size. My guess is that you can get away with a smaller STN, although it isn't something I've tried before.
Finally, I would recommend using our training scripts which use LPIPS as the perceptual loss. It seems that LPIPS may be more stable at small batch sizes than the SimCLR VGG perceptual loss, although from my experiments it still has degraded performance with a batch size of 5.

I'll go ahead and close this issue now but feel free to reopen or make a new issue if you have other questions.


Better way to make tcp_info stats available in access logger

Title: Better way to make tcp_info stats available in access logger
Description:
We need a way to log tcp_info stats in access logger, we thought of putting these stats as DYNAMIC_METADATA on connections stream_info object. Please suggest if there is a better way to do this?

Thanks, will add option for this.


[HW] Atharva Patil

Name: Atharva Patil
Email<EMAIL_ADDRESS>Linkedin Profile: https://www.linkedin.com/in/atharva-patil-1a341a174
Attach the homework step I screenshots below:

-- image removed by team --

Attach the homework step II screenshots below:

Thank you for the Amazing Workshop!

Hey Atharva Patil, great job! Congrats! Here is your badge! https://api.badgr.io/public/assertions/0JTEyeRcTUG705-n6ayMkw

Nice job Atharva, looks good, thank you.  Please delete your token from the Astra UI to be safe.


Use native Array.isArray

Hello @kt3k, thanks for contributing. LGTM -- I'll merge, but before I do a couple things:

How about instead of repeating Array.isArray we do this:
- var isarray = require('isarray');
+ var isArray = Array.isArray;

Then of course change all isarray => isArray.

I see that you retained isarray in package.json and bumped the version. Did you try to determine whether there's anything else still using that dep? Maybe we can just get rid of it now.

Notes: according to my research v8 added Array.isArray in v1.3.16, which would make it available from Node v0.1.15.

var isArray = Array.isArray;

OK. That looks better. I'll change it.

I see that you retained isarray in package.json and bumped the version. Did you try to determine whether there's anything else still using that dep? Maybe we can just get rid of it now.

I keep isarray in devDependencies because it's used in test/array.js as an example of bundleing of npm modules. It seems purely used as an example of npm module, and I think it's ok to keep it (because it only affects tests).

@kt3k Thanks!

I keep isarray in devDependencies because it's used in test/array.js as an example of bundleing of npm modules. It seems purely used as an example of npm module, and I think it's ok to keep it (because it only affects tests).

Oh ok, sorry, I hadn't noticed that you moved it to devDeps. Nonetheless, if that's the only thing it's used for I'm going to see if there's something else that we're depending on for some other reason that can be substituted so we can just eliminate isarray.

I replaced test cases which use 'isarray' by ones using 'xtend' instead, and removed 'isarray' from devDeps. Looks cleaner!

@kt3k Thanks! I had actually done that part myself locally yesterday but hadn't had the chance to push yet. I took a look at your latest commit and I certainly appreciate you doing that, but for that part I used what I committed yesterday only because it's a smaller diff. I replaced isarray with defined which ended up with the same assertion methods and number of assertions (and even expected value!).
I squashed your first 2 commits together and merged via #1555. Thanks again!

My pleasure! Thanks for merging 😄
Appreciate your quick response!

Published in 13.0.1. Not to discourage you from contributing, but just FYI -- responses usually aren't so quick.


Proposal: podcast:events tag, enabling podcast publishers to receive unforgeable inbound events from other entities in the open podcast ecosystem

I've been listening and thinking about several conversations in the podcast world recently, and I think it would great if we could come up with a solid way of letting podcasters receive inbound events from verifiable 3rd party actors (podcast apps, directories, other hosts) with useful information for them.  For example, a way for trusted podcast apps to send unforgeable information about subscribers/play information back to the show.
Ideally, we could do this in a way similar to the other podcast namespace tags, that kept the podcast feed publisher in control, and as the source of truth, while keeping things as simple as possible and without needing preauthorization or shared secrets, or certifications, etc.
Instead of a wall of markdown here in the github issue, I've started a quick Google doc with the general overview of a system I think could work well, and am publishing it publicly to see what everyone thinks and get the conversation started:
[Google Doc] Proposal: <podcast:events> tag, enabling podcast publishers to receive unforgeable inbound events from other entities in the open podcast ecosystem
It's meant to be readable by anyone, so I tried to avoid rabbit-holing on a bunch of detailed example code, but there should be enough in there for folks to get the idea.
2022-10-06: Published a new skymethod/podcast-events GitHub repo with a full implementation of both the sending side and the receiving side.  Also a validator to test client implementations at: https://podcast-events-validator.op3.dev/. Both linked in the proposal doc

I foresee some feet-dragging for app-developers to support this, but I do really like the idea and see a ton of potential for it!
I have a few thoughts for potential improvement.

I suggest making it <podcast:webhook> instead of <podcast:events>. I think "webhook" communicates its purpose better, avoids potential confusion, and leaves the "events" word usable for other plain-English uses (like a way for a podcaster to indicate layman "events," like a meetup, a live-stream, etc.). And before anyone says, "It doesn't matter what we call it, and only developers will see it," I think it does matter what we call it because podcasters need to be able to easily understand it, ask their publishing tools about it, and for it to be easy to educate them.
Your example showed this in <channel>. I think there could be some uses for putting it in <item> tags, too.
For any kind of episode event, I think we should send the item GUID instead of the enclosure URL because the GUID is supposed to never change for an episode, but enclosure URLs can change at any time.

I could see such a webhook receiving all kinds of information (not necessarily that these are the best ways to send such information, but this is only for brainstorming), whenever:

someone follows or unfollows the show,
someone comments on an episode,
someone shares an episode,
someone deletes an episode without finishing it,
the podcast is featured in the app catalog,
the podcast receives a rating or review,
the podcast's catalog listing is visited from an in-app ad (paired with when someone follows a show, this can provide great conversion data!)

This seems quite close in functionality to NPR's RAD proposal, and I wonder whether it's worth reviewing that for ideas and gotchas of how it was proposed there. Particularly, the opportunity to request the app to "tell me if this point was played" was helpful for program research and potentially for ad delivery.
Improvements on it are the use of signed data, and as @theDanielJLewis says, it could be extensible with a number of different ideas.
I like the webhook name suggestion. Like switching text to txt, I think it's important to use unambiguous language that helps a rookie get up to speed as fast as possible.
I would also suggest it's unlikely to get take-up in apps: unless a) someone pays them to implement it; or b) someone associated with this proposal builds an app that we can get behind

Some differences between this and RAD (spec):

RAD is primarily designed to report advertising events (see their event properties), this proposal is primarily designed to raise the level of quality listen metrics available to publishers without taking a step back on listener privacy in any way
It is not helpful to advertisers directly, only in the aggregate sense that publishers will have a better idea how many of their downloads were actually listens, for participating podcast apps
RAD events can be easily forged, the 'listen' events in this proposal cannot
RAD events require embedding information into ID3 tags (!), this proposal does not

More context around this proposal:  there has been some indications by some of the large podcast apps that they would be willing to send this listen data (which they already collect) securely to publishers in order to help raise the quality bar and prevent overbilling, but will only do it if it does not take a step back on privacy in any way.
The mechanism this proposal uses was specifically chosen to be something that Apple, for example, would be willing to implement - once a consensus is reached on the payload.  The auth mechanism (JWT with public JWK url sets) in particular was chosen since it's not only an industry standard these days, but also used by Apple in other callback apis where security is key (Sign-in with Apple / App Store In-app purchase callback notifications), ie their security team has already blessed something similar.

Re: embedding in apps, this proposal is strictly server-to-server, since each request needs to be signed with the private side of the RSA keypair.  The private side would never leave the sender's servers or be embedded in a client app or website, etc.
The flow would be that the app sends listen data down to their servers in the same way they do today, then periodically send it server-to-server for feeds that declare <podcast:events>

After reading through everything, I wonder would this be doable as podping notifications?  Same payload, but without the need for JWT since Hive does the authentication and it requires staking.  Perhaps you are worried about message volume overwhelming Hive?

Bringing up the difference with RAD inspired a thought. This kind of webhook indicator would receive anything the app sends it. But should we consider a way for podcasts to indicate (probably in the episode JSON data) particular triggers? Like ping the webhook if X chapter is completed, or if XX:XX point is played, or if X% is played.

I wonder would this be doable as podping notifications?

The thing is I think we want something that the app sends securely, and targeted only to the show, and the show's declared third parties.  Hive would leave the listen data wide open, right?
I guess there would be nothing preventing someone from starting a service that implemented the receiving side of this protocol (something DJ could do in a weekend!), got used by podcasters in their feeds, and turned around and republished that data back out to Hive or anywhere else - but it's the podcaster's choice in this case.

Listening to the board meeting with Dave and Adam.
I like the idea of a different example rather than starting with "listens", given the almost instant "but NPR RAD, blah blah" reaction.
One that seems to be a great and simple usecase is "reviews". When I get a review on a podcast in Podverse, send that review back to me using a webhook.
I can absolutely build a receiver for Podnews (it would be super dumb, just capturing the info in a database) if you need a podcast using one.

One that seems to be a great and simple usecase is "reviews"

I like this idea! I'll try to compile a quick summary of all of the great ideas from folks I've heard so far and put them in the doc, and provide ratings payload examples alongside listen info so it doesn't seem so tied to one use case.
Strawman ratings item-level payload(s):
Like all event payloads, fields common to all events can be included once in the top-level object instead of repeated in every item-level payload.
All fields are required except the ones marked optional.
Rating summary
Could be sent daily/weekly if the app doesn't want to send every rating/review.
{
  "kind": "rating-summary",
  "asof": "2022-10-08T16:00:00.000Z",
  "rating": 4.7,
  "ratingScaleMin": 1,
  "ratingScaleMax": 5,
  "count": 955,
  "episodeUrl": "<episode url that was downloaded>", // optional, only for apps that support episode-level reviews
  "feedUrl": "<feed of the subject show in question>",
  "userAgent": "<user agent that downloaded the episode>",
  "referer": "<referer that downloaded the episode>", // optional, only applicable for webapps
}

Rating
For apps willing to send every rating/review.
{
  "kind": "rating",
  "time": "2022-10-08T16:49:00.226Z",
  "rating": 4,
  "ratingScaleMin": 1,
  "ratingScaleMax": 5,
  "review": "Hats off to Jane Doe who excels in character development and tells a great story with brilliant historical context.",
  "author", "JohnDoe",
  "episodeUrl": "<episode url that was downloaded>", // optional, only for apps that support episode-level reviews
  "feedUrl": "<feed of the subject show in question>",
  "userAgent": "<user agent that downloaded the episode>",
  "referer": "<referer that downloaded the episode>", // optional, only applicable for webapps
}

Consider what could be done with a tool like Zapier or IFTTT. Both services can offer webhooks for listening. So doing something in the podcast app could trigger a Zapier or IFTTT action.
One (not recommended) example could be a system to auto-tweet whenever my show gets a new follower, and then promotes my follow page.
@johnspurlock, I have a ton of thoughts of how to best format rating and review data (since that is literally my business), but we can discuss that later.

The key thing about this proposal is that the requests are signed with strong encryption and need to be verified by the receiver before being processed or forwarded on.  It's a way to setup an unforgeable/unspoofable channel from the podcast app -> feed owner (podcaster/podcast hosting company).
So as long as first receiver (the inboxUrl declared in the tag) can do the encryption necessary to verify the signature of the payloads coming in, it would be ok.
I'm thinking that the flow would be that the podcast hosting company (or 3rd party) declared as the inboxUrl in the tag would be the first receiver, trusted to properly verify the events and handle policy around which apps are trusted.  Then provide checkboxes/fields in their podcaster-facing CMS for various event hooks like: "rating webhook url", which could then be ifttt/zapier/webhook-co-of-the-month

Perhaps there could be a public key, or some simple authentication process, that podcasters could give to various apps, or else there's a central list of approved apps allowed to send payloads to the webhook URL.
For example, Libsyn would already know have the signature stuff in place to authenticate anything from Podverse, Overcast, Apple Podcasts, and such. But for any fringe app or service, the podcaster needs to get a public API key from their hosting provider (or Zapier) that they provide to that app in order to give them permission to send the data. Anything gets block that comes in that's not preapproved by the webhook provider or using a key the podcaster has authorized.

Hello
Coming to this late conversation but I am working no a new app that has an ActivityStream with specific verbs like Follow, Play, Clip, Comment etc.  - see image below.  This was going to remain internal to the app but with Podcast Events I am very happy to  adapt the stream to use this tag.  So Daniel's brainstorm above is exactly what we have developed.  Once it has been agreed as a tag it will probably only take a few days to adapt the code.

Can we please quickly change this to <podcast:webhook> before we go much further?
"Events" is a geeky term, and I believe the intention of RSS was to maintain some level of human readability. But more important than that, I'd like us to reserve the "events" name for a different use, like allowing podcasters to promote actual events (conferences, live-streams, meetups, giveaways, promotions, deadlines, etc.) in their podcasts through a special tag.
I also think we could simplify the tag by changing inboxUrl to simply url. After all, wouldn't any URL we use be an inbox URL? It's fewer characters and more straightforward.

@theDanielJLewis I'm happy to bikeshed the names later, but right now let's keep the documentation/artifacts stable while people are reading it - waiting to get feedback from a wider group of people.  One thing to keep in mind is that while the tag only has one attribute today, it might have additional useful properties in the future (like the receiver's verifiable identity etc) so I wouldn't want to limit it to a single url (hence inboxUrl in my proposal).
@samsethi Sending user activities down to the show is a really neat idea.  This looks a lot like activitypub, but perhaps this is a quicker way to jumpstart, the publisher could always turn around and resyndicate over activitypub or whatever they like.  A big advantage this proposal has over activitypub is that the events are verified to be coming from an app with a strong public identity. So apps have the incentive not to spam, since the receiver can easily decide to drop all incoming events from their app, or events of a certain type.
Would love to brainstorm with you about what the payload for these user activity events would look like.   How to identity a user, and keep the actions freeform enough to support verbs not yet considered.

@johnspurlock yes it was the old ActivityPub standard but I am not wedded to it.  Likewise love to brainstorm this further with you. Will DM you.

Worthwhile linking to this today - a feature that I think podcast:events could help with.
https://wearebumper.com/blog/2023/01/16/listen-time/
I've pointed the author here, too.


Add support for Arduino, Teensy 4 and 4.1

What's new
Added lots of changes to allow the stack to build without threading under the Arduino IDE. Added CAN plugin for Teensy hardware using the FlexCAN T4 api.
Changes include:

Added single threaded CANHardwareInterface option. Changed dynamic casts to static casts to completely eliminate rtti, which shouldn't be an issue since the type can be inferred in those cases where we were using dynamic_pointer_cast.
Added preprocessor checks to optionally compile out all threading. Added a script to automatically generate an Arduino library from the content of the repo.
Added Arduino example files.
Added a python script to automatically generate an Arduino library based on the content of the repo.

Tested with Teensy 4.1 hardware and SN65HVD230 transceiver.
Follow-up tasks

[x] Create new repository to store the latest-generated Arduino library version of the stack
[x] Submit library to the Arduino library registry
[ ] Add CI pipeline for building with Arduino
[ ] Investigate adding pipeline that can automatically generate the Arduino library and commit it to the other repo, plus open a PR
[ ] Create documentation for using the library on Arduino

How to use/test this right now
Until all the tasks above are done, here's how to try this on Teensy 4.1 on Windows.

Install Arduino IDE
Add Teensy board support to Arduino IDE by following these instructions
Open the example file in your Arduino IDE.
Select your board as shown:

Ensure optimizations are set to "Smallest Code"

There's some kind of bug in the Teensy's core, where if you are not building with the default optimization settings, the board fails to start up. Some attempts appear to have been made to fix this but it remains an issue for me at least. To ensure your board starts up, add #pragma GCC optimize ("O2") to the teensy's startup.c file which is likely located at the following location: C:\Users\<your user name>\AppData\Local\Arduino15\packages\teensy\hardware\avr\1.58.1\cores\teensy4

Run the generateArduinoLibrary.py script with Python in the root of the repo. I usually do this in IDLE because I'm a Python neophyte basically. I know C++ things, what can I say.

Copy the newly created arduino_library folder that the script creates to your Arduino libraries directory. You may need to create this directory if it doesn't exist, but it's usually located at C:\Users\<your user name>\Documents\Arduino\libraries. Maybe rename the arduino_library folder to AgIsoStack or something if you want.
Build the sketch, and upload it to your board 🙂

You'll also need a CAN transceiver connected to your Teensy's CAN1 pins.
Closes #264

Added to Arduino library registry here: https://github.com/arduino/library-registry/pull/3320


Typos and code style improvements

There is a lot of changes but most of them are about improving code style and typos.
Also as you can see I am pushing the ShotGrid namespace rather then Shotgrid. It looks nicer but this is really just my opinion ;)

please don't try merge this PR until the bigger PRs https://github.com/ynput/ayon-shotgrid/pull/72, https://github.com/ynput/ayon-shotgrid/pull/79 are merged. And like @m-u-r-p-h-y said, if anything we should update to Flow now

We are not planning to rename it any time soon to Flow

I don't get why did you merge this? Didn't I ask yesterday to please hold on this until the bigger PRs are merged? now it makes this pending PR https://github.com/ynput/ayon-shotgrid/pull/79 harder to merge and https://github.com/ynput/ayon-shotgrid/pull/72 contains a lot of things not specific to those changes


fix Elgin, Ontario

new source:
https://gisdata.elginmapping.ca/arcgis/rest/services/Public/MapServer/1
changed broken issue:
http://results.openaddresses.io/sources/ca/on/elgin

Having a hard time caching this one :(


App throws error in Outlook

Reporting on behalf of partner Typeface. Typeface app shows in Outlook even though it doesn't have supported capabilities and throws an error:

To repro:

Install and sign into Typeface app from Teams
Open Outlook (with targetted release turned on)
Compose a new email and try click on the Typeface app from the app menu in the new email

@ktcodebite - Thanks for reporting your issue. We will test this and update you soon.

@ktcodebite - Thanks for reporting your issue. Could you please track this issue here: https://github.com/MicrosoftDocs/msteams-docs/issues/9335

Just a heads up that the issue you linked is a different problem but with the same app

@ktcodebite - We are checking this internally with engineering team, we will keep you posted on the updates. Thank you.


Documentation/Maintenance/Dashboard.md refers to a link that is 404

In Documentation/Maintenance/Dashboard.md there is the command:
$ git remote add -t dashboard origin https://itk.org/ITK.git
But that fails.
Where has this moved? To github somewhere I guess?

Yes, we have the same dashboard branch on GitHub.


cm8jl65: move to PX4Rangefinder and cleanup

Can you explain if there is any functional change, that we expect to address #12987? I mean, besides the refactor to PX4Rangefinder? It's tricky to see since the diff is mostly entire files moving around, even if I look at the commits individually. Did you lower the uORB priority?

Can you explain if there is any functional change, that we expect to address #12987? I mean, besides the refactor to PX4Rangefinder? It's tricky to see since the diff is mostly entire files moving around, even if I look at the commits individually. Did you lower the uORB priority?

For the cm8jl65 (and all drivers) I was already going to do this refactor eventually, but thought it would be worth a try for #12987. Although now that I've looked in more detail I have other questions. https://github.com/PX4/Firmware/issues/12987#issuecomment-533814751
There's no intentional functional change, but there are often unintended eccentricities in each driver that "shake out" after a refactor like this. The goal is to try and distill each driver down to the simplest form where we can mainly just get the raw data out to the rest of the system. PX4Rangefinder is a place to consolidate all the orb and metadata boilerplate common between classes of driver that is often handled inconsistently.

The changes all look fine to me and are functioning as expected when tested on the bench.  I understand there are intermittent issues with Catkin right now, so I think you're probably good to go when you see fit to merge in your work!


buy and sell stock trades information and plot

How to get buy and sell stock trades information and plot after training for getting obvious results about what have done during training and for backtest part.

There are some Jupiter notebooks tutorials in FinRL. I think your question is covered in the notebooks. For example, the codes below in FinRL_stock_trading_NeurIPS_2018.ipynb can achieve this for backtest part:
df_account_value, df_actions = DRLAgent.DRL_prediction(
    model=trained_a2c,
    environment = e_trade_gym)
df_actions.head()

If you want to see the actions in the agent's learning (training) process, you can call Env.actions_memory, e.g., StockTradingEnv.actions_memory.
Hope this helps.


Add .DS_Store to .gitignore

Consider adding .DS_Store to .gitignore file.

The only reason it's not there already is that our old Boxen config added a .gitignore file to everyone's home directories, which some of us have continued using: https://github.com/tylersticka/dotfiles/blob/master/home/.gitignore