Storage chains part 1

Initial groundwork towards supporting storage chains in substrate. This includes transactions addressable in the database and block/transaction pruning.
Future PRs will include:

Serving transactions over IPFS bitswap
Ref-counted transactions and renewals.
Offchain worker API
Runtime module

polkadot companion: paritytech/polkadot#2253

Can you explain what "storage chain" means?  Why is IPFS involved?  What purpose it serves, aka what is the threat model?

Can you explain what "storage chain" means?  Why is IPFS involved?  What purpose it serves, aka what is the threat model?

The idea is to maximize on-chain storage efficiency. Here's Gav's writeup on the feature.
v1.0: Finite history direct storage chain:

6 second blocks (10/min), 4 MB max extrinsics/block
IPFS-addressable extrinsics (new feature)
This chain doesn't have its own internal economy but rather uses a DOT derivative.
State only handles data market and maintains upcoming renewals. Each block has an array of sizes, one for each extrinsic, rounded up to 256 bytes. Assuming max 16MB extrinsics a 24-bit size), we can drop the lower 8-bits to get a Vec<u16>. Assuming 40KB average item size, then ~100 items per block, =~ 200 bytes state per block. 100,000 state entries for each block in 7 day period, and ~20 MB state.
Each block from the last 7 days also has a 32-byte Merkle root stored in state. The trie is formed (in-memory) from the 32-byte chunked extrinsics (or perhaps use some other chunking means), indexed by (extrinsic_index, chunk_index). This should be checked as part of block verification.
Creating a new block requires solving a short proof-of-work. The seed for the PoW is derived from several randomly selected bytes from the block 7 days prior (and which is now going out of scope); the proof of these bytes as well as the PoW solution is included on-chain. The risk of short-term forks may first be reduced through a proof-of-stake mechanism which applies different difficult requirements to each potential block author.
Block authors are paid out 50% of the fees immediately, and 50% is placed in a pot for them to split pro-rata weekly.
New data extrinsics charged per-byte (to pay for network storing a byte for 7 days); renewals charged per-byte on their original data (the size is in state). A single fee for the book-keeping of the size payable, too. Since everything is fixed-timespan, there's no issue with fee-charged (as opposed to deposit-taking).
Network maintainers (i.e. full nodes) are required to retain
Assume only 7-day block history (and all current state) needed for sync of full nodes.
All data (extrinsics) kept for minimum 7 days. Data can be renewed with a simple on-chain tx (no need to resend data).
Nodes throw out old (> 7 days) blocks and their data.
Total max block data needed to be stored by full-nodes: 7*24*60*10*4MB =~ 400GB
Needs fast-sync (warp sync not so important) along with only partial block history download. Syncing beyond 7 days of history is designed to be needless for running a block-authoring full-node.
An optimised publication of blocks allow extrinsics whose hash already appears in chain history to be ommitted. For normal sync-behaviour, the full block (all extrinsics) remains (to ensure that nodes which are not yet synced and don't have full history can still sync properly)

The idea is to maximize on-chain storage efficiency. Here's Gav's writeup on the feature.
v1.0: Finite history direct storage chain:

6 second blocks (10/min), 4 MB max extrinsics/block
IPFS-addressable extrinsics (new feature)
This chain doesn't have its own internal economy but rather uses a DOT derivative.
State only handles data market and maintains upcoming renewals. Each block has an array of sizes, one for each extrinsic, rounded up to 256 bytes. Assuming max 16MB extrinsics a 24-bit size), we can drop the lower 8-bits to get a Vec<u16>. Assuming 40KB average item size, then ~100 items per block, =~ 200 bytes state per block. 100,000 state entries for each block in 7 day period, and ~20 MB state.
Each block from the last 7 days also has a 32-byte Merkle root stored in state. The trie is formed (in-memory) from the 32-byte chunked extrinsics (or perhaps use some other chunking means), indexed by (extrinsic_index, chunk_index). This should be checked as part of block verification.
Creating a new block requires solving a short proof-of-work. The seed for the PoW is derived from several randomly selected bytes from the block 7 days prior (and which is now going out of scope); the proof of these bytes as well as the PoW solution is included on-chain. The risk of short-term forks may first be reduced through a proof-of-stake mechanism which applies different difficult requirements to each potential block author.
Block authors are paid out 50% of the fees immediately, and 50% is placed in a pot for them to split pro-rata weekly.
New data extrinsics charged per-byte (to pay for network storing a byte for 7 days); renewals charged per-byte on their original data (the size is in state). A single fee for the book-keeping of the size payable, too. Since everything is fixed-timespan, there's no issue with fee-charged (as opposed to deposit-taking).
Network maintainers (i.e. full nodes) are required to retain
Assume only 7-day block history (and all current state) needed for sync of full nodes.
All data (extrinsics) kept for minimum 7 days. Data can be renewed with a simple on-chain tx (no need to resend data).
Nodes throw out old (> 7 days) blocks and their data.
Total max block data needed to be stored by full-nodes: 7*24*60*10*4MB =~ 400GB
Needs fast-sync (warp sync not so important) along with only partial block history download. Syncing beyond 7 days of history is designed to be needless for running a block-authoring full-node.
An optimised publication of blocks allow extrinsics whose hash already appears in chain history to be ommitted. For normal sync-behaviour, the full block (all extrinsics) remains (to ensure that nodes which are not yet synced and don't have full history can still sync properly)

Wondering a bit about addressing by something like 'BlockID ++ Extrinsic index' instead of extrinsic hash.
But I guess it doesn't make sense if extrinsic/transaction hash is going to be replaced with ipfs address and support ref counting?

Right, IPFS content is addressable by content hash. And the idea with renewal transactions is that that they put the same content on the chain by increasing the reference counter.
'Transaction' usage in naming and description instead of block extrinsic confused me a little, but it is probably related to the targeted functionalities (which I am unaware of).
My understanding that we use the term "transaction" for user generated pieces of block body. And "extrinsic" should not be user facing. So I've used "transaction" in the database layer to underline the fact that this is intended for storing blocks of user data, addressable by data hash.

Wondering a bit about addressing by something like 'BlockID ++ Extrinsic index' instead of extrinsic hash.
But I guess it doesn't make sense if extrinsic/transaction hash is going to be replaced with ipfs address and support ref counting?

Right, IPFS content is addressable by content hash. And the idea with renewal transactions is that that they put the same content on the chain by increasing the reference counter.
'Transaction' usage in naming and description instead of block extrinsic confused me a little, but it is probably related to the targeted functionalities (which I am unaware of).
My understanding that we use the term "transaction" for user generated pieces of block body. And "extrinsic" should not be user facing. So I've used "transaction" in the database layer to underline the fact that this is intended for storing blocks of user data, addressable by data hash.

The idea is to maximize on-chain storage efficiency. Here's Gav's writeup on the feature:

What does it store?  And why?
Ignoring parachains internal archetecture, there are three storage security models proposed for Polkadot:

Relay chain blocks and state known by all validators.
Availability store data guaranteed available by validators, but efficiently erasure coded across all validators.  It expires data after 24 hours currently.
XCMP messages provided by a sending block's approval checkers.
Archival long-term storage where we make zero security guarantees and data could simply vanish forever, but it theoretically exists for curious people.

If I understand, you've designed an archival-ish storage system with zero security guarantees, meaning PVFs cannot ever depend upon data stored here, yes?  In other words, anytime a PVF wants data stored here it must slurp said data back into its own block, which then makes the data available for validators.  If so, why should this require Polkadot integration?
Also, you expire data after 7 days.  Why archive such a short time?
You're doing so as a parachain.  Why?  As a parachain, this pushes its data into the availability store, meaning this costs a parachain exactly what placing data into its own blocks costs.

* IPFS-addressable extrinsics (new feature)

What does this mean?

* State only handles data market and maintains upcoming renewals.

So there is zero user data in these block?  Only IPFS addresses for which storage nodes store the data?  If so, yes this makes sense as a storage model.  :)
Why would this require any special polkadot integration though?  We cannot use this data from polkadot directly so why not do everything inside the parachain's runtime?
Is the 7 days just an accounting feature?

* Each block from the last 7 days also has a 32-byte Merkle root stored in state. The trie is formed (in-memory) from the 32-byte chunked extrinsics (or perhaps use some other chunking means), indexed by `(extrinsic_index, chunk_index)`. This should be checked as part of block verification.

What does this mean?

* Creating a new block requires solving a short proof-of-work. The seed for the PoW is derived from several randomly selected bytes from the block 7 days prior (and which is now going out of scope); the proof of these bytes as well as the PoW solution is included on-chain. The risk of short-term forks may first be reduced through a proof-of-stake mechanism which applies different difficult requirements to each potential block author.

I'm confused:  A priori, I'd envision this being some parathread, so surely paying the parathread fee mostly suffices, no?  Are we talking huge amounts of data here?  Like what?

* Block authors are paid out 50% of the fees immediately, and 50% is placed in a pot for them to split pro-rata weekly.

Is the block author hosting all the user data?  If not, they're not doing much work, so no reason to pay them.  We've no mechanism to enforce they serve the data obviously, but that's fine since we'll never depend upon the data.

* Network maintainers (i.e. full nodes) are required to retain

Who are these?  Not Polkadot full nodes obviously.  It's whoever joins this parachain?

* Assume only 7-day block history (and all current state) needed for sync of full nodes.

* All data (extrinsics) kept for minimum 7 days. Data can be renewed with a simple on-chain tx (no need to resend data).

That's useful.

In short, I'm unsure why this requires any polkadot integration, given that polkadot cannot depend upon archival-like guarantees.
Just fyi, we'll later expand availability store usage for required data:  At present, anytime a parachain calls set_code we require all polkadot nodes see its code immediately.  In theory, parachains could post_code with their own parachain candidate block, which then stores the new code into the erasure coded availability store.  After we finalize the relay chain block including post_code parachain candidate, then the parachain could switch_code into the new code.  At this point, we've perhaps only one polkadot validator who possesses the code.  Yet, any who require this code could fetch it like approval checkers fetch the blocks they check, thanks to the availability store.  We'll require this for hierarchical/multiple relay chain operation, and maybe it'd make polkadot more efficient even now, but currently we're hoping code churns slowly enough to make it worth storing parachain code on the relay chain.

The idea is to maximize on-chain storage efficiency. Here's Gav's writeup on the feature:

What does it store?  And why?
Ignoring parachains internal archetecture, there are three storage security models proposed for Polkadot:

Relay chain blocks and state known by all validators.
Availability store data guaranteed available by validators, but efficiently erasure coded across all validators.  It expires data after 24 hours currently.
XCMP messages provided by a sending block's approval checkers.
Archival long-term storage where we make zero security guarantees and data could simply vanish forever, but it theoretically exists for curious people.

If I understand, you've designed an archival-ish storage system with zero security guarantees, meaning PVFs cannot ever depend upon data stored here, yes?  In other words, anytime a PVF wants data stored here it must slurp said data back into its own block, which then makes the data available for validators.  If so, why should this require Polkadot integration?
Also, you expire data after 7 days.  Why archive such a short time?
You're doing so as a parachain.  Why?  As a parachain, this pushes its data into the availability store, meaning this costs a parachain exactly what placing data into its own blocks costs.

* IPFS-addressable extrinsics (new feature)

What does this mean?

* State only handles data market and maintains upcoming renewals.

So there is zero user data in these block?  Only IPFS addresses for which storage nodes store the data?  If so, yes this makes sense as a storage model.  :)
Why would this require any special polkadot integration though?  We cannot use this data from polkadot directly so why not do everything inside the parachain's runtime?
Is the 7 days just an accounting feature?

* Each block from the last 7 days also has a 32-byte Merkle root stored in state. The trie is formed (in-memory) from the 32-byte chunked extrinsics (or perhaps use some other chunking means), indexed by `(extrinsic_index, chunk_index)`. This should be checked as part of block verification.

What does this mean?

* Creating a new block requires solving a short proof-of-work. The seed for the PoW is derived from several randomly selected bytes from the block 7 days prior (and which is now going out of scope); the proof of these bytes as well as the PoW solution is included on-chain. The risk of short-term forks may first be reduced through a proof-of-stake mechanism which applies different difficult requirements to each potential block author.

I'm confused:  A priori, I'd envision this being some parathread, so surely paying the parathread fee mostly suffices, no?  Are we talking huge amounts of data here?  Like what?

* Block authors are paid out 50% of the fees immediately, and 50% is placed in a pot for them to split pro-rata weekly.

Is the block author hosting all the user data?  If not, they're not doing much work, so no reason to pay them.  We've no mechanism to enforce they serve the data obviously, but that's fine since we'll never depend upon the data.

* Network maintainers (i.e. full nodes) are required to retain

Who are these?  Not Polkadot full nodes obviously.  It's whoever joins this parachain?

* Assume only 7-day block history (and all current state) needed for sync of full nodes.

* All data (extrinsics) kept for minimum 7 days. Data can be renewed with a simple on-chain tx (no need to resend data).

That's useful.

In short, I'm unsure why this requires any polkadot integration, given that polkadot cannot depend upon archival-like guarantees.
Just fyi, we'll later expand availability store usage for required data:  At present, anytime a parachain calls set_code we require all polkadot nodes see its code immediately.  In theory, parachains could post_code with their own parachain candidate block, which then stores the new code into the erasure coded availability store.  After we finalize the relay chain block including post_code parachain candidate, then the parachain could switch_code into the new code.  At this point, we've perhaps only one polkadot validator who possesses the code.  Yet, any who require this code could fetch it like approval checkers fetch the blocks they check, thanks to the availability store.  We'll require this for hierarchical/multiple relay chain operation, and maybe it'd make polkadot more efficient even now, but currently we're hoping code churns slowly enough to make it worth storing parachain code on the relay chain.

@burdges This PR does not mention polkadot or parachains anywhere. This is to add generic support for such chains in substrate. The idea is to store user data not in the state merkle tree, but rather as block bodies. Runtime only keeps track of data hashes but has no access to the data itself. This way storage would be way less expensive. @gavofyork can probably explain rationale better than myself.
Users send data as transaction payload, which is added to the block database (hosted by all full nodes) and may be retrieved by CID (data hash) over IPFS.
The payload is removed from the database after 7 days unless there's another renewal transaction that references existing data hash. The runtime only validates the merkle tree route of all the added/renewed data hashes in the block.

@burdges This PR does not mention polkadot or parachains anywhere. This is to add generic support for such chains in substrate. The idea is to store user data not in the state merkle tree, but rather as block bodies. Runtime only keeps track of data hashes but has no access to the data itself. This way storage would be way less expensive. @gavofyork can probably explain rationale better than myself.
Users send data as transaction payload, which is added to the block database (hosted by all full nodes) and may be retrieved by CID (data hash) over IPFS.
The payload is removed from the database after 7 days unless there's another renewal transaction that references existing data hash. The runtime only validates the merkle tree route of all the added/renewed data hashes in the block.

Runtime only keeps track of data hashes but has no access to the data itself.

Alright sounds fine then.  I'm still unclear on why this needs special support, but anyways it breaks nothing afaik.  :)

Runtime only keeps track of data hashes but has no access to the data itself.

Alright sounds fine then.  I'm still unclear on why this needs special support, but anyways it breaks nothing afaik.  :)

What purpose does the PoW serve here?  Any block production constraint suffices, no?

What purpose does the PoW serve here?  Any block production constraint suffices, no?

@burdges right. Initial implementation will use existing consensus at least. Not sure why @gavofyork mentioned PoW there, but it is out of the scope for now.

Alright sounds fine then. I'm still unclear on why this needs special support, but anyways it breaks nothing afaik. :)

Special support would be in the database/networking layer to enable querying and reference-counting individual transactions.

@burdges right. Initial implementation will use existing consensus at least. Not sure why @gavofyork mentioned PoW there, but it is out of the scope for now.

Alright sounds fine then. I'm still unclear on why this needs special support, but anyways it breaks nothing afaik. :)

Special support would be in the database/networking layer to enable querying and reference-counting individual transactions.

bot merge

bot merge

Is there a general level tracking issue for the storage chain? I think it's very helpful for our use case: We need to store a series of "snapshot" for the encrypted confidential contract states and do re-encryption periodically. Since they are just snapshots, only the recent ones are actually useful, and therefore we would like to just prune the older snapshots by default for all the full node. It can make our blockchain much much lighter and more efficient.

Is there a general level tracking issue for the storage chain? I think it's very helpful for our use case: We need to store a series of "snapshot" for the encrypted confidential contract states and do re-encryption periodically. Since they are just snapshots, only the recent ones are actually useful, and therefore we would like to just prune the older snapshots by default for all the full node. It can make our blockchain much much lighter and more efficient.


Bump pyupgrade from v3.17.0 to v3.18.0

Bumps pre-commit hook for pyupgrade from v3.17.0 to v3.18.0 and ran the update against the repo.

CI Failure due to beeware/briefcase#2026.


Fix panic trying to get a param value

This should fix the panic reported by rancher/os#624

@ibuildthecloud take a look

I'm more a fan of go-check, just FYI.  I know we use testify in other places.

@imikushin can you just explain the basic problem?  Like what was the code doing before that blew up.

The problem was in trying to get parts[2] (the 3rd element) when there are only 2 elements in parts


Update akka-actor, akka-testkit to 2.6.18

Updates

com.typesafe.akka:akka-actor
com.typesafe.akka:akka-testkit

from 2.6.8 to 2.6.18.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "com.typesafe.akka" } ]

labels: library-update, early-semver-patch, semver-spec-patch, commit-count:1

Superseded by #90.


fix self signed certificate compatibility

Closes #17
Default keySize is 1024 and algorithm which is not enough nowadays.
Default algorithm is sha1 which is deprecated. https://security.googleblog.com/2014/09/gradually-sunsetting-sha-1.html
See https://github.com/jfromaniello/selfsigned/blob/c5ac42bdb5949bce47679221284331cab71a1e1e/README.md#options

Unfortunately https://github.com/jeansaad/chalet/issues/17 is silent to the user and because of the current design it's difficult to make it explicit.
Ideally chalet would self-heal in this case but that would introduce quite a lot of complexity.
Maybe I can add a note to the README.md or create an issue that you can pin ?
What do you think is best?

Self signed certificate works fine with these new settings:
Chromium 90 / Fedora 34

Firefox Developer Edition 89.0b15 / Fedora 34


Update dynamic-configuration-1.md

The existing link is self-referential; I believe the intended target is the file in the parent directory.

LGTM, thanks @jasonsommerfield !


devicetree.h: switch to DT_IRQ_INTC_BY_IDX for L2 and L3 INTID encodings

Using DT_IRQ_INTC() to fetch the interrupt controller associated with a node works well for nodes which consume interrupts from a single aggregator. However, when specifying multiple (and different) interrupt aggregators via the interrupts-extended property, the L2 and L3 interrupts will no longer be encoded properly. This is because DT_IRQ_INTC(node_id) uses DT_IRQ_INTC_BY_IDX(node_id, 0) so all the interrupts will use the first aggregator as their parent. To fix this, switch from using DT_IRQ_INTC() to DT_IRQ_INTC_BY_IDX().
This can be tested using configurations such as the following:
intc1: interrupt-controller@cafebabe {
    /* some more properties here */
    interrupt-controller;
    #interrupt-cells = <1>;
};

intc2: interrupt-controller@deadbeef {
     /* some more properties here */
    interrupt-controller;
    interrupts-extended = <&intc1 14>;
};

intc3: interruptc-controller@babebeef {
     /* some more properties here */
    interrupt-controller;
    interrupts-extended = <&intc1 15>;
};

consumer_node: ip@cafecafe {
    /* some more properties here */
    interrupts-extended = <&intc2 16>, <&intc3 15>;
};

Without this patch the interrupts from consumer_node will be encoded as follows: 14 | ((16 + 1) << 8) and 14 | ((15 + 1) << 8)

will take a look soon, but is it possible to extend the existing devicetree test for this?

will take a look soon, but is it possible to extend the existing devicetree test for this? i.e. with the DT configuration in the OP

created https://github.com/zephyrproject-rtos/zephyr/pull/69711

LGTM, thanks. Could you please create a bug report for this so that it can be backported?

created https://github.com/zephyrproject-rtos/zephyr/issues/69820


Long response times from the API

For the past couple of hours LRCLib api takes a while to respond, for example:
$ count 'curl https://lrclib.net/api/search --url-query track_name=Mamontovas -vv'
* Host lrclib.net:443 was resolved.
* IPv6: (none)
* IPv4: <IP_ADDRESS>
*   Trying <IP_ADDRESS>:443...
* Connected to lrclib.net (<IP_ADDRESS>) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / id-ecPublicKey
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: CN=lrclib.net
*  start date: Jul 31 22:40:54 2024 GMT
*  expire date: Oct 29 22:40:53 2024 GMT
*  subjectAltName: host "lrclib.net" matched cert's "lrclib.net"
*  issuer: C=US; O=Let's Encrypt; CN=E5
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
*   Certificate level 1: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/1.x
> GET /api/search?track_name=Mamontovas HTTP/1.1
> Host: lrclib.net
> User-Agent: curl/8.9.1
> Accept: */*
>
* Request completely sent off
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/1.1 200 OK
< Server: nginx/1.22.1
< Date: Sun, 08 Sep 2024 17:42:14 GMT
< Content-Type: application/json
< Content-Length: 2
< Connection: keep-alive
< vary: origin, access-control-request-method, access-control-request-headers
< access-control-allow-origin: *
<
* Connection #0 to host lrclib.net left intact
[]
23s 186ms 276us

Is this expected?

This is not expected, normally most requests should be served in under 500ms. It seems lrclib.net is receiving a huge number of search requests recently, but it doesn't appear to be an attack. Maybe this is a sign that the server needs to be scaled up.

Same, takes around 18 seconds for fetching lyrics

Much better now, around 6 seconds on average

I'm seeing a lot of "connection closed before message completed", "operation timed out", and "connection reset by peer" messages in LRCGET, so something's definitely wacky. In fact, every request I've made this morning has failed in one of these three ways.

It seems part of the problem is on Hetzner's side. I think they are short on shared resources for their Intel-based servers, so the CPU performance seems to be heavily throttled.
Anyway, I've just upgraded the server to a more powerful AMD-based plan, so everything should be much better now.
The monthly payment for lrclib.net is now around $15. Hopefully, more people will step up to make monthly sponsorships so that I can comfortably cover the cost.

The wait time is now ~300ms, the issue has been fixed


Docs don't contain conf location for Docker

I'm not entirely sure where the doc source is located, so I'm posting here. Not a very big issue to fix, but the current docs for finding the config file are located here: https://docs.fluentd.org/v0.12/articles/config-file
And are also the top Google result if you Google fluent config file
The current docs list:
/etc/td-agent/td-agent.conf

and
/etc/fluent/fluent.conf

As the only two places the config file might be.
However when using the Docker container, the config file is located in /fluentd/etc which is not documented there and is hiding somewhere in the middle of the official DockerHub repo: https://hub.docker.com/r/fluent/fluentd/ only briefly mentioned in a docker mount command:
docker run -ti --rm -v /path/to/dir:/fluentd/etc fluentd -c /fluentd/etc/ -v

I spent a good 30 minutes poking around for this non-standard config location. Since the place in docker containers for the config file is... uh... interesting, could it please be added to https://docs.fluentd.org/v0.12/articles/config-file ?
Thanks :)

Since the place in docker containers for the config file is... uh... interesting, could it please be added to https://docs.fluentd.org/v0.12/articles/config-file ?

This is docker image specific directory, this is why we need -c option to tell it,so if we add note for conf location, it should be in docker image README.

I disagree, by doing that you're pretty much saying Docker is not a first class citizen (which I think most people using Fluentd would disagree with).
Take a competitor - Filebeat for instance: https://www.elastic.co/guide/en/beats/filebeat/current/directory-layout.html
Default paths for all installation types are listed. There's even an explanation of what each dir does, and how they can autostart!

I think most people using Fluentd

You have a point. Fluentd with docker is now popular so adding it to the fluentd-docs seems reasonable.
Could you send a PR?

Sure :)
https://github.com/fluent/fluentd-docs/pull/614

Merged fluentd-docs patch. Thanks


docker pull error

I docker pull a image, I docker login first and return Login Succeeded,but then I use docker  pull ,return
WARNING: Error loading config file:/home/dnet/.docker/config.json - EOF
Pulling repository xxxxx
unauthorized: authentication required
that all I wirte in python script
and finally I docker pull in bash,it works successful, why? can any ideas have?

please keep in mind that the GitHub issue tracker is for tracking bugs and feature requests, and not a general support forum; since this is a support question for running docker, could you use one of the following?;

https://forums.docker.com
the docker community slack channel https://blog.docker.com/2016/11/introducing-docker-community-directory-docker-community-slack/
StackOverflow

I'll close this issue here because of the above, but hope this helps you further

sorry ,I use above  and follew with interes feature requests,thanks !


Document what the trunk-share-with-everyone action does

I can't find anything explaining what the trunk-share-with-everyone action does.

Hi! trunk-share-with-everyone is a built-in Trunk Action that generates notifications to turn off single player mode. All it does is check for the single player mode status and generate a notification in VSCode, suggesting for users to run trunk config share. We get that this can be confusing, and we have plans to improve our documentation around single player mode.
Let me know if that helps!
Also, out of curiosity, how did you come across the trunk-share-with-everyone action? It's not something that we advertise as we consider it to be built-in functionality.

@error9900 did you encounter this originally while running trunk actions list or did you see an error show up in VSCode somewhere?

@error9900 did you encounter this originally while running trunk actions list or did you see an error show up in VSCode somewhere?

Yes, trunk actions list is how I came across it. I didn't notice any errors related to that action.


Get rid of center field in SpikeTrian type

I used the center field to specify the start of each trial. I think a better event type could completely replace this field

See #3


Program crashed, help!

Why did he crash?
https://user-images.githubusercontent.com/109384420/187411316-acf8f3c9-2b31-41db-a916-6be269d4e2e4.png
                   **ERROR**

Game name: (please, specify the game name)
Engine version: (specify engine version, if known)
Platform: (specify platform, if known)
Error message: SerializedFile with index 5015 was not found in dependencies
Stack trace:    в uTinyRipper.SerializedFile.FindAsset(Int32 fileIndex, Int64 pathID, Boolean isSafe)
в uTinyRipper.Converters.ProjectAssetContainer.FindAsset(Int32 fileIndex, Int64 pathID)
в uTinyRipper.Classes.PPtr1.FindAsset(IAssetContainer file)    в uTinyRipper.Classes.PPtr1.ExportYAML(IExportContainer container)
в uTinyRipper.Game.Assembly.SerializableField.ExportYAML(IExportContainer container, Field& etalon)
в uTinyRipper.Game.Assembly.SerializableStructure.ExportYAML(IExportContainer container)
в uTinyRipper.Classes.MonoBehaviour.ExportYAMLRoot(IExportContainer container)
в uTinyRipper.Classes.Object.ExportYAMLDocument(IExportContainer container)
в uTinyRipper.Converters.YAMLAssetExporter.Export(IExportContainer container, IEnumerable1 assets, String path)    в uTinyRipper.Project.AssetExportCollection.Export(ProjectAssetContainer container, String dirPath)    в uTinyRipper.Converters.ProjectExporter.Export(String path, GameCollection fileCollection, IEnumerable1 files, ExportOptions options)
в uTinyRipper.GameStructure.Export(String exportPath, Func`2 filter)
в uTinyRipperGUI.MainWindow.ExportFiles(Object data)

uTinyRipper has been abandoned, but you can try my AssetRipper fork. In the event that AssetRipper encounters issues while ripping your files, feel free to post an issue on that repository.
AssetRipper is actively being developed with the intent to recover games as best as possible. Weekly releases are normally on Sunday or Monday. I encourage you to help out if you're able and willing. Code contributions can be submitted via pull requests.
If you like the work I've done and want to see it continue, please consider sponsoring me. It really does help with development. These are some of the notable things I've implemented since taking over development:

2020+ support
APK, XAPK, and IPA reading with SharpZipLib
Dummy shaders (so everything isn't pink when you open the project)
IL2Cpp support via an integration of Cpp2IL
Better game platform support
Full script decompilation with the ILSpy decompiler
Cross-platform support
Multiple configuration options
Numerous performance improvements
Support for all asset types

Sponsorship Links:

GitHub Sponsors
Patreon
Paypal

Thank you!☺


Incorrect "class variables can only be annotated with ThreadLocal"

When using anykey/clear with annotations on a model type (include Clear::Model) there's a strange error:
require "clear"

@[Deprecated] # Error: class variables can only be annotated with ThreadLocal
class Foo
  include Clear::Model
end

It seems the annotation (@[Deprecated]) is for some reason attached to a class variable declared in Clear::Model's included macro hook. This only happens when include Clear::Model is the first expression in the class body. Inserting anything before it fixes this.
Maybe there's something weird going on in clear (@anykeyh), but in any case, this behaviour looks very much like a compiler bug.
--error-trace output:
$ crystal test.cr  --error-trace
In test.cr:5:3

 5 | include Clear::Model
     ^
Error: expanding macro

There was a problem expanding macro 'included'

Called macro defined in macro 'included'

 1 | macro included # When included into final Model

Which expanded to:

 >  1 |       # Define on which connection the model is living. Useful in case of models living in different databases.
 >  2 |       #
 >  3 |       # Is set to "default" by default.
 >  4 |       #
 >  5 |       # See `Clear::SQL#init(URI, *opts)` for more information about multi-connections.
 >  6 |       #
 >  7 |       # Example:
 >  8 |       # ```
 >  9 |       #  Clear::SQL.init("postgres://postgres@localhost/database_1", connection_pool_size: 5)
 > 10 |       #  Clear::SQL.init("secondary", "postgres://postgres@localhost/database_2", connection_pool_size: 5)
 > 11 |       #
 > 12 |       # class ModelA
 > 13 |       #   include Clear::Model
 > 14 |       #
 > 15 |       #   # Performs all the queries on `database_1`
 > 16 |       #   # self.connection = "default"
 > 17 |       #   column id : Int32, primary: true, presence: false
 > 18 |       #   column title : String
 > 19 |       # end
 > 20 |       #
 > 21 |       # class ModelB
 > 22 |       #   include Clear::Model
 > 23 |       #
 > 24 |       #   # Performs all the queries on `database_2`
 > 25 |       #   self.connection = "secondary"
 > 26 |       #
 > 27 |       #   column id : Int32, primary: true, presence: false
 > 28 |       # end
 > 29 |       #
 > 30 |       # ```
 > 31 |       class_property connection : String = "default"
 > 32 |
Error: expanding macro

In test.cr:3:1

 3 | @[Deprecated]
     ^
Error: class variables can only be annotated with ThreadLocal

Duplicate of https://github.com/crystal-lang/crystal/issues/8314.

Oops, I missed that. Thanks.


Gitlab Support

This tool looks amazing. Are there any plans to add support for gitlab?

Gitlab support would be great! Feel free to open a pull request :) Otherwise maybe I can poke around and figure out Gitlab support if I find some free time.

You'd want to add it here: https://github.com/guykisel/inline-plz/tree/master/inlineplz/interfaces


Use Ejson Body Instead of Compressed Body for External size

Overview

In two places where we calculate the ExternalSize of the document body,
we use the Summary which is a compressed version of the doc body. We
change this to use the actual ejson body. In copy_docs we don't have
access to the #doc record so we can't access the meta where we store
the ejson body. Unfortunately, this means we have to decompress the
document body after reading it from disk.
Testing recommendations
Performance tests were conducted to compare pre-fix External Size values and post-fix External Size Values. Here are the results:
Version        Compression    DB Dize      DB External Size

Pre-Fix         snappy            70 MB        46249702
Pre-Fix         snappy            700 MB      462495533
Pre-Fix         snappy            2.1 GB<PHONE_NUMBER>

Post-Fix       snappy            70 MB        83950133
Post-Fix       snappy            700 MB      839500133
Post-Fix       snappy            2.1 GB<PHONE_NUMBER>

Pre-Fix         deflate_6        70 MB        46249702
Pre-Fix         deflate_6        700 MB      413395945
Pre-Fix         deflate_6        2.1 GB       994598087

Post-Fix       deflate_6        70 MB        83950133
Post-Fix       deflate_6        700 MB      430495496
Post-Fix       deflate_6        2.1 GB<PHONE_NUMBER>

GitHub issue number
PR is number.
Related Pull Requests

Checklist

[X] Code is written and works correctly;
[X] Changes are covered by tests;
[ ] Documentation reflects the changes;

Not sure what your test was there, but that's a pretty big red flag that the external sizes don't match between different compression algorithms as it should not be affected by that. You also probably want to do a pre/post compaction measurement as well to make sure compaction isn't affecting it either.

@davisp: I made the changes you suggested. Also, I think the performance benchmarks were using port 5984 instead of 5986, so it was influenced by the clustered nature of things. Let me know if this looks good, and I'll create a temporary build to check the performance numbers before merging.


Don't include bulk for modalities on Project

Context
Came up in High review of Browse page. The initial designs lumped CITE-seq, Spatial and Bulk under modalities. But now, we would like to lump CITE-seq and Spatial under Modalities and Bulk on its own.
Problem or idea
The backend should reflect that change
Solution or next step
tagging @kurtwheeler for next steps

Addressed in previous PR


readonly on property level not 100% working

number and integer are readwrite
save the JSON looses some readonly-properties

still some properties lost when saving JSON


[HW] Rounak Tadvi

Name: Rounak Tadvi
Email<EMAIL_ADDRESS>Linkedin Profile: www.linkedin.com/in/rounaktadvi
Attach the homework step I screenshots below:

Attach the homework step II screenshots below:

Hey Rounak Tadvi, great job! Congrats! Here is your badge! https://api.badgr.io/public/assertions/0smUHnbtRGiAGNFPCjxIUA


1.0.0-rc.4 release

Added new project approvers
Fixed gometalinter
Increased unit test coverage
Added "pc" machine type for Qemu hypervisor
Fixed "kill" and "stop" container
Shortlog:
1140aca hypervisor: export SerializeParams and DeserializeParams
89577b2 pod: Allow to stop a pod in ready state as a special case
8fdcb90 pod: Ignore container ready or stopped when stopping the pod
72a0be6 pod: Factorize pod stop() function
7f61c04 container: Don't issue an error if trying to stop a stopped container
10c592f container: Signal a ready container is valid but not stopping it
ae8fcbb api: Update the container status when retrieving its status
4e5128a create: Split hyperstart init function to perform setup only at create
57bec7d stop: Wait for end of container process before to remove
284dd91 qemu: Add pc machine type as supported
8d8ccbd hyperstart: Move generic pod stopping code
b22c37f tests: Increase unit test coverage for hook.go
1b3d2f0 tests: Add option to panic mock hook
6b29134 tests: Increase unit test coverage for cc_shim.go.
8b7fe13 gometalinter: Enable some missing checks
daa17ff gometalinter: Fix gocyclo issues
62ddd81 gometalinter: Fix go vet issues
b37d339 virtc: Remove blank lines
82e42b7 CI: Add a new reviewer to pullapprove
1c8c7e6 shim: Stop forwarding output in detach mode
41ad8af CI: Add additional approver
ceb467e hypervisor: Add Default{VCPUs, MemSz} to HypervisorConfig
427b84f CI: Add additional project approver

Coverage remained the same at 66.502% when pulling 2018d1a6501f233f5a5dc9734596a79e147ef06c on sboeuf/new_release into 17f778503274ed939ebeea62ab315b2a7fa31cfd on master.

lgtm


Add SVNU

South Valley National University
http://www.svnu.edu.eg/

@mokhtar0v The first verification stage is complete: the official domain for  South Valley National University is www.svu.edu.eg, to the best of my knowledge. This is just an informational note. The pull request is still under review. The review may take some more time. I greatly appreciate your patience. Here is the proof of domain ownership: https://www.svu.edu.eg/ar/media_files/2023/11/Course-Specification-Template-Competencies.pdf


Suggestion for GPU CI

suggested updates to #103
This is based on #85 and extends those workflows

This one is not longer needed as we went with separate integration tests for gpu instance


chore: reduce features

Did a quick pass and removed as many features as I could. Everything still seems to work and CI passes so...
Fixes RES-112

This should be merged only when we verify that it doesn't break bindings

This should be merged only when we verify that it doesn't break bindings

How do you think we should do that?

How do you think we should do that?
We should do that locally per platform or test releases


Add files via upload

Contributing to the Azure SDK
Please see our CONTRIBUTING.md if you are not familiar with contributing to this repository or have questions.
For specific information about pull request etiquette and best practices, see this section.

Closing as spam


Add method CairoRunner::finalize_segments

Depends on #550
Add method MemorySegmentManager::finalize()
Add method CairoRunner::finalize_segments()

Codecov Report

Merging #542 (d910ad5) into get_used_cells_and_allocated_size-fix (ff41dc7) will decrease coverage by 0.82%.
The diff coverage is 5.00%.

@@                            Coverage Diff                            @@
##           get_used_cells_and_allocated_size-fix     #542      +/-   ##
=========================================================================
- Coverage                                  91.84%   91.01%   -0.83%
=========================================================================
  Files                                         58       58
  Lines                                       3750     3786      +36
=========================================================================
+ Hits                                        3444     3446       +2
- Misses                                       306      340      +34

Impacted Files
Coverage Δ

src/vm/runners/builtin_runner/mod.rs
72.78% <0.00%> (+0.49%)
:arrow_up:

src/vm/runners/cairo_runner.rs
85.96% <3.33%> (-7.95%)
:arrow_down:

src/vm/vm_memory/memory_segments.rs
92.85% <14.28%> (-5.24%)
:arrow_down:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


An error occurred while attempting to switch this system to Oracle Linux

Looking inside the script I made an attempt to understand at which point the script wanted to exit, but I failed.
I was hoping this info & log would be helpful to the devs here to share some possibilities as to what "An error occurred..." that would cause this break at this point in the script.
I have run this wonderful script successfully on many very different VMs
about vm

dnf upgraded
space made in /boot for more kernels
disabled non-natives repos
no automatic updates or crons

[ansible@zabbix-proxy1 ~]$ cat /etc/*-release
CentOS Linux release 8.5.2111
NAME="CentOS Linux"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
CentOS Linux release 8.5.2111
[ansible@zabbix-proxy1 ~]$

script run
[root@zabbix-proxy1 ansible]# chmod +x centos2ol.sh
[root@zabbix-proxy1 ansible]# bash centos2ol.sh -V
Checking for required packages...
Checking your distribution...
Creating a list of RPMs installed before the switch
Verifying RPMs installed before the switch against RPM database
package epel-release is not installed
Checking for yum lock...
Checking for required python packages...
Identifying dnf modules that are enabled
Finding your repository directory...
Learning which repositories are enabled...
Repositories enabled before update include:
appstream
baseos
extras
atomic
ol8_baseos_latest
ol8_appstream
ol8_UEKR6
Looking for yumdownloader...
Backing up and removing old repository files...
An error occurred while attempting to switch this system to Oracle Linux and it may be in an unstable/unbootable state. To avoid further issues, the script has terminated.
[root@zabbix-proxy1 ansible]#

Do you have the list of RPMs installed prior to the attempt? Can you add -x to the set -e line so we can get a more verbose output too?

@Djelibeybi
Avi set -x was the ticket to find the issue! Awesome mate.
Turns out in my case
[root@zabbix-proxy1 ansible]# rpm -q --whatprovides /etc/redhat-release
centos-linux-release-8.3-1.2011.el8.noarch
centos-linux-release-8.4-1.2105.el8.noarch

So I dnf remove centos-linux-release-8.3-1.2011.el8.noarch
And the script happily went on it merry way

Huzzah! It's weird that you managed to get two centos-linux-release packages installed, but given one was from 2011, I guess the reason is lost to history. I'm going to close this issue, but feel free to open a new one if anything else occurs.

BTW, since centos8 repos have move to vault I also
Change to following line to each respective file:
[root@etp-monitor1 ansible]# vi /etc/yum.repos.d/CentOS-Linux-BaseOS.repo
[root@etp-monitor1 ansible]# vi /etc/yum.repos.d/CentOS-Linux-AppStream.repo

CentOS-Linux-BaseOS.repo
baseurl=http://vault.centos.org/$contentdir/$releasever/BaseOS/$basearch/os/
CentOS-Linux-AppStream.repo
baseurl=http://vault.centos.org/$contentdir/$releasever/AppStream/$basearch/os/

Comment the mirrors

Oh, good point. Let me fix that up.


Squash does not remove trailing slash in ui-sref

If I define a state as below:
	$stateProvider.state('productsByCategory', {
	    url: '/products/:catSlug/:subCatSlug?',
	    controller: 'MyController',
	    params: {subCatSlug: {value:null, squash: true}},
	    templateUrl: '/app/products/list.html'
	})

I expect this to match the URL path /products/watches, which it does (no trailing slash).
However, if I do the below methods, I get the URLs with the trailing slashes:
	Go to Watches

I get the below compiled tag:
	Go to Watches

(The slash is not squashed)
Is this an expected behavior? And if so, is there a way to retrieve the squashed version via ui-sref?

Here is a reproduction - this appears to be a bug.

is it because of the last ? ?

+1
@nateabele this was flagged as a bug on October 2015, it doesn't seem like a very complicated thing to fix. Any chance someone could have a look at it?

You could always file a PR :)


Proxy authentication not working

I've been trying to run the Vulnz 5.0.1 CLI to create a cache of the NVD data. I am running behind a corporate HTTP proxy, which uses basic authentication. I'm running this in Java 11 and I have configured the following _JAVA_OPTIONS (which I see are being picked up by the application):
-Dhttps.proxyHost=http-proxy -Dhttps.proxyPort=8080 -Dhttps.proxyUser=<user> -Dhttps.proxyPassword=<password> -Djdk.http.auth.tunneling.disabledSchemes="" -Djdk.http.auth.proxying.disabledSchemes=""
I am getting a 407 from the proxy, and when I point this at Fiddler I don't see a Proxy-Authorization: header coming through.
I'm wondering if more configuration is needed in the RateLimitedClient on line 114.
I hope I'm overlooking something, and someone can set me on the right path.

There are users of the client library (same one that is used by vulnz) that have reported using the JAVA_TOOL_OPTIONS has worked for them: https://github.com/jeremylong/DependencyCheck/issues/6127

Is it as simple as adding useSystemProperties() to the end of the builder chain on line 115? HttpClientBuilder suggests that it will not use the configured properties otherwise. I suspect that the SystemDefaultRoutePlanner is only taking care of sending the requests to the proxy, not managing the authentication.

Yes - it looks like it is as simple as adding useSystemProperties()... at least I hope ;)

Thank you! That seems to have done it. I see a 407 response followed by a repeated request with the Proxy-Authorization header.

Thank you for confirming!

I should be able to release the updated version in 2-3 days.


Did you see Jisp?

Hi Frederic!
Did you see Jisp? If yes, why do you decide to write yet another Lisp syntax for javascript?
By the way, as I know, original jisp developer lost interest to his project. May be it is good moment to take over and continue to evolve it?

Closing this old issue. Thanks to https://12forks.com MVP for pointing to it.


KeyError: 'O"'

Hi @Hironsan
thanks for your effort in building such a fine tool.
I'm using anaGo for my school Project, for the Arabic language, but it seems there is a problem.
and this is the dataset I used :

WikiFANE_Gold: Gold standard Wikipedia-based Fine-grained Arabic Named Entity Corpus, ~500K tokens
NewsFANE_Gold: Gold standard Newswire-based Fine-grained Arabic Named Entity Corpus, ~170K tokens.

this is the problem :
Epoch 1/15
789/789 [==============================] - 5196s 7s/step - loss: 337.9646
Traceback (most recent call last):
File "C:/Users/Osama Bani Salameh/PycharmProjects/anaGoTest/anaGoTestMainFile.py", line 11, in
model.train(x_train, y_train, x_valid, y_valid)
File "C:\Program Files\Python36\lib\site-packages\anago\wrapper.py", line 50, in train
trainer.train(x_train, y_train, x_valid, y_valid)
File "C:\Program Files\Python36\lib\site-packages\anago\trainer.py", line 51, in train
callbacks=callbacks)
File "C:\Program Files\Python36\lib\site-packages\keras\legacy\interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "C:\Program Files\Python36\lib\site-packages\keras\engine\training.py", line 2262, in fit_generator
callbacks.on_epoch_end(epoch, epoch_logs)
File "C:\Program Files\Python36\lib\site-packages\keras\callbacks.py", line 77, in on_epoch_end
callback.on_epoch_end(epoch, logs)
File "C:\Program Files\Python36\lib\site-packages\anago\metrics.py", line 124, in on_epoch_end
for i, (data, label) in enumerate(self.valid_batches):
File "C:\Program Files\Python36\lib\site-packages\anago\reader.py", line 137, in data_generator
yield preprocessor.transform(X, y)
File "C:\Program Files\Python36\lib\site-packages\anago\preprocess.py", line 112, in transform
y = [[self.vocab_tag[t] for t in sent] for sent in y]
File "C:\Program Files\Python36\lib\site-packages\anago\preprocess.py", line 112, in
y = [[self.vocab_tag[t] for t in sent] for sent in y]
File "C:\Program Files\Python36\lib\site-packages\anago\preprocess.py", line 112, in
y = [[self.vocab_tag[t] for t in sent] for sent in y]
KeyError: 'O"'
by the way, the dataset has O label within it, so do you know what's going on?
regards,
Osama Bani Salameh

Did you use Fine-grained Arabic Named Entity Corpora for training?

A Hybrid Approach to Features Representation for Fine-grained Arabic
Named Entity Recognition

I trained the model with the dataset and I could do it:
venv ❯ python train_example.py --data_path=WIKIFANE_selective.txt
Loading datasets...
Transforming datasets...
149369
Building a model...
Training the model...

Epoch 1/1
1607/1607 [==============================] - 13867s 9s/step - loss: 492.3222
 - f1: 72.06
Saving the model...

Try to use code in ver1.0.0 branch.

examples/train_example.py


Augustine's Panacea - not importing EA bonuses

#Augustine's Panacea is not showing the bonuses for elemental arrow after importing a character, not using them do display the damage calculation.
I'm testing with this DH: http://us.battle.net/d3/en/profile/Machado-1633/hero/28453583
Evidence: On the left the original item, in Battle.Net website. On the right, the imported one in D3Planner:
http://www.softbr.com/img/diablo3/AugustinesPanacea.jpg
I took the liberty to download the item Json from Battle.Net API, and it seems to bring the value of EA bonuses as a ring type. I've put the JSON here:
http://www.softbr.com/img/diablo3/AugustinesPanacea.json
The EA bonuses seem to be in this key: [Item_Power_Passive#P41_ItemPassive_Unique_Ring_011]. The only thing I couldn't find easy is the Ball Lightning speed.

I don't look at the actual attributes and instead parse the tooltip lines, as its significantly easier than mapping all attributes to their actual names (although I probably have the tools to do that instead now).
I can fix the bug, but I can't check it as your profile no longer has the item equipped :|

Sorry bro, I've been playing this weekend and I forgot to re-equip it after finishing. It should be on now.
I've put a 250% bonus version.

Thanks, fixed.

 Thank you very much!


Specify Usage in README.md.

cc @zgalor @jhernand @elad661 @yaacov @cben this is a good time to suggest interface changes if you have any ideas (They won't be done here tough).


the issue with seg_instance in obs_modalities

@cgokmen Hello! I tried running the demo python -m omnigibson.examples.robots.robot_control_example. This demo runs fine with Isaac Sim version 2023.1.1 and the main branch. However, with Isaac Sim version 4.1 and the og-develop branch, there is an issue with the seg_instance modality, and the error message is:
  File "/home/gpu/.local/share/ov/pkg/isaac-sim-4.1.0/OmniGibson/omnigibson/envs/env_base.py", line 499, in get_obs
    obs[robot.name], info[robot.name] = robot.get_obs()
  File "/home/gpu/.local/share/ov/pkg/isaac-sim-4.1.0/OmniGibson/omnigibson/robots/robot_base.py", line 283, in get_obs
    obs_dict[sensor_name], info_dict[sensor_name] = sensor.get_obs()
  File "/home/gpu/.local/share/ov/pkg/isaac-sim-4.1.0/OmniGibson/omnigibson/sensors/sensor_base.py", line 78, in get_obs
    obs, info = self._get_obs()
  File "/home/gpu/.local/share/ov/pkg/isaac-sim-4.1.0/OmniGibson/omnigibson/sensors/vision_sensor.py", line 304, in _get_obs
    self._remap_modality(modality, obs, info, raw_obs)
  File "/home/gpu/.local/share/ov/pkg/isaac-sim-4.1.0/OmniGibson/omnigibson/sensors/vision_sensor.py", line 323, in _remap_modality
    obs[modality], info[modality] = self._remap_semantic_segmentation(obs[modality], id_to_labels)
  File "/home/gpu/.local/share/ov/pkg/isaac-sim-4.1.0/OmniGibson/omnigibson/sensors/vision_sensor.py", line 381, in _remap_semantic_segmentation
    assert set(image_keys.tolist()).issubset(
AssertionError: Semantic segmentation image does not match the original id_to_labels mapping.

I printed out image_keys and replicator_mapping in the _remap_semantic_segmentation function of vision_sensor.py and found that replicator_mapping is missing 9: 'window' and 10: 'picture'. What could be the reason for this?
image_keys: tensor([ 2,  3,  4,  5,  6,  8, 10, 11], dtype=torch.int32)
replicator_mapping: {0: 'background', 1: 'unlabelled', 2: 'bottom_cabinet', 3: 'agent', 4: 'straight_chair', 5: 'walls', 6: 'breakfast_table', 8: 'floors', 11: 'electric_switch'}

Same Problem Here:
I have no idea how the semantic segmentation code is working. Here part of my code to save the images and segmentation masks:
def camera_saving(camera_mover:VisionSensor,
                  env: og.Environment,
                  saving_path: Path,
                  position: np.ndarray,
                  orienation:np.ndarray,
                  debug: bool = False,
                  debug_path: Path = None):
    # update camera mover location
    camera_mover.set_position_orientation(
        position=position,
        orientation=orienation
    )
    for _ in range(4):
        env.step(np.array([0,0]))  # update the environment
    current_pose, current_orienation = camera_mover.get_position_orientation()
    np.allclose(position, current_pose, atol=1e-2) & np.allclose(orienation, current_orienation, atol=1e-2), \
    f"The camera setting is not correct, we have current_pose: {current_pose} set pose {position}, current_orientation: {current_orienation}, new orientation {orienation}"

    obs = camera_mover.get_obs()[0]
    saved_dict = {
        'position' : position,
        'orientation' : orienation,
        'rgb' : obs["rgb"][..., :3][..., ::-1],
        'depth' : obs["depth_linear"],
        'semantics' : obs["seg_semantic"]
    }
    np.savez_compressed(saving_path, **saved_dict)
    if debug and debug_path:
        rgb_image = saved_dict['rgb']
        cv2.imwrite(f'{debug_path}', img = rgb_image)

And here is how to enable the camera:
    cam_mover = VisionSensor(
            prim_path="/World/viewer_camera",  # prim path
            name="my_vision_sensor",  # sensor name
            modalities=["rgb","depth_linear", "seg_semantic"],  # sensor mode
            enabled=True,  # enale sensor
            image_height=480,  #
            image_width=640,  #
            focal_length=17,
            clipping_range=(0.01, 1000000.0),  # vision distance,
        )

And the error is:
  File "/butian/free_exploration/way_points.py", line 70, in camera_saving
    obs = camera_mover.get_obs()[0]
  File "/omnigibson-src/omnigibson/sensors/sensor_base.py", line 76, in get_obs
    obs, info = self._get_obs()
  File "/omnigibson-src/omnigibson/sensors/vision_sensor.py", line 274, in _get_obs
    obs[modality], info[modality] = self._remap_semantic_segmentation(obs[modality], id_to_labels)
  File "/omnigibson-src/omnigibson/sensors/vision_sensor.py", line 313, in _remap_semantic_segmentation
    assert set(np.unique(img)).issubset(set(replicator_mapping.keys())), "Semantic segmentation image does not match the original id_to_labels mapping."

Hello all! This is a regression introduced by recent isaac sim version updates. Here's the fix to this problem, and it is already merged into og-develop last Friday: https://github.com/StanfordVL/OmniGibson/pull/885

Sorry, it still seems to have issues. But everything works well under Isaac Sim version 2023.1.1 and the main branch.
 File "/home/gpu/.local/share/ov/pkg/isaac-sim-4.1.0/OmniGibson/omnigibson/envs/env_base.py", line 569, in _post_step
    obs, obs_info = self.get_obs()
  File "/home/gpu/.local/share/ov/pkg/isaac-sim-4.1.0/OmniGibson/omnigibson/envs/env_base.py", line 499, in get_obs
    obs[robot.name], info[robot.name] = robot.get_obs()
  File "/home/gpu/.local/share/ov/pkg/isaac-sim-4.1.0/OmniGibson/omnigibson/robots/robot_base.py", line 281, in get_obs
    obs_dict[sensor_name], info_dict[sensor_name] = sensor.get_obs()
  File "/home/gpu/.local/share/ov/pkg/isaac-sim-4.1.0/OmniGibson/omnigibson/sensors/sensor_base.py", line 78, in get_obs
    obs, info = self._get_obs()
  File "/home/gpu/.local/share/ov/pkg/isaac-sim-4.1.0/OmniGibson/omnigibson/sensors/vision_sensor.py", line 307, in _get_obs
    self._remap_modality(modality, obs, info, raw_obs)
  File "/home/gpu/.local/share/ov/pkg/isaac-sim-4.1.0/OmniGibson/omnigibson/sensors/vision_sensor.py", line 328, in _remap_modality
    obs[modality], info[modality] = self._remap_instance_segmentation(
  File "/home/gpu/.local/share/ov/pkg/isaac-sim-4.1.0/OmniGibson/omnigibson/sensors/vision_sensor.py", line 453, in _remap_instance_segmentation
    semantic_label = semantic_img[img == key].unique().item()
RuntimeError: a Tensor with 2 elements cannot be converted to Scalar

Hi @RookieXwc! Sorry for the trouble! I just tried reproducing this error on my end, but haven't been able to do so. Would you mind sending me a repro script for this? Alternatively, if it's easier, could you please specify how you got to this error (in which scene, with which objects, and with which modality - instance_seg or instance_id_seg). Thanks!

Hi @RookieXwc , sorry for the trouble! This seems to be another isaac sim regression that we are not aware of. This is going to need a more involved fix, but for now, here's a hot fix that should help: https://github.com/StanfordVL/OmniGibson/pull/900
I have reproduced your error with the robot control example in Rs_int, and have fixed it with the above branch. Please let me know if this works!

Thank you for the timely fix. It seems that there are only some visualization related issues left here.
2024-09-23 07:27:29 [132,903ms] [Warning] [omnigibson.sensors.vision_sensor] Some semantic IDs in the image are not in the id_to_labels mapping. This is a known issue with the replicator and should only affect a few pixels. These pixels wilPressed None. Action: [0.0, 0.0]
Traceback (most recent call last):
  File "/home/gpu/.local/share/ov/pkg/isaac-sim-4.1.0/OmniGibson/omnigibson/utils/ui_utils.py", line 810, in keyboard_event_handler
    self.robot.visualize_sensors()
  File "/home/gpu/.local/share/ov/pkg/isaac-sim-4.1.0/OmniGibson/omnigibson/robots/robot_base.py", line 391, in visualize_sensors
    ob = segmentation_to_rgb(ob, N=256) / 255.0
  File "/home/gpu/.local/share/ov/pkg/isaac-sim-4.1.0/OmniGibson/omnigibson/utils/vision_utils.py", line 179, in segmentation_to_rgb
    use_colors = randomize_colors(N=N, bright=True)
  File "/home/gpu/.local/share/ov/pkg/isaac-sim-4.1.0/OmniGibson/omnigibson/utils/vision_utils.py", line 159, in randomize_colors
    colors[0] = [0, 0, 0]  # First color is black
Pressed None.

@RookieXwc The visualization thing should be fixed here: https://github.com/StanfordVL/OmniGibson/pull/901
We will soon have another code release, but before that happens, I'll push bug fixes to this branch.


Chinese translation broken

After the roulette loc update, the Chinese translation has gone back to English.

I think the translation issues broke in this commit mostly, see changes to i18n/zh/docusaurus-plugin-content-docs...

Looks like it's because we enabled traditional chinese. Looking into fixing it

Fixed


Select and open multiple files at once

In some other fuzzy finders that I've used in the past, you can use shift + down to select multiple files, and then press enter to open them all at once. Here is a quick mockup of what it would look like:

This is particularly useful for opening related files that only differ by extension (MyClass.h/.cpp, Options.html/.js/.css, etc).
Thanks! :sparkles:

:+1:

:+1: for this. With ST3 it is also possible to open a file with right arrow, while keeping search results still open.

:+1:

As @hannupekka mentioned, having the ability to open multiple files but not necessarily in the same order as they appear in the search results would be great.

If I'm not mistaken, I think Helm has a feature where you can mark files to be opened and do so simultaneously once you've made your selection(s).

I think it would be super useful to implement either opening a file from suggestions without closing search results or selecting several files to open (e.g. mark a file to open with some shortcut) or even better to have both possibilities.

:+1:

:+1:

:+1:

:+1:
It's a useful feature.

Came here to request the "right arrow" functionality from Sublime 3. Shift+down would achieve the same thing in some cases but when you have the ability to press right arrow, you can at least skip items you don't want to open.
if I've got 10 projects and I want to edit the bower.json within them all, I have to cmd+p and search 10 times. The most annoying part of this is that the search results are randomised each time so I have to open them in a particular project order to make sure I've opened all of the files I need.
Would be good to:

cmd + p > bower.json
go down and the list and press right-arrow x 10

To illustrate how the current implementation is quite wasteful.
# of key presses  to open n files in a fuzzy finder search after typing query or initial fuzzy-search session
1 file = 1 keys
2 files = 4 keys
3 files= 8 keys
4 files = 13 keys

with @intellix's solution and what sublime uses the same opening of files would look like
1 file = 1 key
2 files = 3 keys
3 files = 5 keys
4 files = 7 keys

and because the keys are mainly the down and right arrow, opening many files would be both easy and quick because of the proximity.

I would like this feature as well, but would also be content with ctrl-clicking on files in the fuzzy finder to open them in the background, as long as it keeps the fuzzy finder open.

+1 this would be great to have and save me a lot of time.

👍

👍

this was just implemented in VSC: https://code.visualstudio.com/updates/v1_7#_open-multiple-files-from-quick-open

We need the Sublime's right arrow guys !

+1

+1, Will be a super useful feature

+1

I really miss this from Sublime. +💯
I poked around in the source code; the confirm call didn't jump out at me. I would be happy to try a PR if someone familiar would like to do a quick pairing / walkthrough.

XOB Fuzzy Finder implemented a ctrl+enter to open and move to the next item:
https://github.com/atom/fuzzy-finder/commit/a4e7e3c4149d7e6e0ed728d0ce91837c564dc10d#diff-083e5c843fa1e05655435c5a18417940
I like the idea but I'd like to keep using the official package.

XOB Fuzzy Finder implemented a ctrl+enter to open and move to the next item:
a4e7e3c#diff-083e5c843fa1e05655435c5a18417940
I like the idea but I'd also like to keep using the official package.

Since XOB is a fork of fuzzy-finder, wouldn't just be a matter of merging the changes back to the mainstream?

sorry i can't understand how to fix this problem :(


Add 90/95/99% to report

This PR also moves the Docker run command to analyze a report to a separate shell script, so it is easier to generate reports on past results.

@brunoborges Thank you!


Display problem with Firefox 35.0.1

Hello guys,
There is a main problem with the script and the new version of Firefox.
When user is scrolling, some div are not displayed correctly and we only see the top of the div. When we stop scrolling, the div appear correctly so the problem is just when div are on movement.
I don't really know javascript langage so maybe it is just a bit mistake which can be resolve easily.
Thx :)

Can't reproduce on http://prinzhorn.github.io/skrollr/ Does it happen there for you?

Nope, maybe it's a problem with img,
look at there : http://baptistekucharski.com/
or there : http://www.emiliegenovese.fr/

Can't reproduce on Ubuntu or Windows 7 both with Firefox 35.0.1


Error:Cannot find module

I've install "mdns" but the program can't find "dns_sd_bindings"
module.js:472
throw err;
Error: Cannot find module '../built/Release/dns_sd_bindings'
What should I do？cp the files to the correct position？
Please help me.

Since you closed this issue, I assume you found a solution?? Care to share with the rest of the group in case someone else is having this same problem?
Thanks

reinstall node and nodejs then reinstall homeassistant.I think “node”or“nodejs”may cause the issue.

@roadiemoose


Issue with Field named "Code"

Hi
I created a new simple table like this:
`
table 70001000 "Poison Class"
{
captionML = ENU='Poison Class';
DataPerCompany = true;
fields
{
    field(1;"Code";Code[10])
    {
        CaptionML=ENU='Code';
       // CaptionML=DES='Code';
    }
    field(2;Description;Text[50])
    {
        Captionml=ENU='Description';
       // Captionml=DES='Beschreibung';
    }
}
keys
{
    key(1;PrimaryKey;"Code")
    {
        Clustered=TRUE;
    }
    key(2;SecondaryKey;Description)
    {
        Clustered=False;
    }
}

}`
The I added a page
page 70001000 "Poison Classes" {     PageType=List;     SourceTable="Poison Class";     layout     {         area(content)         {             repeater(Steps)              {                 field("Code";"Code"){}                 field(Description;Description){}             }         }     } }
Sometimes VS don't know the field "Code". I tried it a couple of times. After I changed the name to "No" it worked. Then I rechanged it back to "Code". Very strange, now it works and I can't repro it in the same project. But when I create a new project and do the same steps I end in the same error.

This was a bug in the incremental parsing. It is fixed in the upcoming update.


Documentation might need correction (YAML file role)

I'm new to copier, but it looks this sentence in docs may need correction:

If a YAML file named copier.yml is found in the root of the project (alternatively, a YAML file named copier.yaml), the user will be prompted to fill in or confirm the default values.

https://copier.readthedocs.io/en/stable/creating/
Also substituting values seems a part of core logic of copier, yet there is nothing about it on the front page of the docs. You provide for copying examples (like shutil.copy), but the user like myself is more interested how to provide custom values for package name, author etc. It would be great if starting examples helped with that.

Isn't what you're searching for in the next chapter, customization?

Isn't what you're searching for in the next chapter, customization?

Isn't what you're searching for in the next chapter, "configuring a template"?

Indeed, but why spend frontpage examples on just copying directories? Anyways, it is just an enhancement suggestion.

Isn't what you're searching for in the next chapter, "configuring a template"?

Indeed, but why spend frontpage examples on just copying directories? Anyways, it is just an enhancement suggestion.

Oh I see what you mean now. In the overview we only show how to generate a project, and you suggest we should add a small example of how to write a template. Maybe something with a folder tree, and one file using Jjinja templating. What do you think @Yajo 🙂 ?

Oh I see what you mean now. In the overview we only show how to generate a project, and you suggest we should add a small example of how to write a template. Maybe something with a folder tree, and one file using Jjinja templating. What do you think @Yajo 🙂 ?

Yes, the landing page is not so nice.

Yes, the landing page is not so nice.

Maybe something with a folder tree, and one file using Jjinja templating.

My idea of what people look at on the first page: oh, it is a coockiecutter alternative? Is it easy to use? How can I mark the template? How can I provide the values to fill? Are there any special behaviours?
Similar page: https://cookiecutter.readthedocs.io/en/1.7.2/first_steps.html#your-first-cookiecutter

Maybe something with a folder tree, and one file using Jjinja templating.

My idea of what people look at on the first page: oh, it is a coockiecutter alternative? Is it easy to use? How can I mark the template? How can I provide the values to fill? Are there any special behaviours?
Similar page: https://cookiecutter.readthedocs.io/en/1.7.2/first_steps.html#your-first-cookiecutter

Even a couple of links to some actual copier template repos would go a long way.
Somewhat related, could you add a copier github tag to the project? and add the same tag to a good example template repos to make them more discoverable?

You probably overlooked this section: https://copier.readthedocs.io/en/stable/#browse-or-tag-public-templates
However, it's true that I could tag Copier itself with that. It's not exactly a template, but it's certainly related to that topic.

I just looked at the copier tag/topic on GitHub, and I wonder if copier-template would not be better, because there are quite a number of unrelated projects using the copier tag. I'm already using both of them in my templates, and copier-extension for my Copier+Jinja extension.


Upgrade opencensus go to 0.22.1

Upgrades opencensus to clear up deprecated paths
Allow websocket to build proper path in dev mode

I just pulled down this branch, re-built everything, and confirmed that fixes the issue I had in https://github.com/vmware/octant/pull/289. :+1:


update readme.md

Hi,

you could even remove the semicolons

Good changes! I would only remove the $ because it may confuse beginners.

Okay, I understand 👌🏼


POST / PUT "Keep-alive" set in header replies 501 Not Implemented

So if I do a request to a route over postman with keep alive set it responds at first Ok and works as expected but after that the webserver responds with Not Implemented for some reason...
Why is that?

Hi @Xentox-Phil,
What do you mean by "after"?
What happens between the 1st call and the next?

And just, do you have le last version of the repository ?
I've looked at the code and tried with Postman, but I can't see anything at the moment.


the email property should be a list of emails

Other fields like phone-numbers and enckeys already are lists. Making the email field an "emails" list field would be a good idea as there might be more than one email per organisation (and therefore a reason for multiple enckeys). You could store multiple emails comma separated but then a client has to parse this list if he wants to know which encryption key is used for a specific email address, etc.

agreed, makes sense.

Let's do it then


Docs required on "InstanceIds" field syntax under EC2 start instances

Hi,
Getting "failed: InvalidParameterType: Expected params.InstanceIds to be an Array" error while using StartInstance Operation under AWS EC2. Tried all possible sytaxs, but could not get anything working.
Examples tried: ['i-abcd1234'] ; {"InstanceIds":["i-abcd1234"],"DryRun":false,"Force":true} ; {"InstanceIds":["i-abcd1234"]} ; "InstanceIds":["i-abcd1234"]
Please help with the solution.
Thanks,
Renjith.

Needs to be looked at. Ref https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/EC2.html#startInstances-property

Also, please note, on manually editing the 'InstanceIDs' field in node-red config file, the StartInstance node seems to work. Seems like some mapping issue. edited to something like this and it worked - ""payload": "[i-abcd1234]","


Countries supported

Check the latest comment here:
https://github.com/bithyve/internal-docs/issues/172

Verified this on Google Play console.


Unified AdminSite imports in docs.

While reading the Django admin site docs, noticed that all the other code snippets on this page use from django.contrib import admin.
Feel free to close the PR if you think it doesn't worth merging.
Thanks!

@hramezani Thanks :+1:


No related posts were found. Check the index.

Hello, can you help me please? I am getting the following message:
"No related posts were found. Check the index."
My version of ghost is 0.7. The theme I'm using is "anima" http://themeforest.net/item/anima-responsive-pet-rescue-and-shelter-template/11172622
This is my web http://tecneofito.com.

Do you have any posts with the same tag as the one you're viewing?
Sent from my iPhone

On Oct 8, 2015, at 3:10 PM, giancta92<EMAIL_ADDRESS>wrote:
Hello, can you help me please? I am getting the following message:
"No related posts were found. Check the index."
My version of ghost is 0.7. The theme I'm using is "anima" http://themeforest.net/item/anima-responsive-pet-rescue-and-shelter-template/11172622
This is my web http://tecneofito.com.
—
Reply to this email directly or view it on GitHub.

Yes, there are more posts with the same tag.

@giancta92 can you link a paste to your config and markup

Hi, excuse me, Where can I get this information?

@giancta92 Make sure your titleClass and tagsClass are correct.


Tweak rails DVM setup to be more like erlang

Be able to start process in foreground or background
Support symlinking of configurable files
Fix bugs preventing oc-id from loading properly

:+1: I like the consistency with erlang services


changed getting filename

Q
A

Bug fix?
no

New feature?
no

BC breaks?
no

Deprecations?
no

Fixed tickets
fixes #issuenum

Related issues/PRs
#issuenum

License
MIT

What's in this PR?
for dots in filename better to rstrip by last dot

Thank you for your contribution.


Cleanup Legacy Package

Description
We had a legacy package which only had the build system in it. We have recently chanaged the build system enough that it has diverged from the actual legacy code.
Moved it into Improbable.Gdk.Core.BuildSystem inside the Core package. Renamed all namespaces. Updated build scripts to invoke correct function.
Tests
Ran ci/build-client.sh and ci/build-gamelogic.sh.
Documentation
N/A - docs on build system are WIP
Primary reviewers
If your change will take a long time to review, you can name at most two primary reviewers who are ultimately responsible for reviewing this request. @ mention them.
@jessicafalk @jared-improbable

Please move the build system into its own package so it isn't manditory


AppSync Lambda mock is broken

Describe the bug
Using the callback() function in lambdas no longer works with the AppSync simulator.
Example:
export default function (event, context, callback) {
  setTimeout(() => {
    callback(null, "response");
  }, 1000);
}

The response is always null.
The issue seems to have been introduced here
The reason is that the above function returns undefined, and context.done(null, null); is called immediately. When `callback()´ is finally called, it is already too late.
Amplify CLI Version
Starting at<EMAIL_ADDRESS>To Reproduce
Use the example lambda above with the AppSync simulator.
Expected behavior
invokeFunction() should wait for the lambda to call the callback.
Additional context
Introduced in #3254
cc: @ammarkarachi

Hi @bboure
You have to return a promise here.
return new Promise(() => setTimeout(() => {
callback(null, "response");
}, 1000));

Thank you @akshbhu
I know I can just return a promise 😄
Problem is, I am using https://github.com/middyjs/middy and internally it uses the callback instead of promises.
So that is kinda out of my hands.

I just want to echo that I am encounter this issue too and because I am using Clojurescript it is much more ergonomic for me to use the callback argument instead of a promise

@bboure This would break users that return null/undefined after a lambda function is invoked. Right now the way your code is structured you are still returning an undefined. I have no hook to wait on for your use case

@ammarkarachi Thank you for your feedback.
I think there should be basically 2 use cases:

null/undefined is be returned by the callback (callback(null, null);)
a promise is returned by the lambda, and that promise resolves as null/undefined

What is the use-case of returning null/undefined directly ?

@bboure In issue #2502 the GraphQL API when returning no data would face this problem

@ammarkarachi I am not sure in what circumstances a lambda would return anything else than a promise.
I see that on this line we are assuming that the lambda is returning a promise (use of await). Which is not always the case.
I assume that your use case in #2502 is expecting a promise resolving to something falsy?
So, in order to fix this issue, what if we checked the returned value of the lambda first?
The current logic should only be executed in case of a promise.
If not, then, the callback is assumed to be called by th e lambda itself.


Load all available tenant options

Right now the plugin only loads the first page with a page size of 2000 for the tenant options.
tenant-option-management\tenant-option-management.service.ts
  getAllOptions() {
    return this.tenantOption
      .list({
        pageSize: 2000,
      })
      .then((res) => res.data.map((o) => ({ id: `${o.category}-${o.key}`, value: o.value })));
  }

It's unusual, but there could be use cases, which have more than 2000 tenant options. In this case, it might happen that not all options are displayed correctly, because they can't be found on the page loaded.
Proposed Solution
Update the query to include query parameter withTotalPages and load all available pages for tenant options.

I will create a PR for this issue

Added as part of v1.0.3


Allow restricting Label to single line

Often times you want to restrict the size of a label to a single line. While it is possible to configure a height this may have undesired effects when different platforms use different vertical spacings.

Ideally we should provide a property to set the number of lines to be shown (#111).

Closing in favor of #111.


added override for hparams in load_from_ckpt

Fixes #1474
Enable the following... (ran into a bunch of issues with transfer learning when moving across clusters)... ugh.
I think @tullie has suggested something like this?
# override some of the params with new values
MyLightningModule.load_from_checkpoint(
    PATH,
    hparam_overrides={'num_layers': 128, 'pretrained_ckpt_path': NEW_PATH}
)

This doesn't work, if hparams is a nested dict, right?

good point. i guess we can support replacing the nested hparams by passing nested updates here? match key by key within the tree

Should work and probably is the same effort as having to flatten the dict.
While we're on it: Can tensorboard log nested hparams? If not, we may want to flatten the dict anyway after the network is initialized?

you mean in the setter for hparams in the pl module?

Yes maybe and undo also save it flattened (then you could acess the entries with something like hparams['a.b'] in the example above) and unflatten it right before model init when laoding from checkpoint...

@Borda is this how we get doc tests?

@justusschock but if we flatten for the user it might be unexpected no?

maybe a better behavior is that on hparams setter in the module it becomes a namespace so that users don't have to use ['notation'].

for loggers we flatten all of them with the dot notation?

why is the override necessary? This already works by passing a dict to kwargs, and then it will be merged / updated with the one in the checkpoint and passed to the LightningModule.
Except for the flattening, I would expect that to work already

@Borda is this how we get doc tests?

What do you mean?

I added the >>> on the docs.
Do I need something special to run as doc tests?

I added the >>> on the docs.
Do I need something special to run as doc tests?

it depends in it is python file, you do not need anything extra
if it is the rest file then you do not use >>> but put it to testcode
.. testcode::

    a = 1

cc: @awaelchli


Fix CodeQL code scanning warnings introduced by PR #6823

Description of the issue
The update of codecov/codecov-action in ci.yml from version 4 to version 5 (a result of PR #6823 ) is now leading to warnings during PR checks:

It's not immediately clear what is causing this, because we don't actually seem to have a CodeQL config file.

Discussed during Cirq Cynq 2024-12-11: we should check previous PR checks to see if they showed up before the recent dependabot configuration change. This might still be a side-effect of the dependabot.yml configuration.

The discussion at https://github.com/github/codeql-action/issues/1179 may yield some clues about what's going on.


dotnet test with .NET Framework on Linux/macOS broken w/ .NET SDK 7.0.400

Description
Running dotnet test --framework net472 with xunit/xunit.analyzers:

Using .NET SDK 7.0.400 on Linux or macOS is broken
Using .NET SDK 7.0.307 (or 6.0.413) on Linux or macOS is fine
Using .NET SDK 7.0.400 on Windows is fine
Using .NET 6 instead of .NET Framework 4.7.2 is fine

This appears to be a non-Windows + .NET Framework (Mono) problem, only with the latest .NET SDK 7.
(It may be coincidence, but I noticed that .NET SDK 7.0.400 ships Microsoft (R) Test Execution Command Line Tool Version 17.7.0-preview-23364-03+bc17bb9693cfc4778ded51aa0ab7f1065433f989 which feels suspect given that it's a preview version and not an RTM version. I have verified, though, that this same version is used on Windows as well.)
Example GitHub Actions failure run: https://github.com/xunit/xunit.analyzers/actions/runs/5882040465
Steps to reproduce
From Linux or macOS:

Install .NET SDK 7.0.400
git clone --recurse-submodules https://github.com/xunit/xunit.analyzers
cd xunit.analyzers
git switch --detach 513113a87fbbe57246d852e68b2074318a1da748
dotnet test src/xunit.analyzers.tests --framework net472

Expected behavior
With 7.0.307:
Test run for /home/bradwilson/dev/1/xunit.analyzers/src/xunit.analyzers.tests/bin/Debug/net472/xunit.analyzers.tests.dll (.NETFramework,Version=v4.7.2)
Microsoft (R) Test Execution Command Line Tool Version 17.6.3 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.

Passed!  - Failed:     0, Passed:  1477, Skipped:     0, Total:  1477, Duration: 1 m 25 s - xunit.analyzers.tests.dll (net472)

Actual behavior
With 7.0.400:
Test run for /home/bradwilson/dev/xunit/analyzers/src/xunit.analyzers.tests/bin/Debug/net472/xunit.analyzers.tests.dll (.NETFramework,Version=v4.7.2)
Microsoft (R) Test Execution Command Line Tool Version 17.7.0-preview-23364-03+bc17bb9693cfc4778ded51aa0ab7f1065433f989 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Testhost process for source(s) '/home/bradwilson/dev/xunit/analyzers/src/xunit.analyzers.tests/bin/Debug/net472/xunit.analyzers.tests.dll' exited with error: Unhandled Exception:
System.TypeInitializationException: The type initializer for 'Microsoft.VisualStudio.TestPlatform.ObjectModel.EqtTrace' threw an exception. ---> System.TypeLoadException: Could not load type of field 'Microsoft.VisualStudio.TestPlatform.ObjectModel.PlatformEqtTrace:TraceSourceLevelsMap' (1) due to: Could not resolve type with token 01000019 from typeref (expected class 'System.Diagnostics.TraceLevel' in assembly 'System, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b77a5c561934e089') assembly:System, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b77a5c561934e089 type:System.Diagnostics.TraceLevel member:(null)
   --- End of inner exception stack trace ---
  at Microsoft.VisualStudio.TestPlatform.TestHost.Program.Main (System.String[] args) [0x00033] in <6059d047aaf14e749a5c8d73c69afc94>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.TypeInitializationException: The type initializer for 'Microsoft.VisualStudio.TestPlatform.ObjectModel.EqtTrace' threw an exception. ---> System.TypeLoadException: Could not load type of field 'Microsoft.VisualStudio.TestPlatform.ObjectModel.PlatformEqtTrace:TraceSourceLevelsMap' (1) due to: Could not resolve type with token 01000019 from typeref (expected class 'System.Diagnostics.TraceLevel' in assembly 'System, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b77a5c561934e089') assembly:System, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b77a5c561934e089 type:System.Diagnostics.TraceLevel member:(null)
   --- End of inner exception stack trace ---
  at Microsoft.VisualStudio.TestPlatform.TestHost.Program.Main (System.String[] args) [0x00033] in <6059d047aaf14e749a5c8d73c69afc94>:0
. Please check the diagnostic logs for more information.

Test Run Aborted.

Diagnostic logs
https://gist.github.com/bradwilson/be699930236685260ced3eb003c27697
Environment
dotnet --info
.NET SDK:
 Version:   7.0.400
 Commit:    73bf45718d

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  22.04
 OS Platform: Linux
 RID:         ubuntu.22.04-x64
 Base Path:   /usr/share/dotnet/sdk/7.0.400/

Host:
  Version:      7.0.10
  Architecture: x64
  Commit:       a6dbb800a4

.NET SDKs installed:
  6.0.413 [/usr/share/dotnet/sdk]
  7.0.400 [/usr/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.21 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.10 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.21 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.10 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  None

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

mono --version
Mono JIT compiler version <IP_ADDRESS> (tarball Tue Jul 11 21:37:50 UTC 2023)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           __thread
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  amd64
        Disabled:      none
        Misc:          softdebug
        Interpreter:   yes
        LLVM:          yes(610)
        Suspend:       hybrid
        GC:            sgen (concurrent by default)

This is probably the same issue as #4549 and #4627, we inserted fix for that into servicing on 7.0.4xx, and it should be in the next service release.  (this has links to the insertion pr https://github.com/microsoft/vstest/issues/4549#issuecomment-1662494569)
Could you try the preview build of 7.0.401 from here please? https://github.com/dotnet/installer#table

Wait, looking at the binaries in 7.0.401, the have the same version as what you show in the console. Something is wrong.

The installer pipeline probably does not produce new builds of 7.0.4xx right now, our insertion is in sdk source code, but the latest insertion from sdk to installer is pretty old. https://github.com/dotnet/installer/pull/16954
So right now you don't have a way to test it I think. 😞

@nohwnd No problem, let me know if/when you have something for me to test.

Also agree that this seem to be the same as #4549.

Glad I found this, just saw our Azure DevOps build pipeline fail on the tests for one project. If it is of any interest then here's a short snippet of our pipeline yaml:
pool:
  vmImage: 'macos-latest'

steps:
- task: DotNetCoreCLI@2
  displayName: Restore App.Test
  inputs:
    command: restore
    projects: '...csproj'
    vstsFeed: '...'

- task: DotNetCoreCLI@2
  displayName: Build App.Test
  inputs:
    projects: '...csproj'
    arguments: '--no-restore'

- task: DotNetCoreCLI@2
  displayName: Run App.Test
  inputs:
    command: test
    projects: '...csproj'
    arguments: '--no-build'

Checked it again today. The installer build is still old.

Strange, our app build pipeline failed last week on tests, then started working.
But now today I got the same error again:
[ERROR] FATAL UNHANDLED EXCEPTION: System.TypeInitializationException: The type initializer for 'Microsoft.VisualStudio.TestPlatform.ObjectModel.EqtTrace' threw an exception. ---> System.TypeLoadException: Could not load type of field 'Microsoft.VisualStudio.TestPlatform.ObjectModel.PlatformEqtTrace:TraceSourceLevelsMap' (1) due to: Could not resolve type with token 01000019 from typeref (expected class 'System.Diagnostics.TraceLevel' in assembly 'System, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b77a5c561934e089') assembly:System, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b77a5c561934e089 type:System.Diagnostics.TraceLevel member:(null)

Are you building on hosted azdo images? Maybe you've hit an agent that already updated, and then hit another one that was not updated yet, and then an updated one again?

Yeah, Azure DevOps build pipeline and using:
pool:
  vmImage: 'macos-latest'

I got this exception on my source tests:
https://github.com/EasyMicroservices/Serialization/pull/56

7.0.401 was released, I checked the nuget and it has  17.7.1 vstest.console, tried running net481 tests on ubuntu (WSL) and it works. So imho fixed, please someone confirm :)
Test run for /mnt/c/Users/jj/Downloads/dotnet-sdk-7.0.401-linux-x64_.tar/dotnet-sdk-7.0.401-linux-x64_/test/bin/Debug/net481/test.dll (.NETFramework,Version=v4.8.1)
Microsoft (R) Test Execution Command Line Tool Version 17.7.1 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.

Passed!  - Failed:     0, Passed:     1, Skipped:     0, Total:     1, Duration: 121 ms - test.dll (net481)

👍🏼


Please Support Other Source Control Providers

Anytime I look for help regarding deploying Azure Static Web Pages, it's all GitHub (do this in GitHub, do that in GitHub). Has it occurred to anyone that some of us might be unlucky enough to use Bitbucket or even Azure DevOps?

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: c1fbc397-f3ea-dc49-ec99-a77e824752e0
Version Independent ID: 7524eee9-a9a6-07f6-acb5-b008a02d6078
Content: Review pull requests in pre-production environments
Content Source: articles/static-web-apps/review-publish-pull-requests.md
Service: static-web-apps
GitHub Login: @sinedied
Microsoft Alias: yolasors

@Alfetta159
Thanks for your feedback! We will investigate and update as appropriate.

Using the Azure Static Web Apps extension in VSCode:

#verticalintegration

Hi @Alfetta159, while the VS Code extension only supports GitHub repos, the steps you're looking for is Deploy to Azure Static Web Apps with external providers | Microsoft Learn. This is a general doc to connect to external sources but also contains a note that links you over to how to configure Bitbucket and GitLab.
#please-close


Add countries property

:tada: This PR is included in version 1.2.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


AWS Linux

The node-cron works on my local ubuntu and my local mac but it does not work on the Amazon Linux ( which is what Beanstalk uses)
Anyone else facing the same issue?

For anyone else who ran into this issue. This alternative npm package does work on AWS Linux
https://www.npmjs.com/package/cron

Hey  @godsmustbcrazy
Actually I never tried to run it on at Amazon Linux.
I've used node-cron inside a docker image at EB, so, my env was a debian...
So, Thanks for your alternative!
I'm closing the issue for now.


Responsive UI clarification

So I just want to clear things out in terms of responsive usage, because I have been facing some issues with responsiveness lately and I was thinking what I am doing wrong,
Text Fonts: I have seen that you have added a list of responsive fonts with certain values corresponding to the usual ones, how can I add a specific font value in the custom theme that is responsive ?
Images: I am not really sure how to make images responsive, if you can please show small example?
Paddings/Margins: I have seen that you have added the scaleWidth and  scaleHeight functions.
Are the scaleWidth and scaleHeight used for responsive spacing only or can they be used in other cases which I am not aware of, if yes where?
Also, if I am missing other tools or facilities that can help for responsiveness, please let us know.
Thank you

hi @toras123 , do you have discord? I want to discuss more before releasing v0.7.0

hi @toras123 , do you have discord? I want to discuss more before releasing v0.7.0

Yes I do, here is my discord: Toras#2778

hi @toras123 , do you have discord? I want to discuss more before releasing v0.7.0

Yes I do, here is my discord: Toras#2778

Friend request sent, from Han Ji-Pyeong


Support for Non-Regional Forms and Gender Differences

Was making my Living Dex and noticed the site does not support forms besides regional forms. Unown, Vivillon, Alcremie and gender differences are the big form changes I would be looking for. Perhaps have it as an option to display and/or have the option to display forms/gender differences as seperate boxes.

Hello, I'm already looking for it but it doesn't seem to be about it. please

In contrast to the previous post, I think all "other" forms should be in their own box. This is entirely because what happens when they add another hat for pikachu? You will need to move 1300+ pokemon in HOME to keep the order straight. If they are in a separate box, the worst case is you need to move the whole box one over to make room for another.

This appears to be another duplicate of #256?
(The checkboxes are probably kinda implied, due to being available for Gmax forms)


Update ByteBuffer docs to advance write index on set operations

Motivation:
Docs give unexpected results since behavior was changed in NIO 2
to only see readable bytes. Fixes #1232.
Modifications:
Documentation updated.
Result:
Documentation gives expected results with no errors.

@glbrntt I believe the other docs are all correct

@glbrntt I believe the other docs are all correct

Awesome -- thanks for checking!


Chore/add unittests

I added some unit tests to the project for watch, unwatch, and unwatchAll
They mock the fs and the epoll dependency so they can be ran on non-raspberry pis.
I'm unsure how familiar y'all are with unit tests, so if there are any questions let me know.
I am happy to add more if we think they are a good idea. 😄
Feel free to pull my branch and make code style changes or whatever

@pizzaisdavid I'll take a look at this asap an let you know if I've any questions.

@pizzaisdavid Thank you very much and sorry for taking so long to merge.

I am happy to add more if we think they are a good idea. :smile:

I definitely think they are a good idea and would be more than happy if you add more :smile: