Error loading script api.py when installing Dreambooth Ext

This is such a great notebook. I'm currently experimenting with training Dreambooth with a custom SD checkpoint but I can't load the Dreambooth extension as I get these errors
What I've tried

Reading through other issues I tried to add !pip install -U diffusers in a different code block
Deleting the runtime and doing this again

Anyone have any ideas or encountered anything similar?
Error loading script: api.py
Traceback (most recent call last):
  File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/scripts.py", line 195, in load_scripts
    module = script_loading.load_module(scriptfile.path)
  File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/script_loading.py", line 13, in load_module
    exec(compiled, module.__dict__)
  File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/sd_dreambooth_extension/scripts/api.py", line 24, in <module>
    from extensions.sd_dreambooth_extension.dreambooth.finetune_utils import FilenameTextGetter
  File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/sd_dreambooth_extension/dreambooth/finetune_utils.py", line 257, in <module>
    class ImageBuilder:
  File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/sd_dreambooth_extension/dreambooth/finetune_utils.py", line 330, in ImageBuilder
    def generate_images(self, prompt_data: list[PromptData]) -> [Image]:
TypeError: 'type' object is not subscriptable

Error loading script: dreambooth.py
Traceback (most recent call last):
  File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/scripts.py", line 195, in load_scripts
    module = script_loading.load_module(scriptfile.path)
  File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/modules/script_loading.py", line 13, in load_module
    exec(compiled, module.__dict__)
  File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/sd_dreambooth_extension/scripts/dreambooth.py", line 17, in <module>
    from extensions.sd_dreambooth_extension.dreambooth.finetune_utils import ImageBuilder, PromptData
  File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/sd_dreambooth_extension/dreambooth/finetune_utils.py", line 257, in <module>
    class ImageBuilder:
  File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/sd_dreambooth_extension/dreambooth/finetune_utils.py", line 330, in ImageBuilder
    def generate_images(self, prompt_data: list[PromptData]) -> [Image]:
TypeError: 'type' object is not subscriptable

have you check in the extension repo : https://github.com/d8ahazard/sd_dreambooth_extension
if there is no similar error, I'll see where is the problem from the colab's side

Thanks @TheLastBen - Part of the issue definitely seems to be coming from the extension
(see: https://github.com/d8ahazard/sd_dreambooth_extension/issues/593) because of a Python 3.8 problem
Others in the community are also experiencing this so I'm going to close this issue and continue the discussion on here:
Referencing: https://github.com/TheLastBen/fast-stable-diffusion/issues/914


Deprecate cardano-api's ProtocolParameters

From @Jimbo4350 here on Slack (context: https://github.com/input-output-hk/cardano-cli/pull/455 caused https://github.com/input-output-hk/cardano-cli/issues/472 and https://github.com/input-output-hk/cardano-cli/issues/474):

A potential way forward would be to:
• Define JSON instances for cardano-api's LedgerProtocolParameters that are identical to what we have for ProtocolParameters.
• We should be able to begin the deprecation of cardano-api's ProtocolParameters .
• Talk to QA about relying fully on the ledger's PParams JSON instances for Conway on wards.

cc @mkoura

@smelc @mkoura @Jimbo4350 just to give you also some context about external tools. we heavily use the protocol-parameters.json output to use it for the build-raw command and calculate things and switch tool behavior on protocol version...
For example, during the pool registration certificate generation query the parameters, and than look at the current minPoolCost value to validate that the generated certificate is within the allowed boundaries. Later on when it comes to the pool registration, query the parameters and look at the stakePoolDeposit parameter to calculate the right amount for the build-raw command. Also for example i am using my own minOutUTXO value function i created to be independend from the inbuilt one of the cli. Also looking at values like protocolVersion.major, utxoCostPerByte, minUTxOValue, utxoCostPerWord. When it comes to the stake key registration, looking for the stakeAddressDeposit parameter to calculate the correct amounts for the transaction. And so on ...
All these actions must be possible on complete offline mode without a node running.
So whatever change will be made in the future, please keep such things in mind. CLI tools like Jormanager, CN-Tools, StakepoolOperatorScripts are building transactions via build-raw to do explicit era transaction (often needed for hw-wallet restrictions) and so on. We are not using the "easier" build command for this.
Best regards, Martin

Thanks @gitmachtl. Just to confirm you would like us to maintain cardano-api's existing JSON serialization of ProtocolParameters

Thanks @gitmachtl. Just to confirm you would like us to maintain cardano-api's existing JSON serialization of ProtocolParameters?
yes. but it would be good if we could use unified names for the parameters era-independent. just in case parameter names are changing in conway+ era, it would be nice to also backport those name changes starting from shelley. for the tools its not important if a parameter is called poolPledgeInfluence or a0 in the json. but it would be nice if its the same one for all eras. so we change it all one time in the tooling, and no need to always check if a parameter is named that or that way.

Why are you still interested in utxoCostPerWord? It was deprecated in Babbage.
utxoCostPerWord was just an example for pre babbage eras, we don't have it often now, but for the testnets we are going thru all the eras over and over again. i am using utxoCostPerByte starting for protocol major 7+ yes.

This is parked until after Chang hard fork.

Thanks @gitmachtl. Just to confirm you would like us to maintain cardano-api's existing JSON serialization of ProtocolParameters?

yes. but it would be good if we could use unified names for the parameters era-independent. just in case parameter names are changing in conway+ era, it would be nice to also backport those name changes starting from shelley. for the tools its not important if a parameter is called poolPledgeInfluence or a0 in the json. but it would be nice if its the same one for all eras. so we change it all one time in the tooling, and no need to always check if a parameter is named that or that way.

Why are you still interested in utxoCostPerWord? It was deprecated in Babbage.

utxoCostPerWord was just an example for pre babbage eras, we don't have it often now, but for the testnets we are going thru all the eras over and over again. i am using utxoCostPerByte starting for protocol major 7+ yes.

So the plan is to eventually deprecate the JSON serialization of ProtocolParameters in cardano-api in favour of PParams in cardano-ledger. Probably some time after the Chang hardfork. This should make life easier for everybody as there will be a single JSON serialization for the protocol parameters (in our libraries anyways).

I am quite happy with the latest update tbh, were we have the "normal" and the "governance" parameters both in the protocol-parameters input/output. Just saw, my last post was from Nov2023 😄


Does KUDO support handling scaling up/down pods automatically ?

I'm asking incase kudo doesn't support it
What if Kudo controller can check pods logs/status for doing something.
Like: we have service A that at sometime because of heavy load, it produces logs that it needs More Resources ( Memory/CPU).
What if Kudo can listen to that kind of situation and immediately scale number of pods
I don't know if kudo can handle

@AhmedMItman to me that sounds more like a use case for autoscaler. Perhaps you can check out https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/

@alenkacz I was asking if there is a way in kudo to for example watch logs in specific POD to trigger the required task depend on the logs


chore(whale-api): migrate whale to apps/whale-api

What this PR does / why we need it:
Final part of whale migration to jellyfish moving whale/src to apps/whale-api.
Which issue(s) does this PR fixes?:

Fixes #977
Additional comments?:
Please note the following required changes to make this migration work. While this PR aims to change as little as possible to avoid exhaustive review requirements some differences between whale and jellyfish requires changes to be made. I will make an effort here to point out changes that required these minor changes to occur.
First change is to update all imports pointing to:

@src => ../relative/path
@whale-api-client => @defichain/whale-api-client
@defichain/*/dist => @defichain/*/src

This is done through a migration script I quickly wrote so that we could quickly port any ongoing updates to whale into jellyfish without having to go file by file each time. You can see the migration script used here.
This change is needed because whale was doing its own internal pathing as seen here. This is an issue because you can't do these sort of sub pathing in individual application projects. While it can technically work from a typescript build perspective, it starts conflicting with the pathing requirements in our jest setup. Trying to consolidate the differences between whale and jellyfish in this manner would be too cumbersome and require us to make root jellyfish changes to adhere to a single application package. So the optimal solution here is simply to just fix the imports, which is done in this PR.
The second change that has occurred in this PR is rooted in the different typescript rules between jellyfish and whale. In jellyfish all catch (err) casts err as unknown. In whale err is any, this requirement forces us to cast all the errors caught in whale to be cast to a correct error instance. The following files has been modified for this requirement.

module.api/loan.collateral.controller.e2e.ts:218:14
module.api/loan.scheme.controller.e2e.ts:138:14
module.api/loan.token.controller.e2e.ts:187:14
module.api/loan.vault.controller.e2e.ts:130:14
module.api/loan.vault.controller.e2e.ts:141:14
module.api/masternode.controller.e2e.ts:97:14
module.api/poolpair.controller.e2e.ts:270:14
module.api/poolpair.service.ts:64:11
module.api/poolpair.service.ts:75:11
module.api/rawtx.controller.e2e.ts:58:14
module.api/rawtx.controller.e2e.ts:76:14
module.api/rawtx.controller.e2e.ts:122:14
module.api/rawtx.controller.e2e.ts:140:14
module.api/rawtx.controller.ts:58:11
module.api/rawtx.controller.ts:61:11
module.api/rawtx.controller.ts:66:40
module.api/rawtx.controller.ts:85:11
module.api/rawtx.controller.ts:88:11
module.api/rawtx.controller.ts:92:40
module.api/transaction.controller.e2e.ts:88:14
module.database/provider.level/level.database.ts:77:11
module.indexer/rpc.block.provider.ts:86:11

I considered changing the typescript rule to cast errors back to any, but since that would be specifically to fix whale I decided against this. It's better to do these very minor updates to whale rather than changing all our jellyfish approaches to adhere to whale which is being deprecated.
Third change was fixing JSON import in the app.module.ts as apps are now sharing the same package.json file.

https://github.com/DeFiCh/whale/blob/main/src/app.module.ts#L1

That should conclude the changes made to the original whale src code. Outside of this nothing else has been modified and is a 1:1 representation of the original whale implementation. This should ensure that we do not need to review the code, but simply ensure that it builds and tests reliably. And any future refactoring and restructuring efforts can be delegated as tasks within jellyfish going forward.

Non src changes:

https://github.com/DeFiCh/jellyfish/pull/1248/files#diff-b55cdbef4907b7045f32cc5360d48d262cca5f94062e353089f189f4460039e0R13

To support functionality that resolves this in class variable assignments, whale uses es2019 but I see no reason not to bump this to es2020 since we need to change it regardless.

https://github.com/DeFiCh/jellyfish/pull/1248/files#diff-b55cdbef4907b7045f32cc5360d48d262cca5f94062e353089f189f4460039e0R18

To support json imports used in whale for package versioning.

https://github.com/DeFiCh/jellyfish/pull/1248/files#diff-1e058ca1442e46581b13571fb8d261f9e1f5657e26c96634d4c1072f0f0347f1R6

To support previous /dist targets to /src which is used extensively in whale.

Note that this was not actually needed in #1005 but seems to break with the latest main and whale src. So this additional rule was added to fix pathing errors in jest.

This PR is a updated version of:

https://github.com/DeFiCh/jellyfish/pull/1005

This PR outdates the following PRs:
These PRs can be made into refactoring tickets post migration

https://github.com/DeFiCh/jellyfish/pull/1201
https://github.com/DeFiCh/jellyfish/pull/1239

This PR builds on the following PRs:

https://github.com/DeFiCh/jellyfish/pull/1166
https://github.com/DeFiCh/jellyfish/pull/1167
https://github.com/DeFiCh/jellyfish/pull/1170
https://github.com/DeFiCh/jellyfish/pull/1241

thanks for the detailed write-up! will do some sanity checks:

[ ] tests pass locally
[ ] contents are indeed the same, apart from the minor changes mentioned

Checks
1. Tests all pass

whale $ jest src
jellyfish $ jest whale-api

2. Same tests

No diff in jest --listTests

3. src/ content

Manually diffed jellyfish/apps/whale-api/src against whale/src locally with intellij diff viewer, and can confirm that only (1) imports are changed and (2) error type assertions are added
only exception is poolpairs.service.ts, which needs syncing with whale's latest changes

Nice! @kodemon

We need to bump the CI test instance count for this soon.

whale-api-client unit tests from #1005 has been moved to follow up PR and placed in its original location from where they originated in whale repository.

https://github.com/DeFiCh/jellyfish/pull/1250

A minor suggestion:

rename apps/whale-api to apps/whale as it encompasses more than just the api layer

Needs syncing up:

poolpair.service.ts

Some notes:

the relative imports can be shortened in a subsequent PR
just FYI - building and running whale's entrypoint main.js throws: node:internal/modules/cjs/loader:936 ; Error: Cannot find module '@defichain/jellyfish-transaction/src/script/_buffer', also to be fixed in a subsequent PR

Updated as per your suggestions, thanks for the review efforts thus far 👍🏻


Suppress the Java reflective access errors in the log

When EthSigner is run old versions of Java e.g. Java 8 then we get a lot of errors like "java.lang.UnsupportedOperationException: Reflective setAccessible(true) disabled" in the logs. These can be ignore and can be suppressed by tweaking the Java options in gradle.
This was fixed in Besu so we can copy what was done there.

#262 merged.


Changing the Favicon

Instead of default favicon, we could put up a custom one!

Please assign this issue to me. I can work on this.

Instead of default favicon, we could put up a custom one!

@Nimit1775 do assign me this, Quick Fix!

Instead of default favicon, we could put up a custom one!

@Nimit1775 do assign me this, Quick Fix!

hey ,  i am not the pa of this project you have tagged the wrong guy


Danger when try to change or add a new item

Two issues here:

When I try to rename a tag name that is a node in treeview, will replace all inside elements into JSON.
If I try to add a new item the same thing happens.

Will be more secure if we cannot to change or add new itens with the same same of the node.

Thanks for reporting. Which version are you using? The latest version should merge the trees when renaming a tag to an already existing node (exactly the same nodes will then be replaced by the one you are renaming).
Adding a new node with a dialog will indeed replace the existing, it seems like a bug to me, should have the same functionality as the field at the left bottom where pressing enter to an existing key will not recreate it, but just selecting the existing one.

I´m using v0.7.0. Thank you for your help, this tool is very useful and is helping us alot!

Hmm, 0.7.0 should be merge the trees just fine. I'm not able to reproduce it locally, do you have any reproduction steps (maybe a simple tranlations example) where this happens?
About the other issue, I have a fix ready for this which will be in the next release as soon as the above issue gets fixed.

Hi @jcbvm
In the first example I changed the name to a same name used as a node. I think that here should have a question if I wish to replace them. That would help when you is editing a lot of tags and forget that you already have one with the same name.

On the second example is the same problem, but now I tried to Add a new item and they disappeared. The same case of show a question here or just block this function will resolve. So, the user will see and will delete it before the add a new.

Below is my test folder with the JSON files
locales.zip
Thank you!

Thanks for your great response.
I see whats happening here, I thought that this was going about renaming a tree to an exiting tree, but here you are renaming a value (single node) to a tree, in which case the result is what is expected, because the single node will replace the whole tree (there is no way we can merge here, otherwise we get invalid JSON). I think it is indeed a good idea to add a confirmation here.
The second issue is actually an issue with the tree (which I got fixed already).
Will make a fix for the problem mentioned above.


Change sensor enum from ALL_CAPS to Pascal style

Motivation and Context

How Has This Been Tested

Types of changes

[x] Docs change / refactoring / dependency upgrade
[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist

[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[x] I have completed my CLA (see CONTRIBUTING)
[ ] I have added tests to cover my changes.
[ ] All new and existing tests passed.

@vauduong Try pulling from master and see if that solves the CircleCI issue.

@vauduong Try pulling from master and see if that solves the CircleCI issue.


tgi can use transformers==4.35.0? because if i use prompt_tokens>40k,transfromers has some bug if version is lower.

System Info
tgi latest version
Information

[X] Docker
[ ] The CLI directly

Tasks

[ ] An officially supported command
[ ] My own modifications

Reproduction
tgi can use transformers==4.35.0? because if i use prompt_tokens>40k,transfromers has some bug if version is lower.
Expected behavior
i just see in requestments.txt, transfomers is require 4.33.3 .

#1351 updates transformers to 4.36


Ksync files watch needs to be restarted

Ksync watch needs to be restarted for it to pick a new spec
if i have started a ksync watch and then i create a spec it sometimes does not pick up changes and pick up and starts sync only after i restart ksync watch
✋ i can work on fixing this(if there is an issue) if given some help

Not able to reproduce will reopen when see it again

Yes i changed the selector in ksync yaml and it did not pick up my pod. I had to restart for changes to be picked up by the watch

@alok87 It's not really designed for editing of the status files under the hood, generally you would need to delete and re-add the spec via the cli.

Is it a good thing to have? any issues you see if we it auto reloads spec list based on yaml change?

I believe it is listening for file changes though, https://github.com/vapor-ware/ksync/blob/master/cmd/ksync/watch.go#L72

It is not working sometimes. I would try to reproduce this and come back.

I'll close this for now. Reopen if you see it again.


Encoding error in row names

https://cirm-dev.isrd.isi.edu/chaise/record/#1/Microscopy:Experiment/ID=20160915-H%26ENone-GA-0
the & gets displayed as &
The annotation is:
('Microscopy', 'Experiment', 'tag:isrd.isi.edu,2016:table-display', '{
    "row_name" :{"row_markdown_pattern":"{{{ID}}}"},
    "compact": {"row_order": [{"column":"Experiment Date","descending":true}]}
}')

The database contents can be seen here of course
https://cirm-dev.isrd.isi.edu/ermrest/catalog/1/entity/Microscopy:Experiment/ID=20160914-H%26ENone-JAM-0

which are
[{"ID":"20160914-H&ENone-JAM-0","Experiment Date":"2016-09-14","Initials":"Jill McMahon","Disambiguator":"0","Comment":"","Experiment Type":"H&E","Probe":"None","Number of Slides":4,"Number of Scans":4,"Probes":["None"]}]

The offending line.
https://github.com/informatics-isi-edu/chaise/blob/master/record/index.html.in#L40

I will assign this issue to @robes once the code is merged.

Assigning to @robes for review.


[AdditionalVectorCrypto] adding section on addition vector crypto extensions

This pull request is the riscv-isa-manual version of a pull request started on riscv-crypto: https://github.com/riscv/riscv-crypto/pull/362.
/!\ This pull request is a draft for the future fast track extensions.
This pull requests draft the changes associated with two fast track extensions for vector crypto.
During the specification process for vector crypto 1.0.0 a few items had to be discarded because they appeared too late in the process. This fast track extension tries to address some of them.
The official demand that will be discussed in the Task Group and submitted to the Unpriv Committee is being drafter here: https://docs.google.com/document/d/1zpYhnZi2NxhjfcBGvPOy0oDhx6lTXchscG17Qcl6wv8/edit?usp=sharing
New features:

Zvbc32e: Extending vclmul[h].v[vh] instruction to support SEW=32-bit value

should be available standalone (ELEN >= 32) or in addition to Zvbc (ELEN >= 64)
no new encoding

Zvkgs: Adding .vs variants to vghsh and vghmul

should depend on Zvkgs
new encodings

Open questions:

[X] Should Zvbc32e be allowed when ELEN >= 32 without depending on Zvbc ? (Answer: YES)
[ ] Should Zvbc32e support SEW=16 ? (SEW=8 ?)
[x] Find encodings
[x] How to name the two new extensions
[ ] Do we need to define a Zvkt(bc/bc32e) to extend Zvkt to the extension of vclmul[h/] defined in Zvbc32e ?

Related changes:

[x] spike-isa-sim modifications

[x] add support for Zvbc32e and Zvkgs: https://github.com/riscv-software-src/riscv-isa-sim/pull/1748

[x] encoding proposal in riscv-opcodes repo: https://github.com/riscv/riscv-opcodes/pull/270
[ ] adapt vector crypto code samples for Zvbce32 https://github.com/riscv/riscv-crypto/blob/main/doc/vector/code-samples/zvbc-test.c
[ ] adapt vector crypto code samples for Zvkgs https://github.com/riscv/riscv-crypto/blob/main/doc/vector/code-samples/zvkg-test.c
[ ] Sail models

[ ] Zvkgs
[ ] Zvbc32e

Draft versions:

Version
pdf

v0.0.1 (August 31st 2023)
https://github.com/riscv/riscv-crypto/files/12487628/riscv-crypto-spec-vector-extra.pdf

v0.0.2 (January 17th 2024)
https://github.com/riscv/riscv-crypto/files/13970691/riscv-crypto-spec-vector-extra.pdf

v0.0.3 (February 1st 2024)
https://github.com/riscv/riscv-crypto/files/14146438/riscv-crypto-spec-vector-extra_v0.0.3.pdf

v0.0.4 (February 6th 2024)
riscv-crypto-spec-vector-extra_v0.0.4.pdf

v0.0.5 (March 7th 2024)
riscv-crypto-spec-vector-extra_v0.0.5.pdf

v0.0.6 (June 19th 2024)
riscv-crypto-spec-vector-extra_v0.0.6.pdf

Changelogs

from v0.0.5 to v0.0.6:

adding vs2 / vd overlap as reserved encoding for new vghsh.vs / vgmul.vs instructions (review feedback from @QJtaibai )

Original Plan for the fast track schedule

References

project announcement  on the unpriv mailing list (https://lists.riscv.org/g/tech-unprivileged/message/568) and the crypto TG mailing list (https://lists.riscv.org/g/tech-crypto-ext/message/944)

Will it be reserved or not for instructions (vghsh and vgmul) , if the vd register group overlaps with the vs2 register group.

Good question @QJtaibai , this is not listed but should be to aligned with the other .vs instructions (and since overlapping those does not make sense functionally and can make implementation harder).

Hello! I have a question that in the instruction part of document vghsh.vs shares same encoding with vghsh.vv, and has func6:101100, while in Appendix A it looks vghsh.vs have func6:100011. I wonder which one is correct?

Hello! I have a question that in the instruction part of document vghsh.vs shares same encoding with vghsh.vv, and has func6:101100, while in Appendix A it looks vghsh.vs have func6:100011. I wonder which one is correct?

Appendix A and the riscv-v opcode Pull request (https://github.com/nibrunieAtSi5/riscv-opcodes/pull/1/files) are right, the instruciton part of the document was incorrect. I have fixed it.
Note that those opcodes are simply suggestion at this point (but that does not mean the suggestions should not be consistent).
Thank you for pointing that out.

I have developed a small example to demonstrate how Zvbc32e can be leveraged to implement CRC (with the folding method): https://github.com/nibrunie/rvv-examples/blob/e55d529fe54316a95b733aea82cadbfbfbf08e67/src/crc/vector_crc_be_zvbc32e.c
The current implementation provides "performance" of about 0.65 instruction per bytes (on 1MB input buffer).

riscv-crypto-spec-vector-extra_v0.0.7.pdf

Will it be reserved or not for instructions (vghsh and vgmul) , if the vd register group overlaps with the vs2 register group.

Good question @QJtaibai , this is not listed but should be to aligned with the other .vs instructions (and since overlapping those does not make sense functionally and can make implementation harder).

We don't have this constraint on vghsh.vv and vgmul.vv? Is it OK to make it stricter?

Will it be reserved or not for instructions (vghsh and vgmul) , if the vd register group overlaps with the vs2 register group.

Good question @QJtaibai , this is not listed but should be to aligned with the other .vs instructions (and since overlapping those does not make sense functionally and can make implementation harder).

We don't have this constraint on vghsh.vv and vgmul.vv? Is it OK to make it stricter?

For LMUL>1, the operations likely take place over multiple cycles or uops. The restriction on .vs ensure that the scalar element in the first register is not overwritten before all LMUL parts have been processed. This is the same reason we have rules on widening and narrowing instructions. This restriction is not needed for .vv since the input bits and output bits from the same LMUL piece.

Will it be reserved or not for instructions (vghsh and vgmul) , if the vd register group overlaps with the vs2 register group.

Good question @QJtaibai , this is not listed but should be to aligned with the other .vs instructions (and since overlapping those does not make sense functionally and can make implementation harder).

We don't have this constraint on vghsh.vv and vgmul.vv? Is it OK to make it stricter?

For LMUL>1, the operations likely take place over multiple cycles or uops. The restriction on .vs ensure that the scalar element in the first register is not overwritten before all LMUL parts have been processed. This is the same reason we have rules on widening and narrowing instructions. This restriction is not needed for .vv since the input bits and output bits are from the same offset in the register group. Overwriting an earlier part of the register doesn't affect operations on later parts of the register.

Yeah, thanks! I get it! But it seems that vd can overlap vs1 for reductions, aren't they the same?

Will it be reserved or not for instructions (vghsh and vgmul) , if the vd register group overlaps with the vs2 register group.

Good question @QJtaibai , this is not listed but should be to aligned with the other .vs instructions (and since overlapping those does not make sense functionally and can make implementation harder).

We don't have this constraint on vghsh.vv and vgmul.vv? Is it OK to make it stricter?

For LMUL>1, the operations likely take place over multiple cycles or uops. The restriction on .vs ensure that the scalar element in the first register is not overwritten before all LMUL parts have been processed. This is the same reason we have rules on widening and narrowing instructions. This restriction is not needed for .vv since the input bits and output bits are from the same offset in the register group. Overwriting an earlier part of the register doesn't affect operations on later parts of the register.

Yeah, thanks! I get it! But it seems that vd can overlap vs1 for reductions, aren't they the same?

Maybe the assumption for reductions is that vd would only be updated after all of vs1 is read since you need to see all elements before you know the result.

What about the VPR alignment rules for vghsh.vs / vgmul.vs?
As I mentioned in https://github.com/riscv-software-src/riscv-isa-sim/issues/1548

Possible alignment value: 1, 2, 4, 8 (1 for any VPR, 2 for V0, V2, ...)
Prerequisite: VLEN * LMUL >= EGW

Single key in .vs inst: max(EGW/VLEN, 1)
Others: max(LMUL, 1)

The non-overlap constraint for vector element group is part of the vector crypto specification
https://github.com/riscv/riscv-isa-manual/blob/04179addb9725a0b253689ce59235041161ea832/src/vector-crypto.adoc#L311

In the case of the .vs instructions defined in this specification, vs2 holds a 128-bit scalar element group.
For implementations with VLEN ≥ 128, vs2 refers to a single register. Thus, the vd register group must not
overlap the vs2 register.
However, in implementations where VLEN < 128, vs2 refers to a register group comprised of the number
of registers needed to hold the 128-bit scalar element group. In this case, the vd register group must not
overlap this vs2 register group.

[%autowidth]
[%header,cols="4,4,4"]
|===
| Instruction
| Register
| Cannot Overlap
| vaes*.vs      | vs2      | vd
| vsm4r.vs      | vs2      | vd
| vsha2c[hl]    | vs1, vs2 | vd
| vsha2ms       | vs1, vs2 | vd
| vsm3me        | vs2      | vd
| vsm3c         | vs2      | vd

It was defined as no overlapping between vector register groups.
This was not necessarily repeated for each operation
https://github.com/riscv/riscv-isa-manual/blob/04179addb9725a0b253689ce59235041161ea832/src/vector-crypto.adoc#L1530

Only for the .vs form: the vd register group overlaps the vs2 scalar element group

I think this constraint makes sense here as well since allowing overlapping would constraint uarch which cannot save/stage the scalar element group before finishing the operation (as stated by @topperc it is highly likely that for LMUL > EGW / VLEN implementations will iterate to compute the operation over the full vector register groups in a few cycles). Note that .vs makes sense for vl / EGS > 1 since this is where we can actually exploit the fact of having a scalar element group.
The size of vs2 vector register group should be understood as with EMUL = min(EGW / VLEN, 1).
The example in the base vector crypto spec seems to hint this way:

However, in implementations where VLEN < 128, vs2 refers to a register group comprised of the number
of registers needed to hold the 128-bit scalar element group.

but should maybe be generalized to EGW.

Zvkgs: Adding .vs variants to vghsh and vghmul
should depend on Zvkgs

Is the second line supposed to say Zvkg rather than Zvkgs?

Yes, you are right @topperc , this is a typo dependency to Zvkg was intended (the assumption is that the vector-scalar(element group) variant do not make sense without the vector-vector variant.

In the v0.0.7 spec, I have a question of the operation code of the vghsh.vs and vgmul.vs. Take vghsh.vs for example, It says H is common to all element groups, and you put the "let H = brev8(...)" code outside the loop. I suspect the H of all the group should be the same, and is from the first 128bit of vs2. But inside the each loop, H will be changed.
So if you write just like the operation code in the spec, the H of the next loop will not be the first 128bit of the vs2. I want to ask it is just the purpose of the instruction or is just a miss?
If you want the H to be the same value in the beginning of the each loop, you should put "let H = brev8(...)" inside the loop, so in the beginning of the loop, we can get the H value from vs2 again.

In the v0.0.7 spec, I have a question of the operation code of the vghsh.vs and vgmul.vs. Take vghsh.vs for example, It says H is common to all element groups, and you put the "let H = brev8(...)" code outside the loop. I suspect the H of all the group should be the same, and is from the first 128bit of vs2. But inside the each loop, H will be changed. So if you write just like the operation code in the spec, the H of the next loop will not be the first 128bit of the vs2. I want to ask it is just the purpose of the instruction or is just a miss? If you want the H to be the same value in the beginning of the each loop, you should put "let H = brev8(...)" inside the loop, so in the beginning of the loop, we can get the H value from vs2 again.

@httoracle ,
I am not sure I understand your question.
The goal of vghsh.vs and vgmul.vs is to use the same H across all elements groups.
This is why H is extracted initially (and once and for all) through:
  // H is common to all element groups
  let helem = 0;
  let H = brev8(get_velem(vs2, EGW=128, helem)); // Hash subkey

H also corresponds to the element group of index helem=0. H must not depend on i (else the instructions become equivalent to vghsh.vv and vgmul.vv already present in Zvkg).
The are two loops in the behavior of vghsh.vs and vgmul.vs, the outer loop iterates on the element groups and the outer loop iterate on the bits of each element groups (to perform the local carry-less multiplication and reduction). Extracting the H element group must be external to both those loops.

In the v0.0.7 spec, I have a question of the operation code of the vghsh.vs and vgmul.vs. Take vghsh.vs for example, It says H is common to all element groups, and you put the "let H = brev8(...)" code outside the loop. I suspect the H of all the group should be the same, and is from the first 128bit of vs2. But inside the each loop, H will be changed. So if you write just like the operation code in the spec, the H of the next loop will not be the first 128bit of the vs2. I want to ask it is just the purpose of the instruction or is just a miss? If you want the H to be the same value in the beginning of the each loop, you should put "let H = brev8(...)" inside the loop, so in the beginning of the loop, we can get the H value from vs2 again.

You are right,
H needs to be reset inside the outer loop. this is a mistake in the current behavior code. I will correct it.

Additional Vector Crypto extension v0.0.8 (September 9th 2024):
riscv-crypto-spec-vector-extra_v0.0.8.pdf
Changelog from v0.0.7:

Fixing behavior of vghsh.vs and vgmul.vs (thanks to @httoracle )
Upgrading alias for 0x77 opcode from OP-P to OP-VE (thanks to @wangpc-pp )

Maybe a silly question: do we need zvbc8e/zvbc16e? I feel it can be useful, but I have no data to show how common CRC8/CRC16 are

@wangpc-pp

Maybe a silly question: do we need zvbc8e/zvbc16e? I feel it can be useful, but I have no data to show how common CRC8/CRC16 are.

That is definitely no a silly question and it has been appeared a few times.
There are a few use cases for 8-bit and 16-bit vector carryless multiply (as you mentioned at least 8 and 16-bit CRCs) but when I asked in the crypto task group for motivations, no one was able to say this was a key use cases which would justify specific extensions (rather than relying on widening and existing instructions from Zvbc[32e], it seems 8 and 16-bit CRCs are less used than the 32-bit versions).

From a hardware point of view, I have run a few experiments and supporting 8 and 16-bit vclmul[h] on top of the 32-bit and 64-bit variants would have a minimal cost
No new opcode would need to be allocated since this would simply be extending the range of valid vsew for vclmul[h] to cover 0 and 1
Another question was: if we go down that route do we need to specific Zvbc[8,16,32]e independently or to avoid some fragmentation they would better be specified in a single extension.

v 0.0.9:
Changelog: Integration feeback, including ARC's:

change name OP-P to OP-Ve
adding justification for Zvbc32e and Zvkgs

riscv-crypto-spec-vector-extra_v0.0.9.pdf
Note:: part of the justification for Zvkgs relies on the fact that a scalar element group in a .vs vector instruction has EMUL=EGW/VLEN. This fact implies that when VLEN >= 128, one single vector register is required to hold a scalar element group for Zvkgs and it does not need to have an index aligned on LMUL (similar to what should apply for Zvkned or Zvksed). Although this can appear intuitive, I don't think this is ever mentioned explicitly in the original vector crypto specification.

@wangpc-pp

Maybe a silly question: do we need zvbc8e/zvbc16e? I feel it can be useful, but I have no data to show how common CRC8/CRC16 are.

That is definitely no a silly question and it has been appeared a few times. There are a few use cases for 8-bit and 16-bit vector carryless multiply (as you mentioned at least 8 and 16-bit CRCs) but when I asked in the crypto task group for motivations, no one was able to say this was a key use cases which would justify specific extensions (rather than relying on widening and existing instructions from Zvbc[32e], it seems 8 and 16-bit CRCs are less used than the 32-bit versions).
* From a hardware point of view, I have run a few experiments and supporting 8 and 16-bit `vclmul[h]` on top of the 32-bit and 64-bit variants would have a **minimal cost**

* No new opcode would need to be allocated since this would simply be extending the range of valid `vsew` for `vclmul[h]` to cover `0` and `1`

* Another question was: if we go down that route do we need to specific Zvbc[8,16,32]e independently or to avoid some fragmentation they would better be specified in a single extension.

I don't know the answer, but here are some investigations:

ARM has 8 bits PMUL.
I tried to find any useful usages via Debian Code Search:

https://codesearch.debian.net/search?q=svpmul&perpkg=1
https://codesearch.debian.net/search?q=vmul_p8&perpkg=1
https://codesearch.debian.net/search?q=vmulq_p8&perpkg=1

Here are two usages that I think is useful:

RAID6 syndrome calculation in Linux kernel: https://sources.debian.org/src/linux/6.10.11-1/lib/raid6/neon.uc/?hl=54#L54
Erasure Code based on GF(8): https://sources.debian.org/src/ceph/18.2.4+ds-7/src/erasure-code/jerasure/gf-complete/src/neon/gf_w4_neon.c/?hl=64#L64

If the cost of supporting 8/16 bits won't be large, personally, I would like to have them added.

I don't know the answer, but here are some investigations:

ARM has 8 bits PMUL.

I tried to find any useful usages via Debian Code Search:

https://codesearch.debian.net/search?q=svpmul&perpkg=1
https://codesearch.debian.net/search?q=vmul_p8&perpkg=1
https://codesearch.debian.net/search?q=vmulq_p8&perpkg=1

Here are two usages that I think is useful:

RAID6 syndrome calculation in Linux kernel: https://sources.debian.org/src/linux/6.10.11-1/lib/raid6/neon.uc/?hl=54#L54
Erasure Code based on GF(8): https://sources.debian.org/src/ceph/18.2.4+ds-7/src/erasure-code/jerasure/gf-complete/src/neon/gf_w4_neon.c/?hl=64#L64

If the cost of supporting 8/16 bits won't be large, personally, I would like to have them added.

That is very interesting, it would be also interesting to check if anyone has ever ported jerasure/gf-complete over RISC-V (Vector) and what would be the expected benefit overall.

Is there any concern that the name Zvkgs is too similar to the existing Zvksg? Though this isn't the first time we had such similar names, we have Zbc and Zcb.

Is there any concern that the name Zvkgs is too similar to the existing Zvksg? Though this isn't the first time we had such similar names, we have Zbc and Zcb.

No concerns were voiced so far, but if you think this could be an issue we can try to come up with another name. (Zvkga for additional Zvkg instructions ?)

Zvbc32e may not be suitable now if we extend SEW to 8/16/32 bits. Maybe Zvbcxw(Extended Width), Zvbcxe(Extended EEW)?

Zvbc32e may not be suitable now if we extend SEW to 8/16/32 bits. Maybe Zvbcxw(Extended Width), Zvbcxe(Extended EEW)?

I discussed this with @aswaterman and he suggested that the suffix 32* has already been used to mean any element width 32-bit and lower, e.g. in Zve32f, so I think Zvbc32e still works.

Agreed

I don't agree this wording, I strongly recommond that we should change to another name.
My thinking is that 32 in Zve32x/f is Minimum VLEN, not Supported EEW. For Zvbc32e, the postfix e indicates that it is about EEW, so people may think only 32 bits is supported.

I don't agree this wording, I strongly recommond that we should change to another name. My thinking is that 32 in Zve32x/f is Minimum VLEN, not Supported EEW. For Zvbc32e, the postfix e indicates that it is about EEW, so people may think only 32 bits is supported.

I understand your concern @wangpc-pp
Even if we interpret Zve has meaning  is the minimum VLEN then it also works here for Zvbc(32e/e32/e32x)
We could switch to e32 rather than 32e and have Zvbce32 (IIRC we need to avoid ending an extension name with a number to facilitate parsing ISA string and this is why I used 32e) to mean that the minimum VLEN supported by the extension is 32 and it has the same supported list of EEW as Zve32(x/f): that is 8, 16, and 32.
If I summarize, the options are:

Zvbc32e
Zvbcxw (Extended Width),
Zvbcxe (extended EEW)
Zvbce32
Zvbce32x

I would be in favour of keeping 32 somewhere in the name because at least it makes explicit one of the constraint (VLEN/ELEN) but I agree that Zvbc was not explicit on listing the only EEW it supports and having only 32 in the name does not make it explicit that it supports 8 and 16 bits SEW values as well.

I don't agree this wording, I strongly recommond that we should change to another name. My thinking is that 32 in Zve32x/f is Minimum VLEN, not Supported EEW. For Zvbc32e, the postfix e indicates that it is about EEW, so people may think only 32 bits is supported.

I don't think the 32 in Zve32x/f is about VLEN. The focus on the Zve* extensions is about reducing ALU cost by limiting which element sizes are supported or what operations are supported on large elements. The removal of 64-bit integers elements is the important difference between Zve32x and Zve64x not the minimum VLEN.

I don't agree this wording, I strongly recommond that we should change to another name. My thinking is that 32 in Zve32x/f is Minimum VLEN, not Supported EEW. For Zvbc32e, the postfix e indicates that it is about EEW, so people may think only 32 bits is supported.

I don't think the 32 in Zve32x/f is about VLEN. The focus on the Zve* extensions is about reducing ALU cost by limiting which element sizes are supported or what operations are supported on large elements. The removal of 64-bit integers elements is the important difference between Zve32x and Zve64x not the minimum VLEN.

That was my initial interpretation: that the VLEN constraint relaxation was actually a consequence on the limitation of ELEN in general or on the largest SEW supported for floating-point elements.

Thanks for these explanations! I am still not convinced but I will let it be since it seems I'm in the minority. 😄

The 32 in Zve32 represents ELEN.  The minimum VLEN is set to be the same as ELEN for these extensions (Table 62 in current vector chapter).

New version: v0.0.10:
Changelog:

Changing Zvbc32e definition to include 8-bit and 16-bit SEW values
Fixing typos (including unifying to "carry-less" (1))

riscv-crypto-spec-vector-extra_v0.0.10.pdf
(1): it seems the original vector crypto specification uses both "carryless" and "carry-less". Looking at wikipedia, both seem valid (https://en.wikipedia.org/wiki/Carry-less_product, https://en.wikipedia.org/wiki/Finite_field_arithmetic#Carryless_multiply). I chose to unify to "carry-less" but I am open to suggestion.

New version of the spec draft (v0.0.11 Dec5th 2024):
riscv-crypto-spec-vector-extra_v0.0.11.pdf
Changelog:

implementing ARC feedback:

explicit mention of "reserved" behavior for vl / vstart not multiple of EGS
split into two chapters (one per extension)

multiple typo fixes + switch to in "development" state


Support Arithmetic function SQRT

Motivation

Support an arithmetic function SQRT.

Modifications

Add jpql model and serializer for SQRT

Commit Convention Rule

Please commit your modification based by conventional commit:

Commit type
Description

feat
New Feature

fix
Fix bug

docs
Documentation only changed

ci
Change CI configuration

refactor
Not a bug fix or add feature, just refactoring code

test
Add Test case or fix wrong test case

style
Only change the code style(ex. white-space, formatting)

chore
It refers to minor tasks such as library version upgrade, typo correction, etc.

If you want to add some more commit type please describe it on the Pull Request

Result

The arithmetic function sqrt is added.

Closes

#596

Codecov Report
Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (0b08f5a) 91.97% compared to head (e45b69e) 91.97%.

Files
Patch %
Lines

.../render/jpql/serializer/impl/JpqlSqrtSerializer.kt
83.33%
1 Missing :warning:

Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #612      +/-   ##
===========================================
- Coverage    91.97%   91.97%   -0.01%
===========================================
  Files          297      299       +2
  Lines         3215     3227      +12
  Branches       195      195
===========================================
+ Hits          2957     2968      +11
- Misses         204      205       +1
  Partials        54       54

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

please resolve conflicts.


Support modifying Base URL

Currently, the only mechanism to change the base url is the following:

a) install configure, setup etc an IPT
b) go to administration, change the base url to the new version
c) stop the IPT, change the server settings (ports, virtual hosts, DNS etc) and restart
d) if necessary (e.g. a new war deployed) set the location of the data directory to
be that of the original

Some things to consider:
- has the GBIF registry been notified of the change in URL in this procedure?  If so,
when was it validated?
- one cannot alter the order of this, since it is not possible to log in if the configuration
is changed.  This can be demostrated by the following procedure using maven:

a) start an IPT on port 7001, configure etc
b) stop jetty, change the pom to use port 7002
c) start the server and try and log in

One solution would be to solve this through documented procedures in the user guide,
and make sure that it is possible to initiate a "change the IPT baseURL" request to
the registry through the administration section.

Original issue reported on code.google.com by timrobertson100 on 2010-10-18 18:24:07

These week, we migrated our baseUrl to https in ipt.gbif.es. Reading the documentation and this issue I think we should also:
Admin > IPT settings > ￼Publish all resources
Just to be sure we don't duplicate things, etc.


Infinite loop when installing custom app with MS Edge

Expected Behavior
The app extension is installed. Infinite loop is not happening. "itp" cookie is set.
Current Behavior
When I click "Install app" in Shopify partner dashboard, an infinite loop starts. It seems to be caused by missing "itp" cookie.
Failure Information
Read from bottom to top (bottom goes first) and so on, I stop it manually
GET / 					302 Found
GET /itp 					302 Found
GET /authenticate/oauth 	200 OK
GET / 					302 Found
GET /itp 					302 Found
GET /authenticate/oauth 	200 OK
GET /                                       302 Found
GET /itp                                   302 Found
GET /authenticate/oauth        200 OK

Steps to Reproduce
Please provide detailed steps for reproducing the issue.

Create custom app on Shopify, using Laravel and your package.
Try to install app from step 1 on test shop using MS Edge browser in private mode (other browsers not tested)
Infinite loop starts, observe server logs, browser traffic etc.

Context
Please provide any relevant information about your setup. This is important in case the issue is not reproducible except for under certain conditions.

Package Version: v16.0.0
Laravel Version: v8.11.2
PHP Version: 7.4
Using a toolset: Docker, ngrok

@osiset Issue is fixed on https://github.com/osiset/laravel-shopify/releases/tag/v16.0.1 Closing.


With global-hl-line-mode enabled, tern-find-definition doesn't highlight the line it jumps to

Hey, I am unsure if this a tern issue or a hl-line issue. Posting here to sort this out collaboratively as I am not very good with emacs internals and debugging in it (especially post command hooks) in general.
Observed behaviour:
When I run tern-find-definition (bound to M-.) on a symbol, the cursor correctly jumps to the location where the symbol was declared. But, the line the cursor jumped to is not highlighted. Instead the line the cursor jumped from remains highlighted.
Steps to reproduce

Create a js file with the following contents:

const a = 42
module.exports = a

Move the cursor on the usage of symbol a
Invoke tern-find-definition

Metadata
Tern - v0.21.0
Emacs - Commit 0943cc1 (brew installed using --HEAD)

Thanks in advance folks!

I'd say that is an issue with hl-line, but I'm also not super knowledgeable about all the obscurities of elisp programming. Still, going to close this since I'm not planning to debug it myself. If you ever find a solution that requires a change to tern.el, feel free to open a PR.


[DO NOT MERGE] Include ExtendedResourceToleration plugin

Testing to see if the plugin can be enabled by default

/retest

/retest


added Youtube-Clone under React-JS-Projects/Intermediate

Closes:  #751

Title :  Youtube-Clone
Name: : Sourabh Singh Rawat
Idenitfy yourself: :  GSSOC Contributor

Describe the add-ons or changes you've made 📃
Added a project under React-JS-Projects/Intermediate named "Youtube-Clone".
Updated React-JS-Project/README.md file, Adding my project to the list.
How Has This Been Tested? ⚙

The code has been tested under various conditions :

WebBrowser view
Mobile View
slow net case

Screenshots 📷

-> mobile view:

-> Browser view ( laptop, computers, etc ) :

Type of change ☑

What sort of change have you made:

[ ] Bug fix (non-breaking change which fixes an issue)
[X] New feature (non-breaking change which adds functionality)
[ ] Code style update (formatting, local variables)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

Checklist: ☑

[X] My code follows the Contributing Guidelines & Code of Conduct of this project.
[X] This PR does not contain plagiarized content.
[X] I have performed a self-review of my own code.
[X] I have commented my code, particularly wherever it was hard to understand.
[X] My changes generate no new warnings.

Note to reviewers 📄

Thanks @Avdhesh-Varshney 💯


How can remove the unwanted part on the top navigation bar

Hi,
I want to remove the talk section in the picture below, which is the top navigation bar from the academic resume template.
thank you.

Hey this is likely something that can be done inside of the template, generated content or in the worst case scenario config.yml , have you tried to check if theres a place to remove them there? Also please provide a repo, or link to it so anyone can review the code, I believe this is not a bug but would like to see the code for it, thanks

If this has anything to do with netlify-cms then it can be re-opened.

Ok, thank you. Let me try. I am really new to web programming. Is there a forum to ask such questions?


optimize: optimise way of init seata

What this PR does:
optimize to initialize the project manually
Which issue(s) this PR fixes:

Fixes #174
Special notes for your reviewer:
Does this PR introduce a user-facing change?:

hey guy, the CI failed

done

Pls add comment on every func. If you do not want add comment, "// nolint" should be enough.

Comments have been added to the new method this time, and the rest will be handed over to PR separately in the future


Converting auth section Firebase Android codelab

The sections of the Firebase FriendlyEats Android codelab up through and including Authentication should be updated for the Kotlin version of this app. I'll add the rest of the codelab sections in another PR

@racheldoshcollins looks like the auto-preview is working but if you want to get a local preview server running you can look at the little "devserver" I made for another codelab:
https://github.com/firebase/emulators-codelab/tree/master/scripts
The key is the claat tool although I added some JS on top to give me compile-on-save behavior.


Allow javac options to have directories that end in .java

I ran into a problem where the output directory getting passed to Errorprone as a javac option ended in .java and it was mistakenly getting used for a source file.
e.g.
javac -d ouptut/com/hello.java com/exmplar/HelloWorld.java

I looked at using the Java Main or Arguments class to do the parsing via processArgs() instead but they changed it between Java 8 and Java 9 and say the API is use at your own risk.

BaseErrorProneCompiler is deprecated (https://github.com/google/error-prone/commit/e17752771ea771cc044768a95575682135b71657), I recommend using BaseErrorProneJavaCompiler instead, which among other things does flag handling in a more robust way.


npm install needs to point explicitly to newest version to work

Please provide a link to the documentation page and section
https://blacksmithgu.github.io/obsidian-dataview/plugin/develop-against-dataview/
Describe the problem
A clear and concise description of what is unclear about the documentation
I tried to use the API in my own plugin but it failed. (see more info here: https://forum.obsidian.md/t/i-would-like-to-use-the-dataview-api-in-my-own-plugin/48622), lettig me know that getAPI is not available.
I found out that the npm install obsidian-dataview just installed version 0.4.15 for some reason.
only by explicitly requesting
npm install<EMAIL_ADDRESS>could I get the newest package which included the getAPI in its export. Either something needs to be fixed in the  npm repository or this should be stated in the documentation.
Describe the solution you'd like
A clear and concise description of what you want to happen. Feel free to suggest wording, full sentences, etc.
replace
npm install -D obsidian-dataview
by
npm install -D<EMAIL_ADDRESS>Additional context
Add any other context about the problem here.
see https://forum.obsidian.md/t/i-would-like-to-use-the-dataview-api-in-my-own-plugin/48622

I cannot reproduce this. Leaving out the version on a npm install always installs the latest tag, and the latest is pointing to 0.5.51: https://www.npmjs.com/package/obsidian-dataview?activeTab=versions
I tried myself on a new repository and it installed 0.5.51 like expected. Did you really run npm install -D obsidian-dataview ?

yes, I did (actually numerous times). I also involved a more advanced developer in this who was very suprised by this behavior. If this does not reoccur for anyone, I am happy, but as I spend quite some hours finding out what was going on I wanted to report this, at least someone might come across this issue.
For me it showed up initially only in that getAPI was not available in the obsidian-dataview.  @blacksmithgu : For me it would be fine to close it, except you see anything that should be done beyond reporting this instance.

yes, I did (actually numerous times). I also involved a more advanced developer in this who was very suprised by this behavior. If this does not reoccur for anyone, I am happy, but as I spend quite some hours finding out what was going on I wanted to report this, at least someone might come across this issue.
For me it showed up initially only in that getAPI was not available in the obsidian-dataview. @blacksmithgu : For me it would be fine to close it, except you see anything that should be done beyond reporting this instance.

This behaviour is really odd. I found an issue on npm itself that others experience that as well but it doesn't include a solution. Maybe invalidating your npm cache can help. It seems like the cache is not retrieving newer versions, having the latetst tag pointing to an older one.
In any case, this seems to a npm related issue, not one caused by dataview, so I'd like to close the issue. I am sorry though it took so much of your time. I still think it's best to leave the concrete version out of the documentation, otherwise this needs to be touched on every release. However, if you have the chance to try and the problem is still reproducible for you, could you check if npm i obsidian-dataview@latest installs 0.5.51 ? If this workarounds the issue, I'll open a PR to use this syntax instead. Internally, it does the very same like npm i obsidian-dataview, but on the issue mentioned above it seems like specifying the tag fixed the issue for some.

Hi, here are the latests results that overall confirm your findings and ideas:
a) i went back to version 0.4.15 by doing npm i<EMAIL_ADDRESS>-> this broke my plugin as expected
b) i then updated using npm i obsidian-dataview@latest
-> this restored it by going back to 0.5.51
c) I went back again to 0.4.15 as in step a)
d) I updated using npm i obsidian-dataview
-> this moved it up only to 0.4.25 (which was the one I had previously, again plugin still broken)
e) I update again using  npm i obsidian-dataview@latest
this brought it back 0.5.51 again restoring the plugin.
So overall: yes, i believe to add @latest seems the more robust way of getting the latest package. So would be great if you could change that in the documentation. Thanks for your support on this issue @s-blu .


Decreased result summary length

The maximum length of the result summary was set to 750 earlier. Since, max allowed size of embed is 6000, all the 10 results were not able to fit in a single embed. Setting maximum result summary to 350 allows all the results to fit in the embed.
This resolves the issue #3

rebased and merged in bee0da729b07131559a53d8975bb30d86283f7ff
Thanks for your contribution


common_sub_expression_eliminate  exists bug

Describe the bug
It will make some strange problem.
Error: Internal("Optimizer rule 'common_sub_expression_eliminate' failed, due to generate a different schema, original schema:

DFSchema { fields: [
DFField { qualifier: Some(\"t1\"), field: Field { name: \"t1_id\", data_type: UInt32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} } },
DFField { qualifier: Some(\"t1\"), field: Field { name: \"t1_name\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} } },
DFField { qualifier: Some(\"t1\"), field: Field { name: \"t1_int\", data_type: UInt32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} } },
DFField { qualifier: Some(\"t2\"), field: Field { name: \"t2_id\", data_type: UInt32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} } },
DFField { qualifier: Some(\"t2\"), field: Field { name: \"t2_name\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} } },
DFField { qualifier: Some(\"t2\"), field: Field { name: \"t2_int\", data_type: UInt32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} } }], metadata: {} }, new schema:

DFSchema { fields: [
DFField { qualifier: None, field: Field { name: \"t2.t2_int < UInt32(10)UInt32(10)t2.t2_int\", data_type: Boolean, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} } },
DFField { qualifier: Some(\"t1\"), field: Field { name: \"t1_id\", data_type: UInt32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} } },
DFField { qualifier: Some(\"t1\"), field: Field { name: \"t1_name\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} } },
DFField { qualifier: Some(\"t1\"), field: Field { name: \"t1_int\", data_type: UInt32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} } },
DFField { qualifier: Some(\"t2\"), field: Field { name: \"t2_id\", data_type: UInt32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} } },
DFField { qualifier: Some(\"t2\"), field: Field { name: \"t2_name\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} } },
DFField { qualifier: Some(\"t2\"), field: Field { name: \"t2_int\", data_type: UInt32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} } }], metadata: {} }")

original:
"Projection: t1.t1_id, t1.t1_name, t1.t1_int, t2.t2_id, t2.t2_name, t2.t2_int
  Filter: t2.t2_int < Int64(10) OR t1.t1_int > Int64(2) AND t2.t2_name != Utf8("w")
    Left Join: t1.t1_id = t2.t2_id
      TableScan: t1
      TableScan: t2"

after:
"Filter: (t2.t2_int < UInt32(10)UInt32(10)t2.t2_int AS t2.t2_int < UInt32(10) OR t1.t1_int > UInt32(2)) AND (t2.t2_int < UInt32(10)UInt32(10)t2.t2_int AS t2.t2_int < UInt32(10) OR t2.t2_name != Utf8("w"))
  Projection: t2.t2_int < UInt32(10) AS t2.t2_int < UInt32(10)UInt32(10)t2.t2_int, t1.t1_id, t1.t1_name, t1.t1_int, t2.t2_id, t2.t2_name, t2.t2_int
    Inner Join: t1.t1_id = t2.t2_id
      TableScan: t1 projection=[t1_id, t1_name, t1_int]
      TableScan: t2 projection=[t2_id, t2_name, t2_int]"

To Reproduce
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
Additional context
Add any other context about the problem here.

multiple_or_predicates() generate nasty result, look like this bug cause it. https://github.com/apache/arrow-datafusion/pull/4487#discussion_r1040001890 it's a related comment

Current this method is depend on the Self::desc_expr(expr)
let desc = Self::desc_expr(expr);

It will cause some strange expr.
id_array:

[
(15, "(t2.t2_int < UInt32(10) OR t1.t1_int > UInt32(2)) AND (t2.t2_int < UInt32(10) OR t2.t2_name != Utf8(\"w\"))t2.t2_int < UInt32(10) OR t2.t2_name != Utf8(\"w\")t2.t2_name != Utf8(\"w\")Utf8(\"w\")t2.t2_namet2.t2_int < UInt32(10)UInt32(10)t2.t2_intt2.t2_int < UInt32(10) OR t1.t1_int > UInt32(2)t1.t1_int > UInt32(2)UInt32(2)t1.t1_intt2.t2_int < UInt32(10)UInt32(10)t2.t2_int")
, (7, "t2.t2_int < UInt32(10) OR t1.t1_int > UInt32(2)t1.t1_int > UInt32(2)UInt32(2)t1.t1_intt2.t2_int < UInt32(10)UInt32(10)t2.t2_int")
, (3, "t2.t2_int < UInt32(10)UInt32(10)t2.t2_int")
, (1, "")
, (2, "")
, (6, "t1.t1_int > UInt32(2)UInt32(2)t1.t1_int")
, (4, "")
, (5, "")
, (14, "t2.t2_int < UInt32(10) OR t2.t2_name != Utf8(\"w\")t2.t2_name != Utf8(\"w\")Utf8(\"w\")t2.t2_namet2.t2_int < UInt32(10)UInt32(10)t2.t2_int")
, (10, "t2.t2_int < UInt32(10)UInt32(10)t2.t2_int")
, (8, "")
, (9, "")
, (13, "t2.t2_name != Utf8(\"w\")Utf8(\"w\")t2.t2_name")
, (11, "")
, (12, "")
]

id_array/arrays_list -> affected_id -> project_exprs/predicate
it will cause strange column/expr

problem in #3635 look like exist multiple bug.
cc @liukun4515 @andygrove @alamb @alex-natzka

cc @waynexia  whom I think contributed the optimization originally, and perhaps might have some suggestions

Hi, I'm afraid I won't have time to look into this. I did go over the error message though, here's what I think is happening:

I think some other optimization rule than common_sub_expression_eliminate transforms the filter
t2.t2_int < Int64(10) OR t1.t1_int > Int64(2) AND t2.t2_name != Utf8("w")
into
(t2.t2_int < UInt32(10) OR t1.t1_int > UInt32(2)) AND (t2.t2_int < UInt32(10) OR t2.t2_name != Utf8("w"))
which is weird but technically not wrong.
Then common_sub_expression_eliminate sees that t2.t2_int < UInt32(10) occurs twice, so it adds a projection before the filter where this expression gets computed. (This is correct and expected.) The projection creates the additional column t2.t2_int < UInt32(10)UInt32(10)t2.t2_int as alias for t2.t2_int < UInt32(10).
This changes the output schema, because we don't project out the column again.

The last point is the problem IMO. I guess common_sub_expression_eliminate would need to add another projection after the filter that gets rid of the extra column.

Appreciate the investigation @alex-natzka :heart:

This changes the output schema, because we don't project out the column again.

The last point is the problem IMO. I guess common_sub_expression_eliminate would need to add another projection after the filter that gets rid of the extra column.

This point sounds reasonable to me. I revisited the unit tests in common_sub_expression_eliminate, and looks like I only used the plans that have their own output schema like Projection or Aggregator. Which will ignore the additional column by itself. I plan to look into it this week (and sorry for the late reply 🥲).

Hi, I submit https://github.com/apache/arrow-datafusion/pull/4733 to fix this. Please check it out 🥰


Different addresses of P->meth of two different points on the same curve

Hello.
I am using low level openssl API for elliptic curves and faced with strange behavior
at this line. It performs address comparison of two objects.
My code:
EC_KEY *eckey1 = NULL; // private
EC_KEY *eckey2 = NULL; // public

// open input files. in1 - EC private key, in2 - EC public key
in1 = bio_open_default(path1, 'r', informat);
in2 = bio_open_default(path2, 'r', informat);

eckey1 = PEM_read_bio_ECPrivateKey(in1, NULL, NULL, passin);
eckey2 = PEM_read_bio_EC_PUBKEY(in2, NULL, NULL, NULL);

// create context
BN_CTX *ctx = NULL;
if ((ctx = BN_CTX_new()) == NULL)
    perror("can't create ctx");

// compare groups. they should be in the same group
if(-1 == EC_GROUP_cmp(eckey1->group, eckey2->group, ctx))
    perror("keys are in different groups");

// create new point and copy public key there
EC_POINT* Z1  = EC_POINT_new(eckey1->group);
if(!EC_POINT_copy(Z1, eckey1->pub_key))
    perror("error at Z1=pub"); // this returns 0 and perror is invoked

EC_POINT_copy returns 0, because address of Z1->meth is different than address of eckey1->pub_key->meth, but every member of Z1->meth is the same as according member of eckey1->pub_key->meth, including addresses.
I don't understand why addresses of Z1->meth and eckey1->pub_key->meth are different. It leads to perror in my code. Should I ignore it? I made a mistake somewhere?
P.S. Thanks for great lib! And happy New year :)

Hey, Happy New Year! :-) It looks like filing issues while everyone is on holiday doesn't work too well!! Sorry this got lost.
Anyway, it looks to me like you are reaching inside OpenSSL internals here. The EC_KEY structure is opaque and its members (like pub_key) are not exposed via the public API (did you include the internal header file ec_lcl.h???). I suspect this is the cause of your issue. Closing.


'linuxmusl-x64' binaries cannot be used on the 'linux-x64' platform.

I'm getting an error:
'linuxmusl-x64' binaries cannot be used on the 'linux-x64' platform.
the container linux kernal is 3.10.0 x86_64.

If you're using a glibc-based Linux at runtime, you'll need to ensure a glibc-based Linux is used at npm install time (it looks like you're installing on a musl-based Linux such as Alpine).

https://sharp.pixelplumbing.com/install#common-problems

The architecture and platform of Node.js used for npm install must be the same as the architecture and platform of Node.js used at runtime.

I hope this information helped. Please feel free to re-open with more details if further assistance is required.

If it's helpful to anybody in the future, this is what our Dockerfile looked like before and after to get sharp to work on Node 14 Alpine.
Before (Sharp didn't work)
FROM osgeo/gdal:alpine-normal-3.1.3

WORKDIR /usr/local/src/server

ARG NODE_ENV=production
ENV NODE_ENV=${NODE_ENV}
ARG PORT=80
ENV PORT=${PORT}
EXPOSE ${PORT}

# nodejs-current means nodejs Current, currently at 14.x
RUN apk add --no-cache --update nodejs-current ghostscript graphicsmagick

COPY ./dist/test ./test
COPY ./dist/src ./src

COPY ./dist/scripts ./scripts

CMD ["node", "src/main"]

After (Sharp works)
FROM node:14-alpine as deps

WORKDIR /deps

COPY ./dist/src/yarn.lock .
COPY ./dist/src/package.json .
RUN yarn install --production

FROM osgeo/gdal:alpine-normal-3.1.3

WORKDIR /app

ARG NODE_ENV=production
ENV NODE_ENV=${NODE_ENV}
ARG PORT=80
ENV PORT=${PORT}
EXPOSE ${PORT}

# nodejs-current means nodejs Current, currently at 14.x
RUN apk add --no-cache --update nodejs-current ghostscript graphicsmagick

COPY ./dist/src .
COPY --from=deps /deps/node_modules ./node_modules

CMD ["node", "main"]

problem is if you do an npm install when your node is v16 for example and then you start up your docker with an image from alpine v14 the docs clearly state that it's going to fail.
If you want to get it to work.
run npm install when your local node used to run npm install is the same version as the docker specified container. This was my problem
nvm use v14.x
rm -rf node_modules
npm install
docker-compose up << with alpine-v14


[Frontend][Core] Add Guidance backend for guided decoding

This pull request extends guided decoding capabilities

Add guidance backend
Process logits in parallel with threadpool

guidance backend supports regex, choice, json and grammar.
relevant: https://github.com/vllm-project/vllm/issues/5245
Usage

JSON Generation

from pydantic import BaseModel, ConfigDict

model = "Qwen/Qwen2.5-0.5B-Instruct-GPTQ-Int4"
llm = LLM(model=model)

class UserProfile(BaseModel):
    name: str
    age: int
    email: str

    model_config = ConfigDict(extra="forbid")

sampling_params = SamplingParams(
    temperature=0.0,
    top_p=0.95,
    max_tokens=512,
    guided_decoding=GuidedDecodingParams(
        json=UserProfile,
        backend="guidance",
    ),
)

outputs = llm.chat(
    messages=[
        [
            CustomChatCompletionMessageParam(
                role="system", content="You are a helpful assistant."
            ),
            CustomChatCompletionMessageParam(
                role="user",
                content="Tell me something about yourself (name, age, email) in JSON format.\n",
            ),
        ],
    ],
    sampling_params=[sampling_params],
)

Choices Generation

sampling_params = SamplingParams(
    temperature=0.0,
    top_p=0.95,
    max_tokens=512,
    guided_decoding=GuidedDecodingParams(
        choice=["3","4","5","6"],
        backend="guidance",
    ),
)

outputs = llm.chat(
    messages=[
        [
            CustomChatCompletionMessageParam(
                role="system", content="You are a 5 years-old helpful assistant."
            ),
            CustomChatCompletionMessageParam(
                role="user",
                content="How old are you?",
            ),
        ],
    ],
    sampling_params=[sampling_params],
)

Regex Generation via OpenAI Client

model = "Qwen/Qwen2.5-0.5B-Instruct-GPTQ-Int4"
client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="NOKEY",
)

completion = client.chat.completions.create(
    model=model,
    messages=[
        {
            "role": "user",
            "content": "You are a 5 years-old helpful assistant. information.",
        },
        {
            "role": "user",
            "content": """How old are you?""",
        },
    ],
    extra_body={"guided_regex": "\\d+", "guided_decoding_backend": "guidance"}
)

Benchmark
Model: QWEN2.5-7B-GPTQ-INT4
Dataset: GSM8K
Guided Type: JSON

Metric
Outlines
Guidance

Accuracy
1023/1318 (77.62%)
1032/1318 (78.3%)

Average Output tokens
166 (+/- 83)
195 (+/- 69)

Average Latency in ms per Request (1 concurrent req)
2567 (+/- 976)
1799 (+/- 466)

Average Latency in ms per Request (4 concurrent reqs)
8697 (+/- 3866)
3655 (+/- 1154)

Average Latency in ms per Request (8 concurrent reqs)
17370 (+/- 8139)
5997 (+/- 1991)

Thanks @njhill for your quick review. Really appreciate it.

Presumably the parallelization speedup is due to the fact that the pytorch ops involved release the gil?
That's one reason, another one is the parser (llguidance) used in guidance was implemented in Rust, and it automatically releases GIL when called. So it would be more efficient to run guidance in parallel.
Were your outlines measurements also using the threadpool?
Yes, experiments were done using threadpool
It would be good to also try with the latest outlines 0.1.x if possible which is apparently much faster than < 0.1. We would want to upgrade to that too in any case.
I haven't tested outlines 0.1.x yet, just used the current version in VLLM. However, I am not focusing too much on the benchmark for this PR. The focus is to make guidance available as another guided decoding backend to VLLM's community so people can choose what's best for them. :)

I also figured out lm-format-enforcer is not thread-safe. It failed some tests when number of threads is larger than 1.
@njhill any suggestions for this?

I also figured out lm-format-enforcer is not thread-safe. It failed some tests when number of threads is larger than 1. @njhill any suggestions for this?

Decided to rollback to single threaded version to not break lm-format-enforcer. Hope the current code is good for merging :)

@mgoin Resolved conflict with newly merged xgrammar

Resolved conflict with newly merged xgrammar

@njhill @mgoin


Rhobeg and rhoend arrays

Hi, I'd like to make a feature request, namely rhobeg and rhoend made into arrays where each element of the array corresponds to the parameter index, as in xl and xu. Thanks.

Hi @asenzz ,
Thank you for the comments. It is a great idea.
However, this will not be done, for the following reasons.

Allowing rhobeg to be a vector will lead to (slightly) new algorithms (the novelty is not sufficient for a paper, but the implication on the coding is nontrivial). The purpose of PRIMA is to provide the reference implementation for Powell's method (see the README).  We have no intention to implement new algorithms.

It is equivalent to scaling the variables, which is already available in the MATLAB, Julia, and (the to-be-released) Python interfaces.

Thank you anyway.
Best regards,
Zaikun


timestamp error

This happend right after recording a new dataset on a freshly flashed nicla. First dataset had incorrect timestamp. Not exactly sure how to reproduce.

Dataset is initialized with a start time 0, only once data is received the timstamp changes.
possible solution will be to set the current time during init

Could not reproduce this error:  Dataset is initialized with the current time + an offset.


Type 'any' is not assignable to type 'never' with boolean in interface in TS 3.5

TypeScript Version:  3.5.1

Search Terms:

Type 'any' is not assignable to type 'never'

Code
interface T {
  str: string;
  bool: boolean;
}

const initState: T = {
  str: 'date',
  bool: true,
};

   let k: keyof T;
   initState[k] = 'test' as any;
// ~~~~~~~~~~~~ Type 'any' is not assignable to type 'never'.ts(2322)

This seems to happen specifically with boolean properties. If I get rid of the boolean-valued property from the interface it works fine. If I have a larger interface and select non-boolean properties, it also works fine. But boolean and another type produces this error.
This error happens with TS 3.5 but not 3.4 (see playground link).
Expected behavior:
No error.
Actual behavior:
Type 'any' is not assignable to type 'never'
Playground Link:  link
Related Issues:

This is a documented breaking change; see "Fixes to unsound writes to indexed access types" in the release notes https://devblogs.microsoft.com/typescript/announcing-typescript-3-5/

Thanks @RyanCavanaugh. But I don't follow why this is unsound. k is keyof T so it should be a valid index. I can use it to read from initState just fine:
const v = initState[k];  // ok

The right-hand side has an any type, so it should be fine to assign to any property in T.

Just to elaborate on why this seems odd... changing boolean to number in the interface makes the error go away, which doesn't seem like it should affect the soundness of the assignment.
interface T {
  str: string;
  num: number;
}

const initState: T = {
  str: 'date',
  num: 2,
};

let k: keyof T;
initState[k] = 'test' as any;  // ok

I guess it comes down to this, which also doesn't make a ton of sense to me (also in earlier TS versions):
type T1 = number & string;  // number & string
type T2 = string & boolean;  // never

The break is desired but the inconsistency between boolean (which is secretly a union) and number is not.

This has caused a very large number of errors for us with insufficient guidance of how to fix (from the release notes)...

Most instances of this error represent potential errors in the relevant code. If you are convinced that you are not dealing with an error, you can use a type assertion instead.

OK, so we have large numbers of a pattern that looks like (from the OP):
initState[k] = v; // where k: keyof T and v: T[k]

And these all fail because we have lots of boolean properties.
Is the solution:

Remove boolean from all interfaces? What should be used instead? Is boolean deprecated now?
Some way to explicitly handle when the value of property k is boolean that then gates it out of the initState[k] check?
Switch all instances that use keyof to just deal with any instead and lose type checking?

@KeithHenry the gist of the breaking change was that you never had type safety in the assignment to begin with. If you want the old behavior, you can put an as any around the initState[k]:
interface T {
  str: string;
  bool: boolean;
}

const initState: T = {
  str: 'date',
  bool: true,
};

declare let k: keyof T;
initState[k] = 'test';  // Type '"test"' is not assignable to type 'never'.
initState[k] = 'test' as any;  // Type 'any' is not assignable to type 'never'.
(initState[k] as any) = 'test';  // ok

Alternatively, you can construct a narrower type than keyof T which consists of only those keys whose values are assignable to boolean. You can do this quite generally with a conditional type:
interface T {
  str: string;
  bool: boolean;
  bool2: boolean;
  d: Date;
}

type KeysOfType<T, U> = { [k in keyof T]: T[k] extends U ? k : never }[keyof T];

const initState: T = {
  str: 'date',
  bool: true,
  bool2: false,
  d: new Date(),
};

declare let k: KeysOfType<T, boolean>;  // let k: "bool" | "bool2"
initState[k] = 'test';  // Type '"test"' is not assignable to type 'boolean'.
initState[k] = true;  // ok

@danvk Thanks for the clarification, that helps.
Our current solution is to replace all initState[k] = 'test' with (initState as any)[k] = 'test', it sounds like that's the correct practice - we can then use KeysOfType where we need to check.

@KeithHenry if you put the as any after the property access, TypeScript will check that k is a valid key. This is more similar to what TS did before 3.5.
(initState[k] as any) = 'test';  // index is checked
(initState as any)[k] = 'test';  // index is not checked

@danvk Cheers, thanks for the clarification!

@danvk

type KeysOfType<T, U> = { [k in keyof T]: T[k] extends U ? k : never }[keyof T];

The type returned by this is a union of keys of T, but it ends with  | undefined which is not a valid index key.
Any recommendations to avoid undefined being part of the returned keys union?
Below is a simplified example of my code which suffers from undefined being in the list of keys.
class LargeType extends SomeOtherType {
  // ... SomeOtherType includes other types of mostly optional properties (number, boolean, etc...)
  //     and a couple conversion methods that take no parameters but return objects.
  prop1?: string,
  prop1Op?: myEnum,
  prop2?: string,
  prop2Op?: myEnum,
  // ... long list of similar property pairs
}

myFunction(
  myValue: any,
  propKey: KeysOfType<LargeType, string | undefined>,
  myObject: LargeType
) {
  if (typeof(myValue) === 'string') {
    myObject[propKey] = myValue;
    //       ^^^^^^^   Type 'undefined' cannot be used as an index type.
  }
}

@Zarepheth
Try this:
type KeysOfType<T, U> = { [k in keyof T]-?: T[k] extends U ? k : never }[keyof T];

(I added a -? in the mapped type to remove the optional-ness of the properties.)

@danvk Thanks! That works!

I'm running into the issue with a Promise that should return a True. This seems like a bug in Typescript. I literally can't set Boolean as a type.


Website improvements

What type of PR is this?
improving Website
What this PR does / why we need it:

Adding Slack link to Header (desktop/mobile)
Adding Slack link to Footer
Removing some useless files (*/**/.DS_Store)
Updating .gitignore (ignoring .DS_Store)
Updating Twitter link to https://twitter.com/oss_gr
Adding video to demo page
Adding Armadactl section to  libraries page

Which issue(s) this PR fixes:

Some suggestions from the Slack channel

Screenshots
Desktop Header

Mobile Header

Footer

libraries page

Demo page

┆Issue is synchronized with this Jira Task by Unito

@richscott @dave-gantenbein I am happy to merge this. Let me know if you agree


[Improve] Optimize jobsOverview for null to be the starting NPE excep…

…tion.

What changes were proposed in this pull request
Issue Number: close #xxx

Brief change log

Verifying this change

This change is a trivial rework / code cleanup without any test coverage.
(or)
This change is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:

Does this pull request potentially affect one of the following parts

Dependencies (does it add or upgrade a dependency): (yes / no)

There are some code style issue, you can run mvn spotless:apply  to fixed them


Support AD when nimbleFunction objects are used in models

We recently added support (for the imminent release) for using methods of nimbleObjects as user-defined distributions and functions in models.
There were shortcomings in support for AD with these new features. This PR attempts to clean up those gaps and also add tests, which were obviously missing or we would have noticed the shortcomings earlier.
I have locally successfully run test-ADbasics, where most of the new testing resides. Let's see any other issues with CI testing.
Incidental to its main purpose, this PR has several other pieces:

add tests for the recent features of having modelDef objects retain the user environment (previously it was retained only in BUGSdecl objects)
modify distClass and how its objects are populated so that they get the buildDerivs field from any user-defined distribution (and TRUE or FALSE for built-in distributions).
modify and add testing for modelDef method for checking for AD support in a distribution.

@perrydv ok to merge given the Win/MacOS failures are unrelated?

@paciorek I'm more cautious about this branch since it does change some pervasive internals like the distClass objects, even if only slightly.  I might have missed the test outcomes. At the moment they are showing as canceled. Did they otherwise run and pass except for the current general test issues we're seeing?

@perrydv I think my commit was from the GH browser just dealing with a trivial merge conflict.
Linux tests passed on your last commit - Win/Mac failures were unrelated.
So that sounds like we are ready to merge.


[bugfix] new TransitionDrawable memory leak

[bugfix] new TransitionDrawable point to old drawable array, which cause memory leak
As we know, TransitionDrawable is a LayerDrawable which contains a array of two drawables need to do transition.  But directly use old drawable to create new TransitionDrawable, let the the newly created TransitionDrawable hold a reference to the old array rather than the only drawable.
placeholder = imageView.getDrawable();

So I break the reference chain and fix like this:
                                Drawable oldDrawable = imageView.getDrawable();
                                if (oldDrawable != null) {
                                    if (oldDrawable instanceof LayerDrawable) {
                                        LayerDrawable oldLayerDrawable = (LayerDrawable) oldDrawable;
                                        placeholder = oldLayerDrawable.getDrawable(0);
                                    } else {
                                        placeholder = imageView.getDrawable();
                                    }
                                }

Please review that, and I hope you may accept this pr.

Hi buptfarmer, thanks for the PR, will review in the next couple of days :)
Dont worry about the failing build, currently PR branches miss the private keys. Have to find a workaround in the future.


Improve interactive docker/bash.sh

Some fixes to hopefully improve docker/bash.sh for use in local development:

properly quote command-line arguments
mount repo at $(pwd) by default; fixes problems when using git-worktree.

Don't know how the latter change will impact the CI, but let's see if it breaks it too badly first.

closing for now as per inactive status, feel free to reopen a new one


ChromaDB Chroma.from_documents error:   AttributeError: 'Collection' object has no attribute 'upsert'

System Info
windows 11, python 3.9.16 , langchain-0.0.215
Who can help?
No response
Information

[X] The official example notebooks/scripts
[ ] My own modified scripts

Related Components

[ ] LLMs/Chat Models
[ ] Embedding Models
[ ] Prompts / Prompt Templates / Prompt Selectors
[ ] Output Parsers
[ ] Document Loaders
[X] Vector Stores / Retrievers
[ ] Memory
[ ] Agents / Agent Executors
[ ] Tools / Toolkits
[ ] Chains
[ ] Callbacks/Tracing
[ ] Async

Reproduction
The loaded documents succesfully get indexd in FAISS, Deeplake and Pinecone, so I guess the issue is not there. When using  Chroma.from_documents the error is thrown:
  File ~\...langchain\vectorstores\chroma.py:149 in add_texts
    self._collection.upsert(
        AttributeError: 'Collection' object has no attribute 'upsert'

Code:
with open('docs.pkl', 'rb') as file:
    documents= pickle.load(file)
text_splitter = RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=200)
docs = text_splitter.split_documents(documents)
chroma_dir = base_dir + "ChromaDB" + "_" + str(chunk_size) + "_" + str(overlap)
db2 = Chroma.from_documents(docs, embeddings, persist_directory=chroma_dir)

Uploaded the pickle zipped as .pkl is not supported for upload.
docs.zip
Expected behavior
Index the docs

@luca-git What version of chroma do you have installed?
Fix:  Update Chroma and install lark.


Cow: Watch zimed.com

Cow requests the watch of the watch_keyword zimed\.com. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

zimed\.com has been seen in 1 true positive, 0 false positives, and 0 NAAs.

Approved by Jeff Schaller in Charcoal HQ


return error when master node failover

hello,
i have a redis cluster with three master and three slave. when one master or slave node is down, function command() will return error "server closed connection".  i try to set retry option but useless, is there any ways to solve it? thanks (my request way is Synchronous blocking )

Hi You.
The option max_retry_count is only used for redirects and -TRYAGAIN responses. There is no retry for other errors.
Why? Because we don't know if the command has been executed on the Redis node or not before the error happened. For some commands like INCR or LPUSH it is not safe to run them twice.
Maybe we can implement automatic retry for some commands where it it safe to do it, but for now I think you will need to do retry in a loop in your code.

ok ,thank you, i wll try it


Bazarr media volume condition issue

Under the Servarr chart the Bazarr deployment volume mount is referencing jellyfin media PVC regardless if it's enabled or not here:
https://github.com/kubitodev/helm/blob/f4429b260729b92cf9c7b1d5cfd57dd70d77efb0/charts/servarr/templates/bazarr/deployment.yaml#L53
compared to Radarr:
https://github.com/kubitodev/helm/blob/f4429b260729b92cf9c7b1d5cfd57dd70d77efb0/charts/servarr/templates/radarr/deployment.yaml#L61
or Sonarr:
https://github.com/kubitodev/helm/blob/f4429b260729b92cf9c7b1d5cfd57dd70d77efb0/charts/servarr/templates/sonarr/deployment.yaml#L61

Thank you for noticing that! Fixed in release 1.0.5.


Add missing group by in the feedQuery on the Thread model (result in an error on MariaDB)

After running the project I got the error:

SQLSTATE[42000]: Syntax error or access violation: 1055 'laravelio.threads.author_id' isn't in GROUP BY (SQL: select count(*) as aggregate from (select `threads`.*, CASE WHEN COALESCE(MAX(replies.created_at), 0) > threads.created_at THEN COALESCE(MAX(replies.created_at), 0) ELSE threads.created_at END AS latest_creation from `threads` left join `replies` on `threads`.`id` = `replies`.`replyable_id` and `replies`.`replyable_type` = threads group by `threads`.`id`) as `aggregate_table`)

This PR adds the missing group by columns.

Hmm well, the project isn't really meant to be optimized for MariaDB as it uses MySQL. Everything works fine on MySQL? I don't want to overcomplicate things just for an engine we're not using anyway.

As far as I know MySQL currently ignores it and in this query it will not result in different responses. So I understand it completely if you don't want to merge it 😃

Yeah think we're going to leave this as is if there's no tangible benefit for us in MySQL. Thanks though!


Publish vendor and source code tarballs together when doing releases.

When building for distros such as Fedora and RHEL, there is usually no access to the internet, causing errors when cargo vendor is run. In order to fix this, people maintaining such packages have to run the command themselves and then patch the original source code, adding the vendored code.
Other projects such as kata-containers,  and mdevctl already publish both source and vendor codes in separate tarballs, ensuring that code is buildable even though some other project used gets deprecated and/or deleted.

Closing this as a dup of https://github.com/confidential-containers/guest-components/issues/554


Notes about String.replace

(Moved from rom https://github.com/microsoft/TypeScript-DOM-lib-generator/issues/1562)
I am using TypeScript 5.0.4.
I have noticed the typing of String.replace seem oddly convoluted.
From lib.es5.d.ts:
interface String {
    replace(searchValue: string | RegExp, replaceValue: string): string;
    replace(searchValue: string | RegExp, replacer: (substring: string, ...args: any[]) => string): string;
}

From lib.es2015.symbol.wellknown.d.ts:
interface String {
    replace(searchValue: { [Symbol.replace](string: string, replaceValue: string): string; }, replaceValue: string): string;
    replace(searchValue: { [Symbol.replace](string: string, replacer: (substring: string, ...args: any[]) => string): string; }, replacer: (substring: string, ...args: any[]) => string): string;
}

Like the first parameter is declared as an union, I would expect the second to be the same, at least in the lib.es5 version.
Notably, this makes the following code not compile:
// The second argument is an union of replaceValue and replacer from lib.es5
const replacementSteps: ([RegExp, string | ((substring: string, ...args: any[]) => string)])[] = [
    [/(a|b)/gi, "a"],
    [/(c|d)/gi, match => match.toUpperCase()],
];

let myString = "abcdef";

for (const [searchValue, replacement] of replacementSteps)
    myString = myString.replace(searchValue, replacement); // TS2345 - the second parameter is considered invalid

By adding this into my declaration, the code works:
declare interface String {
    replace(searchValue: string | RegExp, replaceValue: string | ((substring: string, ...args: any[]) => string)): string;
}

I cannot propose a solution other than my last hack, and of course the variants with Symbol would have to be taken into account. Please consider this just a note.
See also this: https://github.com/microsoft/TypeScript/issues/54223

I believe the use of overloads here instead of a union was a holdover from when union types couldn't provide contextual types, or maybe even when we didn't have unions at all. I think this can be updated without breaking anything.

I am happy to modify #50452 to fix this.

This definition allows the use of a union while preserving the Symbol.replace bit:
replace<T extends string | ((substring: string, ...args: any[]) => string)>(searchValue: { [Symbol.replace](string: string, replaceValue: T): string; }, replaceValue: T): string;


Realm "defaultDefaultClientScopes" and "defaultOptionalClientScopes" are ignored

Describe the bug
The top level realm properties "defaultDefaultClientScopes" and "defaultOptionalClientScopes" are ignored.
I found this out because I wanted to remove the default "web-origins" client scope as it substantially increased the size of my access token size which went over the chromes 4kb cookie size limit.
To Reproduce

Create test.json import file

{
  "id": "test",
  "realm": "test",
  "defaultDefaultClientScopes": ["email"],
  "defaultOptionalClientScopes": ["address"]
}

Spin up fresh keycloak

docker run --rm \
    -p 8080:8080 \
    -e KEYCLOAK_USER=admin \
    -e KEYCLOAK_PASSWORD=admin \
    quay.io/keycloak/keycloak:13.0.1

Run keycloak-config-cli

docker run --rm \
    --network="host" \
    -v ${PWD}:/config \
    -e IMPORT_PATH=/config/test.json \
    -e KEYCLOAK_URL=http://localhost:8080/auth \
    -e KEYCLOAK_USER=admin \
    -e KEYCLOAK_PASSWORD=admin \
    adorsys/keycloak-config-cli:latest-13.0.1

Observe the Default Client Scopes in Keycloak

Expected behavior
I expect the Assigned Default Client Scopes to be only set to "email" and nothing less
Environment:

Keycloak Version: 13.0.1
keycloak-config-cli Version: 13.0.1
Java Version: 11
Docker Tag: latest-13.0.1

Additional context
This is due to https://issues.redhat.com/browse/KEYCLOAK-13951. The HTTP requests for getting and updating realms ignores the defaultDefaultClientScopes in the RealmRepresentation. This is also an issue with the client org.keycloak.admin.client.Keycloak.
This could be worked around by manually adding/removing the default client scopes when updating a realm. The org.keycloak.admin.client.resource.RealmResource already has the methods to do it:
@GET
@Produces(MediaType.APPLICATION_JSON)
@Path("default-default-client-scopes")
List<ClientScopeRepresentation> getDefaultDefaultClientScopes();

@PUT
@Path("default-default-client-scopes/{clientScopeId}")
void addDefaultDefaultClientScope(@PathParam("clientScopeId") String clientScopeId);

@DELETE
@Path("default-default-client-scopes/{clientScopeId}")
void removeDefaultDefaultClientScope(@PathParam("clientScopeId") String clientScopeId);

I may add a lot of test case here.
For example. If an existing client has defaultClientScopes: group and keycloak-config-cli tries to add defaultDefaultClientScopes: group, then this be result in a 409 Conflict and the keycloak log reports a SQL violation ...


DataTable width doesn't seem to work if empty:true

hello,
setting the DataTable width in the headers doesn't seem to work if the empty property is set to true.
On version 0.64.1
A workaround is by using slotted cells

Thanks for reporting. Fixed in v0.64.2.


Exception: "x not finite"

Description (Required)

Recently, the console keeps throwing this exception (x not finite).
All information I can provide is the server console, because no players have noticed this.
At the error log, it said something about BarbedWire, I will soon give more details.
If you close this issue, please let me know if I can have chance to re-open this issue or contact you.
I don't know whether this affect gameplay or not, still waiting on detail reports from players.
Thank you in advance.
Steps to reproduce the Issue (Required)

Working on it.
Expected behavior (Required)

Console should not throw exception.
Working on it.
Server Log / Error Report
https://pastebin.com/hM1xPu0G

Environment (Required)

Minecraft Version: Paper v1_16_R2
CS-CoreLib Version: vDEV - 92 (git e9150d72)
Slimefun Version: vDEV - 774 (git e66af680)
Plugin Version: vDEV - 12 (git fc82651c)

Fixed, forgot to check if they values were finite. Thank you for the bug report :bow:

Fixed, forgot to check if they values were finite. Thank you for the bug report :bow:


支持实时数据写入和查询

增加kafka bs reader插件，支持实时数据写入和查询

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Combine key and cert fails with wildcard certificates

If  ssl_certs_common_name contains an asterisk for getting a wildcard certificate (like "*.mydomain.com") the step "Combine key and cert" will fail because the wildcard in the regex isn't properly escaped.

Here is a pull request
https://github.com/jdauphant/ansible-role-ssl-certs/pull/41

Thanks ;)


Removed unused local variable in PerfTrace for gfx6 and gfx9

NumShaderEngines does not seem to be used locally.

Can one of the admins verify this patch?

ok to test

test this please

Test results for commit 1666d3ef23e702e0975e72922485d3ad4fa391aa:Test run totals:Passed:        183806/340842 (53.9%)Failed:        4/340842 (0.0%)Not supported: 157031/340842 (46.1%)Warnings:      1/340842 (0.0%)Expand for test failuresFAILURE: dEQP-VK.memory.pipeline_barrier.transfer_src_transfer_dst.1024Stack trace: 14:BufferCopyToImage Result differs from reference, Expected: 0x0000003b, Got: 0x0000000d, At offset: 0FAILURE: dEQP-VK.memory.pipeline_barrier.transfer_src_transfer_dst.8192Stack trace: 14:BufferCopyToImage Result differs from reference, Expected: 0x0000003b, Got: 0x0000000d, At offset: 0FAILURE: dEQP-VK.memory.pipeline_barrier.transfer_src_transfer_dst.65536Stack trace: 10:BufferCopyToBuffer Result differs from reference, Expected: 0x00000014, Got: 0x00000059, At offset: 24384FAILURE: dEQP-VK.memory.pipeline_barrier.all_device.65536Stack trace: 5:SubmitRenderPass Image comparison failed

retest this please


Amend landing page words & respond to points raised by comms

Pre-reqs:

[x] Follow up with Comms @ YFF to ensure we have a list of their points
[ ] Ensure we have access to updated wording for home page which Comms has been working on.

Other tickets may fall out of this.

Emailed comms - waiting for feedback now. Moved it back to backlog

New landing page text:
Youth Futures Foundation is the What Works Centre for youth employment.
Our aim is to reduce gaps in data and research to improve employment outcomes for young people facing the greatest challenges. The Data Dashboard presents the latest youth labour market data and our latest research.
Please could the first sentence be in bold and a slightly larger font size?

Added text, picked up points about bold in a separate ticket #156