createRule message no longer works with $params

I just upgraded to 0.4.0 from 0.2.11 and noticed that there have been some breaking changes but I can't figure out exactly how to make this work. My custom rule message is now throwing errors and I'm not sure about the proper way to write this now.
Reproduction:
https://github.com/martinszeltins/regle-testing
Previously I had this rule:
const minWeightRule = createRule({
    validator(value: Maybe<number | string>, min: number, condition: boolean) {
        return Number(value) >= min && condition === false
    },

    message: (_, { $params: [min, condition] }) => {
        return t('min_weight_message', { min: min, condition: condition.toString() })
    }
})

But now that I upgraded to 0.4.0 I am getting errors:
Cannot destructure property '$params' of 'undefined' as it is undefined.

Type '(_: any, { $params: [min, condition] }: { $params: [any, any]; }) => string' is not assignable to type '(metadata: RegleRuleMetadataConsumer<string | number, [min: number, condition: boolean], boolean>) => string | string[]'.
    Target signature provides too few arguments. Expected 2 or more, but got 1.ts-plugin(2322)

I tried reworking it like this but it also did not work:
const minWeightRule = createRule({
    validator(value: Maybe<number | string>, min: number, condition: boolean) {
        return Number(value) >= min && condition === false
    },

    message: ({ min, condition }) => {
        return t('min_weight_message', { min: min, condition: condition.toString() })
    }
})

Hey!
Yep sorry 0.4.0 introduced a breaking change into the message declaration!
https://github.com/victorgarciaesgi/regle/releases/tag/v0.4.0
you can rewrite it like this, this first parameter have just been moved alongside $params.
const minWeightRule = createRule({
  validator(value: Maybe<number | string>, min: number, condition: boolean) {
    return Number(value) >= min && condition === false;
  },

  message: ({ $params: [min, condition] }) => {
    return t('min_weight_message', { min: min, condition: condition.toString() });
  },
});

Now if your want the value, you can just do
message: ({ $value }) => {
    return $value
  },

It was better as it's not used a lot of times and forced to declare a first parameter

Thanks! 👍🏻


Automated Package Upgrade

:tada: This PR is included in version 7.7.259 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


error prints “do hw ciphers failed” when 'openssl speed -multi 80'

On a machine with 96 cores, when running openssl speed -multi 80 test, it creates 80 processes to do the encryption in parallel. During the test, a lot of error prints “do hw ciphers failed”.
Expected: There should be no error prints in screen. If the hw ciphers fails, the uadk_provider should automatically switch to the default provider (software implementations).
Reproduce:
openssl speed -provider uadk_provider -provider default -multi 80 -seconds 20 -elapsed -evp sm4-ctr

The /var/log/uadk.log generated by this command is more than 132MB.
It cannot be uploaded. :(
Anyway, it's easy to reproduce the issue, as it's only one command. See above.

need enable  sw via cnf


Cannot manually start a mapping pass, and rest call returns "Cannot PUT /api/v2/robot/capabilities/MappingPassCapability" with Conga 5090

Hi,
I have a setup with CongaTudo 2022.2 enabled via HomeAssistant addon, and a Conga 5090 connected to it.
I can't invoke a mapping pass "cleaning". Valetude UI doesn't offer such option, and, when I try to invoke it by a rest API call, I obtaing a "Cannot PUT /api/v2/robot/capabilities/MappingPassCapability" message:
CALL:
curl -X 'PUT'
'https://mySuperHouse/api/hassio_ingress/541jCKfsFvX0qfofDn2IBt9rTDbPit8jLLRoGD4z6AY/api/v2/robot/capabilities/MappingPassCapability'
-H 'accept: /'
-H 'Content-Type: application/json'
-d '{
"action": "start_mapping"
}'
RESPONSE:

Error

Cannot PUT /api/v2/robot/capabilities/MappingPassCapability

Vacuum Model
Conga 5090
Valetudo Version
2022.2.0

Please, try the beta addon.
If you still have problems, open a issue in Congatudo repository: https://github.com/congatudo/Congatudo


fix: insert whitespaces into assoc items for assist when macro generated

This is obviously only a temporary hack which still produces ugly looking items, but as long as the syntax is valid one can at least have rustfmt fix the formatting again.
Fixes https://github.com/rust-analyzer/rust-analyzer/issues/6588
bors r+

bors retry


GO annotation import pipeline

Aim: Import high confidence GO annotations (GO term to gene) for GO terms directly linked to CL terms - bidirectional by and object property restriction.
The links from GO terms to Genes (proteins) should follow standard GO semantics.  There are 3 possible sources:

QuickGO API:- Python script to generate ROBOT template --> RDF
AMIGO API: - Python script to generate ROBOT template --> RDF https://amigo.geneontology.org/amigo (Ask Chris/Seth about API)
OAK apparently can generate an RDF file with GO annotations for any set of GO terms.  This may be ideal if the library is robust enough and this scales well. https://incatools.github.io/ontology-access-kit/guide/associations.html

Filter on species (mouse + human) + direct experimental evidence ECO:0000269
Example of GO annotation query with filters through QuickGO GUI:
https://www.ebi.ac.uk/QuickGO/annotations?goUsage=descendants&goUsageRelationships=is_a,part_of,occurs_in&goId=GO:0097208&taxonId=9606&taxonUsage=descendants&evidenceCode=ECO:0000269&evidenceCodeUsage=descendants
https://www.ebi.ac.uk/QuickGO/api/index.html#!/annotations/downloadLookupUsingGET

This requires finding direct links GO -> CL.  The simplest way to get these would be from UberGraph queries

non-redundant graph simple triples

Hi @dosumis,
I've made some initial progress on this issue. I retrieved GO annotations from CL_KG using the following query:
MATCH (c:Cell)-[]-(g)
WHERE g.curie STARTS WITH 'GO:'
RETURN g.curie

I then queried the QuickGO annotation service with all the GO terms, except for GO:0030154 and GO:0005622 (which caused a timeout error that I'll investigate further).
Here’s the preview of the resulting table. How should we proceed from here?

Click to expand table

GO Name
Gene Product DB
Gene Product ID
Symbol
Qualifier
GO Term
GO Evidence Code
Reference
With/From
Taxon ID
Assigned By
Gene Product Name

retinal cone cell apoptotic process
UniProtKB
P37242
Thrb
acts_upstream_of_or_within
GO:0097474
IMP
PMID:20203194

10090
MGI
Thyroid hormone receptor beta

retinal cone cell apoptotic process
UniProtKB
Q91ZI8
Dio3
acts_upstream_of_or_within
GO:0097474
IMP
PMID:20203194

10090
MGI
Thyroxine 5-deiodinase

gamma-delta T cell differentiation
UniProtKB
A7TZE6
Skint1
acts_upstream_of
GO:0042492
IDA
PMID:21300860

10090
MGI
Selection and upkeep of intraepithelial T-cells protein 1

gamma-delta T cell lineage commitment
UniProtKB
B3F5L4
Scart2
acts_upstream_of_or_within
GO:0002365
IDA
PMID:18641307

10090
MGI
SRCR domain-containing protein

gamma-delta T cell lineage commitment
UniProtKB
B3F5L5
Scart2
acts_upstream_of_or_within
GO:0002365
IDA
PMID:18641307

10090
MGI
SRCR domain-containing protein

gamma-delta T cell differentiation
UniProtKB
P06240
Lck
acts_upstream_of
GO:0042492
IMP
PMID:7807014
MGI:MGI:1857211
10090
MGI
Proto-oncogene tyrosine-protein kinase LCK

gamma-delta T cell differentiation
UniProtKB
P06800
Ptprc
acts_upstream_of
GO:0042492
IMP
PMID:7807014
MGI:MGI:2181288
10090
MGI
Receptor-type tyrosine-protein phosphatase C

CD8-positive, gamma-delta intraepithelial T cell differentiation
UniProtKB
P26715
KLRC1
involved_in
GO:0002305
IDA
PMID:18064301

9606
UniProt
NKG2-A/NKG2-B type II integral membrane protein

gamma-delta T cell differentiation
UniProtKB
P42230
Stat5a
acts_upstream_of
GO:0042492
IGI
PMID:15294943
MGI:MGI:103035
10090
MGI
Signal transducer and activator of transcription 5A

gamma-delta T cell differentiation
UniProtKB
P42232
Stat5b
acts_upstream_of
GO:0042492
IGI
PMID:15294943
MGI:MGI:103036
10090
MGI
Signal transducer and activator of transcription 5B

gamma-delta T cell differentiation
UniProtKB
P48025
Syk
acts_upstream_of
GO:0042492
IMP
PMID:7477352
MGI:MGI:2384078
10090
MGI
Tyrosine-protein kinase SYK

gamma-delta T cell differentiation
UniProtKB
P48025
Syk
acts_upstream_of
GO:0042492
IMP
PMID:8790395
MGI:MGI:1857421
10090
MGI
Tyrosine-protein kinase SYK

gamma-delta T cell activation
UniProtKB
P97792
Cxadr
involved_in
GO:0046629
IDA
PMID:20813954

10090
UniProt
Coxsackievirus and adenovirus receptor homolog

gamma-delta T cell differentiation
UniProtKB
Q00417
Tcf7
involved_in
GO:0042492
IMP
PMID:30413363

10090
UniProt
Transcription factor 7

gamma-delta T cell activation
UniProtKB
Q03526
Itk
involved_in
GO:0046629
IMP
PMID:23562159

10090
UniProt
Tyrosine-protein kinase ITK/TSK

gamma-delta T cell differentiation
UniProtKB
Q04891
Sox13
involved_in
GO:0042492
IMP
PMID:17218525

10090
UniProt
Transcription factor SOX-13

gamma-delta T cell differentiation
UniProtKB
Q04891
Sox13
involved_in
GO:0042492
IMP
PMID:30413363

10090
UniProt
Transcription factor SOX-13

gamma-delta T cell activation
UniProtKB
Q29980
MICB
involved_in
GO:0046629
IDA
PMID:9497295

9606
UniProt
MHC class I polypeptide-related sequence B

gamma-delta T cell activation
UniProtKB
Q29983
MICA
involved_in
GO:0046629
IDA
PMID:9497295

9606
UniProt
MHC class I polypeptide-related sequence A

gamma-delta T cell proliferation
UniProtKB
Q3TSG4
Alkbh5
acts_upstream_of
GO:0046630
IMP
PMID:35939687

10090
UniProt
RNA demethylase ALKBH5

gamma-delta T cell activation
UniProtKB
Q80UL9
Jaml
involved_in
GO:0046629
IDA
PMID:20813954

10090
UniProt
Junctional adhesion molecule-like

gamma-delta T cell lineage commitment
UniProtKB
Q8C9T4
Scart2
acts_upstream_of_or_within
GO:0002365
IDA
PMID:18641307

10090
MGI
SRCR domain-containing protein

CD8-positive, gamma-delta intraepithelial T cell differentiation
UniProtKB
Q8K1Z6
Gpr18
acts_upstream_of_or_within
GO:0002305
IDA
PMID:25348153

10090
MGI
N-arachidonyl glycine receptor

CD8-positive, gamma-delta intraepithelial T cell differentiation
UniProtKB
Q8K1Z6
Gpr18
acts_upstream_of_or_within
GO:0002305
IMP
PMID:25348153
MGI:MGI:5708707
10090
MGI
N-arachidonyl glycine receptor

gamma-delta T cell differentiation
UniProtKB
Q9QYE5
Jag2
involved_in
GO:0042492
IMP
PMID:10383933

10090
UniProt
Protein jagged-2

CD8-positive, gamma-delta intraepithelial T cell differentiation
UniProtKB
Q9WUT7
Ccr9
acts_upstream_of_or_within
GO:0002305
IMP
PMID:25348153
MGI:MGI:3654108
10090
MGI
C-C chemokine receptor type 9

erythrocyte maturation
UniProtKB
A8VU90
Ankle1
NOT
involved_in
GO:0043249
IMP
PMID:27010503

10090
UniProt

erythrocyte development
UniProtKB
F8VPQ2
Arid4a
acts_upstream_of_or_within
GO:0048821
IMP
PMID:18728284
MGI:MGI:3687004
10090
MGI
AT-rich interactive domain-containing protein 4A

erythrocyte development
UniProtKB
G5E8Q8
Adgrf5
acts_up

Converting QuickGO ouput to RDF

GO NAME
GENE PRODUCT DB
GENE PRODUCT ID
SYMBOL
QUALIFIER
GO TERM
GO EVIDENCE CODE
REFERENCE
WITH/FROM
TAXON ID
ASSIGNED BY
GENE_PRODUCT_NAME

retinal cone cell apoptotic process
UniProtKB
P37242
Thrb
acts_upstream_of_or_within
GO:0097474
IMP
PMID:20203194

10090
MGI
Thyroid hormone receptor beta

(GO:Class)-[:<QUALIFIER> { has_reference: '<REFERENCE>', evidence: '<GO EVIDENCE CODE>'  }]->(protein:Class { label: '<GENE PRODUCT NAME>', symbol: '<SYMBOL>')-[]->(Species:Class}

## iri/curie mappings:
annotation properties --> literals
label: rdfs:label
symbol: http://purl.obolibrary.org/obo/IAO_0000028
has_reference: http://purl.org/dc/terms/references
Classes
Protein: https://identifiers.org/uniprot:
e.g. https://identifiers.org/uniprot:P0DP23
Species: http://purl.obolibrary.org/obo/
e.g.
10090
-->
http://purl.obolibrary.org/obo/NCBITaxon_10090


Queue Creation and subscription on startup rather than at first request?

Is your feature request related to a problem? Please describe.
We came across an issue recently within one of our projects where the first request received after the startup of the box would be significantly slower than any other requests, but only when messaging was involved in the request.
We found that most of the response time was taken up by queue creation and subscription, even to messages that weren't involved in that request. After that, all is well 😄we've tried looking at our implementation to see if there's any way to avoid this, but from looking at other implementations, we found that we weren't doing it too differently from anyone else.
Describe the solution you'd like
It would be cool if we could try to invoke this initialisation process sooner, rather than at the point of first request if possible.
Describe alternatives you've considered
There might also be the possibility of only dealing with necessary queues at the above point rather than all at once? From looking at the common implementation I understand this might be a longer-term change.
Additional context

When you say first request, are you talking about publishing? There is a library within Just Eat that wraps JustSaying, this library defers some work until the first published message, it may be this you are seeing rather than a behaviour of JustSaying.

@slang25 yes, this the first SNS/SQS publish, in an API that receives a http request, and publishes a message about it.


Does this work on multi-process environements?

If my apps runs on multiple process,
does e.g. an insert on one process trigger the hook on the other instance as well?

Nope, this only runs in the process which do the insert.
If you need things to happen on all meteor instances whenever something is inserted/updated/removed I suggest doing a find on the dataset and observe the cursor with onAdded etc. and trigger the actions then.

There might be other solutions also.

Closing this as answered, please correct me if I'm wrong.

Maybe some information for the archive ;-)
We used a lot of db observers on one of our projects (where this question here was relevant), but we noticed that it affects the memory consumption and performance of the db and the server if you have a lot of observers and the observed set is large.
We therefore using now more hooks then observers.
In general, there isn't a clear answer to what is the better solution, both solutions have down- and upsides.


switch to zerocopy websocket

As it's tiny and simple and faster than gorilla implementation
https://github.com/gobwas/ws

I Tested it and I think we will have to rewrite a lot of thing for almost no gain. Closing it for now will re-open it if needed.


Delete lab environments that are no longer used

User Story
Delete the following environments and their relevant VPC's

sprinkler production

bench production

heater development

heater production

cooker production

Value
These accounts were created to aid with testing when building the platform, but their names are confusing and they will be replaced by the sandbox, testing and example accounts detailed here: #877 #878 #879
This will also be a good exercise in developing and documenting the process for deleting accounts, which we have not tried yet.
We will keep development environments for sprinkler, bench and cooker for the moment until we have created a test account
Questions / Assumptions

Definition of done

[ ] accounts and vpcs have been deleted
[ ] readme has been updated
[ ] user docs have been updated
[ ] another team member has reviewed
[ ] tests are green

Reference
How to write good user stories

Terraform used to remove resources from below accounts.
Heater - development

This work is now complete, all specified accounts have been deleted


Styling issues with Bootstrap v3.7

I recently upgraded date-time-picker to v6.0.15 and I'm getting a weird styling conflict with Bootstrap v3.7.

As you can see, the picker is much smaller than it's default and this is only happening once I add Boostrap to angular.json.
"styles": [
  "node_modules/bootstrap/dist/css/bootstrap.min.css",
  "node_modules/ng-pick-datetime/assets/style/picker.min.css",
  "src/styles.css"
],

Has anyone else run into this issue with a possible solution?
The picker displays normally without Bootstrap.

this is because the picker is designed responsive. And when you and the bootstrap, the bootstrap change your font size of your web.
Change the font size of the picker could solve the problem.

Thanks @DanielYKPan. In case anyone else stumbles upon this the solution is as follows:
::ng-deep .owl-dt-container {
  font-size: 1.5rem;
}

Hey @atom-morgan  , Are you facing this type of issue ?

I'm facing this issue. Can you help to fixed this issue ?


OverflowError manual.py on Rpi

When trying to run manual.py on a video larger then 2GB I get the following error:
OverflowError: Python int too large to convert to C ssize_t
This does not occur when a file has a different extension like .mkv or .avi.
Trackback:
root@rpi:/mp4automator# python3 manual.py -i /downloads/Jupiters_moon/
Manual processor started.
Python 32-bit 3.6.9 (default, Nov  7 2019, 10:44:02)
[GCC 8.3.0].
Guessit version: 3.1.0.
/usr/bin/python3
Loading config file /mp4automator/config/autoProcess.ini.
Processing file /downloads/Jupiters_moon/Jupites_moon.mp4
Unable to retrieve rating.
Matched movie title as: Jupiter's Moon 2017-11-01 (TMDB ID: 444395)
Proceed using guessed identification from filename?
# [y/n]: y
Processing Jupiter's Moon
Input and output extensions are the same so passing back the original file [process-same-extensions: False].
Bypassing conversion and setting outputfile to inputfile.
Tagging file: /downloads/Jupiters_moon/Jupites_moon.mp4.
Trying to write tags.
There was an error writing the tags.
Traceback (most recent call last):
  File "/mp4automator/resources/metadata.py", line 205, in writeTags
    video.save()
  File "/usr/local/lib/python3.6/dist-packages/mutagen/mp4/__init__.py", line 1099, in save
    super(MP4, self).save(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/mutagen/_util.py", line 140, in wrapper
    return func(self, h, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/mutagen/_file.py", line 133, in save
    return self.tags.save(filething, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/mutagen/_util.py", line 169, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/mutagen/_util.py", line 140, in wrapper
    return func(self, h, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/mutagen/mp4/__init__.py", line 425, in save
    self.__save(filething.fileobj, atoms, data, padding)
  File "/usr/local/lib/python3.6/dist-packages/mutagen/mp4/__init__.py", line 433, in __save
    self.__save_existing(fileobj, atoms, path, data, padding)
  File "/usr/local/lib/python3.6/dist-packages/mutagen/mp4/__init__.py", line 493, in __save_existing
    resize_bytes(fileobj, length, len(ilst_data), offset)
  File "/usr/local/lib/python3.6/dist-packages/mutagen/_util.py", line 916, in resize_bytes
    insert_bytes(fobj, insert_size, insert_at)
  File "/usr/local/lib/python3.6/dist-packages/mutagen/_util.py", line 853, in insert_bytes
    mmap_move(fobj, offset + size, offset, movesize)
  File "/usr/local/lib/python3.6/dist-packages/mutagen/_util.py", line 737, in mmap_move
    file_map = mmap.mmap(fileno, length - offset, offset=offset)
OverflowError: Python int too large to convert to C ssize_t
Relocating MOOV atom to start of file.
QT FastStart did not run - perhaps moov atom was at the start already or file is in the wrong format.

This is a mutagen error due to your sys.maxsize being too low, I would post this over here
https://github.com/quodlibet/mutagen
Not something I have the power to fix


sqlc not working with datatype UUID

Version
1.21.0
What happened?
Since MariaDB 10.7 there is a native UUID data type. Using this data type in the schema definition gives an error when generating code:
sqlc generate failed.
# package db
query.sql:2:9: syntax error near "uuid primary key,"

Workaround
When I change the datatype in the schema.sql from uuid to binary and add an override in the sqlc.yaml the generation works as expected.
updated schema.sql:
create table user (
	id binary default uuid() not null primary key,
	name varchar(50) not null,
	email varchar(255) not null,
	password_hash varchar(60) not null,
	created_datetime timestamp not null default current_timestamp(),
	updated_datetime timestamp not null default current_timestamp() on update current_timestamp()
);

updated sqlc.yaml:
overrides:
  go:
    overrides:
      - db_type: "binary"
        nullable: false
        engine: "mysql"
        go_type:
          import: "github.com/google/uuid"
          type: "UUID"

However, changing the data type in the schema.sql is not desired because it does not reflect reality.

Database schema
create table user (
	id uuid default uuid() not null primary key,
	name varchar(50) not null,
	email varchar(255) not null,
	password_hash varchar(60) not null,
	created_datetime timestamp not null default current_timestamp(),
	updated_datetime timestamp not null default current_timestamp() on update current_timestamp()
);

SQL queries
-- name: listUsers :many
select * from user;

Configuration
{
  "version": "1",
  "packages": [
    {
      "path": "db",
      "engine": "mysql",
      "schema": "schema.sql",
      "queries": "queries.sql"
    }
  ]
}

Playground URL
https://play.sqlc.dev/p/e7301e06df354e56281efe68b535030621c6432705b11e54f542b1e322241858
What operating system are you using?
macOS
What database engines are you using?
MySQL
What type of code are you generating?
Go

we had another issue with uuid on this version.
sqlc is trying to add the import github.com/google/uuid on the generated file querier_sqlc.go but this file does not use it directly - it just define the Querier interface and the go vet goes crazy.
we will try to write a minimal example that trigger this issue

@peczenyj the Querier imports issue was resolved in https://github.com/sqlc-dev/sqlc/pull/2709.
Regarding sqlc support for MariaDB's custom UUID type, I don't think we can add that without introducing a new engine. There may be a point at which there are enough differences between MySQL and MariaDB where we want to introduce a new engine type, and maybe that point is already here. I'm not an expert so can't really say, but this is probably a conversation better handled in a GitHub discussion.

@peczenyj the Querier imports issue was resolved in https://github.com/sqlc-dev/sqlc/pull/2709.
Regarding sqlc support for MariaDB's custom UUID type, I don't think we can add that without introducing a new engine. There may be a point at which there are enough differences between MySQL and MariaDB where we want to introduce a new engine type, and maybe that point is already here. I'm not an expert so can't really say, but this is probably a conversation better handled in a GitHub discussion.


enhance: delete output in olapscan toString()

Proposed changes
Issue Number: close #xxx
Problem summary
delete output in olapscan toString()
Checklist(Required)

[x] Does it affect the original behavior
[ ] Has unit tests been added
[ ] Has document been added or modified
[ ] Does it need to update dependencies
[x] Is this PR support rollback (If NO, please explain WHY)

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall


Add chebyshev Iteration

It adds the Chebshev iteration in https://en.wikipedia.org/wiki/Chebyshev_iteration
The second-order richardson uses a similar formula but the scalars are constant in all iteration. Chebyshev Iteration update the scalar from the previous one (the scalar are the same if upper/lower eigval does not change)

Do you intend adding an eigenvalue estimation? I think that would be very helpful, because most times users don't have that available. I think PETSc is also doing that.
I briefly searched for that and here are some references that might be interesting for that:

https://doi.org/10.1137/0907057
https://doi.org/10.1137/0613050
https://doi.org/10.1007/BF01397475

@MarcelKoch thanks for providing these reference.
I do not think I will put them in this pull request.
From https://doi.org/10.1137/0907057, they introduce the algorithm
adaptive chebyshev iteration
Perform m-step GMRES and use the information from the Hessenberg matrix to get the estimation of eigenvalue (I may be wrong),
and then perform https://link.springer.com/article/10.1007/BF01389971 to get the parameter for chebchev
Repeat these two steps until converge
This will be more standalone solver not smoother for multigrid because it will require several steps on GMRES, which is too expensive IMO.
From https://petsc.org/release/docs/manualpages/KSP/KSPCHEBYSHEV/#kspchebyshev,
More precisely, https://petsc.org/release/docs/manualpages/KSP/KSPChebyshevEstEigSet/
They use Lanczo or GMRES to get the maximum estimation to decide the parameter by following
min-eig-boundary = a * min-eigest + b * max-eigest  = 0.1 max-eigest (default), and
max-eig-boundary = c * min-eigest + d * max-eigest = 1.1 max-eigest (default)
(because min_eigest is usually inaccurate)
It only did once in generation, so this is more usable for Multigrid
For GMRES, we need to get the Hessenberg out and compute eigenvalue on it.
GMRES and Lanczo are related to #135

@upsj To avoid the mutable and reusing value from workspace, I think moving them into generatio such that prefill all alpha and beta before apply. I think it will also give better performance because we do not need to do some tiny fill before the limitation during apply.
Also, refill it when the foci or criterion change (or specfic factor for that).

If those are just static coefficients independent of the matrix, that sounds very reasonable. I tend to agree with what I believe @MarcelKoch said, which is that we shouldn't consider changeable stopping criteria here. We also currently don't seem to have a way to propagate information from other parts of the solver hierarchy to compute appropriate focus values.

It should cover the eigenvalye range from the matrix itself. Without estimation from our side, the value is given by user. They should be changed together with matrices. Something like L1 on SPD matrix can give the value based on the property without knowing the detail of matrices. I will come back for this after the release because it is not the urgent case.


[BEAM-9002] Add test_flatten_same_pcollections to fnapi runner

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Retest this please

R: @aaltay @tvalentyn

LGTM


Install BookStack on Synology NAS

Attempted Debugging

[X] I have read the debugging page

Searched GitHub Issues

[X] I have searched GitHub for the issue.

Describe the Scenario
Hi
I am trying to create a backup install of my main BookStack install on a Synology DS414j NAS. I have two real questions:

How to install BookStack on the NAS

How to 'mirror' my server install to the NAS so they are synchronised

Question 1: I have updated the NAS to the latest allowed DSM version (DSM 7.1.1-42962 Update 5), and updated Web Station. I have also installed PHP 8.0. However, all the tutorials I have found say to install Docker or Portainer to get a working BookStack install.
My problem is that neither package are available (as far as I can see), so I can't even set up a container to install BookStack to! Has anyone recently managed to create a BookStack install on this NAS?
Question 2: Regardless of whether I get the NAs install working, what is the best way of synchronising two installs? Cani I rsync the install from one device to another? Can I restore an sql dump and restore the images, etc? I am mindful of the fact that the .env settings for the db may be different. I am no expert on this stuff, so would welcome some pointers.
Mark
Exact BookStack Version
Latest version
Log Content
No response
PHP Version
8.x
Hosting Environment
Server - Ubuntu 22.04LTS
NAS....?

Hi @techauthoruk

My problem is that neither package are available (as far as I can see), so I can't even set up a container to install BookStack to!

Yeah, the lower value range of Synology boxes (Which I think are those with j at the end of the model name) won't have certain packages due to system limitations.

Has anyone recently managed to create a BookStack install on this NAS?

It likely is possible to get it running, but it might be messy to do outside of docker due to the requirements needed. It's not really what a Synology NAS is suited for tbh.

Regardless of whether I get the NAs install working, what is the best way of synchronising two installs? Cani I rsync the install from one device to another?

What is your actual goal in this? There are methods but no easy way for a bi-directional sync.
For a single-direction sync you can indeed dump the database and copy the files over.
Our backup and restore docs detail the mechanics of this.
As an easier potential option, The latest release includes a system CLI that allows potentially easier backup/restore. It's in early alpha and can still be subject to bugs/issues, but it automates and standardises a lot of the process in the docs linked above.
I'm not confident it'd run without some pains on a Synology system though.

Personally, through experience I've found it's more hassle than it's worth to stretch the use-case of something like a Synology box, which ideally you'd want to remain reliable in it's primary purpose. You'd be going against the grain and there'd be little existing guidance/docs/experience to help when you run into issues (as you're finding now).
I have a separate little intel NUC box to run apps, running proxmox to separate/compartmentalise things via VMs or LXC containers. I have a little overview of my setup here.
If you just need to have a backup of your system though, you could maybe just look to use the new BookStack system CLI to create backups then store the resulting ZIP on your NAS like any other file.

@ssddanbrown thank you for all your comments.
I have abandoned the idea of using the Synology now - there is just no reliable way to get Docker installed. My goal was actually to have a replica of my main bookstack instance installed elsewhere for reference. The sync would only ever be one way (main instance to backup).
The reason behind this - I keep all my notes and instructions for the management of my software apps in bookstack. My server went down a few days ago, and of course I couldn't access my notes to do reinstalls, etc! I wanted a backup on another machine/device so that I could see my notes while I sorted the issues. I do routinely back up the bookstack instance in line with the comments on your site, so it's not a problem restoring bookstack if needed.
I tried this morning restoring the sql backup from my main instance into a bookstack install on another machine, but this didn't work - I get a 419 error when trying to log in. I'm going to play around a little more to see if I can find out why.
Thank you again for your comments.

OK, so resolved now.
My 419 issues seemed to be due to an incorrect database pasword. Changed the password, and everything is working as it should, so closing this.


Training for a large dataset, Java heap space memory out

I trained with Text8 from Eclipse, worked OK. But when I loaded a 2.1 GB text file, OutOfMemoryError occurred even I increased the Eclipse memory to maximum. Any ideas about this?
Thanks.
Pengchu

Hi @pzhanggithub !
Could you tell us, which version of Word2VecJava you used? Do you use the stable 0.9.0 or any development version from this repository?
Additionally, how did you "load" the text file? Can you provide some source code? The training call would be interesting, too.

Thanks for this quick response.
The version I used is what I downloaded from the port a couple of days ago, it is the only version. I Mavened it to Eclipse on my Mac. It passed all tests as you provided. And the tasks of reading, training and interacting were all successfully down.
For test, I used the class of word2vecExamples with any change but the input file name:
File f = new File("/Users/pzhang/Documents/SandReportData/sandreportall/sand25years.txt");
if (!f.exists())
throw new IllegalStateException("Please download and unzip the text8 example from http://mattmahoney.net/dc/text8.zip");
List read = Common.readToList(f);
List<List> partitioned = Lists.transform(read, new Function<String, List>() {
@Override
public List apply(String input) {
return Arrays.asList(input.split(" "));
}
});
	Word2VecModel model = Word2VecModel.trainer()
			.setMinVocabFrequency(5)
			.useNumThreads(20)
			.setWindowSize(8)
			.type(NeuralNetworkType.CBOW)
			.setLayerSize(200)
			.useNegativeSamples(25)
			.setDownSamplingRate(1e-4)
			.setNumIterations(5)
			.setListener(new TrainingProgressListener() {
				@Override public void update(Stage stage, double progress) {
					System.out.println(String.format("%s is %.2f%% complete", Format.formatEnum(stage), progress * 100));
				}
			})
			.train(partitioned);

The error occurs at:
List read = Common.readToList(f);
Thanks.
Pengchu

The file was extracted from report repository and converted into a text file. The same file was used with word2vec.c (by Google) successfully.

Can you try running this with the changes in https://github.com/medallia/Word2VecJava/pull/23?

I try to train on corpus ~2,19 GB. I launch train with next code:
Word2VecModel model =
            Word2VecModel.trainer()
                .setMinVocabFrequency(100)
                .useNumThreads(1)
                .setWindowSize(8)
                .type(NeuralNetworkType.CBOW)
                .setLayerSize(200)
                .useNegativeSamples(25)
                .setDownSamplingRate(1e-4)
                .setNumIterations(5)
                .useHierarchicalSoftmax()
                .setListener(
                    new Word2VecTrainerBuilder.TrainingProgressListener() {
                        @Override
                        public void update(Stage stage, double progress) {
                            logger.info(
                                String.format(
                                    "%s is %.2f%% complete",
                                    Format.formatEnum(stage),
                                    progress * 100
                                )
                            );
                        }
                    }
                )
                .train(new CustomIterable(path));

Code for CustomIterable:
private class CustomIterable implements Iterable<List<String>> {
        private final Path filePath;

        public CustomIterable(Path filePathArg) {
            filePath = filePathArg;
        }

        public Iterator<List<String>> iterator() {
            return new CustomIterator(filePath);
        }
    }

Code for CustomIterator:
    private class CustomIterator implements Iterator<List<String>> {
        private final Path filePath;

        private BufferedReader bufferedReader;

        public CustomIterator(Path filePathArg) {
            filePath = filePathArg;

            try {
                bufferedReader = Files.newBufferedReader(filePath, StandardCharsets.UTF_8);
            } catch (IOException ignored) {
            }
        }

        @Override
        public boolean hasNext() {
            try {
                return bufferedReader.ready();
            } catch (IOException ignored) {
            }
            return false;
        }

        @Override
        public List<String> next() {
            try {
                String line;
                if ((line = bufferedReader.readLine()) != null) {
                    String[] fields = line.split("\\t");
                    String[] words = fields[1].split(" ");
                    return Arrays.asList(words);
                }
            } catch (IOException e) {
                return null;
            }
            return null;
        }

        @Override
        public void remove() {

        }
    }

Also I give to virtual machine -Xmx14g, I use this revision c412fde19c2e63c1f774e5804819273d0853c796
I got OutOfMemoryError.

Tested with fix from #23 and looks like it's work for me. Thank you!


[Feat] Provide libraries for other languages

Is your feature request related to a problem? Please describe.
Currently the libraries for GraphAr are only available for C++ and Spark. But many graph processing systems are implemented by other programming languages (like Neo4j by java). We need to provide libraries for more programming languages.
Describe the solution you'd like
Implement library with

[ ] Java
[ ] Go
[ ] Rust
[ ] Python

Any movement on this? Maybe support a webassembly binding so that we can use it from any other programming language. WASI is now a days is the default binding.

Any movement on this? Maybe support a webassembly binding so that we can use it from any other programming language. WASI is now a days is the default binding.

Hi, @kesavkolla , thanks you for your interest in GraphAr. The work of implement libraries for languages  is not start yet but it would be our next major work of development.
I'm not very familiar with webassembly binding but according to the document it seems to be a great solution to make GraphAr accessible for use with programming languages other than C++ and Spark. It would be very helpful if you can provide some detail idea about webassembly binding and how do you want to use GraphAr?

@acezen I am seeing this as a trend with many open source projects where the base SDK is developed in one language either C or rust and other language SDKs are derived from WASI. WASI has good compatible layer with many implementations like wasmer, wasmtime etc... it will be so easy to use in any programming language. Investing in each programming language as a separate SDK requires lot of effort in each programming language. So IMO wasm is the right choice to go with.

I am actually looking for Rust client so that I can load GraphAr in rust and start leveraging Gremlin path traversal.

@acezen I have asked a question in hugegraph community about support for GraphAr
https://github.com/apache/incubator-hugegraph/discussions/2149
They also say GraphAR to support WASM binding. IMO This is really a huge thing so that many programming languages can start using GraphAR.

Hi @kesavkolla,

I am actually looking for Rust client so that I can load GraphAr in rust and start leveraging Gremlin path traversal.

Yes, we also have the plan to add rust support to read and build graph data in GraphAr format. We currently have a on-going refactor (#116) to have a clear repo layout for multi-languages supported. We could launch the Rust SDK after that PR merged and it would nice if we could have some collaboration to develop the Rust SDK.

I have asked a question in hugegraph community about support for GraphAr

Thanks for introducing. I have left a comment on that thread to discuss the detail.

base SDK is developed in one language either C or rust and other language SDKs are derived from WASI.

May I know more details about this idea? If I understand it correctly, WASM is usually used for porting/migrating existing software developing in C++/Rust to browser or edge computing.

May I know more details about this idea? If I understand it correctly, WASM is usually used for porting/migrating existing software developing in C++/Rust to browser or edge computing.

WASM is for running things in browser but there is WASI which makes thing to run on server side. There are many WASI compatible hosts are avaialble eg: (wasmtime, wasmer, wasmedge etc...). All these support cross language bindings. If we have WASI compatible assembly it can be used in any programming language. WASI makes that portability. To my knowledge WASI has support for file systems, environment variables etc... in addition to standard WASM. There is also experimental proposal for sockets and HTTP too. WASI is growing rapidly.
https://github.com/WebAssembly/WASI

All these support cross language bindings. If we have WASI compatible assembly it can be used in any programming language. WASI makes that portability.

It won't be hard to compile current C++ implementation to WASM with the help of WASI (we only requires apache-arrow and a small YAML parser library). However to make it friend to users the APIs may need carefully design to expose to WASM.
For multiple language support, I think we prefer to implement the SDKs in these languages rather than via WASM to keep a nice language-specific interfaces and deliver good run time performance. There's currently no concrete proposal and needs further discussion, though.


Why the GPU memory of  rednet50 is larger than resnet50?

I used  Tesla V100.
When I train resnet50-retinanet, the gpu memory is 9.26G.
But when I train rednet50-retinanet, the gpu memory is 10.98G.
In your paper, the params and floats of rednet50 is less than  resnet50.

I only change resnet50 to rednet50,  don't change the neck, the head, the input size and other params.
I don't know why. I expect for your reply

I have the same problem.Rednet50 needs larger gpu memory.Have you solved the problem?


BUG:improvment in footer section

Is there an existing issue for this?

[X] I have searched the existing issues

What happened?
In footer section instagram,facbook,twitter,youtube button is not working
either attached a link make this button active or reomve the option
Add ScreenShots

What browser are you seeing the problem on?
No response
Record

[X] I agree to follow this project's Code of Conduct
[X] I'm a GSSOC'23 contributor
[X] I want to work on this issue

please assign this issue to me

issue #889 solved ,  remove "#" in every link with the respective URL .


Replace let with var

Can you please replace let with var in for (let letter of string)?

Hi @Unsfer
Can you say why this would be needed?

Hi.
I have very old environment, which I cannot update, unfortunately.
When building it throws now:
Block-scoped declarations (let, const, function, class) not yet supported outside strict mode
Not a serious issue, downgraded version of postcss-resolve-nested-selector  for now

Which node version?
I don't mind making such changes as long as it doesn't bloat the codebase or introduce extra dependencies. Changing let to var seems fine to me. But I want to test that it then actually works and doesn't just throw again a few lines later :)

node v4.6.1

I have no idea how to test such an extremely old node version.
Nothing supports that.
So this was a best effort patch.
I've released 0.1.5 where I use var instead of let.
Hopefully that works for you 🤞

It's working with the latest version. I'll test it for some time and make a response.
Thanks a ton!

Awesome!
Closing for now, If you still have a problem, please file a new issue and then we will take a look :)
Thank you again for reporting.


option for adding constants when string literal enum option enabled

As per #68, I have made the constant generation for enums opt-in.
mWriter.preferences.useEnumPattern();
mWriter.preferences.createConstantsForStringLiteralTypeEnums();

also the documentation could hint why this could be useful before TypeScript v2.0.2

@atsu85 also tried to document better.

@andyperlitch, thanks a lot :)


adding support for Nestjs

Draft Proposal
New connect-nest package to natively support Buf Connect in Nestjs.
This packages uses NestMiddleware to integrate with Buf Connect with the underlying express or fastify server (agnostic implementation).

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Hey! Thank you for the PR!
We are discussing how to handle framework integrations going forward. I'd suggest holding off until I get back to you with how we want to incorporate such packages.

@stevenceuppens Can you open an RFC similar to this one proposing a new repo for the nestjs framework? We are not familiar with nestjs and would like someone with nestjs experience to be a co-maintainer of that repo, are you open to maintaining it in the long term?

@stevenceuppens closing this issue, please open an RFC if you want to add the package. Thanks!

Good effort but this won't give access to DI to the Connect endpoints, they will end up with undefined injected services because it's currently used as global middleware.
https://docs.nestjs.com/middleware#global-middleware


Adding sorting networks lessons (fixes #458)

Have not added the generated resources yet, the lesson content is ready to review though.
Sorting Network Cards resource has now been updated, @tah942 can you please check the new card types (words, letters, Māori colours) are as you expected when reviewing this PR.
Note @JackMorganNZ this has unveiled a bug where some exercises are being listed multiple times, probably related to Sorting Networks having lessons used in multiple age groups.

@JackMorganNZ this is ready for re-review.
I changed the name of the image used on the homepage to make it more obvious that it shouldn't be moved, hope that's ok?

@JackMorganNZ forgot to move the ct links to a separate file! this will need reviewing again.


Invalid Cookie when trying to get a quote

Bug Report
Describe the bug
I'm getting an error: "Invalid Cookie" when trying to quote.
Minimal Reproduction
// symbol: ATAO
// symbol: AAMATX
// symbol: AAVVF
// Can be any
const symbolResult = await yahooFinance2.quote(symbol);
Environment
$ node -v
v20.0.0
Additional Context

Related to https://github.com/gadicc/node-yahoo-finance2/issues/633.

Hey all, thanks for all the reports!  Looking into this now.

Hey all, working on this... it's not as simple as I'd hoped, unfortunately.  Everyone is welcome to see the progress in #634 and get involved if they'd like to or are able.
However, there is an easy alternative in the meantime.  So far it seems only quote() is affected.  quoteSummary() still works and provides most (but not all) of the same info, so may be able to help you in the meantime.
e.g.
Quote API (raw JSON from Yahoo that I had backed up)
          {
            "language": "en-US",
            "region": "US",
            "quoteType": "EQUITY",
            "quoteSourceName": "Nasdaq Real Time Price",
            "triggerable": true,
            "customPriceAlertConfidence": "HIGH",
            "currency": "USD",
            "postMarketChangePercent": -0.531021,
            "postMarketTime":<PHONE_NUMBER>,
            "postMarketPrice": 159.22,
            "postMarketChange": -0.850006,
            "regularMarketChange": -4.25,
            "regularMarketChangePercent": -2.5864167,
            "regularMarketTime":<PHONE_NUMBER>,
            "regularMarketPrice": 160.07,
            "regularMarketDayHigh": 166.15,
            "regularMarketDayRange": "159.75 - 166.15",
            "regularMarketDayLow": 159.75,
            "regularMarketVolume": 90009247,
            "regularMarketPreviousClose": 164.32,
            "bid": 0,
            "ask": 0,
            "bidSize": 9,
            "askSize": 10,
            "fullExchangeName": "NasdaqGS",
            "financialCurrency": "USD",
            "regularMarketOpen": 164.98,
            "averageDailyVolume3Month": 101720751,
            "averageDailyVolume10Day": 79056160,
            "fiftyTwoWeekLowChange": 43.86001,
            "fiftyTwoWeekLowChangePercent": 0.37742025,
            "fiftyTwoWeekRange": "116.21 - 182.94",
            "fiftyTwoWeekHighChange": -22.869995,
            "fiftyTwoWeekHighChangePercent": -0.12501363,
            "fiftyTwoWeekLow": 116.21,
            "fiftyTwoWeekHigh": 182.94,
            "dividendDate":<PHONE_NUMBER>,
            "earningsTimestamp":<PHONE_NUMBER>,
            "earningsTimestampStart":<PHONE_NUMBER>,
            "earningsTimestampEnd":<PHONE_NUMBER>,
            "trailingAnnualDividendRate": 0.865,
            "trailingPE": 26.611805,
            "trailingAnnualDividendYield": 0.005264119,
            "epsTrailingTwelveMonths": 6.015,
            "epsForward": 6.56,
            "epsCurrentYear": 6.16,
            "priceEpsCurrentYear": 25.985392,
            "sharesOutstanding":<PHONE_NUMBER>6,
            "bookValue": 4.402,
            "fiftyDayAverage": 172.2574,
            "fiftyDayAverageChange": -12.187393,
            "fiftyDayAverageChangePercent": -0.070751056,
            "twoHundredDayAverage": 151.27095,
            "twoHundredDayAverageChange": 8.799057,
            "twoHundredDayAverageChangePercent": 0.058167525,
            "marketCap":<PHONE_NUMBER>776,
            "forwardPE": 24.400915,
            "priceToBook": 36.363018,
            "sourceInterval": 15,
            "exchangeDataDelayedBy": 0,
            "pageViewGrowthWeekly": -0.038373638,
            "averageAnalystRating": "1.8 - Buy",
            "tradeable": false,
            "firstTradeDateMilliseconds":<PHONE_NUMBER>00,
            "priceHint": 2,
            "marketState": "PREPRE",
            "exchange": "NMS",
            "shortName": "Apple Inc.",
            "longName": "Apple Inc.",
            "messageBoardId": "finmb_24937",
            "exchangeTimezoneName": "America/New_York",
            "exchangeTimezoneShortName": "EST",
            "gmtOffSetMilliseconds": -18000000,
            "market": "us_market",
            "esgPopulated": false,
            "displayName": "Apple",
            "symbol": "AAPL"
          }

quoteSummary("AAPL"):
{
  summaryDetail: {
    maxAge: 1,
    priceHint: 2,
    previousClose: 167.63,
    open: 166.09,
    dayLow: 165.91,
    dayHigh: 167.4,
    regularMarketPreviousClose: 167.63,
    regularMarketOpen: 166.09,
    regularMarketDayLow: 165.91,
    regularMarketDayHigh: 167.4,
    dividendRate: 0.92,
    dividendYield: 0.0056,
    exDividendDate: 2023-02-10T00:00:00.000Z,
    payoutRatio: 0.1545,
    fiveYearAvgDividendYield: 0.94,
    beta: 1.296885,
    trailingPE: 28.42347,
    forwardPE: 25.32273,
    volume: 12178317,
    regularMarketVolume: 12178317,
    averageVolume: 63909388,
    averageVolume10days: 49926240,
    averageDailyVolume10Day: 49926240,
    bid: 0,
    ask: 0,
    bidSize: 1000,
    askSize: 1800,
    marketCap:<PHONE_NUMBER>152,
    fiftyTwoWeekLow: 124.17,
    fiftyTwoWeekHigh: 176.15,
    priceToSalesTrailing12Months: 6.823385,
    fiftyDayAverage: 156.0672,
    twoHundredDayAverage: 150.1536,
    trailingAnnualDividendRate: 0.91,
    trailingAnnualDividendYield: 0.0054286225,
    currency: 'USD',
    fromCurrency: null,
    toCurrency: null,
    lastMarket: null,
    coinMarketCapLink: null,
    algorithm: null,
    tradeable: false
  },
  price: {
    maxAge: 1,
    preMarketChangePercent: -0.00954487,
    preMarketChange: -1.60001,
    preMarketTime: 2023-04-20T13:29:59.000Z,
    preMarketPrice: 166.03,
    preMarketSource: 'FREE_REALTIME',
    regularMarketChangePercent: -0.0029827594,
    regularMarketChange: -0.5,
    regularMarketTime: 2023-04-20T14:32:19.000Z,
    priceHint: 2,
    regularMarketPrice: 167.13,
    regularMarketDayHigh: 167.4,
    regularMarketDayLow: 165.91,
    regularMarketVolume: 12178317,
    averageDailyVolume10Day: 49926240,
    averageDailyVolume3Month: 63909388,
    regularMarketPreviousClose: 167.63,
    regularMarketSource: 'FREE_REALTIME',
    regularMarketOpen: 166.09,
    exchange: 'NMS',
    exchangeName: 'NasdaqGS',
    exchangeDataDelayedBy: 0,
    marketState: 'REGULAR',
    quoteType: 'EQUITY',
    symbol: 'AAPL',
    underlyingSymbol: null,
    shortName: 'Apple Inc.',
    longName: 'Apple Inc.',
    currency: 'USD',
    quoteSourceName: 'Nasdaq Real Time Price',
    currencySymbol: '$',
    fromCurrency: null,
    toCurrency: null,
    lastMarket: null,
    marketCap:<PHONE_NUMBER>152
  }
}

@pazer001 everything is working :tada:   (with no changes on our side...
But we'll have to keep an eye on it.  When I was working on the code earlier, I noticed some inconsistencies in the data required to make it work, even in the browser... so, they might have just turned it off while they work out some bugs on their side, and then could turn it on again in the future.  Unfortunately I can't develop / prepare for such an eventuality unless they turn it back on again... so let's just hope it stays off :crossed_fingers:

I have started to notice Error: Invalid cookie again...

oh dear 😔
I'm away for the weekend and will only be and to look at this next week 😕
but presumably if it's more stable now it will be easier to investigate
further.
my prior partial work is still available on the PR if anyone wants to take
a stab before then.
On Thu, 4 May 2023, 20:46 Thomas Kaul, @.***> wrote:

I have started to notice Error: Invalid cookie again...
—
Reply to this email directly, view it on GitHub
https://github.com/gadicc/node-yahoo-finance2/issues/633#issuecomment-1535170162,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAC5IGTRR65IX65TVXQTQULXEPTN3ANCNFSM6AAAAAAXFAHOTU
.
You are receiving this because you were mentioned.Message ID:
@.***>

Yea, its a lot more prominent for me again. Would anyone happen to know of a way to send an array of tickers to quote summary in a similar way to how you can do it with quote? I don't necessarily want to use a for loop to go through them all cos it takes ages

I have notice Exiting with Error: Invalid Cookie only with quote. If I use quoteSummary it works. Unfortunately quoteSummary does not allow multiple symbols
$ yahoo-finance quote MULT3.SA
Exiting with Error: Invalid Cookie

I have notice Exiting with Error: Invalid Cookie only with quote. If I use quoteSummary it works. Unfortunately quoteSummary does not allow multiple symbols
$ yahoo-finance quote MULT3.SA
Exiting with Error: Invalid Cookie

This worked for me! I'm just looking for price data, just spam the api with a message per-ticker and this works great!

can cofirm, error is appearing again out of nothing!

still same problem:
$ yahoo-finance quote AAPL
Exiting with Error: Invalid Cookie

Anyone else getting a invalid JSON response when spanning quotesummary with each ticker?
node_modules\node-fetch\lib\index.js:273 [0]                             return Body.Promise.reject(new FetchError(invalid json response body at ${_this2.url} reason: ${err.message}, 'invalid-json')); [0]                                                        ^ [0] Waiting for the debugger to disconnect... [0] FetchError: invalid json response body at https://query2.finance.yahoo.com/v10/finance/quoteSummary/AAPL?formatted=false&modules=price reason: Unexpected token T in JSON at position 0

quoteSummary() is working for me, but search() is also affected.

yahoo-finance2 @ v2.3.10
"Error: Invalid Cookie" when using .quote()
.quoteSummary() is working as an alternative.
@gadicc it's really cool how involved you are maintaining this.  enjoy your weekend brother

Hi guys, many thanks from me too. I really apriciate the work.
Did you see this thread?
https://stackoverflow.com/questions/76059562/yahoo-finance-api-get-quotes-returns-invalid-cookie
I don't know if its a setting or hardcoded. But for me the switch to quoteSummary() is fine for the moment

Hi all, thanks for all the updates and kind words.
Continuing on from my work from last time, this is now working again with the latest commit to cookies branch which includes the WIP PR https://github.com/gadicc/node-yahoo-finance2/pull/634.
All that's left:

Use only when needed (just quote and search, I think? anyone seen anything else?)
Figure out best way to expire crumb... currently 24hrs are hard coded, but probably we can use the cookie expiration time.
Tests for all the cookie / crumb stuff
Adapt test code to remove crumb from URL to not break cache (and prob same with cookies)

I'm actually away on an overseas trip at the moment, but still going to try to get something out today, as my time for the rest of the week is more limited.

Hi all, out of time for today, but I think we're in the good place.  Updated the above message, only outstanding thing are unit tests for the new code.  Of course I tested by hand that it does work :)
I usually don't like to publish new releases without tests on the new code, but I would like to get a working release out.  In any event, if people can test the code from the cookies branch, that would be amazing, and with a bit of feedback I'll publish a new release even if I don't get a chance to do the tests.
Of note: only quote() uses the new cookie & crumb code.  search() didn't require it in my tests.  So please if anyone can give this a spin and report back on your findings, that would be great :)

Hi again all :)
I know it's easier to test against published packages rather than checking out a development branch... and also wanted to have the published release working on as much as possible.  So, all the above is now published in v2.4.0`.
Your feedback will be helpful to confirm that everything is working in a variety of cases.  I haven't deleted the http cache of all prior tests as it's more than I have time to look into now, but basic testing in a variety of operations seems to all be working well.  So let's see how we go!

:tada: This issue has been resolved in version 2.4.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

Unfortunately Yahoo responses differ depending on origination IP.  So the above fix only works in certain territories and further work will still be needed in EU.

yep!
[Nest] 172  - 05/08/2023, 4:10:55 PM   ERROR [YahooFinanceService] Error: Invalid Cookie
[Nest] 172  - 05/08/2023, 4:10:55 PM    WARN [YahooFinanceService] Fallback to yahooFinance.quoteSummary()
[Nest] 172  - 05/08/2023, 4:10:55 PM   ERROR [YahooFinanceService] Error: Invalid Cookie
[Nest] 172  - 05/08/2023, 4:10:55 PM    WARN [YahooFinanceService] Fallback to yahooFinance.quoteSummary()
[Nest] 172  - 05/08/2023, 4:10:56 PM   ERROR [YahooFinanceService] Error: Invalid Cookie
[Nest] 172  - 05/08/2023, 4:10:56 PM    WARN [YahooFinanceService] Fallback to yahooFinance.quoteSummary()
[Nest] 172  - 05/08/2023, 4:10:57 PM   ERROR [YahooFinanceService] Error: Invalid Cookie
[Nest] 172  - 05/08/2023, 4:10:57 PM    WARN [YahooFinanceService] Fallback to yahooFinance.quoteSummary()
[Nest] 172  - 05/08/2023, 4:10:57 PM   ERROR [YahooFinanceService] Error: Invalid Cookie
[Nest] 172  - 05/08/2023, 4:10:57 PM    WARN [YahooFinanceService] Fallback to yahooFinance.quoteSummary()
[Nest] 172  - 05/08/2023, 4:10:57 PM   ERROR [YahooFinanceService] Error: Invalid Cookie
[Nest] 172  - 05/08/2023, 4:10:57 PM    WARN [YahooFinanceService] Fallback to yahooFinance.quoteSummary()

Tracking in https://github.com/gadicc/node-yahoo-finance2/pull/637.
Feel like I'm one step away... just got to get all the cookie domains correct.
In second last step, all correct cookies are sent, but on the quote request, some are missing, and we get Invalid cookie.
Ironically for whatever reason, in the test specs, it all works.  And maybe there's a clue there.
Unfortunately I'm out of time.  I'm away overseas and not sure when I'll get a chance to work on this again, hence this update... in case anyone else wants to take a look in the meantime.

:tada: This issue has been resolved in version 2.4.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

Working!  Definitely lots of room for improvement but at least things are working again.  All my tests were via a UK VPN.  YMMV.  If things aren't working for you, please report what country you're in :pray:

Just a heads up, using the previous Yahoo API version works just fine:
curl https://query1.finance.yahoo.com/v6/finance/quote?symbols=AAPL vs
curl https://query1.finance.yahoo.com/v7/finance/quote?symbols=AAPL

would like to express our gratitude @gadicc and team for the quick resolution here!

Thank you!

@gadicc You're doing an amazing work on this! Congrats!

I got a lot of Invalid Cookie messages the last days. I use QuoteSummary

I got a lot of Invalid Cookie messages the last days. I use QuoteSummary

I can confirm the error for quoteSummary() (https://github.com/ghostfolio/ghostfolio/issues/2148).

:tada: This issue has been resolved in version 2.4.2 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

Thanks @Luminilion, for the kind words, and @JosHonig and @dtslvr for the updated reports.
So, we only do all the extra cookie and crumb logic when we know we need it, as it requires something like 6 extra requests (in EU at least) before we can start doing the regular queries (once off, per process load; hope to add DB persistance for cookies when I get a chance soon, useful for e.g. serverless).
Up until now, this has been only the quote API, but looks like Yahoo have just rolled this out to quoteSummary too.  Fortunately, the fix is trivial (we simply add { query: { needsCrumb: true } } to quoteSummary.ts's call to modeExec().
So, please continue to report if you notice this on any other APIs... and of course, PRs are welcome as usual... in this case it's a one line change :)

That was quick 🚀 Thank you very much for the new release @gadicc!

Invalid cookie error also occurred on endpoints;

calendarEvents,
balanceSheetHistory,
incomeStatementHistory,
cashflowStatementHistory.

Fixed with upgrade from 2.4.1 to 2.4.3
Nice job!!


Replacing old content with pointers to new data documentation

We've moved all of our data documentation to http://www.measurementlab.net/data/ and that is the new authoritative location, so replacing the old content with links there.

@mtlynch LGTM!


fix: typo

Fixes Or Enhances
Make sure that you've checked the boxes below before you submit PR:

[x] Tests exist or have been written that cover this particular change.

@go-playground/validator-maintainers

Coverage: 74.182% (+0.0%) from 74.182% when pulling 2d96ecdadc79649457b04e5be0f14c648622398e on TcMits:typo into c7e0172e0fd176bdc521afb5186818a7db6b77ac on go-playground:master.


PropertyValueTypeMismatchException

I got PropertyValueTypeMismatchException when running following. I use .net 4.5.2. Could anyone help me to resolve it? Thanks in advance
class Person{
public long? Id { get; set; };
}
List people = new List<Person>{
new Person { Id = 1 },
new Person { Id = null },
new Person { Id = 2 },
}
var filter = new Filter<Person>();
long?[] idList = new long? { 1 };
filter.By("Id", Operation.In, idList);
var result = people.Where(filter); // throw PropertyValueTypeMismatchException

Hi @ChinhDuong,
Good news! I believe I got it right this time (even created a couple of tests specifically for that).
I just published a new version yesterday. Would you mind having a look at that?
Many thanks,
David


Adding qty box and total price in cart page

Till now If the user want to add more quantity of same book it showing in a extra card instead of that I can add qty box with increasing and decreasing within a one card for one book. And also there is no total estimated price showing in cart page it is available only in the payment page, so I can add total price in the cart page also. Kindly assign this issue to me.

Hi. Can I start to work on this issue?

@tharanithar-r Assigned.
Make sure to join the discord server of all contributors - https://discord.gg/fgMHHP4Kcq


How to combine data batch with GCN?

🐛 Bug
Hi!, Thanks for your amazing framework. But recently, I am to using GCN to extract spatial-feature from node features in a batch way, but it got the error like this:
ValueError: `MessagePassing.propagate` only supports `torch.LongTensor` of shape
`[2, num_messages]` or `torch_sparse.SparseTensor` for argument edge_index`.

The code I am using
The edge_index and edge_weight is defined as below:
graph_edges = torch.tensor(graph_edges, dtype=torch.long, device="cuda:0").t().contiguous()
graph_edges_b = torch.stack([graph_edges for x in range(location_t.shape[0])], dim=0)
print("graph_weight", graph_weight.shape)      # [16, 171]
print("graph_edges_b", graph_edges_b.shape) # [16, 2, 171]
print("feature_t", feature_t.shape)                     # [16, 19, 512]

then
enc_feature = self.g_conv1(feature_t, graph_edges_b, graph_weight)

it feeds back the error as below:
ValueError: `MessagePassing.propagate` only supports `torch.LongTensor` of shape `[2, num_messages]` or `torch_sparse.SparseTensor` for argument `edge_index`.

For more details:
The batch_size is 16, and all the graph is fully-connected, but the weights is different for each graph.

MessagePassing indeed only supports as edge_index a shape of [2, num_edges]. In order to allow batch-wise training, you need to make use of the diagonal stacked procedure as described here. The DataLoader does support that automatically when using torch_geometric.data.Data objects.

Thanks for your reply! The graph is dynamically changed through the training, the edge_weight is computed by network, so perheps the DataLoader might not help this situation.  Do you mean I need to use like this?
out_batch = []
for i in range(graph_edges_b.size(0)):
    hidden_enc_feature = F.relu(self.g_conv1(feature_t[i], graph_edges_b[i], graph_weight[i]))
    enc_feature = F.relu(self.g_conv2(hidden_enc_feature, graph_edges_b[i], graph_weight[i]))
    out_batch.append(F.relu(enc_feature))

out_batch = torch.stack(out_batch)

You can concatenate features as follows to avoid the for-loop:
feature = torch.cat(feature_t, dim=0)
graph_weight = torch.cat(graph_weight, dim=0)
cumsum = 0
edge_indices = []
for edge_index, x in zip(graph_edges_b, feature_t):
    edge_indices.append(edge_index + cumsum)
    cumsum += x.size(0)
graph_edges = torch.cat(edge_indices, dim=-1)

hidden_enc_feature = F.relu(self.g_conv1(feature, graph_edges, graph_weight)
enc_feature = F.relu(self.g_conv2(hidden_enc_feature, graph_edges, graph_weight)
out_batch = enc_feature.view(batch_size, num_nodes, -1)

Sorry to bother you again, Could you tell me why use the following codes:
feature = torch.cat(feature_t, dim=0)
graph_weight = torch.cat(graph_weight, dim=0)

Perheps, I do not explain the problem clearly. The shape of feature_t is [batch, 19, 1024], and the shape of graph_weight is [batch, num_edges]. So, when I used torch.cat, it throwed out `TypeError'

Oh, I thought it is a list of tensors. As such, reshaping to [batch * 10, 1024] and [batch * num_edges] is sufficient.

Excuse me, Could you tell me if I have many graphs to do GCN operation, but actually the graphs share same fully-connection(meaning have same graph_index, but different graph_weight)
print(feature.shape)  # [1600, 19, 2048]
print(graph_index.shape)  # [2, 171] just one graph structure
print(graph_weight.shape) #[1600, 171]

When I do the following operation, got error.
gcn_feature = self.g_conv1(feature, graph_index, graph_weight)

I got the error IndexError: The shape of the mask [171] at index 0 does not match the shape of the indexed tensor [1600, 171] at index 0
Do I really need to stack the graph_index to meet  the shape of [1600, 2, 171]?
graph_index = torch.stack([graph_index for i in range(1600)]

But I got error too, RuntimeError: Sizes of tensors must match except in dimension 0. Got 2 and 1600 (The offending index is 0)

Actually, the graphs' graph_index are same, just as image shows.

In case you want to operate on static graphs with varying edge weights, you cannot use the static graph computation scheme that you are currently using. Instead, create a list of data objects and load them via DataLoader:
data_list = []  # Each element in `data_list` describes a data object with x=[19,2048], edge_index=[2, 171] and edge_weight=[171]

loader = DataLoader(data_list, batch_size=128, shuffle=True)

I see! Thanks a lot! Hope you have a good day!


Fixing static references to JRuby runtime state

This PR fixes the following two items:

StringCache is now stored as an internal variable of ArJdbc mod.
Loaded drivers cache is stored as an internal variable, I believe on ArJdbc, but unsure what "self" is in this context.

I request review from @enebo @kares @rdubya.
Specific concerns:

Internal variables use instance variable logic, so they're a bit more overhead than a simple hash map.
Obviously I have to look up ArJdbc module in the StringCache case. Both cases probably should store in same place using same mechanism but the loaded drivers cache has access to a "self" on which the load operation was called.

Unfixed at push time: RubyJdbcConnection has a static IRubyObject defaultConfig used for some JNDI logic. I was not clear what this field is or how to remove it. First instinct is to put it into the ARJDBC top-level module as an internal variable.
Related is the neighboring connectionFactory, which may hold runtime state.

StringCache is now stored as an internal variable of ArJdbc mod.

since its now double lookup it might be worth getting rid of it completely (measuring the impact).

Loaded drivers cache is stored as an internal variable, I believe on ArJdbc, but unsure what "self" is in this context.

WeakHashMap<Ruby, X> was not holding a strong reference to runtime, what has been the issue here?
also I am not sure if its actually used where its mean to be -> establish conn loading drivers every time (e.g. mysql_connection which it used to have an impact on)

RubyJdbcConnection has a static IRubyObject defaultConfig used for some JNDI logic.

that is just a fast compare (caching) mechanism to quickly decide the actual config used.
have been thinking about these leaks as I introduced them e.g. the StringCache did represent a measurable improvement at the time. con being non 'proper' multi-runtime friendliness but than most use-cases I can think of always load ext .class-es with JRuby's class-loader (which is per runtime).
so it never really mattered ... and has proven "fine" since through the years (1.3 has StringCache) no one complained about memory leaking. which doesn't mean that its fine for 100% cases(?) but simply that 'complicating' these needs measuring to decide whether its actually still worth keeping.

I do think we can get the global benefits across connections for stringCache by just making something which is constructed outside connection instances.  This is likely true for caching sites as well (although I really like the idea of rewriting bytecode from callMethod to indy via something like Mjlollnir, but that is a blue sky project near term).

since its now double lookup it might be worth getting rid of it completely (measuring the impact).

I would expect two lookups to still be cheaper than a RubyString allocation (potentially with ByteList and byte[] as well), but I'll leave it up to you if you want to keep it.

WeakHashMap<Ruby, X> was not holding a strong reference to runtime, what has been the issue here?

It's probably nothing. I wanted to be consistent in avoiding static references to runtime state. It didn't seem necessary for it to be global, but I admit I am not clear what it's saving us.
I also agree many risks are mitigated by having arjdbc loaded in a separate classloader, but we don't want to preclude other ways of deploying JRuby.


More Detail on the Parameters

Hi,
May I know the details of the 6 parameters of "add_joint_axis" and 16 parameters of "add_initial_end_effector_pose"?
Thanks!

The add_joint_axis method expects 6 parameters corresponding to the coefficients of the 6D screw axes of the joints in the kinematic chain. These parameters can be found by studying the mechanical structure of the robot. Briefly, the screw axes of a particular joint in the kinematic chain describe the constraint between the end effector pose and the base. Note that the screw axes can be defined in the base frame or the end-effector frame, but for this library make sure to use the screw axes expressed in the base frame (often referred to as the S-frame).
The add_initial_end_effector_pose expects 16 parameters corresponding to the 4x4 homogenous transformation matrix of the end effector frame when the joints are all set to zero.
You can use the following video which explains these parameters in more detail by means of an example: [The add_joint_axis method expects 6 parameters corresponding to the coefficients of the 6D screw axes of the joints in the kinematic chain. These parameters can be found by studying the mechanical structure of the robot. Briefly, the screw axes of a particular joint in the kinematic chain describe the constraint between the end effector pose and the base. Note that the screw axes can be defined in the base frame or the end-effector frame, but for this library make sure to use the screw axes expressed in the base frame (often referred to as the S-frame).
The add_initial_end_effector_pose expects 16 parameters corresponding to the 4x4 homogenous transformation matrix of the end effector frame when the joints are all set to zero.
You can use the following video which explains these parameters in more detail by means of an example: https://youtu.be/cKHsil0V6Qk](https://youtu.be/cKHsil0V6Qk)


Version compatibility warning and dependency conflicts in fresh installation

Hi Muris,
I encountered several warnings when creating a new project with Pruvious. I'm sharing these details to help improve the initialization experience.
Environment:

Windows 10
Node v20.15.1
npm 10.8.2

*I also have a nuxt 3.14.1592 on the same computer.
I'm following your tutorial step by step.
At: npx pruvious@latest init cms, below the details:
√ Which package manager would you like to use?
npm
o Installing dependencies...                                                                                21:48:35
npm warn ERESOLVE overriding peer dependency
npm warn While resolving<EMAIL_ADDRESS>npm warn Found<EMAIL_ADDRESS>npm warn node_modules/pruvious/node_modules/magicast
npm warn<EMAIL_ADDRESS>from<EMAIL_ADDRESS>npm warn   node_modules/pruvious
npm warn<EMAIL_ADDRESS>from the root project
npm warn
npm warn Could not resolve dependency:
npm warn peerOptional<EMAIL_ADDRESS>from<EMAIL_ADDRESS>npm warn node_modules/pruvious/node_modules/c12
npm warn<EMAIL_ADDRESS>from<EMAIL_ADDRESS>npm warn   node_modules/pruvious
npm warn
npm warn Conflicting peer dependency<EMAIL_ADDRESS>npm warn node_modules/magicast
npm warn   peerOptional<EMAIL_ADDRESS>from<EMAIL_ADDRESS>npm warn   node_modules/pruvious/node_modules/c12
npm warn<EMAIL_ADDRESS>from<EMAIL_ADDRESS>npm warn     node_modules/pruvious
npm warn deprecated<EMAIL_ADDRESS>This functionality has been moved to @npmcli/fs
npm warn deprecated<EMAIL_ADDRESS>This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.
npm warn deprecated<EMAIL_ADDRESS>This package is no longer supported.
npm warn deprecated<EMAIL_ADDRESS>This package is no longer supported.
npm warn deprecated<EMAIL_ADDRESS>Rimraf versions prior to v4 are no longer supported
npm warn deprecated<EMAIL_ADDRESS>Glob versions prior to v9 are no longer supported
npm warn deprecated<EMAIL_ADDRESS>This package is no longer supported.
npm warn deprecated<EMAIL_ADDRESS>This package is no longer supported.
npm warn deprecated<EMAIL_ADDRESS>This package is no longer supported.
npm warn deprecated<EMAIL_ADDRESS>This package is no longer supported.

> postinstall
> nuxt prepare

i Nuxt collects completely anonymous data about usage.                                                        21:54:33
  This will help us improve Nuxt developer experience over time.
  Read more on https://github.com/nuxt/telemetry

√ Are you interested in participating?
No

√ Types generated in .nuxt                                                                                21:57:48

added 1065 packages, and audited 1067 packages in 9m

184 packages are looking for funding
  run `npm fund` for details

2 low severity vulnerabilities

Some issues need review, and may require choosing
a different dependency.

Run `npm audit` for details.
√ Installation completed.                                                                                21:57:48

√ Initialize git repository?
No
                                                                                21:57:51
✨ Nuxt project has been created with Pruvious and the v3 template. Next steps:
 › cd cms                                                                                21:57:51
 › Start development server with npm run dev

Then I run npm run dev:
 › cd cms                                                                                21:57:51
 › Start development server with npm run dev                                                                  21:57:51

Nuxt 3.14.1592 with Nitro 2.10.4                                                                              22:04:36
                                                                                22:04:37
  ➜ Local:    http://localhost:3000/ run dev
  ➜ Network:  use --host to expose
> dev
> pruvious dev
[pruvious 22:04:42]  WARN  This version of Pruvious is compatible with Nuxt 3.13.x. You are currently using Nuxt 3.14.1592.

  ➜ DevTools: press Shift + Alt + D in the browser (v1.6.4)                                                   22:04:43

√ Database synced in 935 ms                                                                          pruvious 22:04:51
√ Pruvious built in 6384 ms                                                                          pruvious 22:04:51
√ Vite client built in 296ms                                                                                22:04:54
√ Vite server built in 1268ms                                                                                22:04:55
√ Nuxt Nitro server built in 10439 ms                                                                   nitro 22:05:06
i Vite client warmed up in 1ms                                                                                22:05:58
i Vite server warmed up in 3525ms                                                                             22:06:02
√ Pruvious built in 213 ms                                                                           pruvious 22:06:04
√ Pruvious built in 117 ms                                                                           pruvious 22:06:05
√ Pruvious built in 89 ms                                                                            pruvious 22:06:06
√ Pruvious built in 83 ms                                                                            pruvious 22:06:08
√ Pruvious built in 60 ms                                                                            pruvious 22:06:08
i Compiled plugins.server.mjs in 914.05ms                                                                     22:06:08
i Compiled plugins.client.mjs in 924.01ms                                                                     22:06:08
i Compiled types/plugins.d.ts in 931.09ms                                                                     22:06:09
i Compiled plugins.server.mjs in 642.71ms                                                                     22:06:10
i Compiled plugins.client.mjs in 963.88ms                                                                     22:06:10
i Compiled types/plugins.d.ts in 984.42ms                                                                     22:06:10
√ Pruvious built in 58 ms                                                                            pruvious 22:06:24

❔I have 3 questions:
1/ Do I need to worry about these warning?
2/ Regarding this new app I called cms (this is your tutorial at https://pruvious.com/tutorial/01-project-setup), why after using npx pruvious@latest init cms I get this warning WARN  This version of Pruvious is compatible with Nuxt 3.13.x. You are currently using Nuxt 3.14.1592 whereas the package.json in the cms folder shows "pruvious": "^3.13.4"?
3/ I already have a website running nuxt 3.14.1592, can I use Pruvious with this one? Or do I need to downgrade the nuxt version?
Many thanks for such job!

Hi Wesley! Thank you for your feedback. I've bumped Pruvious to v3.14.0 and updated its dependencies. Most warnings should be gone now. The Pruvious and Nuxt incompatibility warning is just precautionary - I added it because some minor Nuxt updates had previously caused the dashboard to become unresponsive.


Selected Search Engines Reset

Hey mate! Love the extension, easily the most well integrated of the various options out there.
Problem is that when I select the specific search engines to use they always get reset to all 4 after a short period of time. Not a huge issue as it doesn't impact usability, but I really only want to enable DDG and Google.

Hey Jack, stoked to hear you're enjoying the extension first of all!
Secondly, sorry you're having issues with the preferences feature.
Can you give me some information about what browser you're using and how strongly the settings getting forgotten correlates with you closing your browser completely and re-opening (e.g. Between computer shutdowns)?
My guess is that the settings arent persisting properly but that they'd be stable with a browser session.
Keen to hear back,
Cheers!


[Bug]  output successfully yet incomplete(missing the final one-two minutes)

[x] I didn't find a similar issue already open.
[x] I read the documentation (README AND Wiki)
[x] I have installed FFMpeg
[x] My problem is related to Spleeter only, not a derivative product (such as Webapplication, or GUI provided by others)

Description
output successfully yet incomplete(missing the final one-two minutes)

Step to reproduce

Installed using ...
Run as ...
Got ... error

Output
Share what your terminal says when you run the script (as well as what you would expect).

Environment

OS
Windows / Linux / MacOS / other

Installation type
Conda / pip / other

RAM available
XGo

Hardware spec
GPU / CPU / etc ...

Additional context

Hi @gulee100,
Is your processed file longer than 10 minutes? the -d or --duration parameter makes it possible to limit the duration of the processed audio in the input files. Its default value is 600s (=10min) as a safeguard, so if you don't overwrite it, any file longer than 10 minutes won't be processed entirely.  It may explain your issue.

Hi @gulee100,
Is your processed file longer than 10 minutes? the -d or --duration parameter makes it possible to limit the duration of the processed audio in the input files. Its default value is 600s (=10min) as a safeguard, so if you don't overwrite it, any file longer than 10 minutes won't be processed entirely. It may explain your issue.

Thank you so much for your help!


Better markdown docs generation

Hi there, and thanks for your project! He is really awesome!
I could ask what you think about the improvement of markdown CLI docs generation - very often I use the following, hm... "addon" (which was crafted by me) to automatically update the CLI usage section in the project readme file: gh.tarampamp.am/urfave-cli-docs. In a few words, it allows using go generate ./... to update a section in the readme file (between the tags):
<!--GENERATED:CLI_DOCS-->
<!--/GENERATED:CLI_DOCS-->

With something like that:

From the sources:
import (
	"fmt"

	"github.com/urfave/cli/v2"
)

func NewApp() *cli.App {
	return &cli.App{
		Name:  "example",
		Usage: "Example CLI application",
		Commands: []*cli.Command{
			{
				Name:    "hello",
				Aliases: []string{"hi"},
				Usage:   "Prints hello",
				Action: func(c *cli.Context) error {
					fmt.Printf("Hello, %s!", c.String("name"))
					return nil
				},
				Flags: []cli.Flag{
					&cli.StringFlag{
						Name:    "name",
						Value:   "stranger",
						Usage:   "Just a name",
						Aliases: []string{"n"},
						EnvVars: []string{"NAME"},
					},
				},
			},
		},
	}
}

And I found this very useful, and probably other users of this library will find this useful and for them. Should I make PR with appropriate changes to this project, or not?

@tarampampam Please submit a PR . Thank you so much

@dearchap Should I replace the ToMarkdown function (with its template) or make a separate function?

@dearchap @urfave Could anyone review the changes in #1722?


fix dynamo_backend issue when launching using config file

What does this PR do?

fix dynamo_backend issue when launching using config file

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.


Can other transports be used

Such as livekit: End to end stack for WebRTC

Yes, absolutely, as soon as someone adds the LiveKit transport. But currently, that's not implemented.

Big +1 on this request! Would be super interested in livekit support for pipecat!

Gotta say their API sounds familiar:
async def entrypoint(ctx: JobContext):
2  vad = silero.VAD()
3  stt = deepgram.STT()
4  llm = openai.LLM()
5  tts = elevenlabs.TTS()
6  assistant = VoiceAssistant(vad, stt, llm, tts, allow_interruptions=True)
7  assistant.set_system_message(["You are a voice assistant created by LiveKit."])
8  await assistant.say('Hello, how can I help you today?')
9
10  @ctx.room.on("participant_connected")
11  def _on_participant_connected(participant:
12                               rtc.RemoteParticipant):
13    assistant.start(ctx.room, participant)
14

I'm planning to work on LiveKit whenever I have some free time.
Has anybody started working on it already?

I'm planning to work on LiveKit whenever I have some free time. Has anybody started working on it already?

Hi @joachimchauvet Do you have progress, would love to do the same!

I'd love to see this happen as well! @kwindla and I were talking about collaborating on this and would love to help however I can.

#396 I tried adding a simple serializer to help bridge LiveKit Agents via WebSocket.

Hi @joachimchauvet Do you have progress? would love to do the same!

Yes, but I'm not quite there yet. I've been able to connect to a LiveKit room and send TTS audio from my Pipecat pipeline, but I'm running into a few issues with some callback handlers and receiving frames from participants.
Once I've tidied things up and have something decent enough to share, I'll post an update here. Still haven't gotten around to video yet, though.

@davidzhao any plans on shipping livekit support for pipecat? One of the reasons not able to use pipecat & livekit, is primarily because of lack of livekit support and vice-versa.

I would like to add my voice to the others, livekit-transport would be fantastic especially video, that would be the last missing puzzle piece for a fully local solution. I hope I am not missing anything but I think the local transport does not support video or image input.

Livekit  will be good option.

+1 for live kit here, if it is really needed I can help to build livekit pipecat module.

@joachimchauvet : Can you send me a test main.py or so with what you dev livekit it, or a example how to use it. (I know it is still in dev, but I could optimized it with your code and help you)

@joachimchauvet : Can you send me a test main.py or so with what you dev livekit it, or a example how to use it. (I know it is still in dev, but I could optimized it with your code and help you)

Sure! I'm on my phone right now but I'll send that over in the morning (CET) with the latest changes I have locally.

I added an example here: https://github.com/joachimchauvet/pipecat-livekit/blob/main/examples/foundational/01b-livekit-audio.py
It requires livekit-api to generate the token.

We're happy to merge this into pipecat main any time.

I added an example here: https://github.com/joachimchauvet/pipecat-livekit/blob/main/examples/foundational/01b-livekit-audio.py It requires livekit-api to generate the token.

@joachimchauvet s there any method to enable LiveKit transport to transmit images simultaneously, similar to how Daily transport works, by actively retrieving photos from the other participant's camera?

@joachimchauvet Is there any method to enable LiveKit transport to transmit images simultaneously, similar to how Daily transport works, by actively retrieving photos from the other participant's camera?

Right now it only supports audio. It's definitely possible to implement video/images with LiveKit but that's not implemented in my LiveKit transport yet.

I think this can now be closed since we support websockets, fastapi websockets, livekit and daily.


Print a warning if a router is configured with both HTTP and HTTPs entrypoints

Welcome!

[X] Yes, I've searched similar issues on GitHub and didn't find any.
[X] Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you expect to see?
Hello!
I recently spent a few hours troubleshooting something that in the end was very simple.
As I'm new to Traefik, I just followed one of the many guides to set things up. In the beginning everything was straight-forward.
After getting HTTP to work, I proceeded with HTTPs. My usual approach is to get connectivity to work first, and then fixing the actual certificate (and other tuning).
At a first look, it looks very simple:
https://doc.traefik.io/traefik/routing/routers/

If not specified, HTTP routers will accept requests from all EntryPoints in the [list of default EntryPoints]

https://doc.traefik.io/traefik/https/overview/

Traefik supports HTTPS & TLS, which concerns roughly two parts of the configuration: routers, and the TLS connection (and its underlying certificates).
When a router has to handle HTTPS traffic, it should be specified with a tls field of the router definition.

So, I basically just proceeded adding a HTTPs entrypoint, that just as expected was added to my router:

But in reality, the HTTPS traffic is just routed into nowhere, with no clear error message.
It's first when you dive deep into the documentation, reading the TLS part of the router docs that you find that a router can only use HTTP, or HTTPS.
Searching the community forum, I see a large amount of posts related to "HTTPs and 404" messages, so I'm not the first one to stumble upon this.
It would have helped me a lot to get at least some hint about this, and I think that there are many places where this could be checked, but I think these would be the most logical:

During configuration loading: Print a WARN (or at least INFO) level log message to traefik log: "Router configured with incompatible entrypoints"
Add some field on the dashboard (Status: Warning?)

I think this would be a great usability improvement for new users.

Hello @sandnabba,
Thanks for your interest in Traefik!
We are aware that router TLS configuration and entryPoint TLS configuration are sometime confusing, and we would like to improve the documentation in that regard.
Nevertheless, we are unsure to follow every step you took, could please elaborate/be more specific, even by sharing static and dynamic configuration?

Hi, sure!
My static configuration (for the entrypoints):
entryPoints:
  web:
    address: ":80"
  websecure:
    address: ":443"

The base for this configuration is the Routing & Loadbalancing -> Entrypoint documentation page:
https://doc.traefik.io/traefik/routing/entrypoints/
An interesting note on this page is that it specifically states:

If there is no EntryPoint with the AsDefault option set to true, then the list of default EntryPoints includes all HTTP/TCP EntryPoints.

But it does not mention that this might result in "conflicting" configuration.
After setting up the above entrypoints, I proceeded to the "Routers" documentation:
https://doc.traefik.io/traefik/routing/routers/#entrypoints
Again, it's clearly stated that:

If not specified, HTTP routers will accept requests from all EntryPoints in the list of default EntryPoints.

So I basically just copy-pasted the "Listens to Every EntryPoint" example (from the page above) and combined it with the "A Service with One Server" service example from the service documentation:
https://doc.traefik.io/traefik/routing/services/#servers
Result:
http:
  services:
    test-service:
      loadBalancer:
        servers:
        - url: "http://<IP_ADDRESS>:80"
  routers:
    test-router:
      rule: "Host(`test.mydomain`)"
      service: "test-service"

As expected, both entrypoints was connected to the "test-router" (this is when I made the screenshot of the dashboard in the original post).
And here I expected that traffic entering the HTTP endpoint should be routed to the service over HTTP, and that HTTPs traffic should be routed over HTTPS, since the dashboard shows that both entrypoints are connected to the same router. (But in reality they are not).
So I think there are two parts to be done here. First of all the documentation should be updated. Maybe provide a dedicated HTTP to HTTPS example? (Since that is a very common usecase).
By much googling, I went with the following solution (using the almost secret "noop@internal" endpoint).
http:
  middlewares:
    http_to_https:
      redirectScheme:
        scheme: https
        permanent: true
  routers:
    forwarder:
      rule: "HostRegexp(`host.+`)"
      entrypoints: "web"
      middlewares: "http_to_https"
      service: "noop@internal"

But I also think that Traefik should prevent me from using this "broken" configuration. According to the dashboard, it really looks like everything should work, when in reality it does not.

And here I expected that traffic entering the HTTP endpoint should be routed to the service over HTTP, and that HTTPs traffic should be routed over HTTPS, since the dashboard shows that both entrypoints are connected to the same router. (But in reality they are not).

Could you elaborate on this? because we are not sure to understand everything, do you have any errors?

Could you elaborate on this? because we are not sure to understand everything, do you have any errors?

Yes, I'm getting a HTTP 404 trying HTTPS:
mil@emil-work: ~ $> curl -k -v -o /dev/null https://test.mydomain
* processing: https://test.mydomain
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying <IP_ADDRESS>:443...
* Connected to test.mydomain (<IP_ADDRESS>) port 443
* ALPN: offers h2,http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [15 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [879 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=TRAEFIK DEFAULT CERT
*  start date: Jul 23 19:50:51 2024 GMT
*  expire date: Jul 23 19:50:51 2025 GMT
*  issuer: CN=TRAEFIK DEFAULT CERT
*  SSL certificate verify result: self-signed certificate (18), continuing anyway.
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [122 bytes data]
* using HTTP/2
* h2 [:method: GET]
* h2 [:scheme: https]
* h2 [:authority: test.mydomain]
* h2 [:path: /]
* h2 [user-agent: curl/8.2.1]
* h2 [accept: */*]
* Using Stream ID: 1
} [5 bytes data]
> GET / HTTP/2
> Host: test.mydomain
> User-Agent: curl/8.2.1
> Accept: */*
>
{ [5 bytes data]
< HTTP/2 404

Http over port 80 works fine. But according to the dashboard, port 443 is routed to the service as well.
Another interesting note is the access log, where we can clearly see that HTTPS traffic ends up in "nowhere" (no service):
<IP_ADDRESS> - - [23/Jul/2024:20:13:04 +0000] "GET / HTTP/2.0" 404 19 "-" "-" 1691 "-" "-" 0ms
<IP_ADDRESS> - - [23/Jul/2024:20:13:07 +0000] "GET / HTTP/1.1" 200 266613 "-" "-" 1692 "test-router@file" "http://<IP_ADDRESS>:80" 442ms

Hello @sandnabba,
In the example you have shared:
entryPoints:
  web:
    address: ":80"
  websecure:
    address: ":443"

The two entryPoints are just ports on which Traefik will listen.
In the second example configuration you have shared:
http:
  services:
    test-service:
      loadBalancer:
        servers:
        - url: "http://<IP_ADDRESS>:80"
  routers:
    test-router:
      rule: "Host(`test.mydomain`)"
      service: "test-service"

The router has no TLS configuration and is not attached to any entryPoint.
At this point, Traefik is (only) able to handle HTTP (non-TLS) traffic on both entryPoints (the port is not determining if the request should be TLS or not).
In the third example you have shared:
entryPoints:
  web:
    address: ":80"
  websecure:
    address: ":443"
    asDefault: true
    http:
      tls: {}

This makes the router (that is not specifying the entryPoint to which it should be attached) to be attached to the websecure entryPoint, and inherit from it's http configuration, so becomes an HTTPS router.
It seems that your issue is related to a configuration issue and the GitHub issue tracker is dedicated to bug and feature requests.
For help on your configuration, please join our Community Forum and reach out to us on the Traefik v2 section.
We will close this issue accordingly but feel free to re-open it if you think that we missed something.

@rtribotte Well, yes and no. Yes it was a configuration problem that I sorted out before opening this issue. (And this particular issue seem to be very common if browsing the community forums, meaning that many people seem to misunderstand this).
So this is a feature request: I think Traefik should print a warning if a router is configured with incompatible entrypoints

Both in the application log (preferably a WARN)
as well as some notification on the dashboard (since according to the screenshot in my first post, it really looks like it should work).

Hello @sandnabba,

I think Traefik should print a warning if a router is configured with incompatible entrypoints

As explained by @rtribotte, EntryPoints in Traefik are essentially ports that can have one or more HTTP and/or HTTPS routers attached to them. This means that when a request is made to a specific EntryPoint, the corresponding HTTP or HTTPS router will handle the request, depending on whether the request is made over HTTP or HTTPS.
Based on this, it doesn’t seem necessary to print a warning as there are no incompatible entrypoints.