Error running test suite

I tired running the test suite with what I think are working credentials (I just made them and created a service account as a BigQuery User).  I see an error when running tests like the following:

GOOGLE_APPLICATION_CREDENTIALS=creds.json DASK_BIGQUERY_PROJECT_ID=dask-bigquery py.test dask_bigquery -s -v

Error:

*** TypeError: with_scopes_if_required() got an unexpected keyword argument 'default_scopes'

Interestingly, the data is created correctly.  It's only the read I see this error.  Additionally, I am able to read directly with pd.read_gbq

Thanks for the report @quasiben! Is it possible to share more of the traceback? I'm curious where with_scopes_if_required and default_scopes are coming from.
cc @ncclementi

I wonder if this is an authentication issue?  I'm running from my local laptop relying on the GOOGLE_APPLICATION_CREDENTIALS env var and a creds file (full error report in the details below)

```
>       ddf = read_gbq(project_id=project_id, dataset_id=dataset_id, table_id=table_id)
dask_bigquery/tests/test_core.py:56:

dask_bigquery/core.py:119: in read_gbq
with bigquery_clients(project_id) as (bq_client, bqs_client):
../../../miniconda3/envs/dask-dev/lib/python3.7/contextlib.py:112: in enter
return next(self.gen)
dask_bigquery/core.py:40: in bigquery_clients
client_info=bqstorage_client_info,
../../../miniconda3/envs/dask-dev/lib/python3.7/site-packages/google/cloud/bigquery_storage_v1/services/big_query_read/client.py:386: in init
or Transport == type(self).get_transport_class("grpc_asyncio")
../../../miniconda3/envs/dask-dev/lib/python3.7/site-packages/google/cloud/bigquery_storage_v1/services/big_query_read/transports/grpc.py:170: in init
("grpc.max_receive_message_length", -1),
../../../miniconda3/envs/dask-dev/lib/python3.7/site-packages/google/cloud/bigquery_storage_v1/services/big_query_read/transports/grpc.py:221: in create_channel
**kwargs,
../../../miniconda3/envs/dask-dev/lib/python3.7/site-packages/google/api_core/grpc_helpers.py:309: in create_channel
default_host=default_host,

credentials = <google.oauth2.service_account.Credentials object at 0x7fe8fd9bc850>, credentials_file = None
default_scopes = ('https://www.googleapis.com/auth/bigquery', 'https://www.googleapis.com/auth/bigquery.readonly', 'https://www.googleapis.com/auth/cloud-platform'), scopes = None, ssl_credentials = None, quota_project_id = None
default_host = 'bigquerystorage.googleapis.com'
def _create_composite_credentials(
    credentials=None,
    credentials_file=None,
    default_scopes=None,
    scopes=None,
    ssl_credentials=None,
    quota_project_id=None,
    default_host=None,
):
    """Create the composite credentials for secure channels.

    Args:
        credentials (google.auth.credentials.Credentials): The credentials. If
            not specified, then this function will attempt to ascertain the
            credentials from the environment using :func:`google.auth.default`.
        credentials_file (str): A file with credentials that can be loaded with
            :func:`google.auth.load_credentials_from_file`. This argument is
            mutually exclusive with credentials.
        default_scopes (Sequence[str]): A optional list of scopes needed for this
            service. These are only used when credentials are not specified and
            are passed to :func:`google.auth.default`.
        scopes (Sequence[str]): A optional list of scopes needed for this
            service. These are only used when credentials are not specified and
            are passed to :func:`google.auth.default`.
        ssl_credentials (grpc.ChannelCredentials): Optional SSL channel
            credentials. This can be used to specify different certificates.
        quota_project_id (str): An optional project to use for billing and quota.
        default_host (str): The default endpoint. e.g., "pubsub.googleapis.com".

    Returns:
        grpc.ChannelCredentials: The composed channel credentials object.

    Raises:
        google.api_core.DuplicateCredentialArgs: If both a credentials object and credentials_file are passed.
    """
    if credentials and credentials_file:
        raise exceptions.DuplicateCredentialArgs(
            "'credentials' and 'credentials_file' are mutually exclusive."
        )

    if credentials_file:
        credentials, _ = google.auth.load_credentials_from_file(
            credentials_file, scopes=scopes, default_scopes=default_scopes
        )
    elif credentials:
        credentials = google.auth.credentials.with_scopes_if_required(

          credentials, scopes=scopes, default_scopes=default_scopes

        )

E           TypeError: with_scopes_if_required() got an unexpected keyword argument 'default_scopes'
../../../miniconda3/envs/dask-dev/lib/python3.7/site-packages/google/api_core/grpc_helpers.py:233: TypeError
</details>

Interesting, I haven't run into this before.
@quasiben In your account when created it what role did you select?  If you're using a service account, when created you need to select the role of "BigQuery Admin" in the section "Grant this service account access to project".
Something else I'm noticing is that in my end if I inspect the credentials dict
credentials = service_account.Credentials.from_service_account_file(cred_fpath)
>>> credentials.__dict__

I see that
...
 '_scopes': None,
 '_default_scopes': None,
 ...

while in your end it seems they are
default_scopes = ('https://www.googleapis.com/auth/bigquery', 'https://www.googleapis.com/auth/bigquery.readonly', 'https://www.googleapis.com/auth/cloud-platform'), scopes = None, ssl_credentials = None, quota_project_id = None


Flat style badge

How about adding this to the available badges?
https://img.shields.io/badge/analytics-GA-blue.svg?style=flat-square
It looks like

Sure. Want to make a pull request? :)

Sure!

Is there.

Flat style does not work.


Cannot clean schema with FileStream table in MS SQL

Which version and edition of Flyway are you using?

Flyway Community 6.4.1

If this is not the latest version, can you reproduce the issue with the latest one as well?
(Many bugs are fixed in newer releases and upgrading will often resolve the issue)

The bug is also present in flyway 6.4.0

Which client are you using? (Command-line, Java API, Maven plugin, Gradle plugin)

Command-line

Which database are you using (type & version)?

Bug happens with MS SQL 2019 Developper Edition and MS SQL 2016 Standard Edition

Which operating system are you using?

Windows 10 and Windows Server 2012

What did you do?
(Please include the content causing the issue, any relevant configuration settings, the SQL statement that failed (if relevant) and the command you ran.)

Enabled FileStream in MS SQL Server
Created a FileGroup for FileStream
Created a Table with FileStream in a schema :
CREATE TABLE [sadip].TableFileStreamTest (  [Guid] [uniqueidentifier] ROWGUIDCOL NOT NULL UNIQUE,   [Id] INTEGER UNIQUE,  [Binaire] VARBINARY(MAX) FILESTREAM NULL )

When I delete the table from SSMS it works fine.
I run flyway trying to clean schema with following command (don't mind password it's on localhost) :
C:\Dev\SADIP\SADIP\SADIP.Sql\Deploiement\flyway-6.4.1\flyway.cmd -url=jdbc:sqlserver://localhost;databaseName=SADIP -user=APP_SADIP -password=sadip -validateOnMigrate=false -outOfOrder=true -schemas=sadip -table=sadip_flyway_history -placeholders.environnement=local -locations=filesystem:C:\Dev\SADIP\SADIP\SADIP.Sql\Deploiement/sql/sadip clean
What did you expect to see?

Cleaning schema and dropping all objects as usual

What did you see instead?

Flyway Community Edition 6.4.1 by Redgate
Database: jdbc:sqlserver://localhost:1433;useBulkCopyForBatchInsert=false;cancelQueryTimeout=-1;sslProtocol=TLS;jaasConfigurationName=SQLJDBCDriver;statementPoolingCacheSize=0;serverPreparedStatementDiscardThreshold=10;enablePrepareOnFirstPreparedStatementCall=false;fips=false;socketTimeout=0;authentication=NotSpecified;authenticationScheme=nativeAuthentication;xopenStates=false;sendTimeAsDatetime=true;trustStoreType=JKS;trustServerCertificate=false;TransparentNetworkIPResolution=true;serverNameAsACE=false;sendStringParametersAsUnicode=true;selectMethod=direct;responseBuffering=adaptive;queryTimeout=-1;packetSize=8000;multiSubnetFailover=false;loginTimeout=15;lockTimeout=-1;lastUpdateCount=true;encrypt=false;disableStatementPooling=true;databaseName=SADIP;columnEncryptionSetting=Disabled;applicationName=Microsoft JDBC Driver for SQL Server;applicationIntent=readwrite; (Microsoft SQL Server 15.0)
Successfully dropped pre-schema database level objects (execution time 00:00.001s)
WARNING: Unable to drop schema [sadip]. Attempting clean instead...
ERROR:
Unable to clean schema [sadip]

SQL State  : S0002
Error Code : 5505
Message    : Une table comportant des colonnes FILESTREAM doit avoir une colonne unique non Null avec la propriété ROWGUIDCOL.

The message say A table with FILESTREAM columns must have a unique not null column with ROWGUIDCOL property but my SQL seems to meet the requirements. Also I can delete the table with SSMS w/o any problem.

The error occurs when we come to clean constraints; the ROWGUIDCOL UNIQUE constraint can't be dropped independently of the table. A fix will be in the next release.


Make talkback speak out decimal values appropriately (EXPOSUREAPP-4665)

Talkback only works appropriately in German with decimal numbers when setting the text as a localized spannable.
This implementation provides a helper function that converts a string to a localized spannable.

Code looks good. Hard to test - is it applied anywhere, yet?

Code looks good. Hard to test - is it applied anywhere, yet?

@ralfgehrer Unfortunately it is not applied yet so you can't test it yet.

@ralfgehrer Unfortunately it is not applied yet so you can't test it yet.


Add editing property

This gives you the ability to set the editing state via props if desired.

Sorry, I've merged pull request #3 first. Hope nothing breaks.


WIP URI slashes

Description
Fixes some issues related to inconsistencies with slashes in URLs.
Checklist:

[X] Securely signed commits
[X] Component(s) with PHPdocs
[ ] Unit testing, with >80% coverage
[X] User guide updated
[X] Conforms to style guide

This code (in URI::filterPath()) has me scratching my head:
		// Fix up some leading slash edge cases...
		if (strpos($orig, './') === 0)
		{
			$path = '/' . $path;
		}
		if (strpos($orig, '../') === 0)
		{
			$path = '/' . $path;
		}

Paths like "./foo/bar" become "/./foo/bar"? But the rest of the code makes me think the intent was that getPath() would always return a preceding slash.

Consider this test too:
public function testConstructorSetsAllParts()
	{
		$uri = new URI('http://username:password@hostname:9090/path?arg=value#anchor');
...
		$this->assertEquals('/path', $uri->getPath());

It seems to me that the intent was for this method to return a preceding slash. I wouldn't have made that choice myself (maybe it is to ensure redirects, etc., aren't relative?) but I think our solution should aim for that.

Actually, the tests are all over the place. It is hard to tell what the actual intent is. Either way (enforcing a preceding slash or removing all preceding slashes) will result in test failures as it is now.
@lonnieezell Any chance you could weigh in?

Alright, well, I feel like a heel. Here's the comment on getPath() which pretty much debunks everything I've said so far:

The path can either be empty or absolute (starting with a slash) or
rootless (not starting with a slash). Implementations MUST support all
three syntaxes.
Normally, the empty path "" and absolute path "/" are considered equal as
defined in RFC 7230 Section 2.7.3. But this method MUST NOT automatically
do this normalization because in contexts with a trimmed base path, e.g.
the front controller, this difference becomes significant. It's the task
of the user to handle both "" and "/".
The value returned MUST be percent-encoded, but MUST NOT double-encode
any characters. To determine what characters to encode, please refer to
RFC 3986, Sections 2 and 3.3.
As an example, if the value should include a slash ("/") not intended as
delimiter between path segments, that value MUST be passed in encoded
form (e.g., "%2F") to the instance.
@see    https://tools.ietf.org/html/rfc3986#section-2
@see    https://tools.ietf.org/html/rfc3986#section-3.3

I will have to take a fresh look at this tomorrow.

Given my previous comment, I don't think setPath() is the problem. I do still think we need to encourage a trailing slash on baseURL and our current handling does not do a good job handling scenarios where those differ. I will open a new PR to start on this.


test-app-10 to 0.0.1

Promote test-app-10 to version 0.0.1

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approvers:
If they are not already assigned, you can assign the PR to them by writing /assign  in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


fix: corrected some lint issues, and got Karma and all mocha tests passing

I fixed a few of the tslint error messages in hopes to get karma up and running.

Also fixed karma and the mocha test!!!

To run the tests, you need to run karma start. not npm run. this is still broken due to #6

I have added a chat room on gitter for this repo so it is easier to chat. https://gitter.im/aspnetcorespa/Lobby?utm_source=share-link&utm_medium=link&utm_campaign=share-link


ValidationAttribute - How To?

Can someone point me to documentation on ValidationAttributes?
I've applied a DataTypeAttribute to a data member and it doesn't show up anywhere?
Reviewing the InputTagHelper I'm unable to figure out how to add additional validation related 'things' in the Startup.Config().
Any help would be appreciated.

The MVC 5 docs on validation should still apply for the most part: http://www.asp.net/mvc/overview/getting-started/introduction/adding-validation
Basically you just add data annotations to your model properties and they should get honored by the HTML input and tag helpers. To show validation errors use the ValidationSummary and ValidationMessage HTML helpers.

That only contains 11 ValidationAttributes.  It does not contain them all (i.e. DataTypeAttribute).

Since it's been impossible for me to step through the open source code  while debugging I'd love to know how the validation attributes get added.  If need be I can add additional ones.


Website not loading AI

I attempted to use uttt.ai on Microsoft Edge, Firefox on Windows, Firefox on Linux and Firefox on Android. I have Javascript enabled. Each time I get a modal saying that my browser is not supported.

I'm working on a simpler and lighter version of the website that may help in the future.
In the meantime, you can always build and use it locally on your computer.


bigtable: support to scale instance nodes

With the bigtable.InstanceAdminClient, users should be able to both get and change the number of nodes an instance has.

@garye anything i could do to help here? Happy to take assignee, but this initially looks like we need a corresponding bigtable RPC, so i'd need some coaching if i took a shot at this ticket.

Sorry that we haven't worked on this yet. I'll try to get to it in the next week or so. Thanks for your patience.

@garye is this still an issue?

Yep! Closed by:
https://github.com/GoogleCloudPlatform/google-cloud-go/commit/c0822307f5df87d8243dc9e4babcc68dd7b1ea31
https://github.com/GoogleCloudPlatform/google-cloud-go/commit/70c2de4bba3c3dc0bd2927e3e3739704a5693e69


feat: doc comments on notation

This makes it possible to write doc comments on infixl and notation declarations, and in fact makes them required when linter.missingDocs is enabled. (The reasoning is that these are just ways of writing a syntax, and we want to put documentation on syntax since that's what usually shows up in hovers.)

Merged manually because of update-stage0, and added update-stage0 after last commit to make sure we restore stdlib_flags.h


HTTP connection and request timeouts

I've added support for connection and request timeouts to http/request. In this branch there are now three timeouts:

the existing socket-timeout, which limits pool generation time
connection-timeout, which limits time to establish a connection post pool-generation
request-timeout, which limits time for response arrival post connection-establishment

I added these because I was getting deferreds that never yielded a value for unavailable hosts, and I couldn't add a "hard" cumulative request timeout via middleware (without violating the pool contract).
In the cases of connection and request timeouts, I dispose the pooled connection because we won't know when data might be coming back from the server. I wasn't sure if there was a more intelligent way to handle this.
Additionally I've introduced reporting of a :socket-time separate from :connection-time which you may want to use.
I was also wondering if socket-timeout should possibly be renamed to something like pool-timeout? Since "socket timeout" typically refers to a limit on the amount of time without data flowing over the socket.
Let me know if there are any adjustments I can make.

Can I get a single merged commit for all these changes?  I'm having a hard time getting a full picture of what's changed.

Yeah I will clean this up and make the suggested changes. I need to head out right now but I can probably take a look at this later today.
On February 1, 2015 at 2:51:36 PM, Zach Tellman<EMAIL_ADDRESS>wrote:
Can I get a single merged commit for all these changes? I'm having a hard time getting a full picture of what's changed.
—
Reply to this email directly or view it on GitHub.

See cleaned-up PR #140


Disable builds on GKE and remove related files

We have CI builds running on CICO (ci.centos.org) and GKE (fabric8cd).
We DO NOT need both the builds and we should disable builds on GKE.
Tasks -

[ ] Remove/Disable GKE (fabric8cd) github webhook
[ ] Remove Jenkinsfile
[ ] Remove deploy/release.groovy file

@jarifibrahim let us get this done

@jarifibrahim I'm totally up for it as we should move completely away from the GKE build system and free up their resources. I'd say, let's do it.
WDYT @michaelkleinhenz @nimishamukherjee ?


Adopt PEP 563, 585, and 604 for src/lightning/pytorch

What does this PR do?

Try to pyupgrade src/lightning/pytorch to adopt PEP 563, 585, and 604. Ran the following and then manually fixed src/lightning/pytorch/loggers/neptune.py
isort -a "from __future__ import annotations" --append-only src/lightning/pytorch
pyupgrade --py37-plus $(find src/lightning/pytorch -name '*.py')

Part of #11205

Before submitting

[x] Was this discussed/agreed via a GitHub issue? (not for typos and docs)
[x] Did you read the contributor guideline, Pull Request section?
[x] Did you make sure your PR does only one thing, instead of bundling different changes together?
Did you make sure to update the documentation with your changes? (if necessary)
Did you write any new necessary tests? (not for typos and docs)
[ ] Did you verify new and existing tests pass locally with your changes?
Did you list all the breaking changes introduced by this pull request?
Did you update the CHANGELOG? (not for typos, docs, test updates, or minor internal changes/refactors)

PR review
Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the review guidelines. In short, see the following bullet-list:

Reviewer checklist

[ ] Is this pull request ready for review? (if not, please submit in draft mode)
[ ] Check that all items from Before submitting are resolved
[ ] Make sure the title is self-explanatory and the description concisely explains the PR
[ ] Add labels and milestones (and optionally projects) to the PR so it can be classified

For a draft pull request, the test_cli.py tests are run? I couldn't find them.

For a draft pull request, the test_cli.py tests are run? I couldn't find them.

They don't. I marked the pull request as ready and added WIP to the title.


fix(button): theme corp mobile component primary button color changed…

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 82.957%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
8766

Relevant Lines:
9694

💛  - Coveralls


Critical grpc failure when doing large operation

Environment details

OS: Whatever OS is used in the node docker image
Node.js version: Whatever version is used in the node docker image
npm version: 8.19.1
@google-cloud/firestore version: 6.4.1

Steps to reproduce
Seems to be related to this: https://github.com/grpc/grpc-node/issues/1769
On my production server, I am running high throughput code and my kubernetes pod seems to randomly die out of nowhere. There is no indication or error that anything occurred. While trying to run that same code locally, I get the error shown below. I tried to see if any high throughput operation would reproduce the error and I found that the sample code below did.
const { docs } = await firestore.collection('InventoryItems')
  .where('UserId', '==', userId)
  .select().get();
await Promise.all(docs.map(async (doc) => firestore.doc(`InventoryItems/${doc.id}`).delete()));

However, this code produced an error that I was able to catch. On production, the error seems to be silent, and doesn't get caught in try/catch blocks.
Error: 13 INTERNAL: Received RST_STREAM with code 2 triggered by internal client error: 80BA10EC01000000:error:0A0003FC:SSL routines:ssl3_read_bytes:sslv3 alert bad record mac:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1584:SSL alert number 20

    at Object.callErrorFromStatus (/Users/jakeleventhal/Code/Ecominate/node_modules/@grpc/grpc-js/src/call.ts:81:17)
    at Object.onReceiveStatus (/Users/jakeleventhal/Code/Ecominate/node_modules/@grpc/grpc-js/src/client.ts:358:36)
    at Object.onReceiveStatus (/Users/jakeleventhal/Code/Ecominate/node_modules/@grpc/grpc-js/src/client-interceptors.ts:462:34)
    at Object.onReceiveStatus (/Users/jakeleventhal/Code/Ecominate/node_modules/@grpc/grpc-js/src/client-interceptors.ts:424:48)
    at /Users/jakeleventhal/Code/Ecominate/node_modules/@grpc/grpc-js/src/call-stream.ts:330:24
    at processTicksAndRejections (node:internal/process/task_queues:77:11)
for call at
    at ServiceClientImpl.makeUnaryRequest (/Users/jakeleventhal/Code/Ecominate/node_modules/@grpc/grpc-js/src/client.ts:328:30)
    at ServiceClientImpl.<anonymous> (/Users/jakeleventhal/Code/Ecominate/node_modules/@grpc/grpc-js/src/make-client.ts:189:15)
    at /Users/jakeleventhal/Code/Ecominate/node_modules/@google-cloud/firestore/build/src/v1/firestore_client.js:225:29
    at /Users/jakeleventhal/Code/Ecominate/node_modules/google-gax/src/normalCalls/timeout.ts:54:13
    at repeat (/Users/jakeleventhal/Code/Ecominate/node_modules/google-gax/src/normalCalls/retries.ts:104:19)
    at /Users/jakeleventhal/Code/Ecominate/node_modules/google-gax/src/normalCalls/retries.ts:144:7
    at OngoingCallPromise.call (/Users/jakeleventhal/Code/Ecominate/node_modules/google-gax/src/call.ts:81:23)
    at NormalApiCaller.call (/Users/jakeleventhal/Code/Ecominate/node_modules/google-gax/src/normalCalls/normalApiCaller.ts:43:15)
    at /Users/jakeleventhal/Code/Ecominate/node_modules/google-gax/src/createApiCall.ts:118:26
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
Caused by: Error:
    at WriteBatch.commit (/Users/jakeleventhal/Code/Ecominate/node_modules/@google-cloud/firestore/build/src/write-batch.js:433:23)
    at DocumentReference.delete (/Users/jakeleventhal/Code/Ecominate/node_modules/@google-cloud/firestore/build/src/reference.js:351:14)
    at /Users/jakeleventhal/Code/Ecominate/apps/onboarder/src/onboarder.ts:46:97
    at step (/Users/jakeleventhal/Code/Ecominate/apps/onboarder/src/onboarder.ts:33:23)
    at Object.next (/Users/jakeleventhal/Code/Ecominate/apps/onboarder/src/onboarder.ts:14:53)
    at /Users/jakeleventhal/Code/Ecominate/apps/onboarder/src/onboarder.ts:8:71
    at new Promise (<anonymous>)
    at __awaiter (/Users/jakeleventhal/Code/Ecominate/apps/onboarder/src/onboarder.ts:4:12)
    at /Users/jakeleventhal/Code/Ecominate/apps/onboarder/src/onboarder.ts:46:44
    at Array.map (<anonymous>)
    at /Users/jakeleventhal/Code/Ecominate/apps/onboarder/src/onboarder.ts:46:30
    at step (/Users/jakeleventhal/Code/Ecominate/apps/onboarder/src/onboarder.ts:33:23)
    at Object.next (/Users/jakeleventhal/Code/Ecominate/apps/onboarder/src/onboarder.ts:14:53)
    at fulfilled (/Users/jakeleventhal/Code/Ecominate/apps/onboarder/src/onboarder.ts:5:58)
    at processTicksAndRejections (node:internal/process/task_queues:95:5) {
  code: 13,
  details: 'Received RST_STREAM with code 2 triggered by internal client error: 80BA10EC01000000:error:0A0003FC:SSL routines:ssl3_read_bytes:sslv3 alert bad record mac:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1584:SSL alert number 20\n',
  metadata: Metadata { internalRepr: Map(0) {}, options: {} },
  note: 'Exception occurred in retry method that was not classified as transient'
}
```”

Interestingly, when I change .delete() to .update({ test: 1}), I don’t get the error. I get the error again when I change it back to .delete()

Hi @samborambo305 , Thanks for reporting.
There are a few things to consider:

Make sure you're using the latest versions of @grpc/grpc-js and @google-cloud/firestore
Best practices  is to not exceed 1 write per second to a document. Other limits here.
https://cloud.google.com/firestore/docs/best-practices#updates_to_a_single_document

Updates to a single document
As you design your app, consider how quickly your app updates single documents. The best way to characterize your workload's performance is to perform load testing. The exact maximum rate that an app can update a single document depends highly on the workload. Factors include the write rate, contention among requests, and the number affected indexes.
A document write operation updates the document and any associated indexes, and Firestore synchronously applies the write operation across a quorum of replicas. At high enough write rates, the database will start to encounter contention, higher latency, or other errors.

https://cloud.google.com/firestore/docs/best-practices#high_read_write_and_delete_rates_to_a_narrow_document_range

This issue is known as hotspotting, and your application can experience hotspotting if it does any of the following:

Creates new documents at a very high rate and allocates its own monotonically increasing IDs.
Firestore allocates document IDs using a scatter algorithm. You should not encounter hotspotting on writes if you create new documents using automatic document IDs.
Creates new documents at a high rate in a collection with few documents.
Creates new documents with a monotonically increasing field, like a timestamp, at a very high rate.
Deletes documents in a collection at a high rate.
Writes to the database at a very high rate without gradually increasing traffic.

You should also consider making a batch (each batch can perform up to 500 operations). Something like

batch.delete(doc1);
batch.delete(doc2);
// ...
batch.delete(doc100);
await batch.commit();

See: https://firebase.google.com/docs/firestore/manage-data/transactions#batched-writes

For deletion of all documents / subcollections, please also consider https://googleapis.dev/nodejs/firestore/latest/Firestore.html#recursiveDelete

due to the fact that firestore doesn’t support infinitely sized atomic updates, I use the class at the bottom of this comment. essentially I am only using batch updates. this should address all the concerns for hot-spotting, throttling, etc.
the way i am using the class in my code that experiences the issue is like this
const batchUpdates = {};

LARGE_LIST.forEach((item) => {
  // preprocessing
  …

  BatchUpdates.set(item);
})

await BatchUpdates.commit();

i also ensured i am on the latest @google-cloud/firstore and @grpc/grpc-js
// This is the class I'm using

import firestore from './Firestore';

// The max number of writes per batch that firestore allows
// TODO: Remove the batch limit once https://issuetracker.google.com/issues/180813634 is resolved
export const BATCH_LIMIT = 75;

interface BatchUpdateWithPayload {
  payload: any;
  type: 'update' | 'set';
}

interface DeleteBatchUpdate {
  type: 'delete';
}

interface BatchUpdateObject {
  [docId: string]: BatchUpdateWithPayload | DeleteBatchUpdate;
}

export class BatchUpdates {
  static update = (batchUpdates: BatchUpdateObject, docId, updatePayload) => {
    // If the payload is not empty
    if (!_.isEmpty(updatePayload)) {
      // If the doc id has not been encountered before
      if (!(docId in batchUpdates)) {
        batchUpdates[docId] = {
          payload: updatePayload,
          type: 'update'
        };
      } else {
        // Update the existing payload
        (batchUpdates[docId] as BatchUpdateWithPayload).payload = {
          ...(batchUpdates[docId] as BatchUpdateWithPayload).payload,
          ...updatePayload
        };
      }
    }
  };

  static set = (batchUpdates: BatchUpdateObject, docId, setPayload) => {
    batchUpdates[docId] = {
      payload: setPayload,
      type: 'set'
    };
  };

  static delete = (batchUpdates: BatchUpdateObject, docId) => {
    batchUpdates[docId] = {
      type: 'delete'
    };
  };

  static commit = async (batchUpdates: BatchUpdateObject) => {
    const updateEntries = Object.entries(batchUpdates);
    if (!updateEntries.length) {
      return;
    }

    // The number of batches to save in parallel
    const NUM_PARALLEL_BATCHES = 10;

    // Iterate synchronously to help with memory usage
    const increment = BATCH_LIMIT * NUM_PARALLEL_BATCHES;
    for (let i = 0; i < updateEntries.length; i += increment) {
      const entriesSlice = updateEntries.slice(i, i + increment);

      // Construct an array of batches that can hold BATCH_LIMIT updates each
      const batches = [...Array(Math.floor(entriesSlice.length / BATCH_LIMIT + 1))].map(() => firestore.batch());

      // Add each batch update to the correct batch index
      entriesSlice.forEach(([docId, update], j) => {
        const batchIndex = Math.floor(j / BATCH_LIMIT);
        const firestoreDoc = firestore.doc(docId);

        switch (update.type) {
          case 'update':
            batches[batchIndex].update(firestoreDoc, update.payload);
            break;
          case 'set':
            batches[batchIndex].set(firestoreDoc, update.payload);
            break;
          case 'delete':
            batches[batchIndex].delete(firestoreDoc);
            break;
          default:
            break;
        }
      });

      // Commit the batch updates
      await Promise.all(batches.map(async (batch) => batch.commit()));
    }
  };
}

due to the fact that firestore doesn’t support infinitely sized atomic updates, i use the class in the attached file. essentially I am only using batch updates. this should address all the concerns for hot-spotting, throttling, etc.
the way i am using the class in my code that experiences the issue is like this
const batchUpdates = {};

LARGE_LIST.forEach((item) => {
  // preprocessing
  …

  BatchUpdates.set(item);
})

await BatchUpdates.commit();

BatchUpdates.ts.zip

I added throttling to bulkWriter:
firestore.bulkWriter({ throttling: { maxOpsPerSecond: 500 }})

and this error was gone.

I will be forced to move away from Firestore because of this issue.

Closing this issue as it's a duplicate of https://github.com/googleapis/nodejs-firestore/issues/1023 and raising the issue internally again.


Time out when running naive-mpc-optim

I've setup the solcast integration, with which I have the following for sensor.solcast_24hrs_forecast:
[3367, 3430, 3443, 3428, 3342, 3206, 3022, 2782, 2486, 2151, 1776, 1372, 959, 523, 214, 80, 31, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 50, 167, 348, 606, 903, 1206, 1501, 1790, 2060, 2333, 2601, 2775]
I've setup the following shell_command:
  mpc_optim: "curl -i -H \"Content-Type: application/json\" -X POST -d '{\"pv_power_forecast\":{{states('sensor.solcast_24hrs_forecast')}}}' http://localhost:5000/action/naive-mpc-optim"
Which returned the following error when it ran:
Logger: homeassistant.components.shell_command
Source: components/shell_command/__init__.py:86
Integration: Shell Command (documentation, issues)
First occurred: 12:53:05 (1 occurrences)
Last logged: 12:53:05

Timed out running command: `curl -i -H "Content-Type: application/json" -X POST -d '{"pv_power_forecast":{{states('sensor.solcast_24hrs_forecast')}}}' http://localhost:5000/action/naive-mpc-optim`, after: 60s
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/shell_command/__init__.py", line 87, in async_service_handler
    stdout_data, stderr_data = await process.communicate()
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/subprocess.py", line 198, in communicate
    stdin, stdout, stderr = await tasks.gather(stdin, stdout, stderr)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/subprocess.py", line 178, in _read_stream
    output = await stream.read()
             ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/streams.py", line 683, in read
    block = await self.read(self._limit)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/streams.py", line 690, in read
    await self._wait_for_data('read')
  File "/usr/local/lib/python3.11/asyncio/streams.py", line 522, in _wait_for_data
    await self._waiter
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/shell_command/__init__.py", line 86, in async_service_handler
    async with async_timeout.timeout(COMMAND_TIMEOUT):
  File "/usr/local/lib/python3.11/site-packages/async_timeout/__init__.py", line 129, in __aexit__
    self._do_exit(exc_type)
  File "/usr/local/lib/python3.11/site-packages/async_timeout/__init__.py", line 212, in _do_exit
    raise asyncio.TimeoutError
TimeoutError

This seems like some kind of connection problem. The shell command is not able to connect with the add-on server at port 5000. Try just to ping it, are you able to do that?

Not really, sorry. Can you guide me here please?

What's your configuration? Are you using HA OS?
Home Assistant 2023.7.2
Supervisor 2023.07.1
Operating System 10.3
Interface: 20230705.1 - latest

So if you try right now EMHASS is working fine?

So if you try right now EMHASS is working fine?

Yes: all automations are triggering as expected and I can see the forecasts on the UI

Ok this means that from time to time you may have some connection issues but they have nothing to do with EMHASS

probably yes. Will close this ticket


misc fixes

@nicknitewolf Hey I made the requested corrections, thanks for the second set of eyes!

Looks good now, but I bet there are other bad math in this data set. GPT3 is notoriously bad at math


An introspective metadata endpoint

It would be great to have an introspective metadata endpoint, like /about or /info to retrieve all relevant metadata useful for debugging.
This could include:

OTEAPI-services version.
OTEAPI Core version.
List of installed OTEAPI plugins.
List of installed OTEAPI plugin versions.
List of endpoint paths/routes.
Full list of (non secret) configuration settings.
Redis instance information.
...

Regarding exposing the metadata - we need to ensure this information should not be publicly available. Knowing which versions of dependent libraries are running can be useful info for people with harmful intents. If we create an admin/ -endpoint and encourage not to publicly expose it, we can use this for extracting metadata, statistics, etc...


Incorrect package in Storybook integration guide

📃 Documentation Issue
Have you read the Contributing Guidelines on issues?

[X] Yes

Description
Doc URL: Storybook Integration (Deployed) and markdown
While following the getting started guide, suggested command npx test-storybook --eject fails with could not determine executable to run. I believe this command is incorrect
test-storybook is a package that hasn't been updated in ~7 years
Getting Started Step 1 links to Storybook's instructions, which mentions @storybook/test-runner. Switching to npx that package outputs what I assume is the correct behavior per the existing docs, e.g.: "The above should have created a file in the root directory, test-runner-jest.config.js"
Suggested Fix
Update instructions to use storybook's package
- npx test-storybook --eject
+ npx @storybook/test-runner --eject

Documentation Checklist

[x] Typos
[x] Grammar
[x] Incorrect information
[x] Missing information

Hey! Thanks for pointing this out! The current guide works if the package is already installed since it's finding the test-storybook 'binary' installed by @storybook/test-runner at /node_modules/.bin/test-storybook. I think that's a good point though and this should be updated. Let me put a PR together.


Bump API Version (#476)

Forgot to bump the API version.
Fixes #476.

Codecov Report

Merging #513 into master will decrease coverage by 0.14%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             master    #513      +/-   ##
===========================================
- Coverage     82.55%   82.4%   -0.15%
  Complexity      896     896
===========================================
  Files           144     144
  Lines          2224    2228       +4
  Branches        272     272
===========================================
  Hits           1836    1836
- Misses          257     261       +4
  Partials        131     131

Impacted Files
Coverage Δ
Complexity Δ

web-ui/src/client/app/components/editor.jsx
0% <0%> (ø)
0% <0%> (ø)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5d5dea3...b90a954. Read the comment docs.

:+1:

:+1:

:+1:


No longer collecting iteration results automatically.

Fixes #784

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Coverage remained the same at 100.0% when pulling eba7f39459edf923ffdb2565f9509117e3dba07c on rep_opt into dfb6455054caf0c5e4d90aec1989aeffddcf1ab6 on master.


Fix doubled translation from dragging layers sharing an upstream Transform node

Hello,
This fixes above issue, demo:
https://github.com/GraphiteEditor/Graphite/assets/51503822/70b7fccb-8520-45d5-a599-0b4f8ce24ecf
Closes #1529

@0HyperCube you're better than me at understanding this part of the code and the implications of this change. Could you please see if this looks good to go? Thanks.

@shipp02 please leave a comment in the original issue so I can assign it to you.

I finally got a chance to do some QA testing on this. The first thing I noticed is that Dragging individual layers is completely broken. Just draw two layers and try dragging each of them on after the other. This is probably caused by your choice to store the transform in the Select tool's struct. I'd avoid attempting to persist state since this should be a stateless problem as far as I can tell.

@Keavon persisting state is necessary to implement the first of my suggested solutions from the issue:

In select_tool.rs, instead of applying a delta to the current transform, store the transform at the start of the drag and compute the new transform based on that (similar to how dragging the selection bounds works)

Thank you for confirming my approach!
I was in the middle of finals hence the lack of a response from me in the
last week.
I will fix the bug @Keavon found while he was running QA.
On Fri, Mar 22, 2024, 10:02 AM 0HyperCube @.***> wrote:

@Keavon https://github.com/Keavon persisting state is necessary to
implement the first of my suggested solutions from the issue:
In select_tool.rs, instead of applying a delta to the current transform,
store the transform at the start of the drag and compute the new transform
based on that (similar to how dragging the selection bounds works)
—
Reply to this email directly, view it on GitHub
https://github.com/GraphiteEditor/Graphite/pull/1681#issuecomment-2015522280,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AMI6FTXCBTQET5GTNIW5NJDYZRPZZAVCNFSM6AAAAABERUTULCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJVGUZDEMRYGA
.
You are receiving this because you were mentioned.Message ID:
@.***>

@shipp02 just checking if you've been able to continue addressing this.

I was able to reproduce the regression you mentioned but wasn't sure how to fix it.

I'm going to close this for now but feel free to reopen it if you're able to continue making progress.


Link "to execute a failover" seems to be dead

Link "to execute a failover" seems to be dead. Just points to article search overview page.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: b14fdc12-b544-951b-7275-f1e41eb72e1f
Version Independent ID: 84cd2dac-d190-5a20-7dfb-71703016b43b
Content: Business continuity and disaster recovery for Dynamics 365 SaaS apps - Power Platform
Content Source: power-platform/admin/business-continuity-disaster-recovery.md
Service: power-platform
Sub-service: admin
GitHub Login: @MicroSri
Microsoft Alias: sriknair

Thanks for your comment, @peta. This issue has been assigned to the doc author for review.


Incorrect validation message when attempting to login after a loooong time/with wrong pw

Not sure, if it's related to an old password that was changed in the meantime...
OS: macOS High Sierra 10.13.5
Ghost-Desktop: 1.6.0
This happened when after I updated to 1.6 from 1.5.
When trying to login after a long time not being logged in, I see this entry screen:

Clicking the Retry button here didn't change anything. I would expect that the message tells me Wrong password or similar. Re-entering my correct password then worked fine. Again, this being said that I'm not 100% sure if my password here ever changed.
A user reported the same behaviour but encountered additional logs:

I also have. Resetting the password does not help.

I reset the password, made a new user, nothing helped. Through the browser there is access, through the desktop application shows an error.

👍 Just encountered this. Entered a password that was old, just had a progress loader for ages with no feedback. Was very confused as to what the error was, until I realised it was just the wrong password.

My password is correct. I go with him from another computer. There everything works.


Spec changes testing - Changed resolution options.

When you were moving over to using $refs, an option got changed.
In "/state/{state}/renewables/current", the resolution option used to only include 30. Now it includes 30 and 5.
Can you confirm if this was an intended change or not?
Thanks,
Carl.

It's intended.


Verisure not receiving state updates

Suddenly my verisure components stopped receiving updates.
Have no more information right now. Will investigate in the CET evening.

Could get som info from error log. Seems to be connected to when MyPages goes down for maintenance.
First I got a couple of these:
16-02-17 06:09:31 homeassistant.components.verisure: Caught connection error <_sre.SRE_Match object; span=(176, 225), match='<title>My Pages - Maintenance -  Verisure</title>>, tries to reconnect

Then:
16-02-17 06:10:03 homeassistant.components.verisure: Caught connection error <_sre.SRE_Match object; span=(196, 247), match='<title>Choose country - My Pages - Verisure</titl>, tries to reconnect
16-02-17 06:10:06 homeassistant.components.verisure: Caught connection error <_sre.SRE_Match object; span=(196, 247), match='<title>Choose country - My Pages - Verisure</titl>, tries to reconnect
16-02-17 06:10:06 homeassistant.components.verisure: Caught connection error <_sre.SRE_Match object; span=(196, 247), match='<title>Choose country - My Pages - Verisure</titl>, tries to reconnect
16-02-17 06:10:07 homeassistant.components.verisure: Could not login to Verisure mypages, The email address or password you entered is incorrect. Please try again.
16-02-17 06:10:07 homeassistant.components.verisure: Could not login to Verisure mypages, The email address or password you entered is incorrect. Please try again.

After that a huge stack trace:
16-02-17 06:10:09 homeassistant.core: BusHandler:Exception doing job
Traceback (most recent call last):
  File "/home/per/.homeassistant/lib/verisure/session.py", line 167, in json_to_dict
    return json.loads(doc)
  File "/usr/lib/python3.5/json/__init__.py", line 319, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.5/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 34 column 1 (char 33)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/per/home-assistant/homeassistant/components/verisure.py", line 164, in update_component
    for overview in get_function():
  File "/home/per/.homeassistant/lib/verisure/devices/climate.py", line 21, in get
    status = self._session.get(OVERVIEW_URL)
  File "/home/per/.homeassistant/lib/verisure/session.py", line 112, in get
    return self.json_to_dict(UNESCAPE(response.text))
  File "/home/per/.homeassistant/lib/verisure/session.py", line 173, in json_to_dict
    'Error: {0} - Data: {1}'.format(e, doc)))
  File "/home/per/.homeassistant/lib/verisure/session.py", line 201, in raise_response_error
    raise ResponseError(match)
verisure.session.ResponseError: <_sre.SRE_Match object; span=(196, 247), match='<title>Choose country - My Pages - Verisure</titl>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 376, in _make_request
    httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 378, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.5/http/client.py", line 1174, in getresponse
    response.begin()
  File "/usr/lib/python3.5/http/client.py", line 282, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.5/http/client.py", line 243, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.5/socket.py", line 575, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.5/ssl.py", line 924, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.5/ssl.py", line 786, in read
    return self._sslobj.read(len, buffer)
  File "/usr/lib/python3.5/ssl.py", line 570, in read
    v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/requests/adapters.py", line 376, in send
    timeout=timeout
  File "/usr/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 609, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib/python3.5/site-packages/requests/packages/urllib3/util/retry.py", line 247, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3.5/site-packages/requests/packages/urllib3/packages/six.py", line 310, in reraise
    raise value
  File "/usr/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
    body=body, headers=headers)
  File "/usr/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 380, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/usr/lib/python3.5/site-packages/requests/packages/urllib3/connectionpool.py", line 308, in _raise_timeout
    raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='mypages.verisure.com', port=443): Read timed out. (read timeout=3)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/per/home-assistant/homeassistant/core.py", line 779, in job_handler
    func(arg)
  File "/home/per/home-assistant/homeassistant/helpers/event.py", line 183, in pattern_time_change_listener
    action(now)
  File "/home/per/home-assistant/homeassistant/helpers/entity_component.py", line 172, in _update_entity_states
    entity.update_ha_state(True)
  File "/home/per/home-assistant/homeassistant/helpers/entity.py", line 146, in update_ha_state
    self.update()
  File "/home/per/home-assistant/homeassistant/components/sensor/verisure.py", line 78, in update
    verisure.update_climate()
  File "/home/per/home-assistant/homeassistant/util/__init__.py", line 275, in wrapper
    result = method(*args, **kwargs)
  File "/home/per/home-assistant/homeassistant/components/verisure.py", line 139, in update_climate
    update_component(MY_PAGES.climate.get, CLIMATE_STATUS)
  File "/home/per/home-assistant/homeassistant/components/verisure.py", line 171, in update_component
    reconnect()
  File "/home/per/home-assistant/homeassistant/components/verisure.py", line 121, in reconnect
    MY_PAGES.login()
  File "/home/per/.homeassistant/lib/verisure/mypages.py", line 46, in login
    self._session.login()
  File "/home/per/.homeassistant/lib/verisure/session.py", line 92, in login
    self.csrf = self._get_csrf()
  File "/home/per/.homeassistant/lib/verisure/session.py", line 152, in _get_csrf
    timeout=RESPONSE_TIMEOUT)
  File "/usr/lib/python3.5/site-packages/requests/sessions.py", line 480, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python3.5/site-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.5/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.5/site-packages/requests/adapters.py", line 449, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='mypages.verisure.com', port=443): Read timed out. (read timeout=3)

And finally an infinite amount of:
16-02-17 06:10:30 homeassistant.components.verisure: Wrong password

My password still works fine, so the error message is wrong.

Looks similar to mine.
Experienced this twice. Last night, and the day before:
16-02-17 07:11:01 homeassistant.components.verisure: Could not login to Verisure mypages, The email address or password you entered is incorrect. Please try again.
16-02-17 07:11:01 homeassistant.components.verisure: Could not login to Verisure mypages, The email address or password you entered is incorrect. Please try again.
16-02-17 07:11:01 homeassistant.components.verisure: Could not login to Verisure mypages, The email address or password you entered is incorrect. Please try again.
16-02-17 07:11:04 homeassistant.core: BusHandler:Exception doing job
Traceback (most recent call last):
  File "C:\Users\Kontoret\AppData\Roaming\.homeassistant\lib\verisure\session.py", line 167, in json_to_dict
    return json.loads(doc)
  File "C:\Users\Kontoret\AppData\Local\Programs\Python\Python35-32\lib\json\__init__.py", line 319, in loads
    return _default_decoder.decode(s)
  File "C:\Users\Kontoret\AppData\Local\Programs\Python\Python35-32\lib\json\decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Users\Kontoret\AppData\Local\Programs\Python\Python35-32\lib\json\decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 34 column 1 (char 33)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\kontoret\home-assistant\homeassistant\components\verisure.py", line 164, in update_component
    for overview in get_function():
  File "C:\Users\Kontoret\AppData\Roaming\.homeassistant\lib\verisure\devices\climate.py", line 21, in get
    status = self._session.get(OVERVIEW_URL)
  File "C:\Users\Kontoret\AppData\Roaming\.homeassistant\lib\verisure\session.py", line 112, in get
    return self.json_to_dict(UNESCAPE(response.text))
  File "C:\Users\Kontoret\AppData\Roaming\.homeassistant\lib\verisure\session.py", line 173, in json_to_dict
    'Error: {0} - Data: {1}'.format(e, doc)))
  File "C:\Users\Kontoret\AppData\Roaming\.homeassistant\lib\verisure\session.py", line 201, in raise_response_error
    raise ResponseError(match)
verisure.session.ResponseError: <_sre.SRE_Match object; span=(196, 247), match='<title>Choose country - My Pages - Verisure</titl>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Kontoret\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 376, in _make_request
    httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Kontoret\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 378, in _make_request
    httplib_response = conn.getresponse()
  File "C:\Users\Kontoret\AppData\Local\Programs\Python\Python35-32\lib\http\client.py", line 1174, in getresponse
    response.begin()
  File "C:\Users\Kontoret\AppData\Local\Programs\Python\Python35-32\lib\http\client.py", line 282, in begin
    version, status, reason = self._read_status()
  File "C:\Users\Kontoret\AppData\Local\Programs\Python\Python35-32\lib\http\client.py", line 243, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "C:\Users\Kontoret\AppData\Local\Programs\Python\Python35-32\lib\socket.py", line 575, in readinto
    return self._sock.recv_into(b)
  File "C:\Users\Kontoret\AppData\Local\Programs\Python\Python35-32\lib\ssl.py", line 924, in recv_into
    return self.read(nbytes, buffer)
  File "C:\Users\Kontoret\AppData\Local\Programs\Python\Python35-32\lib\ssl.py", line 786, in read
    return self._sslobj.read(len, buffer)
  File "C:\Users\Kontoret\AppData\Local\Programs\Python\Python35-32\lib\ssl.py", line 570, in read
    v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Kontoret\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\adapters.py", line 376, in send
    timeout=timeout
  File "C:\Users\Kontoret\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 609, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "C:\Users\Kontoret\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\packages\urllib3\util\retry.py", line 247, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "C:\Users\Kontoret\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\packages\urllib3\packages\six.py", line 310, in reraise
    raise value
  File "C:\Users\Kontoret\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 559, in urlopen
    body=body, headers=headers)
  File "C:\Users\Kontoret\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 380, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "C:\Users\Kontoret\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 308, in _raise_timeout
    raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='mypages.verisure.com', port=443): Read timed out. (read timeout=3)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\kontoret\home-assistant\homeassistant\core.py", line 789, in job_handler
    func(arg)
  File "c:\users\kontoret\home-assistant\homeassistant\helpers\event.py", line 186, in pattern_time_change_listener
    action(now)
  File "c:\users\kontoret\home-assistant\homeassistant\helpers\entity_component.py", line 172, in _update_entity_states
    entity.update_ha_state(True)
  File "c:\users\kontoret\home-assistant\homeassistant\helpers\entity.py", line 151, in update_ha_state
    self.update()
  File "c:\users\kontoret\home-assistant\homeassistant\components\sensor\verisure.py", line 78, in update
    verisure.update_climate()
  File "c:\users\kontoret\home-assistant\homeassistant\util\__init__.py", line 275, in wrapper
    result = method(*args, **kwargs)
  File "c:\users\kontoret\home-assistant\homeassistant\components\verisure.py", line 139, in update_climate
    update_component(MY_PAGES.climate.get, CLIMATE_STATUS)
  File "c:\users\kontoret\home-assistant\homeassistant\components\verisure.py", line 171, in update_component
    reconnect()
  File "c:\users\kontoret\home-assistant\homeassistant\components\verisure.py", line 121, in reconnect
    MY_PAGES.login()
  File "C:\Users\Kontoret\AppData\Roaming\.homeassistant\lib\verisure\mypages.py", line 46, in login
    self._session.login()
  File "C:\Users\Kontoret\AppData\Roaming\.homeassistant\lib\verisure\session.py", line 87, in login
    response = self._session.send(req, timeout=RESPONSE_TIMEOUT)
  File "C:\Users\Kontoret\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\Kontoret\AppData\Local\Programs\Python\Python35-32\lib\site-packages\requests\adapters.py", line 449, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='mypages.verisure.com', port=443): Read timed out. (read timeout=3)
16-02-17 07:11:30 homeassistant.components.verisure: Wrong password

Seems solved now, right?

Actually no. I got this last night:
16-03-02 23:27:00 homeassistant.components.verisure: Caught connection error Maintenance, tries to reconnect 16-03-02 23:27:00 homeassistant.components.verisure: Caught connection error Maintenance, tries to reconnect
Until:
16-03-02 23:59:03 homeassistant.components.verisure: Could not login to Verisure mypages, The email address or password you entered is incorrect. Please try again. 1 instance
And then:
``16-03-02 23:59:04 homeassistant.components.verisure: Wrong password`
Maybe this is expected?

Should be 2 tries... I think.
Maybe we should send a bugreport to verisure about that they send the wrong error 😀

Yeah! Would be cool to see their response. 😅😂
Could it be a possible solution to catch the maintenance error, and then
give unlimited retries of login, instead of relying on the wrong password
response?
So basically:
If response maintenance error:
set some_flag in wrong_password method = 1
Wrong password method:
if flag==1 then retry login
else wrong password after 2 tries
?
On Thu, Mar 3, 2016 at 6:57 PM, Per Sandström<EMAIL_ADDRESS>wrote:

Should be 2 tries... I think.
Maybe we should send a bugreport to verisure about that they send the
wrong error 😀
—
Reply to this email directly or view it on GitHub
https://github.com/balloob/home-assistant/issues/1289#issuecomment-191887102
.

Yes,  I guess thats the best way to go

Today with latest code:
16-03-08 07:52:10 homeassistant.components.verisure: Caught connection error Unable to validate response form My Pages, status code: 500 - Data: b'{"vector":[{"type":"validation","message":"An error has occurred. Please contact Customer Support.","fieldLabel":null,"label":"error.contactCustomerService","duplicate":false,"isDuplicate":false}],"hasErrorsOrFatals":false,"hasValidations":true}', tries to reconnect
16-03-08 07:52:10 requests.packages.urllib3.connectionpool: Starting new HTTPS connection (1): mypages.verisure.com
16-03-08 07:52:10 homeassistant.components.verisure: Wrong user name or password for Verisure MyPages
16-03-08 07:52:30 homeassistant.components.verisure: Caught connection error <_sre.SRE_Match object; span=(64, 99), match='<title>504 Gateway Time-out</title>'>, tries to reconnect
16-03-08 07:52:30 homeassistant.components.verisure: Caught connection error <_sre.SRE_Match object; span=(64, 99), match='<title>504 Gateway Time-out</title>'>, tries to reconnect
16-03-08 07:52:31 homeassistant.components.verisure: Caught connection error <_sre.SRE_Match object; span=(64, 99), match='<title>504 Gateway Time-out</title>'>, tries to reconnect
16-03-08 07:52:31 requests.packages.urllib3.connectionpool: Starting new HTTPS connection (1): mypages.verisure.com
16-03-08 07:52:31 homeassistant.components.verisure: Caught connection error <_sre.SRE_Match object; span=(64, 99), match='<title>504 Gateway Time-out</title>'>, tries to reconnect
16-03-08 07:53:00 homeassistant.components.verisure: Wrong password for Verisure, update config

It catches a response error and then wrong password.

Any suggestions? I really don't like to have to use up a users password retries and risk to lock their account. But it seems like adding the retry again is the only solution to this.

It seems like any error will eventually lead to wrong password error. What about adding a password check at HA startup? Because it won't be wrong password if you have been logged in for some time.

It might be if the user changes it, and forget to shut down HA before.

Maybe a manual override that can be set in config?
8. mar. 2016 21:53 skrev "Per Sandström"<EMAIL_ADDRESS>
It might be if the user changes it, and forget to shut down HA before.
—
Reply to this email directly or view it on GitHub
https://github.com/balloob/home-assistant/issues/1289#issuecomment-193961224
.

You mean so a user who wants to risk to lock the account might do so, but it isn't the default option?

Yes.
Some other option might be a toggle in the front end to try to login again.
Or give it 2 hours after 1 try with wrong password message. Verisure
schedules their longest maintenance downtime from 0600 to 0900. 3 hours.
Normally everything is back up running by 0800.
8. mar. 2016 22:01 skrev "Per Sandström"<EMAIL_ADDRESS>
You mean so a user who wants to risk to lock the account might do so, but
it isn't the default option?
—
Reply to this email directly or view it on GitHub
https://github.com/balloob/home-assistant/issues/1289#issuecomment-193964717
.

Is it normal to get a lot of these:
6-03-10 15:11:00 requests.packages.urllib3.connectionpool: Resetting dropped connection: mypages.verisure.com
16-03-10 15:11:00 requests.packages.urllib3.connectionpool: Resetting dropped connection: mypages.verisure.com

I have been trying to figure out why they are printed. It seems to be an informational message and nothing to worry about, but it would be nice if it was not printed all the time.

Ok, Nice to know. :+1:
BTW it seems like the module is failing more frequently now because of the last report from me. Any idea?
Does this occour to other user i.e. other home automations using your module?

I have actually not had to restart once myself since the last fix, must be a bit of bad luck on you side...
I will add the retry again and as you suggest wait quite a long time after password error, maybe even add 5 minutes or so on any other error.
I have not had any reports from other users. Could be that the ha implementation is updating very often and thus increase the risk.

Ha is resetting dropped connection every 30 seconds, is that the set intervall?
I've had to restart three times, all times starts with the same error before wrong password:
16-03-08 07:52:10 homeassistant.components.verisure: Caught connection error Unable to validate response form My Pages, status code: 500 - Data: b'{"vector":[{"type":"validation","message":"An error has occurred. Please contact Customer Support.","fieldLabel":null,"label":"error.contactCustomerService","duplicate":false,"isDuplicate":false}],"hasErrorsOrFatals":false,"hasValidations":true}', tries to reconnect

I will keep an eye on it, and report ;)

@turbokongen Do you still get errors?
I'm running a new version that should handle one "wrong password" error, but I still haven't got one.

No errors leading to stop of component anymore. I think this may have been
an issue with the verisure site. Since their release on 16.March it has
been very stable. Going to give it a good test now that I will go away for
5 days. I'll report back when I get home and checked the logs.
On Sun, Mar 20, 2016 at 7:58 PM, Per Sandström<EMAIL_ADDRESS>wrote:

@turbokongen https://github.com/turbokongen Do you still get errors?
I'm running a new version that should handle one "wrong password" error,
but I still haven't got one.
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/balloob/home-assistant/issues/1289#issuecomment-198991390

Sorry for not coming back on this. I have been busy making some zwave support in HA.
I get disconnections usally after 1,5 day of uptime, and tey all looks like this:
16-04-04 23:59:51 homeassistant.components.verisure: Caught connection error <_sre.SRE_Match object; span=(205, 263), match='<title>An error has occurred - My Pages - Verisur>, tries to reconnect
16-04-04 23:59:54 homeassistant.components.verisure: Could not login to Verisure MyPages, HTTPSConnectionPool(host='mypages.verisure.com', port=443): Read timed out. (read timeout=3)
16-04-05 00:00:10 homeassistant.components.verisure: Caught connection error Unable to validate response form My Pages, status code: 500 - Data: b'{"vector":[{"type":"validation","message":"An error has occurred. Please contact Customer Support.","fieldLabel":null,"label":"error.contactCustomerService","duplicate":false,"isDuplicate":false}],"hasErrorsOrFatals":false,"hasValidations":true}', tries to reconnect
16-04-05 00:00:11 homeassistant.components.verisure: Wrong user name or password for Verisure MyPages
16-04-05 00:00:21 homeassistant.components.verisure: Caught connection error Exceeded 30 redirects., tries to reconnect
16-04-05 00:00:25 homeassistant.components.verisure: Could not login to Verisure MyPages, HTTPSConnectionPool(host='mypages.verisure.com', port=443): Read timed out. (read timeout=3)
16-04-05 00:00:30 homeassistant.components.verisure: Wrong password for Verisure, update config
16-04-05 00:00:32 homeassistant.components.verisure: Wrong user name or password for Verisure MyPages
16-04-05 00:01:00 homeassistant.components.verisure: Caught connection error <_sre.SRE_Match object; span=(64, 99), match='<title>504 Gateway Time-out</title>'>, tries to reconnect

And then just wrong password update config from here on.

This is from today after about 11-12hours of uptime:
16-04-06 21:25:31 homeassistant.components.verisure: Caught connection error Not logged in, tries to reconnect
	Line 4064: 16-04-06 21:54:30 homeassistant.components.verisure: Caught connection error Not logged in, tries to reconnect
	Line 4065: 16-04-06 21:54:30 homeassistant.components.verisure: Caught connection error Not logged in, tries to reconnect
	Line 4066: 16-04-06 21:54:31 homeassistant.components.verisure: Caught connection error Not logged in, tries to reconnect
	Line 7843: 16-04-06 23:06:30 homeassistant.components.verisure: Caught connection error Not logged in, tries to reconnect
	Line 40958: 16-04-07 10:02:30 homeassistant.components.verisure: Caught connection error Not logged in, tries to reconnect
	Line 40960: 16-04-07 10:02:30 homeassistant.components.verisure: Caught connection error Not logged in, tries to reconnect
	Line 41207: 16-04-07 10:07:30 homeassistant.components.verisure: Caught connection error Not logged in, tries to reconnect
	Line 41213: 16-04-07 10:07:33 homeassistant.components.verisure: Caught connection error Not logged in, tries to reconnect
	Line 41222: 16-04-07 10:07:37 homeassistant.components.verisure: Could not login to Verisure MyPages, HTTPSConnectionPool(host='mypages.verisure.com', port=443): Read timed out. (read timeout=3)
	Line 41240: 16-04-07 10:08:00 homeassistant.components.verisure: Caught connection error Not logged in, tries to reconnect
	Line 41243: 16-04-07 10:08:01 homeassistant.components.verisure: Caught connection error Not logged in, tries to reconnect
	Line 41245: 16-04-07 10:08:02 homeassistant.components.verisure: Caught connection error Not logged in, tries to reconnect
	Line 44888: 16-04-07 11:23:10 homeassistant.components.verisure: Caught connection error Unable to validate response form My Pages, status code: 500 - Data: b'{"vector":[{"type":"validation","message":"An error has occurred. Please contact Customer Support.","fieldLabel":null,"label":"error.contactCustomerService","duplicate":false,"isDuplicate":false}],"hasErrorsOrFatals":false,"hasValidations":true}', tries to reconnect
	Line 44890: 16-04-07 11:23:10 homeassistant.components.verisure: Wrong user name or password for Verisure MyPages
	Line 44891: 16-04-07 11:23:12 homeassistant.components.verisure: Caught connection error Unable to validate response form My Pages, status code: 500 - Data: b'{"vector":[{"type":"validation","message":"An error has occurred. Please contact Customer Support.","fieldLabel":null,"label":"error.contactCustomerService","duplicate":false,"isDuplicate":false}],"hasErrorsOrFatals":false,"hasValidations":true}', tries to reconnect
	Line 44893: 16-04-07 11:23:12 homeassistant.components.verisure: Caught connection error <_sre.SRE_Match object; span=(205, 263), match='<title>An error has occurred - My Pages - Verisur>, tries to reconnect
	Line 44894: 16-04-07 11:23:12 homeassistant.components.verisure: Wrong user name or password for Verisure MyPages
	Line 44895: 16-04-07 11:23:12 homeassistant.components.verisure: Wrong password for Verisure, update config
	Line 44910: 16-04-07 11:23:30 homeassistant.components.verisure: Wrong password for Verisure, update config
	Line 44912: 16-04-07 11:23:30 homeassistant.components.verisure: Wrong password for Verisure, update config
	Line 44915: 16-04-07 11:23:31 homeassistant.components.verisure: Caught connection error <_sre.SRE_Match object; span=(64, 99), match='<title>504 Gateway Time-out</title>'>, tries to reconnect
	Line 44919: 16-04-07 11:23:32 homeassistant.components.verisure: Wrong user name or password for Verisure MyPages
	Line 44934: 16-04-07 11:24:00 homeassistant.components.verisure: Wrong password for Verisure, update config
	Line 44938: 16-04-07 11:24:00 homeassistant.components.verisure: Wrong password for Verisure, update config
	Line 44940: 16-04-07 11:24:00 homeassistant.components.verisure: Wrong password for Verisure, update config
	Line 44960: 16-04-07 11:24:30 homeassistant.components.verisure: Wrong password for Verisure, update config
	Line 44961: 16-04-07 11:24:30 homeassistant.components.verisure: Wrong password for Verisure, update confi

Thanks @turbokongen. Ill take a look at it in a bit over a week.

Same problem for me now. :)

Seems to be solved


reconnect/connect to libvirt in the loop

There possible situation when libvirt daemon may be stopped during skydive agent startup
but this is not a reason to fail the libvirt probe initialization
Change-Id: Id948e54af4140173154bd829a9c9a61b59f7a22a
Signed-off-by: Sergey Glazyrin<EMAIL_ADDRESS>

run skydive-cdd-overview-tests

Already addressed by https://github.com/skydive-project/skydive/pull/1918


audio: Potential deadlock on Wasm

https://github.com/hajimehoshi/ebiten/runs/990910266
*** Test killed: ran too long (11m0s).
FAIL	github.com/hajimehoshi/ebiten/audio	660.009s

Let's revisit when we find the same issue again.


Question: how to provide nex cargo hashes for python packages with rust dependencies?

I want to provide an overrides along the lines of this but it requires access to additional argument other than final and prev. Unfortunately, because the overrides attribute is a file instead of function I don't have a straightforward way to use the pkgs provided in the kernels module?
final: prev: {
  cryptography =
    let getCargoHash = version: {

            "40.0.1-vendor" = "sha256-gFfDTc2QWBWHBCycVH1dYlCsWQMVcRZfOBIau+njtDU=";

          }.${version} or (
          #How?
            lib.warn "Unknown cryptography version: '${version}'. Please update getCargoHash." builtins.lib.fakeHash
          );
          in
            prev.cryptography.overridePythonAttrs(old: {
                #How?
               cargoDeps =  pkgs.rustPlatgorm.fetchCargoTarball {
                 inherit (old) src;
                 name = "${old.pname}-${old.version}";
                 hash = getCargoHash old.version;
              };
         });

}

Example here: https://github.com/tweag/jupyenv/pull/452/commits/be03f363f46c3a0ad031aa0c90b130a9b3fc8616#diff-cfc9255ab96fac9007f71aac899e2e98bccdb7889d63beadf258c450670477cbR17

Thanks for your pointer. I must have searched only for 'getCargoHash' and tried a few things that were not prev.pkgs.xxx . Let's say it was a long day


Fix Claims

Related Issues
Fixes <link_to_github_issue>
Related PRs

branch
PR

service a PR
Link to PR

service b PR
Link to PR

Description
A few sentences describing the overall goals of the pull request's commits.
Must

[ ] Tests
[ ] Documentation (if applicable)

What's the status here?

@itaihanski can you have a look?


Allow the user to change HTTP request method and headers

This could be achieved by registering a callback to a string and have the user assign a callback via the string to a specific create-select widget.


Automatically Generated GSI is not usable in app

Environment information
System:
  OS: macOS 14.1
  CPU: (12) x64 Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
  Memory: 20.45 MB / 16.00 GB
  Shell: /bin/zsh
Binaries:
  Node: 16.20.2 - /usr/local/bin/node
  Yarn: undefined - undefined
  npm: 8.19.4 - /usr/local/bin/npm
  pnpm: undefined - undefined
NPM Packages:
  @aws-amplify/backend: 1.0.2
  @aws-amplify/backend-cli: 1.0.3
  aws-amplify: 6.3.1
  aws-cdk: 2.141.0
  aws-cdk-lib: 2.141.0
  typescript: 5.3.3

AWS environment variables:
  AWS_STS_REGIONAL_ENDPOINTS = regional
  AWS_NODEJS_CONNECTION_REUSE_ENABLED = 1
  AWS_SDK_LOAD_CONFIG = 1

Description
Often times when creating a relationship between two models, Amplify will automatically generate a GSI. However, you cannot use this GSI in queries in your app, so you need to create another one with the same primary key using .secondaryIndexes(). I feel like this is a waste and there should be a way to use the autogenerated GSI as a query in your code. Also, there is no way to set a sort key on the autogenerated GSI.
In Gen 1, you could create uni-directional relationships with the @key directive so this wasn't a problem. But Gen 2 forces bi-directional relationships which lead to the autogeneration of the GSI.

Hey👋 thanks for raising this! I'm going to transfer this over to our API repository for better assistance 🙂


Resolve alias not working as expected

I have the following setup in my config:
...
	resolve: {
		alias: {
			'app': './src/app.js'
		}
	},
...

With this I'm expecting that I should be able to go var shell = require('shell'); anywhere in my code base and it will include ./src/shell.js.
Instead, when I try and build, I get the following errors:
ERROR in ./build/shell/index.js
Module not found: Error: Cannot resolve module app.js in /Users/anthonyvanderhoorn/Projects/client/build/shell
 @ ./build/shell/index.js 5:14-32

ERROR in ./build/shell/components/shell.js
Module not found: Error: Cannot resolve module app.js in /Users/anthonyvanderhoorn/Projects/client/build/shell/components
 @ ./build/shell/components/shell.js 3:14-32

I've looked through http://webpack.github.io/docs/configuration.html#resolve-alias and think I have everything right but its not working.
Any thoughts?

This worked for me as well thanks.
Why does this not work using (resolve) tried every possible combination?
resolve: {
root: path.resolve('./src/shell'),
modulesDirectories: ['./src/shel'],
extensions: ['', '.js', '.jsx']
},

@wpitallo I can't give you a simple answer because this depends on the path which is passed to require(), your folder structure, other configs, etc.
Given your code snippet, I can tell that you're using the modulesDirectories in a wrong way. modulesDirectories should be an array with just folder names (no paths!), like ["node_modules", "web_modules"]. And for the record, you should also not add "bower_components" to it, as bower_components are not resolved the recursively like node_modules are resolved.


Move to_json and to_yaml to JSON::Serializable and YAML::Serializable

This PR adds the modules JSON::Serializable and YAML::Serializable which define to_json and to_yaml methods as wrappers for to_json(JSON::Builder) and to_yaml(YAML::Nodes::Builder).
Previously these methods were defined in Object which would spill them even to types that don't implement JSON/YAML serialization. Now, they need to be explicitly included. JSON.mapping and YAML.mapping include the resepective module automatically. Therefore most user code will most likely not require any changes.
If the serialization methods are implemented manually, this will require to add include JSON::Serializable/include YAML::Serializable  to the type in order to make the wrapper methods available as well.
Closes  #5695, #3450

I'd also suggest to remove to_pretty_json. This could become a parameter on to_json. I don't think it makes sense to keep these duplicated methods.

Yes, we can just have indent: "" and to have a pretty json you'd just pass ident: "  ".

The specs are failing because Crystal::Exception defines to_json(JSON::Builder) and includes JSON::Serializable. But in some of the sample files for individual compiler features, json is not required and therefore we have an unknown constant.
Other locations in the compiler itself assume an Crystal::Exception to have a to_json method when json is available.
What would be the best way to handle this?
I guess that's not just an issue for the compiler itself, but also for libraries which support serialization but can be used without it, if the application doesn't need it. In that case, you can't just always include Serializable.
One solution could be to include the serialization modules in the core lib. But that sounds very wrong.
This very hacky solution would make it work:
{% if @type.has_constant? :JSON %}
module Crystal
  abstract class Exception
    include JSON::Serializable
  end
end
{% end %}

But I guess this could become somewhat reasonable if it was possible to check for the existance of global namespace constants directly in the type namespace. But I don't think that's currently possible.
Therefore the macro needs to be put in the global namespace (so @type references the program) and reopen the class to add the include.

@straight-shoota Always require "json" in the compiler and tests for the compiler.

I mean, just put require "json" where include JSON::Serializable is needed.
This is a problem in Ruby too, everything is so dynamic that depending on what files you require things work or not (in this case compile or not).
The solution is to have an import system like Go or Python, where to use something you have to import it, every time. It's a bit more tedious, but more robust. But I don't think we'll ever do that in Crystal.

For the compiler and compiler samples, this is probably okay.
But I think it would be feasible to have a more general working solution to easily switch the include statement on or off depending on the existance of a global constant like this:
{% if constant? "JSON::Serializable" %}
include JSON::Serializable
{% end %}

No, that's not good, because maybe you require "json" after requiring that piece of code, and by then it's too late.
What we do with, for example, BigInt, is to have a big/json file that you can require to add JSON support to those types. For the compiler I don't think it's worth it, though.

Does this need any more work?

@asterite
I'm not satisfied with having to require shards differently depending on whether I want to use them with support for JSON and/or YAML serialization or not.
This concerns shards or stdlib components which define data types that support serialization but don't depend on it. The shard can be used without JSON, YAML but in case you want to serialize them, the methods for that should be there and if possible don't require anything more than adding require "json" to be available.
Having these methods defined is not really an issue for the actual converter methods to_json(builder : JSON::Builder) and new(parser : JSON::PullParser) (and equivalent for YAML). These methods can always be defined in the source regardless of JSON usage. They won't be parsed unless they are called. It's just easily pluggable: If you're not using json it shouldn't be an issue having these methods. If you want to use json, they should just work without ado.
The mapping macros are not really affected, they need JSON/YAML always be available anyway. Considering this is a very tight integration of the serialization format in the type, this shouldn't be an issue. In the long term we'll probably end up with something different anyway (see #3620).
The only problem is the definition of the helper methods to_json() and to_json(io : IO) defined in JSON::Serializable (with this PR). It shouldn't be required to redefine these methods on every such type supporting pluggable serialization. Unfortunately, differently from the methods which will not get parsed, include JSON::Serializable can't be used in a type definition if that module is not defined.
So I thought: why not define it always? We could just include dummy modules JSON::Serializable and YAML::Serializable in the core lib. They can be included but don't define anything:
module YAML::Serializable
end

Maybe as an usability impovement they could have the serialization helper methods defined but raise and prompt to require the respective library.
module JSON::Serializable
  def to_json
    {% raise %(missing JSON, please add `require "json"`) %}
  end
end

This solution surely isn't optimal as it add's type definition (even if they are empty) to the core lib which shouldn't be needed to be there. It obviously only works for serialization formats included in the stdlib (although I think JSON and YAML should stay in stdlib) as this won't be a general solution for other serialization formats as well.
But the suggested solution improves ease of use by a great deal and I think that's very important - IMO the benefit should be worth it and I don't see any downsides that would make this impossible.

FYI This now conflicts with #6082 but is undetected in the diff

Yeah, I didn't have time to properly look at #6082 yet and see what from this PR needs to be adapted.

Oh, I didn't mean to close this. Can someone reopen, please?

Reopened. I was only pointing that it was not ok to merge. :-) Not that I was hurry or had any specific input in this PR

With #6082 introducing {JSON,YAML}::Serializable to automatically expose instance variables for serialization, the methods added in this PR should go into a different module in order to separate helper methods like #to_json from autogeneration of to_json(builder).
I opted for {JSON::YAML}::Serializable::Helper, though I'm not sure about that. It a quite long name for being frequently included and add's another included module to JSON::Serializable namespace. Are there any suggestions to rename?

Yesterday I came up with a similar idea and @asterite brought me to #6275 and this PR. I think this is actually good and should be merged eventually. Would it be possible actually to use the same JSON::Serializable module and make it generate or not the to_json(JSON::Builder) depending on wether is already defined? I was playing with JSON::Friendly but not a good name anyway 😆
However, while trying to work on this, I faced this issue: #9173. I wonder if this PR would have the same issue if rebased.


Update dev branch

Update dev branch

why only update this single file (generated automatically)?

why only update this single file (generated automatically)?

The update generated by dependent was merged to the master branch,
the dev branch displays This branch is 150 commits ahead, 4 commits behind master.
So this PR will make the update available to the dev branch


Error when attempting to push results to database

Description of the bug
Hi there, I ran big-bacter for the first time and the main process seems to have completed without error, but when I attempt to push the results to the database, I get the following error message, below. And then, upon attempting to re-run the code, I get a second error during poppunk assignment (also below), which seems to indicate an issue with writing a directory. I checked the database and it looks like all of the isolates have populated under the snippy directory, but not in the assembly directory. Thank you in advance!!
Command used and terminal output
$ nextflow run DOH-JDJ0303/bigbacter-nf -r main -profile docker --input samplesheet.csv --outdir results --db /home/SequencingAdmin/nmdoh_data_storage/big-bacter-db/ --push true -resume
ERROR 1:
ERROR ~ No such file or directory: /home/SequencingAdmin/nmdoh_data_storage/big-bacter-db/Acinetobacter_baumannii/clusters/00001/snippy/2024030700_T1.tar.gz

 -- Check script '/home/mhutchinson/.nextflow/assets/DOH-JDJ0303/bigbacter-nf/./workflows/bigbacter.nf' at line: 142 or see '.nextflow.log' file for more details

ERROR 2:
Assigning clusters with PopPUNK.

Matplotlib created a temporary cache directory at /tmp/matplotlib-cerexqj0 because the default path (/home/mambauser/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
Fontconfig error: No writable cache directories
PopPUNK: assign
	(with backend: sketchlib v2.1.4
	 sketchlib: /opt/conda/envs/poppunk-env/lib/python3.10/site-packages/pp_sketchlib.cpython-310-x86_64-linux-gnu.so)
Mode: Assigning clusters of query sequences

Graph-tools OpenMP parallelisation enabled: with 8 threads
--output and --db must be different to prevent overwrite.

Relevant files
nextflow.log
System information
Nextflow version: 23.04.2
Hardware: Cloud
Executor: Azure VM
Container engine: (e.g. Docker, Singularity, Conda, Podman, Shifter, Charliecloud, or Apptainer)
OS: Ubuntu Linux
Version of nf-core/bigbacter: 1.0.0

Hi @hutchinsonmiri. Sorry to hear you are running into issues. It is not immediately clear to me the source of these issues, so some additional info would be helpful!

Can you share the .nextflow.log file from the first run you performed? It should be named .nextflow.log.2 if you have only run the pipeline 3 times.
Can you confirm the file at path /home/SequencingAdmin/nmdoh_data_storage/big-bacter-db/Acinetobacter_baumannii/clusters/00001/snippy/2024030700_T1.tar.gz exists, has the correct permissions, and is not somehow a symlink.
Can you confirm there is more than one PopPUNK database file in /home/SequencingAdmin/nmdoh_data_storage/big-bacter-db/Acinetobacter_baumannii/pp_db/. This directory should contain<PHONE_NUMBER>.tar.gz and one other file if things were pushed properly.

If this is your first time running BigBacter, then it may be worth deleting the database and starting over. If you run into the same issue again, you could try running the pipeline with --db_info false, as it appears the first error occurred while trying to gather information about your bigbacter database.
Regarding the second error - I am not sure what is going on, but my hunch is that if we can solve the first issue it will go away.

Hi there, thanks for your response!
So the nextflow log above is actually the.nextflow.log.2 but I removed the .2 suffix because github was throwing an error when I tried to upload it. Sorry for the confusion!
And yes, I see the path  /home/SequencingAdmin/nmdoh_data_storage/big-bacter-db/Acinetobacter_baumannii/clusters/00001/snippy/2024030700_T1.tar.gz and it does not appear to be a symlink and I have full permissions. Interestingly, there is no corresponding file here: /home/SequencingAdmin/nmdoh_data_storage/big-bacter-db/Acinetobacter_baumannii/clusters/00001/assembly/ but other samples have populated here.
Also, I can confirm that there are two database files:<PHONE_NUMBER>.tar.gz and one from the day I first ran the pipeline:<PHONE_NUMBER>.tar.gz .
I've tried deleting and reinstalling the Acinetobacter database but encountered the same error. I will say, I am trying to push ~30 samples to the db, so could that be an issue?
Thanks a bunch!!

Thank you for the additional info. Would you happen to have the .nextflow.log file for the first run (the one where you have --push false)? I want to make sure nothing funky is going on during this initial process.
Also, did you get a chance to run with --db_info false? It appears the error is caused by the db_info function looking for a file that "doesn't exist". This is odd because the function will only check files that it has detected are present.
In the meantime, I think I have a solution that will help move things forward. I will let you know when it is ready. I appreciate your patience!

nextflow (2).log
Thanks! Here is the first log file.
And when you suggest to run with --db_info false, would that be while pushing to the db, ie --push true? Appreciate the help!

Yep, I suspect that running --db_info false with --push true may resolve the issue for now. I will hopefully have a long-term solution shortly if that is truly the issue.

That did the trick! All of the samples have been pushed to both the snippy and assembly directories.

Likewise, thanks!!


Use Arcade NuSpec helpers

@dougbu The CI is failing with some provisioning error:

The Windows Release build failure seems to be caused by missing -configuration $(_BuildConfig) argument to eng\scripts\ci-flaky-tests.cmd. The leg builds Release but attempts to run tests from Debug dir.

The Windows Release build failure seems to be caused by missing -configuration $(_BuildConfig) argument to eng\scripts\ci-flaky-tests.cmd. The leg builds Release but attempts to run tests from Debug dir.

That's odd. These test legs have been building and testing just fine up to now. Do the new Arcade scripts effectively make the build configuration a mandatory input?
In any case, we're still hitting dotnet/core-eng#5654 for the Ubuntu jobs…

How did you get the extra information beyond the horrible "There as a failure" message?

That's odd. These test legs have been building and testing just fine up to now. Do the new Arcade scripts effectively make the build configuration a mandatory input?

IDK. It's always defaulted to Debug. I'm not aware of any change in that. Is it possible that he machines were reused and had the Debug build on them accidentally?

Merged these changes manually into darc's branch. Will take this further in #1279. Thanks @tmat❕❕


Docs: Node reserved key

Improved docs of Node runtimes, to mention reserved keyword code. People should avoid using this, as this is not consistent behaviour in all runtimes.

@Meldiron  have we made sure to communicate this in the appwrite docs as well ?