Error while uploading files "Failed to index document"

I have all the packages up to date, review my Weaviate, cloudflare and OpenAI API. Everything seems to be correct.
I see in cloudflare that the files are correctly uploaded.
This is the error in console when uploading a .txt simple file:
`- ready started server on <IP_ADDRESS>:3000, url: http://localhost:3000

info Loaded env from C:\Users\r.ramos\Documents\GitHub\langchain-chat-with-documents.env
"pnpm" no se reconoce como un comando interno o externo,
Error: 'Error in addDocuments' TypeError [ERR_INVALID_URL]: Invalid URL
at WeaviateStore.addVectors (file:///C:/Users/r.ramos/Documents/GitHub/langchain-chat-with-documents/node_modules/langchain/dist/vectorstores/weaviate.js:103:19)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async WeaviateStore.fromDocuments (file:///C:/Users/r.ramos/Documents/GitHub/langchain-chat-with-documents/node_modules/langchain/dist/vectorstores/weaviate.js:155:9)
at async eval (webpack-internal:///(api)/./src/server/api/routers/weaviate.ts:81:13)
at async resolveMiddleware (file:///C:/Users/r.ramos/Documents/GitHub/langchain-chat-with-documents/node_modules/@trpc/server/dist/index.mjs:416:30)
at async callRecursive (file:///C:/Users/r.ramos/Documents/GitHub/langchain-chat-with-documents/node_modules/@trpc/server/dist/index.mjs:452:32)
at async callRecursive (file:///C:/Users/r.ramos/Documents/GitHub/langchain-chat-with-documents/node_modules/@trpc/server/dist/index.mjs:452:32)
at async resolve (file:///C:/Users/r.ramos/Documents/GitHub/langchain-chat-with-documents/node_modules/@trpc/server/dist/index.mjs:482:24)
at async file:///C:/Users/r.ramos/Documents/GitHub/langchain-chat-with-documents/node_modules/@trpc/server/dist/resolveHTTPResponse-e1286cb3.mjs:87:32
at async Promise.all (index 0)
at async resolveHTTPResponse (file:///C:/Users/r.ramos/Documents/GitHub/langchain-chat-with-documents/node_modules/@trpc/server/dist/resolveHTTPResponse-e1286cb3.mjs:84:28)
at async file:///C:/Users/r.ramos/Documents/GitHub/langchain-chat-with-documents/node_modules/@trpc/server/dist/nodeHTTPRequestHandler-f0efcff4.mjs:32:24
at async file:///C:/Users/r.ramos/Documents/GitHub/langchain-chat-with-documents/node_modules/@trpc/server/dist/adapters/next.mjs:42:9
❌ tRPC failed on weaviate.index: Failed to index document`

I have the same issue...

Same here...

Does it fail every time or only with some documents?

Every time unfortunately. With both PDFs and TXT files

The debug errors in the server are the same as the above comment:

info Loaded env from C:\Users\r.ramos\Documents\GitHub\langchain-chat-with-documents.env
"pnpm" no se reconoce como un comando interno o externo,
Error: 'Error in addDocuments' TypeError [ERR_INVALID_URL]: Invalid URL
at WeaviateStore.addVectors (file:///C:/Users/r.ramos/Documents/GitHub/langchain-chat-with-documents/node_modules/langchain/dist/vectorstores/weaviate.js:103:19)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async WeaviateStore.fromDocuments (file:///C:/Users/r.ramos/Documents/GitHub/langchain-chat-with-documents/node_modules/langchain/dist/vectorstores/weaviate.js:155:9)
at async eval (webpack-internal:///(api)/./src/server/api/routers/weaviate.ts:81:13)
at async resolveMiddleware (file:///C:/Users/r.ramos/Documents/GitHub/langchain-chat-with-documents/node_modules/@trpc/server/dist/index.mjs:416:30)
at async callRecursive (file:///C:/Users/r.ramos/Documents/GitHub/langchain-chat-with-documents/node_modules/@trpc/server/dist/index.mjs:452:32)
at async callRecursive (file:///C:/Users/r.ramos/Documents/GitHub/langchain-chat-with-documents/node_modules/@trpc/server/dist/index.mjs:452:32)
at async resolve (file:///C:/Users/r.ramos/Documents/GitHub/langchain-chat-with-documents/node_modules/@trpc/server/dist/index.mjs:482:24)
at async file:///C:/Users/r.ramos/Documents/GitHub/langchain-chat-with-documents/node_modules/@trpc/server/dist/resolveHTTPResponse-e1286cb3.mjs:87:32
at async Promise.all (index 0)
at async resolveHTTPResponse (file:///C:/Users/r.ramos/Documents/GitHub/langchain-chat-with-documents/node_modules/@trpc/server/dist/resolveHTTPResponse-e1286cb3.mjs:84:28)
at async file:///C:/Users/r.ramos/Documents/GitHub/langchain-chat-with-documents/node_modules/@trpc/server/dist/nodeHTTPRequestHandler-f0efcff4.mjs:32:24
at async file:///C:/Users/r.ramos/Documents/GitHub/langchain-chat-with-documents/node_modules/@trpc/server/dist/adapters/next.mjs:42:9
❌ tRPC failed on weaviate.index: Failed to index document`

The browser console shows this:

I have the same error. Trying to follow the pre-signed URL generated [form:  https://doc-user-01234abc.0123456789xyz.r2.cloudflarestorage.com/brief.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&....], I get:
"This XML file does not appear to have any style information associated with it. The document tree is shown below.

SignatureDoesNotMatch
The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
"
I believe it might be related to this bug, but I cannot find a way around it. At Cloudflare Discrord's R2 channel they mention a few things, including this gist for inspiration. This did not help me either, but I am not an expert on the underlying tech, so you might have more luck.
Enabling us to manually upload the documents via CloudFlare could be a workaround - my files end at CloudFlare bucket as intended, but pre-signing fails.

On the R2 account page -> [Manage R2 API Tokens] make sure you have all these permissions enabled:
Edit: Allow edit access of all objects and List, Write, and Delete operations of all buckets

In my case, the R2 details are correct, however I have not been able to successfully parse any documents. All documents are uploaded to R2, but there's a parsing error with Weaviate. Please see: https://share.cleanshot.com/WkMB3LnN

In my case, the R2 details are correct, however I have not been able to successfully parse any documents. All documents are uploaded to R2, but there's a parsing error with Weaviate. Please see: https://share.cleanshot.com/WkMB3LnN

Same. FIles are correctly uploaded it seems

I Found the solution, remove the commented line infront of the WEAVITE_HOST property in .ENV file
WEAVIATE_HOST= xxxxx.weaviate.network # do not use https:// just the domain like bellingcat-xxx.weaviate.network

Unfortunately I'm still getting the same errors. I didn't have that comment in my .env file in the first place.
This is what I'm getting:
Warning: Indexing all PDF objects
Error
    at InvalidPDFExceptionClosure (/path/to/pdf.js:452:35)
    at Object.<anonymous> (/path/to/pdf.js:455:2)
    at __w_pdfjs_require__ (/path/to/pdf.js:45:30)
    at Object.<anonymous> (/path/to/pdf.js:7939:23)
    at __w_pdfjs_require__ (/path/to/pdf.js:45:30)
    at /path/to/pdf.js:88:18
    at /path/to/pdf.js:91:10
    at webpackUniversalModuleDefinition (/path/to/pdf.js:18:20)
    at Object.<anonymous> (/path/to/pdf.js:25:3)
    at Module._compile (node:internal/modules/cjs/loader:1103:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1157:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:168:29)
    at ModuleJob.run (node:internal/modules/esm/module_job:197:25)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Promise.all (index 0)
    at async ESMLoader.import (node:internal/modules/esm/loader:337:24)
    at async PDFLoaderImports [as pdfjs] (/path/to/pdf.js:80:34)
    at async PDFLoader.parse (/path/to/pdf.js:22:42)
    at async eval (webpack-internal:///(api)/./src/server/api/routers/weaviate.ts:73:29)
    at async resolveMiddleware (/path/to/@trpc/server/dist/index.mjs:416:30)
    at async callRecursive (/path/to/@trpc/server/dist/index.mjs:452:32)
    at async next (/path/to/@trpc/server/dist/index.mjs:460:32)
    at async callRecursive (/path/to/@trpc/server/dist/index.mjs:452:32)
    at async resolve (/path/to/@trpc/server/dist/index.mjs:480:24)
    at async /path/to/@trpc/server/dist/resolveHTTPResponse-edf72fc7.mjs:123:32
    at async Promise.all (index 0)
    at async resolveHTTPResponse (/path/to/@trpc/server/dist/resolveHTTPResponse-edf72fc7.mjs:120:28)
    at async /path/to/@trpc/server/dist/nodeHTTPRequestHandler-c9ddbeeb.mjs:61:24
    at async /path/to/@trpc/server/dist/adapters/next.mjs:39:9 {
  message: 'Invalid PDF structure'
}
❌ tRPC failed on weaviate.index: Failed to index document

(/path/to/pdf.js:45:30) this path doesnt look normal to me. Did you add anything to the original project?

I obfuscated the error text to remove the personal details. Everything is untouched from the latest pull.

Ah that makes more sense 😅 well unfortunately I cant help. But a few console.log would help shed a better light. I think logging the client and the method this error is refering to would be a good first step.

Unfortunately I'm still getting the same errors. I didn't have that comment in my .env file in the first place.
This is what I'm getting:
Warning: Indexing all PDF objects
Error
    at InvalidPDFExceptionClosure (/path/to/pdf.js:452:35)
    at Object.<anonymous> (/path/to/pdf.js:455:2)
    at __w_pdfjs_require__ (/path/to/pdf.js:45:30)
    at Object.<anonymous> (/path/to/pdf.js:7939:23)
    at __w_pdfjs_require__ (/path/to/pdf.js:45:30)
    at /path/to/pdf.js:88:18
    at /path/to/pdf.js:91:10
    at webpackUniversalModuleDefinition (/path/to/pdf.js:18:20)
    at Object.<anonymous> (/path/to/pdf.js:25:3)
    at Module._compile (node:internal/modules/cjs/loader:1103:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1157:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:168:29)
    at ModuleJob.run (node:internal/modules/esm/module_job:197:25)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Promise.all (index 0)
    at async ESMLoader.import (node:internal/modules/esm/loader:337:24)
    at async PDFLoaderImports [as pdfjs] (/path/to/pdf.js:80:34)
    at async PDFLoader.parse (/path/to/pdf.js:22:42)
    at async eval (webpack-internal:///(api)/./src/server/api/routers/weaviate.ts:73:29)
    at async resolveMiddleware (/path/to/@trpc/server/dist/index.mjs:416:30)
    at async callRecursive (/path/to/@trpc/server/dist/index.mjs:452:32)
    at async next (/path/to/@trpc/server/dist/index.mjs:460:32)
    at async callRecursive (/path/to/@trpc/server/dist/index.mjs:452:32)
    at async resolve (/path/to/@trpc/server/dist/index.mjs:480:24)
    at async /path/to/@trpc/server/dist/resolveHTTPResponse-edf72fc7.mjs:123:32
    at async Promise.all (index 0)
    at async resolveHTTPResponse (/path/to/@trpc/server/dist/resolveHTTPResponse-edf72fc7.mjs:120:28)
    at async /path/to/@trpc/server/dist/nodeHTTPRequestHandler-c9ddbeeb.mjs:61:24
    at async /path/to/@trpc/server/dist/adapters/next.mjs:39:9 {
  message: 'Invalid PDF structure'
}
❌ tRPC failed on weaviate.index: Failed to index document

Please try updating node to the latest stable release and running npm install pdf-parse@latest to update the library. I have fixed the same error by this way.

In my case, the R2 details are correct, however I have not been able to successfully parse any documents. All documents are uploaded to R2, but there's a parsing error with Weaviate. Please see: https://share.cleanshot.com/WkMB3LnN

Please try updating node to the latest stable release and running npm install pdf-parse@latest to update the library. I have fixed the same error by this way.


Seeing 'no environments' and unable to see insights

Please see the above screenshot.
The environment exists but I am unable to see the data in Digma.
I can see data in the recent activity panel.
https://app.zenhub.com/files/472408329/6d9bcfff-ee47-4b8b-8778-80ce7b117958/download

Couldn't not reproduce
After talking with @doppleware , I'm leaving it until the next occurrence


add selfConsumption (and other calculated values) to influxdb

Is your feature request related to a problem? Please describe.
Calculating the selfConsumption can be done in Grafana e.g. with:
import "math"
import "interpolate"
from(bucket: "evcc")
  |> range(start: v.timeRangeStart, stop:v.timeRangeStop)
  |> filter(fn: (r) => r._measurement == "homePower" or r._measurement == "pvPower")
  |> interpolate.linear(every: 15s)
  |> pivot(rowKey: ["_time"], columnKey: ["_measurement"], valueColumn: "_value")
  |> map(fn: (r) => ({r with selfPower: math.mMin(x:r.homePower, y:r.pvPower) }))
  |> filter(fn: (r) => exists r.selfPower)
  |> integral(column:"selfPower", unit:1h)

but this gets very slow for larger time ranges.
Describe the solution you'd like
Add calculated values that are shown in UI like selfConsumption to influxdb data model
Describe alternatives you've considered
I did calculation in Grafana as shown above, but to slow.
Additional context
Each metric is is reported with own timestamp, so first normalized values have to be created with interpolate to be able to calculate the minumim. I asume the own timestamps are not a bug, but are the basically independent timestamps when the data was collected.
Additionally I store absolute home/pv/self-power meter readings on a daily basis in influxdb, so I have correct values for long term statistics.  Integral of your home/pv/...-power values comes relatively close to these values and easily allows short term analysis.

Hint: dont do it in the grafana query - do it in Transform as binary operation...
I add my PV Power and the measured power of the energymeter - the result is my selfconsumption
(as the value of the measured power is negative...)
Works like a charm for me...

Keine weiteren Erkenntnisse


external-libs for database queries

AFAICT this relates heavily to https://github.com/yaml/yamlscript/issues/140.
I stopped by the matrix chatroom and had a quick chat about this + we decided to move to GH issues so that it isn't lost to chat history.
Summary of the chat:

there isn't a great story today for external libs
the things that work today that are most adjacent to this functionality are curl, yaml/load, json/load, load
yamlscript would like to do better than lein or deps.edn includes
something like babashka pods (which looks like exactly what I am looking for)

Bluntly, I had to look up what a lot of this stuff was. I think most of the value I can provide here is being a stand-in for other people who would also have to look up what this stuff is.

Ultimately, I'm looking to be able to do something like this:
enabled: true
expression:: db.sqlite("select prop1, prop2 from things where [...]", "/path/to/database.db")

Being able to evaluate this locally would be really useful, and the functionality is intuitive.  I provide some path to a sqlite database, the database is queried, and I either get an error or the result.
What happens if I --binary it and then ship that binary to another server? Do I need to also ship the database, or is the result stored in the resulting binary? Maybe the entire database can be stored in the binary?
All 3 of these sound useful, as an end user I think I would just need to know which one will happen so I can build the rest of my contraptions with that understanding.
I think it makes sense to figure out sqlite first since it's very easy to test.  I also suspect that the sqlite external lib would be a sufficiently complicated reference to assist with other external-lib style contributions or plugins.  In particular, I would use it as reference material and attempt to create a similar implementation for an aerospike query.

YS supports pods and iirc I had a demo that used sqlite. Trying to find it.

found it:
$ cat pod-test.ys
!yamlscript/v0

use-pod:
  q(org::babashka/go-sqlite3)
  "0.2.3"

require pod::babashka::go-sqlite3:
  => sqlite

result =:
  sqlite/query(
    ':memory:'
    ["SELECT 41 + 1 AS sum"]
  )

say: result.0.sum
$ ys pod-test.ys
42

What happens if I --binary it and then ship that binary to another server? Do I need to also ship the database, or is the result stored in the resulting binary? Maybe the entire database can be stored in the binary?

Obviously you could base64 encode the db file and then expand it to a place on disk or something like that.
That would work with --binary of course.

Let me know how far that gets you. Still a lot to make this silky smooth...

I had some time to experiment with this today, it not only works perfectly for my needs but the on-ramp for adding more stuff that I would need (other databases, etc) is clear. 🙂

KubeCon dust mostly settled now.
Checking back to see how things are going with your YS usage.


Boolean values not picked up by everything or scalar:true in subscription calls

@remorses , there may have been a change i am not aware of, although i went through the documentation . but recently , i am having to specify each boolean property and setting to true in a subscriptions, even after ...everything as that is now the only way they get pulled into the final result. is this by design ?
let { query, variables } = generateSubscriptionOp({
				employer: {
					__args: { id: appData?.orgId },
					...everything,
					employerLocations: { ...everything },
					employees: {
						...everything,
						isActive:true,
                         isApproved:true,
....

without explicit isActive:true and isApproved:true , those entries get omitted from the result

This is the case with queries too, not just subscriptions

can you paste the graphql schema here?

can you paste the graphql schema here?

Am using Hasura , but below is what the cli generated
"""Where all funds requested created by employee go """
type employeeRequest {
  acceptedTerms: Boolean!
  accessibleEarningsAtContractDate: numeric!
  approvalRequired: Boolean!
  approvedBy: String
  approvedOn: timestamptz
  cancellationReason: citext
  cancelledBy: String
  cancelledOn: timestamptz
  completedOn: timestamptz
  currentStatus: citext

  """An object relationship"""
  employeeRequestApprovedBy: employee

  """An object relationship"""
  employeeRequestCancelledBy: employee

  """An object relationship"""
  employeeRequestRequestedBy: employee!
  id: uuid!
  isApproved: Boolean!
  isCancelled: Boolean!
  isCompleted: Boolean!
  isPaidOut: Boolean!
  paidOn: timestamptz
  requestAmount: numeric!
  requestCode: citext!
  requestDate: timestamptz!
  requestedBy: String!
  totalEarningsAtContractDate: numeric!
}

@remorses ^^


svelte: error: process is not defined

"@urql/svelte": "0.1.3"
Using this project: https://github.com/frederikhors/svelte-urql-app, I had to add in rollup.config.js the below code to make it work:
import replace from '@rollup/plugin-replace'
...
replace({
    'process.env.NODE_ENV': JSON.stringify(
        production ? 'production' : 'development'
    ),
}),

or I had this error:
urql-core.mjs:501 Uncaught ReferenceError: process is not defined
    at new Z$1 (urql-core.mjs:501)
    at initClient (urql-svelte.mjs:18)
    at instance (App.svelte:6)
    at init (index.mjs:1385)
    at new App (App.svelte:12)
    at main.js:3
    at main.js:3

It can be useful?

Hey @frederikhors
This is because most off the time you'll do NODE_ENV=x rollup, this makes process.env defined. We use these checks to allow you to cut more bytes in a production build, this is a standard used in a majority of libraries.
replace({
    'process.env.NODE_ENV': JSON.stringify( 'production'),
}),

Removes all the development warnings saving a lot of bytes since these are just strings.


dont add the orders to table instead of clearing them

Like reported and merged in https://github.com/Laravel-Backpack/CRUD/pull/3428
Backpack were limiting the ordering of the table to 1 column at a time. That PR allowed to order by multiple columns removing the constraint from backpack.
Problem is that in customLogic, backpack still clears any past orders (assuming there would only be one as before), to add only the custom order by.
This PR removed that constraint from the orderLogic and adds the custom orderLogic on top of any orders previously set in crud.
Also, if you were ordering by a foreign column, eg. for monsters:
$this->crud->addColumns([
            'text',
            'textarea',
            [   // 1-n relationship
                'label'     => 'Select', // Table column heading
                'type'      => 'select',
                'name'      => 'select', // the column that contains the ID of that connected entity;
                'entity'    => 'category', // the method that defines the relationship in your Model
                'attribute' => 'name', // foreign key attribute that is shown to user
                'model'     => "Backpack\NewsCRUD\app\Models\Category", // foreign key model
                'wrapper'   => [
                    'href' => function ($crud, $column, $entry, $related_key) {
                        return backpack_url('category/'.$related_key.'/show');
                    },
                ],
                'orderLogic' => function ($query, $column, $columnDirection) {
                    return $query->leftJoin('categories', 'categories.id', '=', 'monsters.select')->orderBy('categories.name', $columnDirection)->select('monsters.*');
                },
            ],
    ]);

select is a column that exists in the monsters table, it's like category_id, but the ordering would be by category.name, not monsters.category_id. So when orderLogic is set, we don't apply backpack logic. This is not a BC because previously backpack would clear this order in customLogic().

A new inspection was created.

The inspection completed: No new issues

@pxpm I've approved the PR, as this was "in progress", I just moved it to "almost ready", please move to "ready to merge" if you think it ready 👌


Remove experimental status of Attachment components

Trello: https://trello.com/c/BQ5bBkMw/881-wrap-up-technical-loose-ends-on-attachments
As these are now used in production by Content Publisher and Govspeak
these are no longer considered experimental. Hopefully there won't be
need for any drastic API changes to them in the near future.
As these are purely internal changes I don't believe they need a changelog entry.

Thanks @alex-ju 👍


TypeError when importing accounts in steempy Python 2.7

In Python 2.7, the steempy cli will crash when calling importaccount with the below stack trace.
Traceback (most recent call last):
  File "/Users/jw/.local/share/virtualenvs/steem-python-Y0u0V5gr/bin/steempy", line 11, in <module>
    sys.exit(legacyentry())
  File "/Users/jw/.local/share/virtualenvs/steem-python-Y0u0V5gr/lib/python2.7/site-packages/steem/cli.py", line 1153, in legacyentry
    steem.commit.wallet.addPrivateKey(active_privkey)
  File "/Users/jw/.local/share/virtualenvs/steem-python-Y0u0V5gr/lib/python2.7/site-packages/steem/wallet.py", line 218, in addPrivateKey
    self.keyStorage.add(self.encrypt_wif(wif), pub)
  File "/Users/jw/.local/share/virtualenvs/steem-python-Y0u0V5gr/lib/python2.7/site-packages/steem/wallet.py", line 158, in encrypt_wif
    bip38.encrypt(PrivateKey(wif), self.masterpassword), "encwif")
  File "/Users/jw/.local/share/virtualenvs/steem-python-Y0u0V5gr/lib/python2.7/site-packages/steembase/bip38.py", line 59, in encrypt
    key = scrypt.hash(passphrase, salt, 16384, 8, 8)
  File "/Users/jw/.local/share/virtualenvs/steem-python-Y0u0V5gr/lib/python2.7/site-packages/scrypt/scrypt.py", line 215, in hash
    password = _ensure_bytes(password)
  File "/Users/jw/.local/share/virtualenvs/steem-python-Y0u0V5gr/lib/python2.7/site-packages/scrypt/scrypt.py", line 93, in _ensure_bytes
    raise TypeError('can not encrypt/decrypt unicode objects')
TypeError: can not encrypt/decrypt unicode objects

This appears to happen due to the handling of the user prompted input and the way it is being sanitized.

@cyon1c was this addressed? IIRC there was another utf fix since this issue was created.

I'm running into the same issue.

@jeffignacio is it possible for you to use Python 3.6? This issue only exists in Python 2.7.


Gauge title spanning two lines?

Just wondering if there is any method of having a gauge title span two lines. For example, referencing the attached image:
Main Bus
Current
Thanks,
Allan.

Change in from ox to ios then creat  a back door to intensefi  the lope to greater then 3.14

您的邮件我已收到，谢谢！


Move package gen to experimental package set

This PR closes #31.
Description of changes
Package gen is incredibly useful for writing twirp client generators. This patch exposes package gen in the experimental prefix /exp/. Think of /exp/ as halfway between internal and public code. No stability guarantees apply.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

We've released https://github.com/twitchtv/protogen to help meet this need. Hooray!


Stop abusing Blueprints._prepConstruction or make it public

Blueprints._prepConstruction does a lot. Its docstring even says as much
https://github.com/terrapower/armi/blob/f318392fd0804adc3a579c5edf6fca1df0e40d0f/armi/reactor/blueprints/__init__.py#L276-L278
Not only does it construct the design of each individual assembly type, it

Does processing on nuclides found in the blueprints.
Checks the area of assemblies.
Conditionally configures the axial snap mesh.
Conditionally performs initial thermal axial expansion on assemblies.
Calls the afterConstructionOfAssemblies plugin hook.

That's a lot! The value there is clear, as all those things must be done in order to make sure the assembly designs are ready to be used.
While I was working on the axial expansion work (#1843), I noticed that this private method is used directly. Which raises some hairs on the back of my neck because, 1) it points out this contradiction in its own docstring
https://github.com/terrapower/armi/blob/f318392fd0804adc3a579c5edf6fca1df0e40d0f/armi/reactor/blueprints/__init__.py#L288
and 2) I am of the belief that non-public methods (single or double leading underscore) should not be called by anything that is not the owning class. Yes, everything in Python is public, somethings just harder to access. But this public/protected/private distinction also allows developers some wiggle room.
If a method or attribute was truly protected, I have more freedom to refactor and break it up or think around that method. Maybe the four things it does don't all need to happen at once can be moved around. But if it's functionally public, it's a heavier lift.
Resolution
I propose that we rethink Blueprints._prepConstruction and determine how to either

stop misusing (in my opinion) a protected method, or
what functionality callers are expecting and make that public.

I've collected the cases where we use this method. Thankfully the majority of them are in testing. I presume that most are needed because we want to build assemblies without spinning up entire reactors from blueprints, but the blueprints class requires the assembly to be processed in _prepConstruction.
Non-test modules
https://github.com/terrapower/armi/blob/f318392fd0804adc3a579c5edf6fca1df0e40d0f/armi/cases/suiteBuilder.py#L168
https://github.com/terrapower/armi/blob/f318392fd0804adc3a579c5edf6fca1df0e40d0f/armi/reactor/reactors.py#L2471
Test modules
https://github.com/terrapower/armi/blob/f318392fd0804adc3a579c5edf6fca1df0e40d0f/armi/reactor/tests/test_blocks.py#L316
https://github.com/terrapower/armi/blob/f318392fd0804adc3a579c5edf6fca1df0e40d0f/armi/reactor/tests/test_assemblies.py#L597
https://github.com/terrapower/armi/blob/f318392fd0804adc3a579c5edf6fca1df0e40d0f/armi/reactor/tests/test_assemblies.py#L1442
https://github.com/terrapower/armi/blob/f318392fd0804adc3a579c5edf6fca1df0e40d0f/armi/reactor/blueprints/tests/test_blueprints.py#L63
https://github.com/terrapower/armi/blob/f318392fd0804adc3a579c5edf6fca1df0e40d0f/armi/reactor/blueprints/tests/test_blueprints.py#L306
https://github.com/terrapower/armi/blob/f318392fd0804adc3a579c5edf6fca1df0e40d0f/armi/materials/tests/test_materials.py#L2013
https://github.com/terrapower/armi/blob/f318392fd0804adc3a579c5edf6fca1df0e40d0f/armi/cases/inputModifiers/tests/test_pinTypeInputModifiers.py#L31
https://github.com/terrapower/armi/blob/f318392fd0804adc3a579c5edf6fca1df0e40d0f/armi/reactor/blueprints/tests/test_materialModifications.py#L62
https://github.com/terrapower/armi/blob/f318392fd0804adc3a579c5edf6fca1df0e40d0f/armi/cases/inputModifiers/tests/test_inputModifiers.py#L131
https://github.com/terrapower/armi/blob/f318392fd0804adc3a579c5edf6fca1df0e40d0f/armi/reactor/blueprints/tests/test_assemblyBlueprints.py#L182
https://github.com/terrapower/armi/blob/f318392fd0804adc3a579c5edf6fca1df0e40d0f/armi/reactor/blueprints/tests/test_blockBlueprints.py#L265

I'm not worried about the times when this pops up in testing.
Unit tests should be able to test private methods.  Just because logic is internal to a class doesn't mean it's not complicated enough, or important enough, to warrant testing.

Unit tests should be able to test private methods

Agree to a point. A unit test on the Blueprints object using and testing protected methods on that object> I can get behind that.
But at test for a materials object needing a private method from `Blueprints?
https://github.com/terrapower/armi/blob/f318392fd0804adc3a579c5edf6fca1df0e40d0f/armi/materials/tests/test_materials.py#L2010-L2014
I think that's stretching the domain of applicability.
I think nearly all of these could be mitigated if we have a Blueprints.buildAssembly method because that's what the majority of these tests need. After some brief skimming at least.

this only pops up once in a non-test environment

Twice
https://github.com/terrapower/armi/blob/f318392fd0804adc3a579c5edf6fca1df0e40d0f/armi/cases/suiteBuilder.py#L168
and
https://github.com/terrapower/armi/blob/f318392fd0804adc3a579c5edf6fca1df0e40d0f/armi/reactor/reactors.py#L2471

I agree, if this is needed to be public in a non-test setting, we should probably make it public. Sure.

FWIW I have been able to have identical success replacing patterns of
bp._prepConstruction(cs)
assem = bp.assemblies[assem_name]

with
assem = bp.constructAssembly(cs, name=assem_name)

The tradeoff is bp.constructAssembly does a deepcopy of an assembly and calls Assembly.makeUnique prior to returning the new assembly. Those may or may not be important for testing but are certainly important for production analysis


Add an Iterator::scan equivalent

fn scan<St, B, F>(self, initial_state: St, f: F) -> Scan<Self, St, F>
    where F: FnMut(&mut St, Self::Item) -> Option<B>
{ ... }

@nikomatsakis's notes from the wiki:

this is generally possible (it will be a bit different; more like reduce), but it is non-trivial
it requires >1 pass and hence produces an intermediate collection
basically you would do a parallel fold operation, performing a sequential scan for each segment

the result of your scan is a VecTree<Vec> of same total length as your input iterator
this gives you partial sums (to consider one concrete application) at each point

you can then traverse the tree (sequentially) to compute, for each vector, the total sum of preceding vectors
you can then re-traverse the tree (in parallel) and add this total sum to each element of the vector (in place)
that last step can itself drive another consumer, so this can still be a parallel adapter

if we are clever, it can even be an exact one, though it will require a nice VecTree implementation

I'd like to try this out, but I'm having a hard time understanding the notes from the wiki.

basically you would do a parallel fold operation, performing a sequential scan for each segment

Ok, so the iterator will be split up to a point, and then we scan each part individually (more or less by using std::iter::iterator::scan), but in parallel? If so, how do we handle the state, as it is &mut? Or do we assume the state is Copy, and copy it for each sequential scan such that all segments starts with a fresh state? The alternative (the one I see, anyways) is to just do everything sequentially, but what do we gain by that?

the result of your scan is a VecTree of same total length as your input iterator

I'm not sure what this means. VecTree? Is this just the "split tree", in which the nodes are the segments as Vecs?
The last three points are also unclear, but I don't want to flat out take a guess at what is meant, as it might be clear if I can understand the above 😄

I also asked @nikomatsakis to clarify, because I don't really get it either. 😄
I can answer about VecTree: that's a hypothetical tree-of-vectors that we had thought about for general purpose collect -- basically folding into a vector locally, then reducing those into a tree that maintains structure.  But we realized that the structure doesn't actually matter, only the order, so today the collect implementations that need it just use LinkedList<Vec<T>>.

Sorry for not responding. I'll try to figure out what I meant and write it up. ;)

Just dropped by to +1 this since I was looking to compute a prefix sum with rayon.

hi, I've done a LOT of prefix sum implementations.
you can find a complex one here : http://www-id.imag.fr/Laboratoire/Membres/Wagner_Frederic/beyond-fold-work.html and another one here : https://github.com/wagnerf42/rayon-adaptive/blob/new_api/examples/cut_prefix.rs with many other prefixes in the same folder.
I also have a "successors" implementation in my crate.

@wagnerf42 There is a 'TODO: wip' in the code you linked to. care to elaborate?
Possibly related/useful is this information about doing this kind of stuff on the GPU: https://people.eecs.berkeley.edu/~driscoll/cs267/papers/gpugems3_ch39.html

It would be excited to see the scan implemented!

I has been 5 years. ^^

It has, but it is never too late to add a cool and useful feature :wink:

I has been 5 years. ^^

It has, but it is never too late to add a cool and useful feature 😉

Agree! It would be excited to see scan implemented in Rayon!

I'm interested in working on this. I've tried to write some code to test out the ideas here and in #393. I think a LinkedList might be sufficient, rather than adding the overhead of a tree.

could you give some details on how you want to do it ? there are several ways, with different pros and cons

I heard from someone who wanted to use my above PR in their project, so I've released it as a separate crate: https://crates.io/crates/rayon-scan
I hope this will be helpful to anyone else in this thread who is looking for this feature!


Push Notification initilization

Hi,
q-municate is really helpful to understand detail of Quickblox and its SDK.
But question appeared after integration of firebase with q-municate.
Firebase authentication via phone number is working but FCM isn't.
I followed the steps provided by your team. (copied sender_id, legacy server key into project; added dependencies firebase-messaging etc. (via Tools -> Firebase in Android Studio))
But notification is not working. How do I solve the issue? Is there any complete documentary or guide to add notifications in q-municate?
Thank you

you shouldn't add dependency to firebase-messaging if you don't need newest version of this lib (in latest version of QuickBlox Android SDK integrated version 11.4.2). If you made all needs from guide, try uninstall Q-municate and install it again. If issue will be actual again, provide there logs from starting your app to getting issue.


Fill match arms assist: add remaining arms for tuple of enums

Fix for #8493
However, the assist is still flaky and does not use hir_ty::diagnostics::match_check

bors r+


issue with thunderbolt patcher

The thunderbolt doesn't see in system report after running a thunderbolt patcher. it runs a script successfully. I can not open update thunderbolt firmware to rollback original on Windows (I have uninstalled patcher without fail before rollback)
HaC Mini version: The last build
OSX version: 10.15.5
NUC model: NUC8i7HVK
Boot SSD model and protocol Samsung 970 Evo
Second SSD model and protocol (if installed): Inland Premium
First DIMM model: Kingston DDR4 8GB
Second DIMM model (if installed): Kingston DDR4 8GB
Display port(s) used [e.g: USB-C, mini DP]: HDMI, USB C, mini DP
Display resolution(s) [e.g: 4K, 5K]: 4K
Any TB3 devices:
Any USB devices: mouse and keyboard
thunderbolt report.zip

I’m confused the second screenshot shows a successful uninstallation and the first screenshot shows that your firmware has been reverted to the original. What’s the issue?

I’m confused the second screenshot shows a successful uninstallation and the first screenshot shows that your firmware has been reverted to the original. What’s the issue?

Both pictures also ran Uninstall to make sure it still has original after uninstall. I just want to report for you clear. I don't know why the patcher runs very quickly. It's not like the first time I ran a patch

It seemed to have uninstalled successfully. I don’t know what your issue is.

It seemed to have uninstalled successfully. I don’t know what your issue is.

yeah me too. At this time, only USB C works on thunderbolt port. I use method thunderbolt legacy and it doesn't work anymore

Does it work on windows?

Does it work on windows?

It doesn't work.

I found Thunderbolt in Windows strange to manage - I had to install/open Thunderbolt Control Center then 'approve' my TB device before it would work - have you tried that?
Also don't forget to set the BIOS Thunderbolt settings back from Legacy Mode, if you made that change.

I found Thunderbolt in Windows strange to manage - I had to install/open Thunderbolt Control Center then 'approve' my TB device before it would work - have you tried that?
Also don't forget to set the BIOS Thunderbolt settings back from Legacy Mode, if you made that change.

yes, I tried but still not open firmware update. Maybe the firmware has a problem

Oh I see you can't run the Intel firmware update on Windows? Yes I have the same issue and I think it's an issue with the updater, not the firmware.  You can use Thunderbolt Control Centre to check the firmware version of the controller and if it's the same as the downloaded on I wouldn't worry about it. (I think v33?)

Oh I see you can't run the Intel firmware update on Windows? Yes I have the same issue and I think it's an issue with the updater, not the firmware. You can use Thunderbolt Control Centre to check the firmware version of the controller and if it's the same as the downloaded one I wouldn't worry about it. (I think v33?)

Have you fixed this problem yet?

Mine shows 33 - will take a picture tomorrow (UK time). Not sure what's going on in your picture, but don't forget you need a device plugged in for certain aspects of the driver to work ...

Do you have the latest drivers / software? That's not the screen I have and both Windows Update and the Intel Driver Update program day I'm on the latest. Attached are the details on my system.

Do you have the latest drivers / software? That's not the screen I have and both Windows Update and the Intel Driver Update software say I'm on the latest. Attached are the details on my system.

thanks for your help. But certainly, I have to flash the firmware with PSI chip

Fair enough, pleased you've worked it out.
Actually I was curious why I myself couldn't run the Thunderbolt FW updater even though my firmware was OK ... turns out that it will only work with the so-called 'legacy' Thunderbolt driver / software and if you've ever installed the newer 'DCH' drivers it's a little tricky to go back to the legacy ones (you have to remove them from Device Manager but install the new ones before Windows attempts to reinstall).
Also, by default it expects a Thunderbolt device plugged in to run the updater, which I was trying with a Thunderbolt monitor but at the end of the update the monitor would reconnect and throw the updater out.  So instead if you start the updater with the Thunderbolt ports empty, then it installs a work around to power the controller on without there being something plugged in and therefore the updater will proceed.
So anyway I re-ran the updater successfully so I am sure I am back to the original firmware now.

fixed with Ch341a flash. thx @osy86 and @hepto support me


how to fix this error

Error detected while processing function xtabline#hi#init[2]..xtabline#hi#apply_theme[7]..
29:
line   13:
E492: Not an editor command:  guifg=#af0000 gui=bold cterm=bold

and the nvim version.

Probably due to colorscheme, I'll fix it asap.

Is it fixed?

yes, it worked. Thanks.


Add feature control configuration in config.yaml

enableFeatures: ["portfolio", "blog", "notes", "documentation"]

Alternatively you could allow to enable them one by one:
portfolio:
  enable: true
blog:
  enable: true
  some_blog_feature: 7

This is how it is done in nixos and it is a cool way to have the configuration of the feature and the enabling of the feature in one place.

@Kaligule Yes. I am probably going this way. This will allow to provide additional configuration for the features.

Already done in main branch.


docs: Fix a few typos

There are small typos in:

dev-docs/RFCs/v4.0.0/error-handling-rfc.md
website/docs/highlights/indexing.md
website/docs/library/errors.md
website/versioned_docs/version-3.x/highlights/indexing.md

Fixes:

Should read preferable rather than preferrable.
Should read hierarchical rather than hierachical.
Should read ambiguous rather than ambigous.

Semi-automated pull request generated by
https://github.com/timgates42/meticulous/blob/master/docs/NOTE.md

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Coverage remained the same at 98.513% when pulling a3440fb6ff370c39f267847309d4979844aca812 on timgates42:bugfix_typos into ea8526f5407ae72ae78265cba68f8a2c5db689c8 on uber:master.

Thanks for spotting and fixing these @timgates42!


problem: access to ledger apps assumes the knowledge of lib internals

solution: refactor and provide a single method to access apps

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 25 (0.0%)  changed or added relevant lines in 2 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage increased (+0.04%) to 19.973%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/ledger/app_ethereum.rs
0
11
0.0%

src/ledger/app_bitcoin.rs
0
14
0.0%

Files with Coverage Reduction
New Missed Lines
%

src/ledger/app_bitcoin.rs
1
41.42%

Totals

Change from base Build<PHONE_NUMBER>:
0.04%

Covered Lines:
450

Relevant Lines:
2253

💛  - Coveralls

Codecov Report

Merging #3 (eccb650) into master (85fc1d9) will increase coverage by 0.13%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master       #3      +/-   ##
==========================================
+ Coverage   30.08%   30.21%   +0.13%
==========================================
  Files          10       10
  Lines        1160     1155       -5
==========================================
  Hits          349      349
+ Misses        811      806       -5

Impacted Files
Coverage Δ

src/ledger/app_bitcoin.rs
41.42% <0.00%> (+0.36%)
:arrow_up:

src/ledger/app_ethereum.rs
28.87% <0.00%> (+0.40%)
:arrow_up:

src/ledger/manager.rs
54.66% <ø> (ø)

src/ledger/traits.rs
0.00% <ø> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Add Plugins

I am very sorry to report that the Add Plugins page is cleaned even when Airplane-Mode is disabled.

I think the number of if ( $this->enabled() )-s should be reduced.

I think the problem here is the last bit of CSS should only be loaded when Airplane Mode is enabled.

https://github.com/norcross/airplane-mode/blob/master/airplane-mode.php#L91-L95
is unconditional.

These new features shouldn't be executed when Airplane Mode is disabled.

I'm working on a patch now

#52 should fix this.

I think add_filter( 'install_plugins_tabs', array( $this, 'plugin_add_tabs' ) ); hides the tabs.

Please move if ( $this->enabled() ) { to line #68

@szepeviktor we are addressing this in a different way

Thank you. Please close it after your commit.

fixed with https://github.com/norcross/airplane-mode/commit/24724297c4f8547703c0950990a31e8e271fa6e1

@norcross Sorry, I do not know what this CSS thing is about.
The original issue was that these are excuted when Airplane Mode is disabled:
https://github.com/norcross/airplane-mode/blob/master/airplane-mode.php#L62-L110

I was reporting the missing tabs of the Add Plugins page.

Does that issue still exist in the current master branch?

YEs.

A screenshot would be very helpful.

Oh! I am very sorry. In the current master everything works.
I just haven't noticed the if ( ! $this->enabled() )-s.
Could all these if-s move into the constructor?

@szepeviktor I have no plans on moving them all to the constructor. since they involve various checks, filters, etc and the plan is for some to have optional setups for user to change status in different ways.


NaN values being saved in Whisper datafiles

hi @lomik , With reference to https://bugs.launchpad.net/graphite/+bug/867882
Somehow I'm finding my whisper files to contain nan, and while this is valid from a python JSON spec, it's breaking the Grafana javascript code which parses the JSON from graphite. And causes an error and does not plot graphs.
Apparently carbon-cache was patched to eliminate such cases of writing nan to the whisper files. Can you look into this?

to further illustrate, here's an example of the cpu data gathered from collectd. This data is replicated and pushed to two hosts, one runs go-carbon, and the other carbon-cache:
with go-carbon gives the below two occurrences:
whisper-dump.py /home/graphite/storage/dc1/apphost00/cpu/0/cpu/interrupt.wsp | grep nan
0:<PHONE_NUMBER>,        nan
0:<PHONE_NUMBER>,        nan
while with carbon-cache gives no output, therefore no nan's. Looking up these timestamps in carbon-cache's wsp files, they're not to be found. It seems that the default behaviour here is to drop any write to whisper containing 'nan'

hello again @lomik i was able to achieve this by adding the below condition in whisper.go before the write happens:
if !math.IsNaN(r.Value) {
    points[i] = &whisper.TimeSeriesPoint{Time: int(r.Timestamp), Value: r.Value}
}

naturally, in the beginning I also had to define importing of math

We are experiencing the same bug in production. Pull #27 fixes it (with tests, too!).
<3


ML: VGG16 Model #1 (24 Layers): Configuration#1: Training model and store the results in the directory for 30 and 50 epochs

Model Architect:

4Conv(Conv+Conv+MaxPool+Dropout)
4Dense(Dense+Dropout)

Put the training on the Grid by using jetscape-ml-tensorflow-cnn.sh
save the model and the evaluation data in the power point slides
update the diagrams data on the power point slides

30 epochs
Epoch 00030: val_accuracy improved from 0.82210 to 0.82210, saving model to /wsu/home/gy/gy40/gy4065/hm.jetscapeml.data/simulation_results/simulation-results-deep-model-cnn-01-1200K-config-01/Models_PP19_vs_LBT_Hadron_ch1/best_model.h5
840000/840000 [==============================] - 309s 368us/sample - loss: 0.8033 - accuracy: 0.8092 - val_loss: 0.7783 - val_accuracy: 0.8221

epoch 50
Epoch 50/50
839936/840000 [============================>.] - ETA: 0s - loss: 0.6099 - accuracy: 0.8146
Epoch 00050: val_accuracy did not improve from 0.82648
840000/840000 [==============================] - 323s 385us/sample - loss: 0.6099 - accuracy: 0.8146 - val_loss: 0.5948 - val_accuracy: 0.8264


Day10

Resolves #21

Pull Request Test Coverage Report for Build 51

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build 46:
0.0%

Covered Lines:
24

Relevant Lines:
24

💛  - Coveralls

Pull Request Test Coverage Report for Build 51

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build 46:
0.0%

Covered Lines:
24

Relevant Lines:
24

💛  - Coveralls


ZIO failure in ZConnectionPool.transaction

I'm getting an InterruptedException in ZConnectionPool.transaction. I am unable to mapError or tapError on the failure. When I run it unsafe, I get a zio.Exit.Failure. Nothing I have tried has allowed me to see the cause of the failure, e.g., the stack trace of the InterruptedException.
I've boiled my problem down to a pretty minimal test case. If I take out the debugging code, which I thought you might find helpful, it comes in well under 50 lines. I am using zio version 2.1.9 and zio-jdbc version 0.1.2
import zio.{ ZIO, ZLayer }
import zio.jdbc.{ ZConnectionPool, ZConnectionPoolConfig, stringToSql }
import zio.test.Assertion.equalTo
import zio.test.{ ZIOSpecDefault, assert }

class Api(val connectionPool: ZConnectionPool) {
  val query     = stringToSql("SELECT 'foo'").query[String]
  val selectOne = connectionPool.transaction(query.selectOne)

  val fooZIO: ZIO[Any, Throwable, Option[String]] = for {
    _   <- ZIO.debug("before selectOne")
    foo <- selectOne.tapError(e => ZIO.debug(s"tapError selectOne $e"))
    _   <- ZIO.debug("after selectOne")
  } yield foo
}

object ApiSpec extends ZIOSpecDefault {
  val connectionPoolConfigLayer = ZLayer.succeed(ZConnectionPoolConfig.default)

  val connectionPoolLayer = ZConnectionPool.postgres(
    "localhost", 5432, "test", Map("user" -> "test", "password" -> "test")
  )

  val apiLayer = ZLayer(ZIO.service[ZConnectionPool].map { connectionPool =>
    new Api(connectionPool)
  })

  val layer = connectionPoolConfigLayer >>> connectionPoolLayer >>> apiLayer

  zio.Unsafe.unsafe { implicit unsafe =>
    val z = zio.Runtime.default.unsafe.run {
      for {
        _ <- ZIO.debug("before")
        z <- ZIO.service[Api].provide(layer).flatMap(_.fooZIO)
          .tapError(e => ZIO.debug(s"EEE $e"))
          .tapDefect(e => ZIO.debug(s"EEE $e"))
          .tapBoth(e => ZIO.debug(s"EEE $e"), e => ZIO.debug(s"EEE $e"))

        _ <- ZIO.debug("after")
      } yield z
    }
    println(z)
  }

  def spec = test("api.fooZIO should be Some(foo)") {
    for {
      api      <- ZIO.service[Api].provide(layer)
      _        <- ZIO.debug("before login")
      response <- api.fooZIO
      _        <- ZIO.debug("after login")
    } yield {
      assert(response)(equalTo(Some("foo")))
    }
  }

}

Here is the output when I run sbt test:
before
before selectOne
Failure(Both(Empty,Both(Empty,Interrupt(Runtime(378775985,1727107547164,zio.jdbc.ZConnectionPool.make.tx(ZConnectionPool.scala:168)),Stack trace for thread "zio-fiber-378775985,2016155100,708221690,403450518":
	at zio.jdbc.ZConnectionPool.make.tx(ZConnectionPool.scala:168)
	at <empty>.Api.selectOne(ApiSpec.scala:8)
	at <empty>.Api.fooZIO(ApiSpec.scala:12)
	at <empty>.Api.fooZIO(ApiSpec.scala:14)
	at <empty>.ApiSpec.<local ApiSpec$>.z(ApiSpec.scala:35)
	at <empty>.ApiSpec.<local ApiSpec$>.z(ApiSpec.scala:36)
	at <empty>.ApiSpec.<local ApiSpec$>.z(ApiSpec.scala:37)
	at <empty>.ApiSpec.<local ApiSpec$>.z(ApiSpec.scala:40)
	at <empty>.ApiSpec.<local ApiSpec$>.z(ApiSpec.scala:41)))))
before login
before selectOne
- api.fooZIO should be Some(foo)
  Exception in thread "zio-fiber-59,58,56,51" java.lang.InterruptedException: Interrupted by thread "zio-fiber-59"
  	at zio.jdbc.ZConnectionPool.make.tx(ZConnectionPool.scala:168)
  	at <empty>.Api.selectOne(ApiSpec.scala:8)
  	at <empty>.Api.fooZIO(ApiSpec.scala:12)
  	at <empty>.Api.fooZIO(ApiSpec.scala:14)
  	at <empty>.ApiSpec.spec(ApiSpec.scala:53)
  	at <empty>.ApiSpec.spec(ApiSpec.scala:54)
0 tests passed. 1 tests failed. 0 tests ignored.

- api.fooZIO should be Some(foo)
  Exception in thread "zio-fiber-59,58,56,51" java.lang.InterruptedException: Interrupted by thread "zio-fiber-59"
  	at zio.jdbc.ZConnectionPool.make.tx(ZConnectionPool.scala:168)
  	at <empty>.Api.selectOne(ApiSpec.scala:8)
  	at <empty>.Api.fooZIO(ApiSpec.scala:12)
  	at <empty>.Api.fooZIO(ApiSpec.scala:14)
  	at <empty>.ApiSpec.spec(ApiSpec.scala:53)
  	at <empty>.ApiSpec.spec(ApiSpec.scala:54)
Executed in 564 ms

[info] Completed tests
[error] Failed tests:
[error] 	ApiSpec
[error] (Test / testOnly) sbt.TestsFailedException: Tests unsuccessful

You will note from the output that the error is completely untappable. It's possible that there is something wrong with my zio-jdbc code and/or Postgres setup, and there is a real error hidden in there somewhere. But I highly doubt it because, inexplicably, I have another test that does nearly the exact same thing, and it passes. And I mean, it passes every time. And this test I shared here fails every time.
The output shows line 168 in ZConnectionPool.scala. That's from zio-jdbc version 0.1.2. Poking a bit in the debugger, the error seems to arise from ZConnection.close.
Whatever is going on, it seems pretty clear that there is a zio-jdbc bug in here somewhere, because I really should be able to map or tap the error.

I've got things working locally by moving the call to .provide(layer) to the end, like so:

  def spec = test("api.fooZIO should be Some(foo)") {
    val testResult = for {
      api      <- ZIO.service[Api]
      _        <- ZIO.debug("before login")
      response <- api.fooZIO
      _        <- ZIO.debug("after login")
    } yield {
      assert(response)(equalTo(Some("foo")))
    }
    testResult.provide(layer)
  }

I still think this is a bug, because no tappable error.
Also, I would love it if someone could explain to me why there is a semantic difference between these two versions of the same test.

Also, I would love it if someone could explain to me why there is a semantic difference between these two versions of the same test.

Is it because I am providing the connection pool only to the service lookup for the Api, and by the time I get around to calling api.fooZIO, that connection pool is already shutting down?
If this is so, shouldn't my for/yield block fail to type-check somehow?

Layers are scoped to the effect/workflow they are provided to, which means the connection pool is released (shut down) immediately after creation, which causes all connection borrow attempts to be immediately interrupted.
I agree that an error or a defect makes more sense in than an interrupt, in this case


Communication with CSV

We should use CSV files and process them in CLI over using custom formats created by daemon or straight up doing the tree processing inside the daemon.

Also communicating entirely through socket would be even better. I will look into it and if possible I will mark this issue as wontfix and open a new one.


Pushed Authorization Requests and Rich Authorization Requests

The purpose of this is to request a potential future feature. The problem statemets are copied as indicated by links augmented with more IdentityServer related information.
Rich Authorization Requests

Expressiveness of scopes is not sufficient for emerging OAuth scenarios, e.g. open banking.
Allocation of requested permissions to resource server specific access token is hard (despite resource indicators).

Source: Rich Authorization Requests (SlideShare) by Torsten Lodderstedt.
Pushed Authorization Requests

OAuth authorization code flow sends parameters as URI query parameters via redirection in the user agent.
Challenges:
There is no cryptographical integrity and authenticity protetion.
There is no mechanism to ensure confidentiality of request parameters.
Authorization request URLs can become quite large, especially in scenarios requiring fine-grained authorization data.

Source: Pushed Authorization Requests (SlideShare) by Torsten Lodderstedt.
Additional context
Rich OAuth 2.0 Authorization Requests (Medium) by Torsten Lodderstedt.
Transaction Authorization or why we need to re-think OAuth scopes (Medium) by Torsten Lodderstedt.
Comments back to “Transaction Authorization or why we need to re-think OAuth scopes” by Nat Sakimura.
IdentityServer for ASP.NET Core 3 by Dominick Baier due to mention of FAPI and its financial use these other APIs closely related, also banking issues and also CIBA that seem to be related to FAPI.

Thanks -
yes there are a couple of things we want to re-work.
This is tracked here
https://github.com/IdentityServer/IdentityServer4/issues/3520
Feel free to continue the conversation on this thread.


Dispatch an event when proxy initialization fails

Supersedes #532, Closes #1258.
This builds on top of the code added in #532. The new DocumentNotFoundEventArgs class contains the proxy object and its identifier. Event subscribers can choose to handle the missing document and call disableException() on the $eventArgs object to prevent the DocumentNotFound exception from being thrown.
As mentioned in the ticket, this event is not available as a lifecycle callback because we don't have a valid object on which to call a lifecycle callback.

To be honest, I'm not sure about the whole "prevent this exception from being thrown" thing. It was an idea that popped in my head last night and I thought I'd go with it. In theory, it makes sense but I think in practice there will be issues with that:

The proxy object is empty, e.g. no initialization has been taken care of
The proxy object has a different constructor, so an event subscriber can't call the original constructor of the class to prepare it as an empty object
There may not be null-protection checks in place so instead of exceptions users might be left with fatal errors

After additional discussion we decided to completely open up the disableException behavior for subscribers to disable and re-enable the exception as they wish. When disabling the exception, a subscriber must ensure the proxy object is properly initialized.

@alcaeus Do you have an example of implementation ?
I would like to disable this exception everywhere, and retrieve a null value when i call the object.

The documentation specifies how you can disable the exception: http://docs.doctrine-project.org/projects/doctrine-mongodb-odm/en/latest/reference/events.html#documentnotfound. Replacing the proxy object with null is not possible: proxy initialization is called from within a proxy object and we can't replace an object itself with null. This is not possible with the current proxy system.

With the exception disabled, what happens if i call for example $this->getImage().
I will have a proxy object of Image class ?
Thanks for your answer =)

Let's say you loaded a document User that has a (non-existing) reference to an Image. Calling $user->getImage() will always yield an instance of an Image: if the Image instance is already managed by the document manager, it will return that instance, regardless of whether it's a proxy or not. If it's not, the document manager will create an instance of a proxy class that extends the original Image. It's important to note that just calling $user->getImage() will not trigger proxy initialization, and neither will calling $user->getImage()->getId().
Only when calling any other method or accessing a public property of $user->getImage() does proxy initialization kick in: the initializer tasks the DocumentPersister with loading data for the ID that is already known. If the initializer doesn't find a document, the documentNotFound event is thrown. If you catch that event, you get the uninitialized proxy instance and can do whatever you want with it. If you call disableException, the initializer will not throw a DocumentNotFoundException, but the proxy object will always be in the state you left it in.
It's important to note that at the time of initialization, we can't reset the invalid proxy instance. The proxy object might be used by multiple documents (think of multiple users holding a reference to the image), and since the proxy object itself is responsible for initializing itself, we can't change anything about the proxy object because you can't overwrite the value of $this.
If you want to control what happens with an invalid reference, initialize it in a controlled way while you still know the owning object and catch the DocumentNotFoundException that is thrown. I hope this helps clear up some confusion.

It is, very detailed answer. Thanks a lot


Bugfix/time_sync

'offset_ns' can be negative

@devbharat Thanks!

Would you mind going over the mavros side as well, so we can have the full loop verified.

I did. It appears to be correctly handled in the mavros plugin. I tested it in the loop and with these fixes it had sensible values for both timestamp_boot and timestamp_computer. I'd still prefer someone else test it too, as my setup manages two pixhawks on one companion computer and I've had to make other changes for that.

Reviewed and applied to master, thanks!


Two Way Binding not working

I'm trying to get 2 way binding to work properly. I setup a test case here, so you can see what we are trying to do.
http://plnkr.co/edit/KjQ9DusaFg2jp1BTFUde?p=preview

you need to specify is="auto-binding" when binding outside of polymer
   <template is="auto-binding">
       <test-app></test-app>
   </template>

:+1: @pflannery - can this be closed?

I don't quite understand where the binding is done outside of polymer in that example - as far as I can see, the binding is attempted inside the test-app polymer element by using the same property for both test-input elements.
IMO, the issue here is that they are attempting to reflect the changes occurring in one of the properties back out to the value of the attribute. I found an example of this here, under 'attribute reflection' :
https://divshot.com/blog/web-components/polymer-features-you-may-have-missed/
I guess using auto-binding might be a 'better way - but is it as simple as wrapping it in an auto-binding template? ...and isn't the example supposed to work too, albeit with the reflect: true addition (and perhaps removal of the *10 which just makes things confusing)?
...or could you do something like this :
<test-input id='one' value='{{$.two.value}}'></test-input>
<test-input id='two' value='{{$.one.value}}'></test-input>

So, when one is updated, the other is too (and vice versa.

The example has changed since I last looked at this. It was just binding directly to the {{value}} not {{hue * 10}}, here is the original http://plnkr.co/edit/LMMrosVjNca2zGFqMsdz?p=preview
I've noticed what's happening now, if the polymer elements in the original example are are swapped around so that test-input comes before test-app then it works.
@Nevraeka perhaps this is still a bug, but it's not something I can clarify. I have a feeling though that dependencies should be defined before using them.

In the case of the hue * 10 binding, I think the issue is that the expression is basically one way. If you use a simple identifier, Polymer sets up a two-way binding, but if you have a more complicated expression, it doesn't. (In this case, you can guess what the user wants to do, but if you consider an expression like hue * multiplier, there's no way for the library to guess what you'd want to assign.
With filters, you can define a reversible transformation by specifying a toModel function. See:
http://plnkr.co/edit/XzCRMxxd30Zu2ssSjJbg?p=preview
@davidmaxwaterman, reflect is different from two-way data binding. reflect causes the corresponding attribute to be set whenever the property value changes. That's useful primarily if you want to style (or querySelect) based on an attribute value.
For more, see:
https://www.polymer-project.org/docs/polymer/polymer.html#attrreflection
And finally, @pflannery is correct that dependencies need to be defined before they're used. Otherwise, data binding may not get set up correctly.

:+1: I was suspicious of the *10 :) anyway, great explanations - thanks guys.

Closing this issue due to age and the release of version 1 of Polymer - please feel free to re-open if this is incorrect.


Documentation update for network timeout

Per the docs for hackney (https://github.com/benoitc/hackney/blob/master/doc/hackney.md) the option for setting the timeout is recv_timeout instead of timeout

Coverage increased (+0.6%) to 92.308% when pulling eb78ca7ad05b19e7f3d7f609e3841e091a31ec81 on matthusby:patch-1 into c0459d7a43bbc43ee2263b05eb1ac13680932120 on scrogson:master.

:+1: thank you


Support chunksize parameter for read_parquet with a single file

I'd like to be able to read a single parquet file into multiple partitions, determined by the chunksize.
Without chunksize
import pandas as pd
import dask.dataframe as dd

df = pd.DataFrame({"a":range(100000), "b":range(100000)})
df.to_parquet("out.parquet")

df = dd.read_parquet("out.parquet",
                     gather_statistics=True, split_row_groups=True)
df.npartitions
1

With chunksize
import pandas as pd
import dask.dataframe as dd

df = pd.DataFrame({"a":range(100000), "b":range(100000)})
df.to_parquet("out.parquet")

df = dd.read_parquet("out.parquet",
                     chunksize="10 MiB",
                     gather_statistics=True, split_row_groups=True)
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-20-25c43bb02cd0> in <module>
      7 df = dd.read_parquet("out.parquet",
      8                      chunksize="10 MiB",
----> 9                      gather_statistics=True, split_row_groups=True)

/opt/conda/envs/rapids/lib/python3.7/site-packages/dask/dataframe/io/parquet/core.py in read_parquet(path, columns, filters, categories, index, storage_options, engine, gather_statistics, split_row_groups, chunksize, **kwargs)
    229     # Parse dataset statistics from metadata (if available)
    230     parts, divisions, index, index_in_columns = process_statistics(
--> 231         parts, statistics, filters, index, chunksize
    232     )
    233

/opt/conda/envs/rapids/lib/python3.7/site-packages/dask/dataframe/io/parquet/core.py in process_statistics(parts, statistics, filters, index, chunksize)
    619         # Aggregate parts/statistics if we are splitting by row-group
    620         if chunksize:
--> 621             parts, statistics = aggregate_row_groups(parts, statistics, chunksize)
    622
    623         out = sorted_columns(statistics)

/opt/conda/envs/rapids/lib/python3.7/site-packages/dask/dataframe/io/parquet/core.py in aggregate_row_groups(parts, stats, chunksize)
    722
    723 def aggregate_row_groups(parts, stats, chunksize):
--> 724     if not stats[0]["file_path_0"]:
    725         return parts, stats
    726

KeyError: 'file_path_0'

cc @rjzamora because of the file_path_0 bug
@beckernick just to be clear the finest granularity that we're able to manage is an individual row group.  Parquet doesn't really support reading data at finer granularities.  In your case I would ask the question of how did pandas.DataFrame.to_parquet choose to save your file?  Did it use row groups?  If not then there is nothing that we can do here.

Thanks @mrocklin , makes sense 👍 . This issue was primarily motivated by the file_path_0 bug that comes up while using chunksize.

Yup.  Thanks for raising!

Thanks for raising @beckernick !  Ill add a test for the case like this where the row-group aggregation is impossible.  Although row-group aggregation cannot work in this case, there should certainly not be an error - Sorry about that!


Add TVL

Working on the app I realized that we had no way to get the TVL, if not monitoring the deposits and saving data in a database or a theGraph node. So, I added a TVL struct and an internal function for that.
I also realized that the contract SeedPool was not upgradeable if adding more storage in SidePool (which is extended by SeedPool) because of the variable factory in SeedPool, locking the contract from upgrading the storage. So, I move the variable in SidePool, even if in SidePool it is never used.
PS > I will merge it later in the day, since tomorrow CertiK will take the contracts for the audit.

since all the synr and passes is transferred to the contract, the TVL can be calculated by looking at the synr/passes hold in the contract?


A edge 05

another round of trying Edge browser against current master

see #116 for the next version


Build spirv-cross in parallel

Building spirv-cross2 in parallel when possible would save significant amounts of initial build time for users of this crate. The related glslang crate would also benefit greatly from this

Fixed in https://github.com/SnowflakePowered/spirv-cross2-rs/pull/6
Fixed for glslang in https://github.com/SnowflakePowered/glslang-rs/pull/7


hashSettled and allSettled do not return string reasons for rejected promises

When I use hashSettled or allSettled to work with many promises at once, I get back objects that look like the following:
{ state: 'rejected',
reason: [TypeError: undefined is not a function] }
The reason is not a string, which leads to errors when I try to do something with this object.

Hi @laurenzlong, thanks for opening an issue. While RSVP and Promises A+ allow for rejecting with any value (like a number, string, etc), it is generally recommended to generate with an Error object. The Error object has a message property, which should be a string. The reason Errors are preferred is they have semantic meaning (something wrong happened) and most runtimes give you access to the stack (where the problem happened).
If a runtime exception happens inside the Promise, that error will be handled by allSettled or hashSettled.
So, I don't think this is a bug, so I'm closing the issue. Let me know if you have any questions and I'll try to answer them.
To properly handle this, you may want to try using the reason.message property instead.

That makes sense, thank you! Also it seems like I must write "reject(err.message)", not "reject(err)". (Where err is an Error object). It would be great if both could work.

Both do work, but I would recommend you reject with an error and then in your handler code, extract the message:
RSVP.reject(new Error('whoops!')).catch(function(err) {
  console.log('was rejected with', err.message);
});

Rejecting with Error objects seems to be the most common pattern used with promises, so it's nice to have consistent code to handle it.

Thanks for the code snippet! I'm still unsure of how this would tie in with using allSettled, and would appreciate your insight.
function operation() {
    var err = new Error('whoops!');
    return RSVP.reject(err);
}

var op1 = operation();
var op2 = operation();

RSVP.allSettled([op1, op2]).then( function(results) {
    console.log(results);
// [{state: 'rejected', reason: [Error: whoops!}, {state: 'rejected', reason: [Error: whoops!}]
});

In the above code snippet, results now contains a non-string for reason. Can you how I can prevent this other than replacing line 3 with 'return RSVP.reject(err.message)'?


Handle socket exceptions within RetryPolicy

Motivation
The current solution provides a retry policy only for cases when WebException happens. However, connectivity issues might also occur on the local infrastructure, for which SocketException is thrown.
Proposed solution
Would be awesome to have an extension possibility for handled exception types and response statuses OR just include SocketException handling into DefaultRetryPolicy along with WebExceptions
Additional context
https://learn.microsoft.com/en-us/dotnet/api/system.net.sockets.socketexception?view=net-7.0

Hi @erikakus. Thank you for the feedback and proposal. We will discuss it and let you know what we can do

Hi @erikakus. Could you please describe your use-case? Just to let us correctly understand the problem which SocketException should solve and to provide a proper solution for you.
And if you want to help us even better, you can include the PR as well 🙂 We welcome the contributions and contributors as well!
Thank you.

Hi!
This issue has gone quiet. 👻
It’s been a while since the last update here.
If we missed anything on this issue or if you want to keep it open, please reply here.
Thanks for being a part of the Kontent.ai community!


[feat] Expose Tao's raw DeviceEvent

Describe the problem
I'm attempting to capture a mouse move event from raw device input, similar to winput. However, it conflicts with Tauri's message loop and so cannot be used.
After some research, I discovered that Tao actually provides a DeviceEvent that exposes raw mouse move events from the OS.
Is there a way to access it?

Platform: Windows
Tauri version: 1.4.0

Describe the solution you'd like
I couldn't find the api that exposes the DeviceEvent from Tauri directly, let me know if there's already one there or other ways (like accessing via, say, a raw window handle?) are available.
I'd like to have the DeviceEvent exposed so that I can utilize it for my usecases that requires getting raw device input.
Alternatives considered

winput - Conflicts with the Tauri's message loop.
multiinput - Not sure what's going on, but it leaks memory.
'mousemove' event from Web api. (see below)

Additional context
What I was trying to achieve is that to capture mouse input while locking the cursor in the center of the app window.
I could use 'mousemove' event from the Web api, but it eventually fails to get relative position because it retreats back to the center position after move, canceling out the points that moved, thus resulting to zero deltas after all.

Ditto on this. I'd like to be able to listen for when devices connect or disconnect for querying hardware/firmware for connection status and raw input. I found a fork which hard-wired this in, but it'd be very helpful for some access to wry::event::DeviceEvent to be natively supported. https://github.com/tauri-apps/tauri/compare/dev...SecretPocketCat:tauri:dev


Implement migration integration test

We should include the CORS update otherwise clients cannot connect

Do you mind opening that as a separate PR?

yes! ill wait till this is merged


How to run the library locally?

I just stumbled on this problem. If I want to change a thing or two in the library and can attach to my app project it directly using the file system like this:
Add to settings.gradle
include ':android-stepper'
project(':android-stepper').projectDir = new File('../android-stepper/stepper')

Add to app/build.dependencies
dependencies {
  implementation project(':android-stepper')

With your library it does not work. Currently I am able to run it from maven version only.
Can you please describe how to a attach Android Stepper to an app project locally?

What is the issue when you try to do this?

Here you are. This error is coming out after performing all operations described above and doing gradle sync:
org.gradle.internal.exceptions.LocationAwareException: Build file '.../android-stepper/stepper/build.gradle.kts' line: 9
Script compilation error:

  Line 9:     id(PluginDependencies.ANDROID_MAVEN)
                 ^ Unresolved reference: PluginDependencies

The problem is obviously, that PluginDependencies is not accessible from the project, only stepper does. PluginDependencies is though defined in buildSrc folder. Should it also be included to the new project, why and how? This is not clear.

Yes, you'll need to add the Dependencies.kt file from the stepper project into your own project.

What do you mean by adding Dependencies.kt to my project?
Do you mean simply copy android-stepper/buildSrc/src/main/java/Dependencies.kt to MyApplication/app/src/main/java/com/example/myapplication/Dependencies.kt? Or there is another way to do it, e.g. as a reference?
Anyway copying does not work.

No, I meant copying it to your buildSrc module (if you have one).

Nope, I don't have buildSrc in my project. I created my project by selecting in Android Studio: New project / Phone or tablet / Empty activity. All other settings are left as suggested by Android Studio. What do you suggest to do?

In that case, I would suggest you create a buildSrc module or copy its contents into the stepper build.gradle.kts

Did anyone manage to import the library folder as an external module successfully? I have the same issues as OP

What exact issues did you have @NageekDC ?


Request : add Date type

Discussed in https://github.com/prisma/prisma/discussions/9809

Originally posted by bmichotte October 17, 2021
Hi there,
If I have the following model
model Foo {
    id        String    @id @default(cuid())
    createdAt DateTime  @default(now())
}

I can, on my pgsql database, query all the Foo created on a certain date with
select * from "Foo" where "createdAt"::date = '2021-10-17'

or with
select * from "Foo" where CAST("createdAt" AS DATE) = '2021-10-17'

Is there any way to do it with prisma ?
Actually, the only way I found is
await prisma.foo.findMany({
   where: {
      AND: [
          { createdAt: { gte: new Date('2021-10-17 00:00:00') }},
          { createdAt: { lte: new Date('2021-10-17 23:59:59') }}
      ]
   }
})

which is not convenient and could lead to errors.
Is the only way to get that query is to create the model with another DateTime @db.Date field which will allow to use { theField: new Date() } ?
This could also be usefull for any birthdate, like we never store a birthdate as a datetime, but as a date
So, is there any way to add a Date type ? And then, with that date field, make any eq/lt/lte/gt/gte queries which will accept a JS Date and do the query with YYYY-MM-DD  ?

Date exists as a native type for DateTime Prisma scalar, @db.Date: https://www.prisma.io/docs/reference/api-reference/prisma-schema-reference#postgresql-6 Is that what you are looking for?

oh, I wasn't aware that DateTime @db.Date worked like that
const users = await prisma.user.findMany({
        where: {
            birthdate: { equals: new Date('1978-10-06') },
        },
    })

Works like a charm.
My bad and thank you for your response

It works but I would like to use date as string.

yes, date string (without time) would make more sense. Now I need to wrap all my Date fields with new Date()


I always have periodic issues with connection azure timeout

For example, connecting to the service every morning is normal, and then there is always a timeout issue in the afternoon
code:

Exception:

Could you help me .Thank you very much.

Thank you for filing this issue @WXCMYDEARMELAN.
Unfortunately, the code snippet doesn't help too much as the issue is that a connection to the remote host can't be made within 10 seconds and is timing out, so some additional details are needed.
Is the application using a proxy?
Given you mentioned it seems to happen at a similar time every day, does anything change in the hosting environment regularly that could coincide with the timeout? Ex, periodic reset of a service, increase in usage, etc.


Adding doc for replication.

@enisoc this is the same doc as the one that was sent for review earlier.

This change is

 with some editing suggestions.
Previously, alainjobart (Alain Jobart) wrote…
> #### Adding doc for replication\.
>
> @enisoc this is the same doc as the one that was sent for review earlier.
>
>
>

Reviewed 3 of 3 files at r1.
Review status: all files reviewed at latest revision, 13 unresolved discussions.

doc/VitessReplication.md, line 5 [r1] (raw file):

## Statement vs Row Based Replication

MySQL supports two primary modes of replication in its binary logs: Statement or

The capitalization of words like "Statement", "Row", "Master", "Pivot" throughout this doc is inconsistent (some are upper, some lower). Capitalizing them at all (except in headings) seems weird to me since I don't think of them as proper nouns.

doc/VitessReplication.md, line 26 [r1] (raw file):

  primary key, and contains the new values for each column, so it’s very fast.
* Each updated row contains the entire row, not just the columns that were
  updated. (this is inefficient if only one column out of a large number has

This is technically configurable with binlog_row_image=minimal but maybe not worth going into that much detail here anyway?

doc/VitessReplication.md, line 41 [r1] (raw file):

  number of rows).
* Schema changes revert to Statement Based Replication.
* If comments are added to a statement, there are stripped from the the

s/there/they/
s/the the/the/

doc/VitessReplication.md, line 43 [r1] (raw file):

* If comments are added to a statement, there are stripped from the the
  replication stream (as only rows are transmitted). There is a debug flag to
  add the original statement to each row update, but it is costly and very

Is there cost for this other than the obvious increase in log data size?

doc/VitessReplication.md, line 74 [r1] (raw file):

## Rewriting Update Statements

Vitess re-writes ‘UPDATE’ SQL statements to always know what rows will be

Just "rewrite" is more consistent with the rest of the usage.

doc/VitessReplication.md, line 85 [r1] (raw file):

SELECT  FROM  WHERE  FOR UPDATE
UPDATE  SET  WHERE  IN  /* primary key values: … */

s/tablet/table/

doc/VitessReplication.md, line 104 [r1] (raw file):

In a Vitess shard, we have a master and multiple slaves (replicas, batch,
…). Slaves are brought up by restoring a recent backup, and catching up on
replication. The Master can be re-parented to a slave at any time. We use

What do you mean by "the master can be reparented to a slave"?

doc/VitessReplication.md, line 118 [r1] (raw file):

* When all slaves are done, reparent to a slave that has applied the change.
* The old master can then be restored from a backup again, and put back into
  service.

"[back into service] as a slave."

doc/VitessReplication.md, line 120 [r1] (raw file):

  service.

With this process, the only guarantees we need is for the change (schema or data) to be backward compatible: the clients won’t know if they talk to a server that has applied the change yet or not. This is usually fairly easy to deal with:

s/guarantees/guarantee/
Needs a line break or two.

doc/VitessReplication.md, line 122 [r1] (raw file):

With this process, the only guarantees we need is for the change (schema or data) to be backward compatible: the clients won’t know if they talk to a server that has applied the change yet or not. This is usually fairly easy to deal with:

* When adding a column, it cannot be referenced by the client until the pivot is

The word "reference" is a noun, in the sense you're intending here. The verb form would be "clients cannot refer to it". I would also suggest "clients cannot use it".

doc/VitessReplication.md, line 124 [r1] (raw file):

* When adding a column, it cannot be referenced by the client until the pivot is
  done.
* When removing a column, the clients should not reference it before starting

You need a stronger guarantee than this statement implies. Something like "all clients must stop referring to it before the pivot begins".

doc/VitessReplication.md, line 135 [r1] (raw file):

Note the real change is only applied to one instance. We then rely on the backup
/ restore process to propagate the change. This is a very good improvement from
letting the changes through the replication stream, where they are applied to

It's also an improvement over the common industry practice of "online schema change", which also must be run on all hosts.

doc/VitessReplication.md, line 159 [r1] (raw file):

We have plans to make this Update Stream feature more consistent, very
resilient, fast, and shard-unaware.

I think "transparent to sharding" would be better.  "Unaware" sounds like it will fail to account for sharding and do something wrong.

Comments from Reviewable

Published.


React library(lib mode) doesn't work with Vite: Invalid hook call

[x] Read the docs.
[x] Use Vite >=2.0. (1.x is no longer supported)
[x] If the issue is related to 1.x -> 2.0 upgrade, read the Migration Guide first.

Describe the bug
Similar to #1401, when using Vite 2.0 in library mode with React, the bundle generated throws "Invalid hook call" when imported in the browser.
Reproduction
I've created a repo with a simple app to demonstrate the issue: https://github.com/thelinuxlich/vite-react-lib
System Info

vite version: 2.0.0-beta.31
Operating System: Ubuntu 20.04
Node version: 12.20
Package manager (npm/yarn/pnpm) and version: yarn 1.22.5

Try  to remove the following code in the index.html file.
  <script type="module">
    import Main from "/dist/react-test.es.js"
    ReactDOM.render(React.createElement(Main), document.getElementById("main"))
  </script>

It will work fine. 😉

Try  to remove the following code in the index.html file.
  <script type="module">
    import Main from "/dist/react-test.es.js"
    ReactDOM.render(React.createElement(Main), document.getElementById("main"))
  </script>

It will work fine. 😉

Try  to replace the following code in the index.html file:
  <script type="module">
    import Main from "/dist/react-test.es.js"
    ReactDOM.render(React.createElement(Main), document.getElementById("main"))
  </script>

with
  <script type="module">
    import Main from "/src/Main.jsx"
    ReactDOM.render(React.createElement(Main), document.getElementById("main"))
  </script>

Try  to replace the following code in the index.html file:
  <script type="module">
    import Main from "/dist/react-test.es.js"
    ReactDOM.render(React.createElement(Main), document.getElementById("main"))
  </script>

with
  <script type="module">
    import Main from "/src/Main.jsx"
    ReactDOM.render(React.createElement(Main), document.getElementById("main"))
  </script>

I created a PR to fix this issue in your repo, check it out for the solutions.
The cause of 'Invalid hook call' error is that you use two react and react-dom module. The first one is bundled from node_modules, and the second one is ones that loaded in html script tags.

I created a PR to fix this issue in your repo, check it out for the solutions.
The cause of 'Invalid hook call' error is that you use two react and react-dom module. The first one is bundled from node_modules, and the second one is ones that loaded in html script tags.

@daychongyang @izznatsir I appreciate your effort, but that doesn't solve the real issue: the bundled library doesn't respect the external rule and can't be imported in a preexisting SPA project.

@daychongyang @izznatsir I appreciate your effort, but that doesn't solve the real issue: the bundled library doesn't respect the external rule and can't be imported in a preexisting SPA project.

Does the module absolutely have to be imported in html script tag with umd build of reat and react dom? Or does the module will be bundled with your SPA?

Does the module absolutely have to be imported in html script tag with umd build of reat and react dom? Or does the module will be bundled with your SPA?

I also check to load the bundled library through script tag in umd format like so (I also change the build global variable name in vite.config.js from 'react-test' to 'ReactTest' due to '-' is invalid char for variable name):
<script src="/react-test.umd.js"></script>
    <script type="module">
      ReactDOM.render(
        React.createElement(ReactTest),
        document.getElementById("main")
      );
</script>

This also works.

I also check to load the bundled library through script tag in umd format like so (I also change the build global variable name in vite.config.js from 'react-test' to 'ReactTest' due to '-' is invalid char for variable name):
<script src="/react-test.umd.js"></script>
    <script type="module">
      ReactDOM.render(
        React.createElement(ReactTest),
        document.getElementById("main")
      );
</script>

This also works.

The bundled library, when using ES format, assumes React to be externalized as a module import. That is, it preserves raw import React from 'react' statements. It will NOT use globals because it's a ESM build, not an IIFE or UMD build.

The bundled library, when using ES format, assumes React to be externalized as a module import. That is, it preserves raw import React from 'react' statements. It will NOT use globals because it's a ESM build, not an IIFE or UMD build.

@yyx990803 I tried importing React as module too and it failed, so maybe I'm skipping some config in Vite?

@yyx990803 I tried importing React as module too and it failed, so maybe I'm skipping some config in Vite?

I don't think you are understanding what the format means. The ES distribution has raw import statements to "react" so it's going to resolve to your local installation of React in node_modules.
If you want to use it with globals you have to build it as UMD or IIFE.

I don't think you are understanding what the format means. The ES distribution has raw import statements to "react" so it's going to resolve to your local installation of React in node_modules.
If you want to use it with globals you have to build it as UMD or IIFE.

Thanks @izznatsir and @yyx990803 and pardon my noobiness in the subject. I updated the repo with the working solution in case someone steps by looking for this answer, I will be using the UMD bundle.

Thanks @izznatsir and @yyx990803 and pardon my noobiness in the subject. I updated the repo with the working solution in case someone steps by looking for this answer, I will be using the UMD bundle.


[bug]dfget's supernode API return err

Ⅰ. Issue Description
the two dfget's supernode api will return a json parsing error，but the coder ignores it.
func (api *supernodeAPI) ServiceDown(node string, taskID string, cid string) (
	resp *types.BaseResponse, e error) {
func (api *supernodeAPI) ReportPiece(node string, req *types.ReportPieceRequest) (
	resp *types.BaseResponse, e error) {

so when dfget use these api,it ignore the error
cw.api.ReportPiece(piece.SuperNode, &types.ReportPieceRequest{
		TaskID:     piece.TaskID,
		Cid:        cw.cfg.RV.Cid,
		DstCid:     piece.DstCid,
		PieceRange: piece.Range,
})
if cost.Seconds() > 2.0 {
		logrus.Infof(
		"async writer and report suc from dst:%s... cost:%.3f for range:%s",
		piece.DstCid[:25], cost.Seconds(), piece.Range)
 }

if ok {
	ps.api.ServiceDown(task.superNode, task.taskID, task.cid)
	serviceFile := helper.GetServiceFile(key.(string), task.dataDir)
	os.Remove(serviceFile)
	logrus.Infof("shutdown, remove task id:%s file:%s",
	task.taskID, serviceFile)
}

Ⅱ. Describe what happened
error happen but we haven't do with it
Ⅲ. Describe what you expected to happen
when we use these api,we can do with the error
Ⅳ. How to reproduce it (as minimally and precisely as possible)

Ⅴ. Anything else we need to know?
Ⅵ. Environment:

dragonfly version:
OS (e.g. from /etc/os-release):
Kernel (e.g. uname -a):
Install tools:
Others:

You should add a code link reference for the code block you attached. Because of that, we can find the  code block's context very easily. @yunfeiyangbuaa

You should add a code link reference for the code block you attached. Because of that, we can find the code block's context very easily. @yunfeiyangbuaa

You should add a code link reference for the code block you attached. Because of that, we can find the code block's context very easily. @yunfeiyangbuaa

Done


Translate into Spanish🇪🇸

Work in progress.

Hey! I started translating into spanish today because I didn't see this pull request so I will stop translating but if you need some help don't be shy and tell me 👍

Hello! ok @JesusMurF, thanks! Do you want to translate from 30 to 41? I´m right now doing 24th question. Let me know :)

It will be a mess because I dont have access to your ES file until Lydia accept your pull request, it will be better if you finish the translation 😊

yes u are right! I will finish it. I'm planning that will be ready for tomorrow. Then you can review it that for sure I will do some mistakes :)

Hey, I see you changed it to a regular PR.  Is the translation finished?

Yes!  I saw that you add more questions. You can merge this PRQ and I'll open another one with new translations (or also other people can work on it, maybe @JesusMurF is interested too)