Typescript should support --lib node

Typescript already has settings to include lib.d.ts for es6, browser, dom e.t.c. Setting up typescript with node is really painful. It seems tutorials on the web are outdated. There is tsd , typings, and supposedly some integration with npm. What is the way to go? I spent a couple of days trying to get node and typescript to work and eventually just gave up and went with javascript.
I feel typescript is alienating a huge part of js node community by adding friction to get things working. It should't be this complicated.
There is already support for moduleResolution: node, adding support for node should be a parameter away in tsconfig.
I'd say when require('fs) or import fs from 'fs' fails, VSCode editor should offer an auto-fix suggestion.

You are raising multiple items here.
First, supporting node typings more "first class" will be occurring with TypeScript 2 and the @typings.  Because type acquisition is hard, this is why the TypeScript team have been working to make it easier.  TS2 needs to be released and the documentation, etc. will need to be flushed through.

It seems tutorials on the web are outdated.

That seems to be a wider community issue.  While there is no plain Node.js tutorial on the official website, I am sure the TypeScript team would gladly accept a submission to the tutorials repo.
At the end of the day, TypeScript will be only one part of an larger ecosystem.  As the design goals for TypeScript state it is a non-goal to:

Provide an end-to-end build pipeline. Instead, make the system extensible so that external tools can use the compiler for more complex build workflows.

Therefore, even when targeting a browser, TypeScript is part of a solution, but is seldom the full solution.

I'd say when require('fs) or import fs from 'fs' fails, VSCode editor should offer an auto-fix suggestion.

Like what?  What helpful information could be provided?  It is informing you already that it cannot find the module you are looking for.
I think you biggest "gripe" is that by default it isn't easy to switch from "assume a browser" to "assume a Node.js environment".  Again that will be addressed in TS2... it should essentially be as simple as:
> tsc --init index.ts
> npm init
> npm install @typings/node --save

Note, I haven't tried it yet and it is my limited understanding of how type acquisition is changing in TS2, as I can't remember if node_modules/@typings are auto-resolved in a project without further configuration.

I agree with some of your points.
Basically typescript already understands browser types because it has a pre-set built of libs e.g.
 --lib                               Specify library files to be included in the compilation:
                                       'es5' 'es6' 'es2015' 'es7' 'es2016' 'es2017' 'dom' 'webworker' 'scripthost' 'es2015.core' 'es2015.collection' 'es2015.generator' 'es2015.iterable' 'es2015.promise' 'es2015.proxy' 'es2015.reflect' 'es2015.symbol' 'es2015.symbol.wellknown' 'es2016.array.include' 'es2017.object'
``

Having node as one of the libs makes sense since node is just another engine.

It also has

--moduleResolution                  Specify module resolution strategy: 'node' (Node.js) or 'classic' (TypeScript pre-1.6).

Which means typescript is node aware at some level. It just doesn't know what node types are.

I am arguing that node is an equal host to the browser host therefore its definition should be included and should be a compiler flag away.

I should be able to take an existing node project and rename all js files to .ts, add --lib node and it should work. It shouldn't complain about core modules not being found.

Hey guys, I was actually pushing for a --lib node pretty recently. It very much does seem to be the ideal workflow for users in some sense.
However, after discussing it with @bowdenk7, we're a bit of the opinion that @types is better suited for this task. The rationale is that the Node API is not necessarily as stable as something like the API surface described in the ES spec, and so versioning matters a lot more. There are potential subtractive changes that we'd have to deal with which are breaking. Our lib.d.ts isn't versioned (which is a bad thing because people can't roll back temporarily to fix potential breaks), and I'm not so sure it would be a good thing to do the same to node.d.ts. Plus, setting node as a custom option for --lib seems to be about the same work as adding @types/node as a dependency and npm install-ing anyway.
All of that said, I would like to see if people hold the opinion that the convenience brought from this fix completely outweighs that problem. If that was the case, we could make our lib files available with TypeScript, but also make them available through @types in a versioned manner if people needed to roll back to something older. I do kind of like that idea personally.

My observations with using typescript@next for both browser and node.js projects:
Browser Project Setup

Done. Source files will have all browser APIs ambiently defined without any config needed.

Node.js Project Setup (with typescript@next):

npm install @types/node --save-dev to get node.d.ts from npm into the project
In tsconfig.json, add "lib": ["es2015"] to opt out of having all the browser APIs ambiently defined
In tsconfig.json, add "types": ["node"] so tsc will look for node.d.ts (this may be needed - see https://github.com/Microsoft/TypeScript/issues/9208#issuecomment-226711625)

So both sides have a point here. Changes with v2.0 do make it pretty easy to set up node.js projects. However, it isn't quite a level playing field, with browser projects still favoured out of the box.

I'm fine with either typings or --lib as long as there is a consistent way
to get it for everything.
Back in the day when I used eclipse with java, it would be smart to
auto-fix my imports.
I'd really love to see the language service offer hints of fixing
automatically. It would really make the experience shine.
e.g

User types: import * as fs from 'fs'
There is a yellow squigly under fs. (yellow for autofixable error)
Hover says: Typings for node available [Install] [Cancel]

I'd say it would be sweet if I could just type away imports and it would
search for modules from npm and offer auto install hints
2nd example

import * as ws from 'ws'
Yellow squigly hint says: Typings and module available in npm
[Install][Cancel]

We live in an age of artificial intelligence. Its bizzare that I have to go
through so many hoops to get typescript and node working. Let's make our
experience painless.
On Friday, July 1, 2016, Daniel Rosenwasser<EMAIL_ADDRESS>wrote:

Hey guys, I was actually pushing for a --lib node pretty recently. It
very much does seem to be the ideal workflow for users in some sense.
However, after discussing it with @bowdenk7 https://github.com/bowdenk7,
we're a bit of the opinion that @types is better suited for this task.
The rationale is that the Node API is not necessarily as stable as
something like the API surface described in the ES spec, and so versioning
matters a lot more. There are potential subtractive changes that we'd have
to deal with which are breaking. Our lib.d.ts isn't versioned (which is a
bad thing because people can't roll back temporarily to fix potential
breaks), and I'm not so sure it would be a good thing to do the same to
node.d.ts. Plus, setting node as a custom option for --lib seems to be
about the same work as adding @types/node as a dependency and npm install-ing
anyway.
All of that said, I would like to see if people hold the opinion that the
convenience brought from this fix completely outweighs that problem. If
that was the case, we could make our lib files available with TypeScript,
but also make them available through @types in a versioned manner if
people needed to roll back to something older. I do kind of like that idea
personally.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/TypeScript/issues/9466#issuecomment-230086346,
or mute the thread
https://github.com/notifications/unsubscribe/AA-JVIoeVZowohRpL2XpWEUjWjNiEXd_ks5qRgHwgaJpZM4JC2vT
.

@nojvek's reasoning echos AWB's tone-deaf tweet that only the web matters. In the end they had to put a member of the node team on the TC39 to work through the module loader debacle. If you haven't noticed, look around, all the JS toolchains use, you guessed it, NodeJS. Say, didn't TypeScript have to adopt the node module resolution scheme? Why yes they did. Node's in your schema in a few different places.  Take a cue from TC39 and get someone from node to work with you. With the near release of Angular 2 and the advent of Universal JavaScript both TypeScript and NodeJS are essential assets of the JavaScript community. Please play together nicely.
I've just spent two days trying to get import * as assert from "assert" to work--one `lib: ["es6", "node"] should have fixed that. I shouldn't have to go searching in GitHub repos to get it working.

@DanielRosenwasser I understand the design to move this to @types. But I am team lib + node :). Its just tooooo common a use case :rose:

I think typescript project should not maintain the lib files. Instead it
should dynamically pull them from as dependencies when installed "npm
install -g typescript".
That means as a user, typescript will work with browser and node out of the
box.
Making changes and maintaining the d.ts files can be done in their
respective repos.
On Wednesday, July 13, 2016, Kenneth Brubaker<EMAIL_ADDRESS>wrote:

@nojvek https://github.com/nojvek's reasoning echos AWB's tone-deaf
tweet that only the web matters. In the end they had to put a member of the
node team on the TC39 to work through the module loader debacle. If you
haven't noticed, look around, all the JS toolchains use, you guessed it,
NodeJS. Say, didn't TypeScript have to adopt the node module resolution
scheme? Why yes they did. Node's in your schema in a few different places.
 Take a cue from TC39 and get someone
from node to work with you. With the near release of Angular 2 and the
advent of Universal JavaScript both TypeScript and NodeJS are essential
assets of the JavaScript community. Please play together nicely.
I've just spent two days trying to get import * as assert from "assert"
to work--one `lib: ["es6", "node"] should have fixed that. I shouldn't
have to go searching in GitHub repos to get it working.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/TypeScript/issues/9466#issuecomment-232550130,
or mute the thread
https://github.com/notifications/unsubscribe/AA-JVIosLzQQnOkA5QsAJjl14nEK1zWxks5qVbJTgaJpZM4JC2vT
.

@clavecoder I am tone deaf now?
You could have offered solutions to peoples problems, but no, you would rather complain that I am tone deaf.  People do take some things personally.

@kitsonk, to me it would be an honor to be equated in any way with @allenwb, who has given so much to the software industry and to the JavaScript community in particular. You, @ahejlsberg, and the entire TypeScript team are similarly improving the lives of JavaScript coders in countless ways. So, no, you are not tone deaf. I attribute AWB's lack of seeing the importance of Node to the mind numbingly fast pace of change of the JavaScript developer experience. Likewise, whereas, TypeScript's original goals were to be a wrapper as thin as possible around JavaScript, it's rise to prominence as a fundamental tool of the development of, by far, the most popular JavaScript development framework, makes it hard not to consider it's role in the developer experience.
Bower's star has faded precisely because npm has become, again by far, the dominant library package management registry on the planet. This has made require('module') and, by implication, from 'module' essential ways to integrate JavaScript in larger projects. I will edit my comment to be more succinct; however, past TypeScript design decisions have demonstrated, in so many ways, that node is not just another library.
And I do most humbly thank you and the entire TypeScript team for their hard work and extraordinary contribution to the JavaScript community.

To avoid any confusion, I am just a member of the wider community, not part of the TypeScript team.  I am the project lead for Dojo 2 which has a vested interest in the roadmap of TypeScript.

TypeScript team, it looks as though the Node.js Foundation folks are interested in working with you: nodejs/LTS/issues/123. You may want to contact @rvagg, @nebrius, or @joshgav to see how they could work with you.

@nojvek, it, perhaps, should be an option of tsc --init where the developer could choose what platform they are targeting. Even if they are installing with npm, it lies with the developer to decide to use module loaders for bundling.

Npm has a notion of peerDependencies.
I would expect typescript to have a default set of typings peerDependencies
for both browser and node.
This means, there are no extra steps to get node working. It works out of
the box. It's just npm "install -g typescript"
If a Dev wants to change versions of the libs, he just npm installs that
version or changes his package.json.
Typescript team also no longer has to worry about maintaining node.d.ts.
It's basically just adding a peerDependencies entry to its package.json.
I feel this is a clean solution and npm does most of the work.
Thoughts?
On Thursday, July 14, 2016, Kenneth Brubaker<EMAIL_ADDRESS>wrote:

@nojvek https://github.com/nojvek, it, perhaps, should be an option of tsc
--init where the developer could choose what platform they are targeting.
Even if they are installing with npm, it lies with the developer to decide
to use module loaders for bundling.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/TypeScript/issues/9466#issuecomment-232664516,
or mute the thread
https://github.com/notifications/unsubscribe/AA-JVDGc_YMLbMp09XhtsUdKpiUUDH0zks5qVjkQgaJpZM4JC2vT
.

peerDependencies are deprecated and have been partially removed from npm 3, so that's a non-starter I'm afraid

npm3 installs in a flat structure so just having dependencies should work
right?
On Thu, Jul 14, 2016 at 11:00 AM, Bryan Hughes<EMAIL_ADDRESS>wrote:

peerDependencies are deprecated and have been partially removed from npm
3, so that's a non-starter I'm afraid
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/TypeScript/issues/9466#issuecomment-232743183,
or mute the thread
https://github.com/notifications/unsubscribe/AA-JVIc3NdkgxJB_pdMAWzslqVSEbHcEks5qVnk6gaJpZM4JC2vT
.

npm3 installs in a flat structure so just having dependencies should work
right?

It doesn't always install flat. If module a depends on version 1.0.0 of module c, but module b depends on version 2.0.0 of module c, then both versions of c are installed, with one of them thrown inside the node_modules folder of either a or b, as npm 2 does. This conflict is why peerDependencies are deprecated BTW, because they don't resolve this problem and break in unexpected/spectacular ways.

peerDependencies are deprecated and have been partially removed from npm 3, so that's a non-starter I'm afraid

peerDependencies is not deprecated.  There was a change in behaviour in npm3 that did not auto-install those dependencies because it causes issues and conflicts.  As far as I can tell this is a myth that has been perpetuated because of the change in behaviour.  I would assume if it were actually deprecated, it would be noted in the documentation.  The change in behaviour means that the developer has to deal with resolving conflicts by themselves, which is appropriate for peerDependencies and was the problem with it.  I am unable to find one authoritative statement that it is deprecated.

It doesn't always install flat.

Correct.  npm3 flattens by default, but has a fairly complex way of dealing with version conflicts.  This is what happens when you build something that has unlimited, low cost access to the local filesystem, like Node.js does, not like how a browser does.  This also leads to building a loader that doesn't allow remapping of packages, because again, relatively costless access to the local filesystem.
Of course, this was attempted to be solved by CommonJS and AMD being all part of a single loader specification back a decade ago with James Burke trying hard to get those of Node.js/CommonJS to have a single loader, but like all good points of hubris, people couldn't agree and so we ended up with CJS and AMD, only a decade later still find ourselves tripping over the same problems.

Typescript team also no longer has to worry about maintaining node.d.ts.

I really don't understand why you think this will magically happen.  If the TypeScript team doesn't maintain them, then who does?
What version of Node.js should be installed?  The version it is running under, is that the final target for the consumable code?
Should it be installed globally so that every project has the same typings?
How does the developer identify a project is for a browser or for node?  What about if they are targeting Rhino, or Electron?
What about the @types with 2.0 do you not like? Pretty much everything you are talking about seems to be essentially that, though in a real maintainable way.

Node.d.ts is currently maintained by a third party.
All I am saying is node is an environment equal to the browser.
When I install typescript. I expect it to understand my node code just the
way it understands my browser code. If it doesn't then I expect a
straightforward hint on how to fix it.
Google/Bing doesn't help because the way to install types is very
fragmented and confusing. A new user might be in for frustration.
On Thursday, July 14, 2016, Kitson Kelly<EMAIL_ADDRESS>wrote:

Typescript team also no longer has to worry about maintaining node.d.ts.
I really don't understand why you think this will magically happen. If the
TypeScript team doesn't maintain them, then who does?
What version of Node.js should be installed? The version it is running
under, is that the final target for the consumable code?
Should it be installed globally so that every project has the same typings?
How does the developer identify a project is for a browser or for node?
What about if they are targeting Rhino, or Electron?
What about the @types
https://blogs.msdn.microsoft.com/typescript/2016/06/15/the-future-of-declaration-files/
with 2.0 do you not like? Pretty much everything you are talking about
seems to be essentially that, though in a real maintainable way.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/TypeScript/issues/9466#issuecomment-232865077,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA-JVCO-ah-aaFsXI3lyNQztiOxnXHMhks5qVx7jgaJpZM4JC2vT
.

@nojvek to repeat myself, what about the @types with 2.0 do you not like?

User scenario
Sample.js
Import * as fs from fs
User tries to compile sample.js. S?he gets the following error:
Compiler error:
Module fs not found. Learn how to install modules at
http://typescriptlang.com/help/modules?module=fs'?
Typescript help site maintains a list of most used modules. Since fs is
passed as parameter.
Typescript help site says:

Run: npm install @typings/node

Add typings to your tsconfig like this

add a /// reference typings/global/index.d.ts

My experience as a user is streamlined. Error messages offer contextual
help. I'm not battling old tutorials and Google.
On Friday, July 15, 2016, Kitson Kelly<EMAIL_ADDRESS>wrote:

@nojvek https://github.com/nojvek to repeat myself, what about the
@types
https://blogs.msdn.microsoft.com/typescript/2016/06/15/the-future-of-declaration-files/
with 2.0 do you not like?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/TypeScript/issues/9466#issuecomment-232958253,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA-JVI90UngiBp2m6dKiseqWX0EzV70-ks5qV5J4gaJpZM4JC2vT
.

@kitsonk @DanielRosenwasser what do you think about my list of reasons for using lib and the possibility of the Node Foundation helping to provide a sanctioned definition file (nodejs/lts#123)?

peerDependencies is not deprecated. There was a change in behaviour in npm3 that did not auto-install those dependencies because it causes issues and conflicts. As far as I can tell this is a myth that has been perpetuated because of the change in behaviour. I would assume if it were actually deprecated, it would be noted in the documentation. The change in behaviour means that the developer has to deal with resolving conflicts by themselves, which is appropriate for peerDependencies and was the problem with it. I am unable to find one authoritative statement that it is deprecated.

I chatted with some npm friends and dug up the original thread on peerDependencies, and you're right, it's not technically deprecated. It very much is persona non grata though, and there is risk that they will be deprecated, as that is the direction the majority of people want to go in. As such, I still think it's not a very good choice, even if it's not officially deprecated (yet).

I definitely consider Node a platform, but platforms also have a version associated with it. Node 10 might not have the same API surface as Node 0.10.33.
I think that a good design decision is better than a convenient-right-now one. Doing "the wrong thing" with declarations for Node.js could lead to a confusing user experience. Like I mentioned in the issue, I think we are very open to the idea of working with Node and the community to improve the workflow, so I'd like to have that discussion before committing to making this a --lib option.

Yup. Sanctioned libs are the best way to go. And while I said "lib": ["node"], obviously you would want node4, node6, etc. But I get your point that any update to node could affect the declaration file. On the other hand, it would be hard to get more complicated than the current es*** picture, and almost anything is better than the current state with nearly undiscoverable access to a community driven declaration file.
In the end, disoverablility of a unique procedure for @types/node is the current issue that, I think, needs to be addressed before 2.0 escapes beta. And by that I mean that you don't just add it to your node tutorial doc page, but you make it discoverable. One way could be an error message for unmatched imports of built-in node modules, say for import * as fs from 'fs', when moduleResolution is node.

If we force people down node v4 vs. node v6 lib etc, then that will increase the learning curve / decrease the usability of ts libs across node version :-/

@basarat, surely it would be nice if the version could be auto-discovered, and I suppose `...some-node-configuration... ['node'] could possibly default to the node environment being run in. But, regardless, a package author may want to target several versions with multiple compilation runs--angular 2 has separate builds for es5 and es2015, for example--so you can't automatically make versioning decisions go away. That's the thing about semantic versioning: it's always so confoundingly semantic. :-)
So, yeah, if the default node configuration could just work, that would be great, but the compiler would always have to allow for specified versions...just as it is doing with es* libs, frankly.

If we force people down node v4 vs. node v6 lib etc, then that will increase the learning curve / decrease the usability of ts libs across node version :-/

To give an idea of the scope of changes, and to have specifics to discuss instead of generic concepts, the node5/6 breaking changes list gives a good idea. As you can see, there's a surprising amount of breaking changes there, the largest of which being the deprecation of the buffer constructor and the addition of a bunch of new buffer allocation methods that don't exist in 4.
I think auto discovery would be nice, as long as specifying specific versions is also easy. It's not uncommon for people to have one version installed locally and to deploy to a different version on the server (not that this is necessarily advisable, but it does happen).

I think auto discovery would be nice, as long as specifying specific versions is also easy. It's not uncommon for people to have one version installed locally and to deploy to a different version on the server (not that this is necessarily advisable, but it does happen).

I would say this happens quite regularly in corporate environments where the CI build pipeline will match production which is more slower to move to modern versions.  So developers may very well be targeting something totally different than what is on their machine.  Also there is a prevalence of n and nvm installed on development machines, which allows quick switching for such purposes.

Any update on this? Wanted to try typescript in a Node.js environment but documentation is lacking

Any update on this? Wanted to try typescript in a Node.js environment but documentation is lacking

All you need to get started is the ts compiler and the node types. e.g.:
npm install typescript @types/node

node_modules\.bin\tsc a.ts --lib es6

https://github.com/Microsoft/TypeScript/issues/12964 tracks adding a tutorial.

versioning applies to everything including es, dom, webworker. I don't know what is your strategy for dom, webworker and similar libs, how are you planning to control versioning when they will be changed? But I do know how you'll handle es. Its simply es5, then es6, then es7 etc. What if same strategy will be applied for node too? Like lib: ["node5"], node6, node7` etc ? I guess node follows semver and critical changes only between major versions?
Its just a suggestion, maybe you can came up with something better and more scalable for lib option for node and all other libs that may come in the future.
I think this is the case when not the best decision in terms of design should be chosen in favour of convenience and easy of use because of popularity of this issue and consumer mass that will need it.

what is the correct target for node 7.6 and native async / await support?

It would be a npm install<EMAIL_ADDRESS>--save and tsc --lib es6.

that generates _async helpers

For me, now that nodejs 7.7 has native async/await support, my tsconfig.json looks like this:
{
  "compileOnSave": false,
  "compilerOptions": {
    "outDir": "out",
    "target": "es2017",
    "module": "commonjs",
    "moduleResolution": "node",
    "sourceMap": true
  },
  "include": [
    "*.ts"
  ]
}

and generated JS does not have any _async helpers, just raw async and await keywords and it runs well.

Had to make a guide for someone, crossposting
Setting up TS for Node
tsconfig.json:
{
    "compilerOptions": {
        "module": "commonjs",
        "moduleResolution": "node",
        "target": **TARGET**,
        ...
    }
    ...
}

Install Node typings (change version to node environment):
npm install --save-dev<EMAIL_ADDRESS>What should TARGET be?
According to the compatibility chart:
Node 7: es2016
Node 8: es2016
Node 9: es2017
Node 10: es2018

@SimonMeskens there's no 100% compatible target. and nor would be. you can use babel preset env if you want to transpile the least possible by node version

I think typescript should include only ECMASCRIPT apis, and put the rest in DT.
Even DOM would have bean better outside

I appreciate your effort :)
I would add that, If one wants to transpile the minimum required, he needs typescript + babel with preset-env and the node target. (babel 7 can make it a bit easier, but some of ts features have issues there )

I don't think I've written a Node project yet that needs transpiling, since anything from es2016 onwards seems minor to me and Node 7 supports es2015 fully, but I'll keep it in mind.

and Node 7 supports es2015 fully

(except modules)

TypeScript supports modules natively, so TypeScript for Node 7 does support modules. Especially with the esModuleInterop flag.

@SimonMeskens thank you.
This issue was raised in 2016, it's 3 years later and still I think your post in this thread is the only thing I found on how to get typescript targetting node.
That is not very good.
I'll try and send a PR against the website to include your handy guide.

@nicferrier Please do!

Sorry to hijack an old thread but this bit me today. Building a node app in TypeScript, buried in the codebase I had written prompt.detail instead of response.prompt.detail.
What should have happened? The TS compiler should have shown an error to indicate that prompt was not a variable.
What actually happened? Unbeknown to me, the default TS settings include lib.dom.d.ts which contains a definition of the browser prompt global function. TS therefore saw prompt and thought I was trying to use the aforementioned function.
Needless to say, a simple mechanism to pick the node libs only would have saved me pain here!

@fiznool there's an issue for that, #18433 it happens a lot, but the timing has been missed for a breaking change

hope one day ts support:
{
  "compilerOptions": {
    "lib": ["DOM", "NODE@10"] // "DOM", "ES2015", "NODE@4", "NODE@6" .....
  }
}

hope one day ts support:
{
  "compilerOptions": {
    "lib": ["DOM", "NODE@10"] // "DOM", "ES2015", "NODE@4", "NODE@6" .....
  }
}

I understand the earlier points about the Node API surface moving too fast compared to the DOM, but the current defaults do skew heavily towards the browser.  I had to go double-check the manual to be sure, but if you omit lib from your compilerOptions, you get dom by default -- as a previous comment says, you have to opt out of the browser explicitly.  It'd be nice to see more first-class support for isometric code.

I understand the earlier points about the Node API surface moving too fast compared to the DOM, but the current defaults do skew heavily towards the browser.  I had to go double-check the manual to be sure, but if you omit lib from your compilerOptions, you get dom by default -- as a previous comment says, you have to opt out of the browser explicitly.  It'd be nice to see more first-class support for isometric code.

Hello from the future!  In light of recent developments, has the team given any new thought to possibly "adopting" a first-class Node lib option? /cc @RyanCavanaugh @DanielRosenwasser @andrewbranch , who I know have been active in other module: node* issues in the last couple months.
I've been trying to divorce @types/node from lib-dom but it's been an uphill climb and I think the community would benefit from some help from the team.  It would be especially great if we could get a lib-node that doesn't conflict with lib-dom...

This is well-supported via the types field. If we could "do it all over again" there arguably won't be even a dom lib; it should "just" be target-associated stuff. The wide range of extant node versions (compared to DOM which is basically "latest") also shows that it's important to keep this out-of-band so it can be selected in an environment-appropriate manner.


Async Task method on Native side hang if awaited

Async Task method on Native side hang if awaited, both in BlazorMobile implementation and ElectronNET (tested with DisplayAlert).

Done


SASS compiler not working on windows machine

I'm trying to get the SASS compiler working on my windows (10) machine. Unfortunately, I get the error [WinError 2] The system cannot find the file specified when I load a page or run py manage.py collectstatic.
This is an extract from my configuration:
STATICFILES_STORAGE = 'pipeline.storage.PipelineStorage'

STATICFILES_DIRS = (
   os.path.join(BASE_DIR, 'common-static'),
)
STATICFILES_FINDERS = (
    'django.contrib.staticfiles.finders.FileSystemFinder',
    'django.contrib.staticfiles.finders.AppDirectoriesFinder',

    'pipeline.finders.PipelineFinder',
)

PIPELINE = {
    'COMPILERS': {
      'pipeline.compilers.sass.SASSCompiler',
    },
    'CSS_COMPRESSOR': 'pipeline.compressors.cssmin.CssminCompressor',
    'JS_COMPRESSOR': 'pipeline.compressors.jsmin.JSMinCompressor',

    'STYLESHEETS': {
        'cssstyles': {
            'source_filenames': (
                'myapp/css/*.css',
                'myapp/css/*.scss',
            ),
            'output_filename': 'css/myapp.min.css',
        }
    }
}

A friend of mine uses the exact same configuration on his linux maching without any problems.
I installed Ruby and SASS, and tried adding the path to the settings:  SASS_BINARY = 'C:/Ruby22-x64/bin/sass' but to no avail

Ok, it turned out to be my mistake: As I wrote, I tried adding a variable SASS_BINARY, but I should have added an entry to the PIPELINE dictionary, as described in the docs.
Also, the correct value would have been C:/Ruby22-x64/bin/sass.bat.


When i go forward i look down and when i go back i look up

Hi. I have this problem with my PC/PS3 Controller ( ShanWan PS3/PC Wired GamePad ) . When i go forward i look down and when i go back i look up. please help

Is this still an issue on the latest version? (0.10.1)

Is this still an issue on the latest version? (0.10.1)


Interpolation

To help process glyphs files outside of Glyphs, the latest versions write the interpolation factors for each instance into the file. Under the key instanceInterpolations you find a dict with the masterID as keys and the interpolation factor as value.

THANK YOU :D


support jackson's fast number parsing/writing features

Jackson's JsonFactory supports these features.

StreamReadFeature.USE_FAST_DOUBLE_PARSER
StreamWriteFeature.USE_FAST_DOUBLE_WRITER
When you create a Jackson ObjectMapper, you can optionally supply a JsonFactory.

Just thought I'd raise awareness of these features of jackson (v2.14.0 and above). With the StreamReadFeature and StreamWriteFeature, you would probably want to make those opt-in - that chesire should probably not use them by default.

Thanks for letting me know about them. Looks like a few other things have changed around here too.


You don't cinst pip

Because pip comes with python.

@majkinetor do you know if this is true of all versions of Python, or only the latest version?

@majkinetor I have just done a quick test, and it looks like when doing choco install python2 the pip.exe is available in the scripts folder.  It isn't directly accessible from the command line.  I am not a Python developer, so I don't know if this is expected or not.  Should the python package create a shim for the pip.exe?
I can take an action to speak with @ferventcoder about deprecating the pip package.

I have removed the choco install pip commands.

Should the python package create a shim for the pip.exe?

Give us a ticket at coreteam-repo for this as everybody expectes pip to just work.

Sounds like a plan! 👍

Issue created here: https://github.com/chocolatey/chocolatey-coreteampackages/issues/1032


Use a SparseArray to save int[] allocations for attributeId.

Save a few int[] object allocations.
Fixes #230.

OK, without any measured performance on allocations vs overhead of method execution and binary search of the SparseArray its hard to say if this is actually better, agreed we would stop allocating memory each time, but that is probably less of hit over what I just mentioned.
But you are right this can be improved and I will come back with some suggestions.
new int[]{android.R.attr.textAppearance} can be static final.
Also the attributeId is derived from the CalligraphyConfig.Builder so. A better use would actually be creating new int[]{attributeId} when constructing the CalligraphyConfig and passing that into the method:
static String pullFontPathFromStyle(Context context, AttributeSet attrs, int[] attributeIdArray) {
}

The other stuff you would need to prove the additional overhead of virtual method calls outweighs some very small allocations.

@chrisjenx:

static final int[] for android.R.attr.textAppearance
pullFontPathFromStyle with int[] array

Will you modify or should I update this PR?

If you can update the PR that would be great. I would drop the sparse array
though. We can look at better ways of improving efficiency with the styles
and textAppearance later after measuring.
Thanks.
On Thu, 10 Dec 2015, 07:37 Sebastian Roth<EMAIL_ADDRESS>wrote:

@chrisjenx https://github.com/chrisjenx:

static final int[] for android.R.attr.textAppearance
pullFontPathFromStyle with int[] array

Will you modify or should I update this PR?
—
Reply to this email directly or view it on GitHub
https://github.com/chrisjenx/Calligraphy/pull/232#issuecomment-163524029
.

@chrisjenx Ready for review.

Cool, looks good. Thanks.

Actually to follow up with what @jw said:

why not a single, static int[1] that you mutate and re-use for each?

The whole inflation process is created on the MainThread, you can just set the ATTR_ARR[0] = attributeId every time.


Recommend --help for getting cli help

Based on an informal internal survey, the cockroach <command> --help syntax seems more standard/intuitive than what's currently in the docs: cockroach help <command>. This PR updates the cockroach command docs accordingly.

This change is

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/5d42ab33159f3b4bf2e7c26ca8cb5c7106679ade/


Which file to use for NPM dependency / injection

When putting together a simple POC to build a LWC OSS page using the lightning-base-components, I am confused as to where to put my NPM dependency. Does it belong in the lwc.config.json file or the lwc-services.config.js file, or both? And what is the purpose of each file?

For consuming LWC npm packages you'll use lwc.config.json. You can read more about it here.


Avoid emulation for ov development environment

Fixes
Fixes #4934 by @dhruvkb
Addresses #4944 by @d3jawu
Problem
I was trying to use the ov environment more proactively in anticipation of the upcoming WordCamp. During this I noticed that ov was considerably slower than it should've been when running under emulation on macOS. This PR aims to fix that.
Description

This PR allows us to go back to running ov in the architecture native to the host machine instead of emulating for linux/amd64. This change significantly improves performance.
For example, it would take many minutes for linting to set up and run earlier, which now only takes a few seconds. This makes commits and pushes so much more frictionless.
To do this,

we replaced the vale Python package with a simpler download script that automatically downloads architecture specific binaries from GH releases. With this change, we can also drop the entire Python project we had scaffolded inside .vale/ for one dependency.
we made k6 optional in the Dockerfile. It will automatically be installed when the architecture is supported and will be skipped otherwise. If we find ourselves needing it more, we can switch to a method similar to Vale above and get it from GH releases.

Testing Instructions

Check out the PR.
If you were using ov before, clean it up:

ov clean
ov init

Check that workflows continue to work normally, for example linting:ov just lint

Checklist

[x] My pull request has a descriptive title (not a vague title likeUpdate index.md).
[x] My pull request targets the default branch of the repository (main) or a parent feature branch.
[x] My commit messages follow best practices.
[x] My code follows the established code style of the repository.
[ ] I added or updated tests for the changes I made (if applicable).
[x] I added or updated documentation (if applicable).
[x] I tried running the project locally and verified that there are no visible errors.
[ ] I ran the DAG documentation generator (ov just catalog/generate-docs for catalog
PRs) or the media properties generator (ov just catalog/generate-docs media-props
for the catalog or ov just api/generate-docs for the API) where applicable.

Developer Certificate of Origin

Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.

Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

I tested this on my Linux machine and while it was extremely slow, which was more likely a fault of my machine than this PR, it completed successfully. If someone has a faster, newer or more capable Linux machine, please test this out!


vscodium: <IP_ADDRESS>09 -> <IP_ADDRESS>22

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for vscodium is: Open source source code editor developed by Microsoft for Windows,
Linux and macOS (VS Code without MS branding/telemetry/licensing)
meta.homepage for vscodium is: https://github.com/VSCodium/vscodium
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found <IP_ADDRESS>16 with grep in /nix/store/vyciw4qphn41dqhhqw8c0iw9x8k9alkl-vscodium-<IP_ADDRESS>16
found <IP_ADDRESS>16 in filename of file in /nix/store/vyciw4qphn41dqhhqw8c0iw9x8k9alkl-vscodium-<IP_ADDRESS>16

Rebuild report (if merged into master) (click to expand)

3 total rebuild path(s)

3 package rebuild(s)

First fifty rebuilds by attrpath
vscode-langservers-extracted
vscodium
vscodium-fhs

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/vyciw4qphn41dqhhqw8c0iw9x8k9alkl-vscodium-<IP_ADDRESS>16 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A vscodium https://github.com/r-ryantm/nixpkgs/archive/9ad6464e7be6324e8974e98c1bd756ddc5b86029.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/vyciw4qphn41dqhhqw8c0iw9x8k9alkl-vscodium-<IP_ADDRESS>16
ls -la /nix/store/vyciw4qphn41dqhhqw8c0iw9x8k9alkl-vscodium-<IP_ADDRESS>16/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

3 packages built:

vscode-langservers-extracted
vscodium
vscodium-fhs

Maintainer pings
cc @Synthetica9 @turion @bobby285271 for testing.

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for vscodium is: Open source source code editor developed by Microsoft for Windows,
Linux and macOS (VS Code without MS branding/telemetry/licensing)
meta.homepage for vscodium is: https://github.com/VSCodium/vscodium
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found <IP_ADDRESS>22 with grep in /nix/store/qhrlzfds2gqzpdvzpllgppzb4ry5v95q-vscodium-<IP_ADDRESS>22
found <IP_ADDRESS>22 in filename of file in /nix/store/qhrlzfds2gqzpdvzpllgppzb4ry5v95q-vscodium-<IP_ADDRESS>22

Rebuild report (if merged into master) (click to expand)

3 total rebuild path(s)

3 package rebuild(s)

First fifty rebuilds by attrpath
vscode-langservers-extracted
vscodium
vscodium-fhs

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/qhrlzfds2gqzpdvzpllgppzb4ry5v95q-vscodium-<IP_ADDRESS>22 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A vscodium https://github.com/r-ryantm/nixpkgs/archive/1186e3a1f6be2401363a6581239083006156173d.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/qhrlzfds2gqzpdvzpllgppzb4ry5v95q-vscodium-<IP_ADDRESS>22
ls -la /nix/store/qhrlzfds2gqzpdvzpllgppzb4ry5v95q-vscodium-<IP_ADDRESS>22/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

3 packages built:

vscode-langservers-extracted
vscodium
vscodium-fhs

Maintainer pings
cc @Synthetica9 @turion @bobby285271 for testing.


Possible improvment to the benchmark

While reading the code, I'm spotting 3 potentials issues:

Your code should be placed in IRAM (using the IRAM_ATTR macro) so it's not evicted from cache when the PSRAM is loading (and the time code doesn't measure slowness of SPI Flash reading).
I'm not sure you're testing what you expect to test. The IRAM->IRAM test isn't doing what you think, since you've allocated the buffer with MALLOC_CAP_DMA and not MALLOC_CAP_32BIT. It's only with the latter that you may get an IRAM buffer (see here). The other flag won't give you IRAM (they might give you DRAM, so you're measuring the SRAM speed here). Please notice that the memory bus of ESP32-S3 is 240MHz (32 bits, so a maximum theorical bandwidth is 960MB/s, a copy should halve that bandwidth so at best, you should get 480MB/s). I don't understand how you could get more than that from the results you've shown. The ESP32 doesn't do any DDR like behavior on the SRAM, only the PSRAM supports it.
If your results are confirmed, then this means that the DSP/PIE can access the SRAM in larger format than 32 bits (and the SRAM supports it), which would be excellent.
PSRAM tests are likely wrong. You need to evince the cache before starting the test, else you'll measure the time it takes to move from the cache to the cache. The default cache size in the ESP32S3 is 32KB (see here, p400 and later). In ESP-IDF, they usually configure it to 64KB. So the 100KB test already include a large part of it from the cache. You can use esp_cache_msync to evict it or simply read 64KB from somewhere here first to prune the cache, and then run the test. The size must also be larger than 128KB so when it finishes, the result is actually stored in the PSRAM and not in the cache. Else, what you'll see is like this:

Source PSRAM buffer
 32KB   |  32KB |  28KB
[   A   |   B   |   C   ]

Dest PSRAM buffer
 32KB   |  32KB |  28KB
[   D   |   E   |   F   ]

Cache
[   x   |   x   ]

After warmup, you'll have this in the cache: (state diagram by time)

|
|   S1    S2  S3    S4
+---*-----*----*----*--------------------> time

S1: Start of benchmark, cache contains [x|x]
S2: Read from PSRAM to cache and write to cache, cache contains [A|D] (actually measuring the PSRAM=>SRAM bandwidth for reading A + SRAM =>SRAM copying)
S3: End of reading A, starts reading B: doesn't fit. cache must evict its data to PSRAM, (actually measuring SRAM =>PSRAM bandwidth for transfering D to PSRAM, then PSRAM=>SRAM for reading B)
S4: End of reading C. Benchmarking stop, yet the data F isn't in PSRAM yet, since you haven't msync'ed yet.

The only part that would be relevant here for benchmark would be around state S3, since it's only there that the PSRAM is read and written. It happens after you've exhausted the cache and before it can leave lingering data in the cache, so after 32KB of source data and 32KB before its end. So in a 100KB test, you only experience this in the 28KB part between both ends. A 192KB test would have measured it in the inner 128KB area and left the 2 ends of 32KB unrelated.
Please notice that you can boost the PSRAM speed. On the ESP32-S3-WROOM-1U-N8R8, the PSRAM is mounted as a Octal SPI, and it's built by ESP-IDF by default as MSPI DDR mode running at 80MHz (see here). So the best bandwidth you could expect is 8bit * 80MHz * 2 = 160MB/s (a lot less, since there's the SPI and address selection overhead). Again, a copy operation will halve that (even more since you need to select the source and destination address for each transaction). So getting more than 50-60MB/s would be very excellent.
Yet, you can configure the MSPI to run at 120MHz (under the CONFIG_IDF_EXPERIMENTAL_FEATURES), with many caveats, but in that case you could expect a theorical bandwidth of 240MB/s or maybe ~100MB/s for a copy.
Anyway, thank you for your repository, it's excellent and I'm sure it can be even better. I've had hard time figuring out actual values for the DSP and SIMD stuff.

While reading the code, I'm spotting 3 potentials issues:

Thanks @X-Ryl669
I modified the code to include the IRAM_ATTR prefix to any of the time sensitive code. This made a small improvement with the largest improvement (2.257%) being to the 'PIE 128-bit (32 byte loop) IRAM->IRAM' copy function. This now goes at 1,789.81 MB/s so not a bad way (its pretty much free) to go that little bit faster. After all every bit counts!
I'm still getting my head around the other two improvements you have suggested but wanted to post this so you knew I appreciated your comment.
I've created this Google Sheet to track improvements..
https://docs.google.com/spreadsheets/d/1A9UKdOb0QqLGQVSIru1gydPLEyCpcejhV0q_KI-OJMs/edit?usp=sharing. Please take a look if you have the time so I know its available to anyone interested.

Please notice that the memory bus of ESP32-S3 is 240MHz (32 bits, so a maximum theorical bandwidth is 960MB/s, a copy should halve that bandwidth so at best, you should get 480MB/s). I don't understand how you could get more than that from the results you've shown. The ESP32 doesn't do any DDR like behavior on the SRAM, only the PSRAM supports it.

If your results are confirmed, then this means that the DSP/PIE can access the SRAM in larger format than 32 bits (and the SRAM supports it), which would be excellent.

@X-Ryl669 That would be the case if the ESP32 only had a 32bit internal memory bus. In fact its a 128bit bus. The only part of the processor that utilizes it is the PIE. Its capable of doing SIMD instructions such as 4 x 32 bit match operations per clock cycle. Or as we use it a 1 x 128bit (16 bytes) load or store operation per clock cycle. This is how we manage to get the 1830.2 MB/s IRAM->IRAM result using the PIE.

I'm not sure you're testing what you expect to test. The IRAM->IRAM test isn't doing what you think, since you've allocated the buffer with MALLOC_CAP_DMA and not MALLOC_CAP_32BIT. It's only with the latter that you may get an IRAM buffer (see here). The other flag won't give you IRAM (they might give you DRAM, so you're measuring the SRAM speed here).

@X-Ryl669 I took a look at the documentation you refer to on the MALLOC_CAP_32BIT. If you look just above it you will see that the DMA version excludes PSRAM but makes no mention of where else it might be. I don't see any mention of DRAM in the ESP32-S3 datasheet so I'm not sure how it could allocate DRAM, which only leaves IRAM. Anyway I tried changing the memory allocation to 32BIT and it made no difference. The results were identical for all the tests as best I can tell.

PSRAM tests are likely wrong. You need to evince the cache before starting the test, else you'll measure the time it takes to move from the cache to the cache. The default cache size in the ESP32S3 is 32KB (see here, p400 and later). In ESP-IDF, they usually configure it to 64KB. So the 100KB test already include a large part of it from the cache. You can use esp_cache_msync to evict it or simply read 64KB from somewhere here first to prune the cache, and then run the test. The size must also be larger than 128KB so when it finishes, the result is actually stored in the PSRAM and not in the cache. Else, what you'll see is like this:

@X-Ryl669 Very interesting and not something I was aware of. I found more information by looking up the esp_cache_msync function here. I'm not sure how to use this function yet as it wants me to specify memory starting addresses and size (no idea why) so I'll have a think on this one. I've also asked Microcontroller as he may have some more insight.

Yet, you can configure the MSPI to run at 120MHz (under the CONFIG_IDF_EXPERIMENTAL_FEATURES), with many caveats, but in that case you could expect a theorical bandwidth of 240MB/s or maybe ~100MB/s for a copy.

@X-Ryl669 I tried the 120 MHz PSRAM on my board. However it failed to get past the RAM detection code. It found it and posted information on the RAM and then hung. :(
I guess this is why its considered experimental. Note the warning says its not stable if the temperature moves by more than 20 degrees so I guess this makes it pretty useless except for some pretty niche use cases (if it worked).
Thanks for the suggestions. I'll let you know if I figure out how to properly test the PSRAM performance taking into account the cache load/flush challenges.

Thanks for your complete answer. As for the PSRAM performance measurement, I would do this way (pseudo code):
char * inBuf = malloc_psram(192kB)
char * outBuf = malloc_psram(192kB)
// Pre fill the buffer
memset(inBuf, whatever)
// Start benchmarking
clock false_start = now
copyBuffer(inBuf, outBuf, 32KB)
clock start = now
copyBuffer(inBuf+32KB, outBuf+32KB, 128KB)
clock end = now
copyBuffer(inBuffer+160kB, outBuf+160kB, 32kB)
clock false_end=now

// Mem benchmark speed:
totalWithCache = 192kB/(false_end - false_start)
totalWithoutCache = 128kB/(end - start)
totalPrePostLoad = 64kB/((false_end - end) + (start - false_start))

I've made the assumption that the cache is smart enough to split a 64kB R/W access in two independent 32kB access (one for read / one for write). Maybe I'm wrong, since there's no documentation about what algorithm is used on the caching process. Maybe the granularity is a lot smaller, so it'll be almost impossible to measure the effect of the caching, or it's larger (a single 64kB block?) in that case, the current test is already measuring the correct value.
One hint however: if you memory map a flash page with MMU (which is, what I think the cache is using for PSRAM), the granularity is 64kB for ESP32S3, IIRC (see here and here for the actual page size)

1. Your code should be placed in IRAM (using the `IRAM_ATTR` [macro](https://docs.espressif.com/projects/esp-idf/en/stable/esp32s2/api-guides/memory-types.html#how-to-place-code-in-iram)) so it's not evicted from cache when the PSRAM is loading (and the time code doesn't measure slowness of SPI Flash reading).

The cache is split into code and data cache, so data cannot push out code, and vice versa. But yes, for maximum accuracy pre-loading the timed code into RAM would be best. Or running the same code more than once and taking the minimum run time.

2. I'm not sure you're testing what you expect to test. ... The other flag won't give you IRAM (they might give you DRAM, so you're measuring the SRAM speed here).

The -S3 actually has no DRAM (data RAM) or IRAM (instruction RAM); all its RAM is byte-addressable and can be executed from. (Except of course the RTC RAM, but we're not interested in that for now :))
So "IRAM" in this benchmark just means "internal RAM" a.k.a. SRAM, as opposed to the external PSRAM.
Adding MALLOC_CAP_INTERNAL to the allocations would make it extra clear that we want internal RAM.

   If your results are confirmed, then this means that the DSP/PIE can access the SRAM in larger format than 32 bits (and the SRAM supports it), which would be excellent.

We confirmed that the results match what the TRM states, i.e. single-cycle 128-bit transfers from and to SRAM :)

I've made the assumption that the cache is smart enough to split a 64kB R/W access in two independent 32kB access (one for read / one for write). Maybe I'm wrong, since there's no documentation about what algorithm is used on the caching process. Maybe the granularity is a lot smaller, ...
It is a lot smaller. The granularity is one cache line, which can be configured to be 32 or 64 bytes.

To avoid the effects of the cache when measuring non-DMA transfers from/to PSRAM, we would have to

esp_cache_msync( sourceBuffer, sourceSize, ESP_CACHE_MSYNC_FLAG_DIR_C2M ); // Make sure the source data we want to copy has actually made it to PSRAM

esp_cache_msync( sourceBuffer, sourceSize, ESP_CACHE_MSYNC_FLAG_DIR_M2C ); // Make the cache forget the cached source data

start timing

copy data from/to PSRAM

esp_cache_msync( destBuffer, destSize, ESP_CACHE_MSYNC_FLAG_DIR_C2M ); // Make sure the data we copied to dest has actually made it to PSRAM

stop timing

and 2. only apply when we copy from PSRAM; 5 only applies when we copy to PSRAM.

It is a lot smaller. The granularity is one cache line, which can be configured to be 32 or 64 bytes.

I think we are not speaking about the same thing here. The cache line is 64 bytes on ESP32-S3. But this doesn't matter for the test. What matter is the cache block size (in SRAM) which is 32K or 64K since when you're benchmarking the memory copy, you're are actually measuring the time it takes to copy from SRAM to SRAM if the data you're copying to/from can fit in this cache block size. You need to overcome the block size to actually measure the PSRAM speed.
When using the CPU (not DMA), you can't skip the caching mechanism, so you must measure the time it takes to fill / empty the cache too, which the current code didn't do (the filling of the cache was here obviously, but the emptying wasn't, since the test stopped without knowing when the cache lines were transmitted). This is relevant to a real world usage. I don't know if the caching algorithm was dumb (like, every time a cache line is touched it'll be marked as dirty and will be transferred later on, and in that case, the benchmark is almost correct, except for the last cache line) or a bit more smarter (like, until all the cache block is used, no cache line is transferred to PSRAM and then only when new data are loaded in the cache that'll trigger transferring the stale/old data to PSRAM, and in that case, the benchmark is irrelevant, since it was fitting in the cache block size anyway).
From the post of Sprite_TM here it seems it's the latter, so it's definitively an issue in the benchmark.
However, since the DMA bypass the cache, the actual PSRAM speed is captured here (althrough it also capture a bit of the FreeRTOS's async scheduling overhead so for such a small buffer, it matters too).
I think to get a better idea of the memcpy speed from & to PSRAM, a larger block is required in all cases (cache or no cache) so that the caching overhead (which is useless in this usage pattern since data is never used twice) or async scheduling overhead is minimized.

To avoid the effects of the cache when measuring non-DMA transfers from/to PSRAM, we would have to [...]

Yes that's correct.

I suspect the "cache flushing" function to simply trash the cache lines by filling them with random memory addresses. I need to do my homework here and read them.
I don't get why you are flushing the memory for async_memcpy since DMA doesn't use the cache anyway.
The value are consistent with the architecture here, PSRAM is using 8 bits/cycle at a 80MHz clock in DDR fashion, so at best it can do 160MB/s (thus at best 80MB/s for a memory copy to/from PSRAM without cache).  Since you need to select the address for transferring from the chip, it's a lot less than that, so a value of ~50MB/s doesn't look too far from the reality.
It's good to see that the effect of caching actually improve performance (38MB/s instead of 32MB/s), since this proves that the address selection overhead in the SPI memory is important. It's not huge but it's nonetheless useful.

I'm only wondering why the DMA transfer would be so much faster than the CPU-driven one.
Seeing that, I added cache prepping around the DMA too, but as expected it made no difference.
I assume the DMA, knowing the size and sequential nature of the transfer beforehand, uses longer bursts than the cache, reducing the relative overhead.

I suspect the "cache flushing" function to simply trash the cache lines by filling them with random memory addresses.

No. The newer ESPs' caches support selectively flushing, invalidating, prefetching, and locking address ranges.

I've merged the code changes and added a flag to compile with the cache code or without it. I get lower results for some of the PSRAM->IRAM copies (to be expected I guess) and no real change in the IRAM->PSRAM (Strange).
Also strange I get significantly faster results for the 8 bit loop copies. Especially for the IRAM->IRAM which makes absolutely no sense. Any ideas?

I think the title "With cache" and "No cache" are inverted in the column, IIRC. The new code tries to disable the effect of the cache, not enabling it.
And for the 8 bit loop copy speedup, I have no conclusions either. You might want to compare the disassembly of the both code to see what has changed. Technically, there's nothing that prevent the compiler to write more efficient code (like ignoring you're doing 8bit->8bit moves/copy and instead write it as a 32bit->32bits move or more). Also, it's possible it is unrolling the loop, or the alignment for the pointers in the new version are now a multiple of 16 bytes allowing more efficient code, or the compiler version increased, or the memory barrier introduced in the new code allowed the compiler to take a shortcut that it couldn't do before (it would have been unsafe doing so before and now it's ok).

Another PR which changes some results: #4 :)

I think the title "With cache" and "No cache" are inverted in the column, IIRC. The new code tries to disable the effect of the cache, not enabling it.

I've sorted this now. I'm using the terms "With Cache Flush" and "No Cache Flush"

Another PR which changes some results: #4 :)

Sorted. I've changed the spreadsheet so the results still make sense historically. The biggest change is to the async_memcpy IRAM->PSRAM. No change in the other direction.
These results make sense now. Going to PSRAM the flush will matter. In the other direction the device has all the time in the world to move the data into IRAM as it arrives and I doubt it even uses the cache.

Makes a lot more sense now.

Thanks @X-Ryl669 and @BitsForPeople. I'll close this issue now.


UPDATE: iPhone Dolby Vision video recordings play in fallback HDR mode (but correctly play as DoVi in native TV player & MX)

Update: see post number 2
I have an iPhone 13 which records videos in h.265 DoVi and outputs streams in MOV containers as is typical of Apple.
The videos I have work on my TV's native media player and MX when in HW mode and I see the TV popup saying DoVI. (making me think the underlying exoplayer supports the file)
Same file played in Just Player, TV says HDR (and contrast looks a bit different).
That said, for comparison, Just Player works absolutely fine with non-Apple DoVi files and the TV correctly shows the DoVi logo when videos start playing. So it's an MOV container/demux issue.
Thanks and hope to see this fixed to make the best player even better.

Did some testing and here's an important update: It's not the MOV container, it's the actual stream / metadata.
I remuxed the MOV file to MP4 and Just Player still uses the fallback HDR metadata as opposed to main DoVi metadata. MX and native TV player both play in DoVi.
Not a deal breaker but as I said in my previous post, if an excellent player can be made better, then I'm all in to try and help.
Happy to share the recording from the iPhone.

Yes, please do share a short sample. There used to be an issue with DV from earlier iPhones in the past  https://github.com/google/ExoPlayer/issues/8530
Also, are the files referenced in that issue playing correctly on your TV?

Here's an untouched short recording from my (regular) iPhone 13 using stock iOS camera app.
On the TV's built-in "media" app, it plays as DoVi
On MX player (HW mode), it plays as DoVi
On Just player, it plays as HDR (and looks a bit washed out, but definitely still useable)
https://drive.google.com/file/d/1Se79BLnirh3Wfk5ebhw8EqCurwWw-JFb/view?usp=sharing


connection refused using ./cmd.sh

after building everything was fine, but when I run ./start.sh nothing happens. executing ./cmd.sh almost every command launch connection refused: getnodes, version....after confirming password i got connection refused exception...

you are need to start NULS restful server using given Github code.

What does "ps aux|grep java" gives you?
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Sunday 25 November 2018 03:43, freemanjackal<EMAIL_ADDRESS>wrote:

after building everything was fine, but when I run ./start.sh nothing happens. executing ./cmd.sh but almost every command launch connection refused: getnodes, version....after confirming password i got connection refused exception...
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.


fixed example formatting

powershell lines wrapped into output lines

Docs Build status updates of commit 3ecd9d0:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docset/windows/dfsr/New-DfsReplicationGroup.md
:white_check_mark:Succeeded
View (win10-ps)

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

#sign-off

@tiburd This ideally would've been submitted through the private repo but let Ned and I know if we need to do anything more with it to get it merged. Thanks for your efforts with this repo! :) - Jason

@nedpyle @JasonGerend PoSh output should be wrapped in ```output. See https://docs.microsoft.com/en-us/PowerShell/scripting/community/contributing/powershell-style-guide?view=powershell-7.1#executable-examples.
Examples should also not use aliases, for example in Format-Table, specify -AutoSize instead of -Auto. https://docs.microsoft.com/en-us/PowerShell/scripting/community/contributing/powershell-style-guide?view=powershell-7.1#formatting-command-syntax-elements

I approve after @tseward's suggestion.

Docs Build status updates of commit 0eb16b1:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docset/windows/dfsr/New-DfsReplicationGroup.md
:white_check_mark:Succeeded
View (win10-ps)

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


Release 0.1.6

Prepare change log and an example

Hi @sakovias and thanks for this awesome library.
I would like to report an issue with the ActiveSupport lock introduced by this PR.
This lock is a problem for people using a version of rails >= 4.x.
Here you can see the issue with bundler:
Bundler could not find compatible versions for gem "activesupport":
  In snapshot (Gemfile.lock):
    activesupport (= <IP_ADDRESS>)

  In Gemfile:
    activemodel-serializers-xml (~> 1.0.0) was resolved to 1.0.0, which depends on
      activesupport (> 5.x)

    rails (~> 5.0.0) was resolved to <IP_ADDRESS>, which depends on
      activejob (= <IP_ADDRESS>) was resolved to <IP_ADDRESS>, which depends on
        globalid (>= 0.3.6) was resolved to 0.3.7, which depends on
          activesupport (>= 4.1.0)

    axlsx_styler (~> 0.1.6) was resolved to 0.1.6, which depends on
      activesupport (~> 3.1)

Thanks @dalpo for reporting this. We'll try to take care of this problem some time this week. Could you report an issue please?
Please use 0.1.5 in the meantime.

Sure! Here you can find the issue: https://github.com/sakovias/axlsx_styler/issues/17
Thanks!


Make paths work on xnix

I'm currently making a deployment of find-that-charity on an Ubuntu box. The paths broke, this fixes them and should be pretty portable :)

Hi @robredpath - hope you've found it useful. Have you made a public-facing deployment of the site? I haven't actually put one together yet but might be worth collaborating on one? (with @BobHarper1 too?)


Support many_to_many keyword

Hello, as a feature request, it would be great if ex_admin could support many_to_many relationships, the "Ecto 2 way".
As far as I know, the Ecto 1 way was
MyApp.User schema :
has_many :groups_users, MyApp.GroupUser
has_many :groups, through: [:groups_users, :group]

MyApp.Group schema :
has_many :groups_users, MyApp.GroupUser
has_many :users, through: [:groups_users, :user]

and now it's
MyApp.User schema :
many_to_many :groups, MyApp.Group, join_through: MyApp.GroupUser

MyApp.Group schema :
many_to_many :users, MyApp.User, join_through: MyApp.GroupUser

I would benefit from this feature as well!
Is there any temporary trick to work with many_to_many?
I tried to add an input
inputs :groups, as: :check_boxes, collection: MyProject.Group.all

but it raises an error when I try to save.

Hello,
It seems that someone worked on this feature...
Haven't tested it though ;-)

@Moxide Thanks, will try it!

I had to add a small fix to fearenales's code to make it work for me.
It seems to work pretty well, could anyone take a look at it and merge this feature to main repo ?
Thanks ;-)


FunctionApp:Unable to show webapp configuration

Describe the bug
Command Name
az webapp show
Errors:
The command failed with an unexpected error. Here is the traceback:
'str' object has no attribute 'value'
Traceback (most recent call last):
  File "/opt/az/lib/python3.6/site-packages/knack/cli.py", line 231, in invoke
    cmd_result = self.invocation.execute(args)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 657, in execute
    raise ex
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 720, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 691, in _run_job
    result = cmd_copy(params)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/__init__.py", line 328, in __call__
    return self.handler(*args, **kwargs)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/command_operation.py", line 363, in handler
    show_exception_handler(ex)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/arm.py", line 429, in show_exception_handler
    raise ex
  File "/opt/az/lib/python3.6/site-packages/azure/cli/core/commands/command_operation.py", line 361, in handler
    return op(**command_args)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/appservice/custom.py", line 605, in show_webapp
    return _show_app(cmd, resource_group_name, name, "webapp", slot)
  File "/opt/az/lib/python3.6/site-packages/azure/cli/command_modules/appservice/custom.py", line 734, in _show_app
    "Unable to find {} '{}', in RG '{}'".format(cmd_app_type.value, name, resource_group_name),
AttributeError: 'str' object has no attribute 'value'

To Reproduce:
Steps to reproduce the behavior.
Run the following command for any valid resource group and function app.

az webapp show -g $RESOURCE_GROUP -n $FUNCTION_APP_NAME

Expected Behavior
Environment Summary
Linux-5.11.0-40-generic-x86_64-with-debian-bullseye-sid, Ubuntu 20.04.3 LTS
Python 3.6.10
Installer: DEB

azure-cli 2.30.0

Extensions:
azure-iot 0.11.0

Additional Context

route to service team

Duplicate of #20437


feat(Keys_Incoming): Decrypt incoming ciphertext

Implementation in typescript
Following #5867 and #5899 we need to also be able to decode. This don't need to happen in a circuit so we simply do it in typescript. This should be able to take a look back at aztec-connect to see if it can get some information from there as it was also using AES.
Take into account #5901.
In aztec connect the ciphertext was called the viewing_key so should be helpful for finding the information.
Very close to #5902

As part of #5899 and #5867 there are typescript parts for encrypting and decryption these two.
We mainly need to deal with the combination of them into something that looks more like the encrypted log buffer that we have currently.

Related information in https://github.com/AztecProtocol/aztec-packages/issues/5901#issuecomment-2100780608.

This should be addressed along with #5901 to not have too many conflicts.
The l1_note_payload should be altered to take the eph_pk, header and body as outlined from the diagram, using the classes to handle decryption of those individually.
There should be a l1_note_payload::fromEncryptedBuffer function that does something along these lines:

  constructor(eph_public_key: Fr, header: EncryptedLogHeader, body: EncryptedLogBody) {}

  static fromEncryptedBuffer(data: Buffer, incomingViewingSecretKey: GrumpkinPrivateKey): L1NotePayload | undefined {
    const reader = BufferReader.asReader(data);
    const ephPK = Fr.fromBuffer(reader);
    const header = EncryptedLogHeader.fromCiphertext(data, incomingViewingSecretKey, ephPK);

    const appSecretKey = Fr.random();

    // slice data
    const body = EncryptedLogBody.fromCiphertext(data, appSecretKey, ephPK);

    return new L1NotePayload(ephPK, header, body);
  }

Was fixed in #6251 and #6411


sliding window ops in c++

I think the big thing is that this pr is trying to add an eager dependency to metal with tensors. Don't really think that is something we want

@tt-aho I changed to use the device buffer instead of tensor and removed the tensor dependency from tt_metal.
I think this should be good for now, if we decide to develop some infra for this on tt_eager side, we can move it then.

Nightly: https://github.com/tenstorrent/tt-metal/actions/runs/9182647129
Model perf: https://github.com/tenstorrent/tt-metal/actions/runs/9182651200

@tt-aho @abhullar-tt Any further objections on the program change?

model perf: https://github.com/tenstorrent/tt-metal/actions/runs/9199932628