A Proposal For Module Resolution

Background
When a user writes a module specifier (the string literal after from in an import declaration) in a TypeScript file, how should the compiler resolve that string to a file on disk to be included in type checking? Because TypeScript never rewrites module specifiers in its JavaScript emit, the only possible answer is that it should mirror whatever resolution behavior the code’s intended runtime module resolver has. I’m using “runtime module resolver” to mean the system whose module resolution behavior has observable effects at runtime: it may be a component of the runtime itself, as in Node, or it may be a bundler that consumes the module specifiers to produce one or more script files. (The “runtime” distinction is made to exclude analysis tools like linters, which may perform module resolution without having any impact on runtime behavior.) TypeScript’s way of handling this has been to say that the user must indicate what their code’s runtime module resolver is via the moduleResolution compiler option so the compiler can mirror it.
As little as five years ago, there were only two places JavaScript could run that were worth mentioning: in Node as CommonJS modules, and in the browser as scripts. For the former, TypeScript had --moduleResolution node. (The latter needed no moduleResolution mode, though you can argue some sort of none value would have been appropriate.) However, bundlers like Webpack were widely used and were themselves module resolvers, perhaps deserving their own moduleResolution setting according to TypeScript’s philosophy. But demand for bundler-specific module resolution was essentially nonexistent, because bundlers mostly just copied Node’s module resolution algorithm, so users were able to get by with --moduleResolution node.
(Note: as of this writing, --moduleResolution node16 and --moduleResolution nodenext are identical in TypeScript. The latter is intended to be updated as Node changes. For brevity, I use node16 in this writing, but both are equally applicable everywhere.)
Over the next few years, though, the landscape changed. Browsers adopted ESM as a natively supported format, and Node added ESM support alongside CJS, with a complex interop system and new features like package.json exports. A new wave of bundlers and runtimes emerged, and many adopted some of the features that Node introduced. But this time, none was similar enough to Node to piggyback on TypeScript’s --moduleResolution node16 option without users noticing problems.
Today
In this new landscape, users have been trying both node and node16 with bundlers and browsers and hitting walls, which I will explore in some detail. In brief, the JavaScript ecosystem is in a phase where we cannot hope to provide a dedicated moduleResolution mode for every runtime and bundler. At the same time, we have resisted allowing resolver plugins for many reasons:

security and performance concerns
implementing correct module specifier generation for a given resolution mode for declaration emit, auto-imports, and path completions is extremely non-trivial
concerns about reliability under file-watching modes (a resolution must return not just what it found, but where it looked and found nothing, and where it looked up auxiliary information used in resolution such as package.json files)
we would like to avoid allowing bespoke per-project resolution simply to reduce chaos

This philosophy has brought TypeScript to a point where we have avoided some significant pitfalls, but have essentially no support for module resolvers that are not Node. To make the issues explicit, let’s examine some hypothetical case studies.
Bundling with Webpack, esbuild, or Vite
These bundlers use a Node-CJS-like resolution algorithm and support package.json exports. If the user chooses --moduleResolution node, any of their dependencies that modify their export structure via package.json exports will be misrepresented by TypeScript—imports from that package may not resolve, they may resolve to incorrect files, and they will receive incorrect auto-imports and path completions. If the user chooses --moduleResolution node16, TypeScript will resolve their imports against dependencies’ package.json exports, but the conditions it uses in the lookup may be wrong: bundlers always set the import condition for imports and the require definition for require calls, but TypeScript believes that import declarations in files that have not been explicitly scoped as ESM will be transpiled into require calls, so it looks up these imports with the require condition, which could lead to incorrect resolutions. Moreover, in these files, TypeScript will prohibit imports (because it think they are actually requires) of ESM-format files. The bundler has no such restriction, as its CJS/ESM divide is purely syntactic. (This is a slight oversimplification and the three bundlers mentioned behave slightly differently, but the simplification is good enough for describing the user experience.) If the user tries to get around this by scoping all of their files as ESM by setting "type": "module" in their own package.json, TypeScript will impose Node’s much stricter ESM resolution algorithm on those files, disabling index-file resolution and extensionless lookups—in fact, the extension the user has to write is .js, which will be nonsensical for the context, where the runtime module resolver (the bundler) only ever sees .ts files. (Vite and esbuild tolerate this extension mismatch out of the box; Webpack has historically required a plugin but just added a config setting for it.) This configuration satisfies both the bundler and TypeScript, but at a high DX cost for the user—TypeScript imposes rules on resolution that are wholly unnecessary for the bundler.
Running in Bun
The situation is exactly the same as the above, since Bun’s module resolver is a port of esbuild’s, and it consumes TS files directly.
Bundling with Parcel or Browserify
These bundlers do not (yet) support package.json exports, so --moduleResolution node is still a reasonably good fit.
Writing ESM for the browser
Every module resolution mode except classic performs node_modules resolution, which does not happen in the browser. classic performs index-file and extensionless lookups, which does not happen in the browser. The closest the user can get is probably to use node16 such that index-file and extensionless lookups are disabled, but they have to take care to avoid node_modules lookups and importing CommonJS dependencies.
Writing ESM for Node, browser, or Deno
We have heard a few arguments recently about the ability to write code targeting multiple runtimes, mostly in the form of “if you let me write my imports with .ts extensions and emit them as .js extensions, my input files will work in Deno and my output files will work in Node” which is not generally true. However, it is true that Node, the browser, and Deno have a small amount of overlap in resolution behavior such that it is possible to write ES modules in JS and publish them both to npm and to a CDN where they can be consumed by browsers or Deno. A user trying to do this today faces the same situation as the case above, since the overlap between these systems is just relative URL imports including extensions: there is no mode restrictive enough to avoid writing imports that will work in Node but not in Deno or the browser. (Note that targeting a single bundler which produces a separate output for each target runtime is, for now, a better approach for multi-platform JS authoring.)
Proposal
Existing module resolution modes (with the exception of classic, whose existence is still a mystery to me) have intended to target one specific runtime and have been named for that runtime—node (v11 and before), node16, and nodenext—and the resolution features they entail are non-configurable implementation details. To move forward, I suggest a strategy of composition: expose some lower-level modes that can be combined with additional options to build up modes that are suitable for a variety of runtime resolvers. If the ecosystem converges on combinations of these settings, we can encapsulate them in a named mode. To start this process, I propose the following reorganization and expansion of options (all names subject to bikeshedding):
Module Resolution Modes

Expose what is now called node as a low-level mode called conventional, with a slight modification necessary to support .ts extension resolution under noEmit, and defaulting esModuleInterop to true. (The name, which I am more than happy to change, is a reference to the fact that most runtimes and bundlers have copied node_modules resolution, extensionless lookups, and special index-file handling from Node to the point where users no longer think of these features as specific to Node. Since we now have node16 which is highly Node-specific, the goal is to create a situation where the only people who should choose a moduleResolution option named after Node are people who are actually using Node.)
Implement today’s node as a composition of conventional and an internal-only option that undoes the modifications mentioned in (1) to preserve backward compatibility. Also, deprecate the name node in favor of node-legacy to encourage users of modern Node to consider migrating to node16, and to encourage users of other runtimes and bundlers to consider migrating to conventional. Today’s node is only accurate to Node v11 and earlier, so we need to start guiding people away from it at some point.
Implement a new low-level mode called minimal which resolves only relative module specifiers including file extensions, and attempts to parse all files as ESM. This can be used as a base for browser module resolution.
Leave classic, node16, and nodenext as they are.

Module Resolution Options

Add an option to enable/disable package.json exports in conventional and add conditions to the resolver. (May also apply to other modes that do node_modules resolution, i.e. everything but classic and minimal.)

That’s it for now—in the future, import maps, HTTP imports, and other features adopted by more than one runtime resolver should be exposed as options. When/if we have support for import maps and HTTP imports specifically, we should consider creating a mode named browser that is a composition of minimal and those options defaulted to true.
Resolution of relative module specifiers ending in .ts
Both minimal and conventional (but not node-legacy) will support resolution to .ts files by specifying a .ts extension in the module specifier. This will be an error, as it is today, unless noEmit is enabled. This allows users who are bundling or directly running their TypeScript source to write relative module specifiers with the extension that their runtime module resolver will actually see, which has always been the underlying goal of telling users to write .js extensions when their runtime resolver will operate on the emitted JS code. Additionally, in these modes, I suggest that it be legal to write an import type of a module specifier ending in .d.ts.
This cannot be supported in today’s node or node16 in a fully backward-compatible way. In these modes, an import of "./foo.ts" will resolve to foo.ts.js or foo.ts.d.ts in the same directory even if foo.ts is also present; unsupported extensions are not probed for existence before moving on to fallbacks. This amounts to a bug in node and node16. I have proposed to leave the bug in place for node-legacy to preserve backward compatibility, but it may be reasonable to try fixing it everywhere and listen for feedback.
It should be noted that composite projects may not disable emit if they are referenced by another project. It may be possible to relax the noEmit restriction to emitDeclarationOnly. The primary challenge here is a portability concern: older moduleResolution modes will not be able to resolve the .ts-suffixed specifiers in those declaration files. I think it’s worth fixing node16 to support this; they would receive the bug fix described above so that they can always resolve .ts-suffixed specifiers, but would continue to issue a checker error. That way, we can safely silence the error in declaration files for better portability, and projects that need to use .ts-suffixed imports could be composite project references. Fixing node16 in this way would also prepare us for the possibility of Node running directly on TS files, transpiling in-memory like ts-node, an idea that has been gaining traction with Node maintainers recently.
However, I think much of the demand we’ve heard so far for being able to use .ts-suffixed module specifiers has been misplaced. Users who tried to use node16 with a bundler may have been prompted to add a .js extension to a module specifier and thought that adding a .ts extension makes more sense, when in actuality they can continue to use extensionless imports in a mode like conventional. Others demand .ts-suffixed imports in combination with module specifier rewriting because they believe that will let them write input code that will run natively in Deno, while tsc’s output will run natively in Node. This is out of scope; the way to write once and ship to multiple environments is to target a bundler and produce multiple bundles. Consequently, I think there are very few users who need to write .ts-suffixed imports (especially in a world with conventional), but they are unobjectionable in noEmit and easy to implement.  The feature is not core to this proposal, but I believe it would be a mistake to create new module resolution modes without at least fixing the aforementioned bug to carve out the possibility of .ts-suffixed imports resolving in the future.
Notes on conventional
This proposal does not allow for a perfect mapping of TypeScript’s resolution behavior onto every bundler, but I think it covers most cases, or what I will call all reasonable cases. If we wanted to be a bit prescriptive, I would be tempted to prohibit .cts and .cjs files, disallow import m = require(...) syntax in TypeScript files, and disable resolution of require calls in JS files. Some of the newer bundlers are explicitly ESM-only, ignoring or prohibiting require calls in user code and converting library dependencies from CJS to ESM. No bundler I tested had separate CJS and ESM resolution algorithms, with the exception of setting import vs. require in the resolver conditions when looking up package.json exports. There seems to be little reason to allow explicitly CJS constructs in implementation files in this mode (while CJS constructs in dependency declaration files obviously need to be consumable). As in TS files today, users will still be free to write require calls, but they will not have special resolution behavior.
Unanswered questions

This proposal exposes users to increased complexity. I don’t see a way around that. I have also proposed deprecating --moduleResolution node, which is the default for --module commonjs. Consequently, many users are using node without realizing it. This raises the question of what defaults we should have in the future, what module settings should be allowed with these moduleResolution modes, and more broadly, how to guide users into selecting the correct settings for their project.

A possible mitigation is to allow multiple tsconfig.json extends and encourage bundlers to publish (or publish ourselves under @typescript) tsconfig bases that reflect the resolution behaviors supported by these bundlers out-of-the-box. That way, an esbuild user could write{
      "extends": ["webpack", "./tsconfig.base.json"],
      "compilerOptions": { /* ... */ }
    }

Should .ts-suffixed resolution be automatically allowed based on noEmit, or should it be gated behind another flag, or should the capability be preserved for the future but not enabled yet? It makes sense to me that noEmit should enable it, because if you’re writing modules but not emitting, it stands to reason that another tool is going to consume the TS modules that you wrote. @DanielRosenwasser raised the idea that this may cannibalize project references usage, which requires declarations to be emitted and can help speed up type checking when splitting large codebases. More thought needs to be put into how .ts imports would work with declaration emit.
For simplicity and consistency, can we fix the resolution bug where files with unsupported extensions do not stop other lower priority extensions from being looked up even in today’s node, breaking backward compatibility? The first time I considered this, I thought this would be an untenable breaking change, because people rely on this behavior to write declarations for files with unsupported extensions, e.g. import styles from "./styles.css" would resolve to styles.css.d.ts because it thinks that’s analogous to styles.css.js, not styles.css. However, @weswigham has proposed a general solution for this at https://github.com/microsoft/TypeScript/issues/50133. Taking some form of that proposal may be key to fixing this “bug” in any resolution mode without effectively losing a feature.

Related: #37582, #49083, #46452, #46334, and probably a dozen others

Add an option to enable/disable package.json exports

I think a very important question for this is: With what default conditions (configurable?) and what behavior for the import/require conditions? Each bundler runs different condition sets from what I know - most have a bundler-specific one, in addition to often using a module condition, on top of import and require.

Also, an important concern for declaration emit is cross-resolution mode compatibility. node and node16 are often close enough that we can gloss over the differences in DT, but with more available configuration, will we potentially need multiple package implementations on DT for different module resolution settings, and, if so, how do we ship (or compose) that?

Also ref https://github.com/microsoft/TypeScript/pull/29353 where I did what I thought would be some minimal emit changes to create a mode about as flexible as bundlers we're at the time. That didn't include any resolution changes :)

With what default conditions (configurable?) and what behavior for the import/require conditions?

import or require based on importing syntax (if we allow require to resolve at all); additional conditions should be configurable.

will we potentially need multiple package implementations on DT for different module resolution settings

This is a problem for us and DT just about as much as it’s a problem for actual runtimes and bundlers consuming implementation packages from npm. What’s in DT should reflect what’s in the implementation package on npm. If the implementation package writes imports that only resolve under bizarre resolvers of the future, it will make sense for the DT package to do the same; it will make sense for users to have broken types under node16 if Node 16 will choke on the implementation package. (This is already basically the world we live in but substitute assumptions about resolution with assumptions about globals.) As of today, nothing I’ve suggested really erodes at the rough cross-compatibility between these existing modes (the changes I proposed to make conventional out of node are very minor). In fact, minimal is the ultimate common-denominator mode, so packages aiming to be as cross-compatible as possible can opt into that (if they take on the extreme limitation of vendoring their dependencies and referencing them with relative paths).

With what default conditions (configurable?) and what behavior for the import/require conditions? Each bundler runs different condition sets from what I know - most have a bundler-specific one, in addition to often using a module condition, on top of import and require.

Just to add onto this (I'm sure this is known from the research put into this proposal, so forgive me if it's already been considered), esbuild has conditions totally configurable (https://esbuild.github.io/api/#conditions, https://esbuild.github.io/api/#main-fields); its behavior was almost changed in the last release which would have changed its defaults too in some scenarios to avoid dual-package hazards (https://github.com/evanw/esbuild/issues/2417), since bundlers can sort of just "import" anything however they want regardless of the syntax (since they control the runtime import behavior internally).

Seems like loaders for Node.js such as ts-node and tsx, or even a future version of Node.js, would possibly want to allow .ts extensions.
Would this be as simple as taking conventional, add the features from node16+, and set noEmit?
I do have some concerns with the approach of enabling .ts extensions without its own flag. Maybe over time noEmit could imply that, the same way that checkJs implied allowJs.

Both minimal and conventional (but not node-legacy) will support resolution to .ts files by specifying a .ts extension in the module specifier. This will be an error, as it is today, unless noEmit is enabled.

I'm curious how this will play out with projects that interpret TS directly; I seem to recall one popular project actually requiring that you didn't set noEmit: false in your tsconfig, but my memory is failing me as to what that is (was it ts-node? a webpack loader? it's been too long ugh)

For anything actually running TS in node at runtime, I think it'd have to be node16 + noEmit allowing direct .ts references.

I do have some concerns with the approach of enabling .ts extensions without its own flag. Maybe over time noEmit could imply that, the same way that checkJs implied allowJs. But maybe my concerns are not well-founded.

Maybe to dovetail with https://github.com/microsoft/TypeScript/issues/50133, we should have a allowedRuntimeExtensions: [".css", ".html", ".ts"] sort of option.

@jakebailey I believe you’re thinking of ts-loader, but only in “full mode” as opposed to transpileOnly, which has been in the back of my mind to look into, but basically everyone uses transpileOnly
@DanielRosenwasser .ts imports in Node proper would be enabled in nodenext under noEmit and/or whatever other options we pick to make it work in any other mode. We will have to fix the resolution bug I mentioned, which I think we should do soon even if we don’t do anything else soon.

basically everyone uses transpileOnly

Not my last team... 😨 (I think for const enum inlining, and one weird dependency that is only a d.ts file and declares a const enum.)

I feel confident that either it will not matter at all or that the combined will and know-how of me and @johnnyreilly will be up to the task.

As a library author, one thing which would be nice is to allow moduleResolution to be an array, which tells typescript that imports must be valid in all of the listed types.  So this would be even more restrictive, as library authors we need to write imports in a way that all the module resolutions work and typescript could help by giving an error.  So we could specify moduleResolution: ["conventional", "node16"] and have typescript verify that our imports are correct no matter if the consumer of our library uses conventional or node16.
There wouldn't really need to be any extra special logic in typescript, if moduleResolution is an array typescript would just process each import twice and error if they are not all equal.

I guess its still time to mention TypeScript Should let the Host System do Specifier Lookup let me explain
The ECMAScript Standard Says Specifiers get Resolved on the Language Embedder Side
In Our Case this is TypeScript it self i my self Authored a Lot of Engines so Host Systems that Embedded ECMAScript
my Proposal is finally Merge Rollup or at last its algo under module Resolution rollup into typescript
Rollup is a loader / bundler and a universal ModuleSystem all in One.
All Existing Host Engines do offer Ways to Manipulate Specifier Resolution

ServiceWorker (Browser) Request Interception rerouting.
NodeJS Loader Hooks
GraalJS Context Filesystem Modification or shim
Deno Bundle everything always before load and this way Configure Specifier Lookup

Advanced
rollup uses Acorn as AST which is more flexible i suggest changing the Typescript AST with the Acorn one for better Interop in general while there exist Shims for both AST's to work with each other.
the missing parts could get created as Acorn plugin
More Big Proposal
Do not Integrate a Plugin system for specifier lookup kick it all out and go back to only support moduleResolution Classic
and then we do in user-space offer Fuse or something else to provide the right file structure.

Hi @andrewbranch this really interesting proposal, we have a recurring meeting at TC39 focusing on Tools with folks from(Deno, Parcel, etc..). It would be nice to bring this to the discussion.  Send me a DM or fill the form in case you want to participate. :)

about DTS resolution, can we emit code like this to let all declaration brings their own resolution mode instead of inheriting from the project?
// my-library/dist/index.d.ts
export { util } from './utils/index.js' asserts { "resolution-mode": [...] }

Thanks for taking the time to study this deeply.  Module resolution is fundamental to the entire web tooling stack so getting it right is valuable and appreciated.
Initial Reaction
We know this is a tough problem space from experience.  Inside Bloomberg we have our own runtime environment (everyone seems to be doing it these days) with its own custom resolution semantics.  As surmised in Background, just like all the other non-Node runtimes we chose to use "moduleResolution": "node" to approximate our system's resolver.  It is not a perfect fit because our resolver is more browser-like and does not do a recursive walk of node_modules directories and does not feature any package.json files.  Therefore the proposed "minimal" mode is a closer approximation to our resolver and we would make use of it 👍
Today we handle resolution of bare-specifiers using "paths" to enumerate the full set can legally be imported by the project.  In our system, each project's tsconfig is tool-generated, so this is easy.  I'm expecting "paths" will continue to compose nicely with the new modes.
Explicitness as a Guiding Principle
The overall direction of this proposal is to permit explicitness in source code as a way of ensuring correctness.  I think this is our North Star and we can use it to resolve some of the consequential issues.
Explicitness matters most obviously in the runtime.  As the web platform evolves to allow more file formats to be directly imported (JS, CSS, JSON, Wasm, et al) extensionless imports become a hazard.  It's hard to know what import x from "./foo" will mean if you have a directory containing:
- foo.js
- foo.css
- foo.json
- foo.wasm
- foo.<whatever is introduced next>

Bundlers answer this question by implementing priority order.  require() also does this.  But for the wider web platform, there is no universal precedence ordering of file types.  Browsers won't answer this question either; they use explicit mime types and regardless, would not tolerate resolution based on multiple round-trips.
So the preference for our codebases is to avoid betting on a specific bundler's convention and instead sidestep this hazard by ensuring that relative specifiers in source code always use the full source code filename explicitly including its extension.
Importing Declaration Files

in these modes, I suggest that it be legal to write an import type of a module specifier ending in .d.ts.

This is an appreciated step forwards towards permitting and encouraging explicitness 👍
Low priority suggestion:  For relative specifiers in "minimal" mode, emitting these extensions in Declaration Emit could also be a useful follow-on step.  It's all more clues for the reader and safeguards against the introduction of new extensions in future.
Emitting relative module specifiers ending in .ts
The proposal states that resolution of .ts files will be supported by minimal, but emitting will not.  Support resolution is highly appreciated and represents another crucial step forwards towards explicitness 👍
However, the ban on emit seems unnecessarily restrictive and is the main thing we would seek to change 🙏
Our internal platform will resolve whatever you point it at.  Just like the web.  So it would be fine to deploy .ts files and have them import each other.  This is primarily desirable from a user's mental model - because the source graph is then coherent, rather than forcing users to reference invisible predicted file names.  I agree that writing anything else (e.g. .js) is "nonsensical for the context".
Beyond developer comprehension, using .ts references will also allow us to reduce tooling complexity by avoiding file & specifier remapping.  In our system we ban the small number of TypeScript concepts that emit runtime code (enums, parameter properties, etc).  So as a concrete example of tooling simplicity, you could imagine our runtime having a development-time file transform loader that makes the TypeScript code executable solely by replacing types with whitespace.  This would make sourcemaps redundant.
The counter arguments in the proposal (see "I think much of the demand we’ve heard so far for being able to use .ts-suffixed module specifiers has been misplaced") don't seem to apply here.  This use-case is not about working around problems with bundlers.  And it's not about shipping the code to multiple environments.  It's purely about having freedom.
If TypeScript enforces the noEmit restriction, we can still workaround it in a few ways.  But the options are not ideal:

Option1 - Fork TypeScript

☹ incurs a small maintenance risk/cost
Our pride in shipping 100% Genuine TypeScript™ so far is mildly damaged

Option 2 - Switch to a third-party TS->JS converter e.g. esbuild or swc

☹ risks divergence from tsc
It feels weird to make a big change to our tooling stack for such a small request
We love having coverage of TypeScript JS emit in our system. It allows us to study the diffs across our entire codebase whenever there is a Beta release, which lets us feedback issues early such as https://github.com/microsoft/TypeScript/issues/46382

Conclusion
To sum up, the core proposal is awesome and includes two key features we value highly:
👍 "minimal" resolution
👍 the ability to reference .ts files
The request to allow emit of .ts specifiers is not a showstopper and could be debated/delivered as a follow-on proposal.

Our internal platform will resolve whatever you point it at. Just like the web. So it would be fine to deploy .ts files and have them import each other

@robpalme Just so I'm clear -- your runtime is capable of handling a situation where there are no .ts files on disk, but two JS files are allowed to refer to each other using their original .ts filenames? e.g.
// a.js
import * as b from "./b.ts";

// b.ts
export { }

How does it know if there was ever a .ts file in the first place? Or is it just totally agnostic?

@RyanCavanaugh I'm describing a different case. Meaning if there is a *.anything on disk, then you can use import foo from "*.anything".  Meaning the file extension is not special.  It's just files referencing files.

@robpalme did you consider using references the tripple slash once to reference files from files? i do that today and it is a win for me. Google tripple slash typescript If you are not aware of that.
for bundling i simply translate the tripple slash references to imports

@robpalme and I discussed the emit vs .ts imports thing offline. TL;DR -- the scenario he describes above would still be well-supported.
The nuance here is that we would still intend to block .ts -> .js emit via tsc in the case where we see .ts imports, since it's unreasonable to think that a runtime could see a .ts -> .ts import in files named .js and treat that correctly (it's not clear how, and especially unclear why, anyone would implement that). However, API-level operations like ts.transpileModule would not be gated off -- tools would still be free to use the TS API to downlevel/type-strip arbitrary content regardless of the file extensions in import statements.

@RyanCavanaugh there is one thing my thing that implements that i have some engines started with integrated Proposal from TC39 and @DanielRosenwasser and other People that i know that are like minded.

https://github.com/tc39/proposal-type-annotations

i for example can take .js and .ts files while .ts files will not work if they use special features like enum but as long as they only use type annotations they will run
In the next JDK Release there will be a Engine Parameter for the JS Interop that enables that for testing. And on the long run i guess more engines will adopt that as it is a clever thing and solves typescripts biggest issue that you can not code something in .ts files that runs as it is.
with that proposal it can run as is as long as it does not use Typescript Superset like enum.

Frank, your linked TC39 proposal doesn't seem to set a convention for file extensions.  It seems like a really bad idea to have one file extension that could mean both:

actual Typescript language complete with all features that have been established over the past decade or so, or
JS/ES with type information only

I would hope that we come up with a better convention for naming typed Javascript files.  (Is .tjs anything yet?)

@thw0rted mope you did not fully understand let me explain there is a ECMAScript Standard it contains 1 module system this module System is Defined in a Specifier Agnostic Way.
In ECMAScript / JavaScript exists only ECMAScript and JavaScript i did not define that this is how it is. NodeJS Implemented a own loader and module system.
@thw0rted there is no need for a extra extension ECMAScript calls everything in the string that you use to import as "Specifier" the Import Specifier needs to get resolved and loaded by the Host Environment (The one that embedded the JS Engine v8)

I think I see what you're saying.  If we look at support for the TC39 erasable type specification proposal the same way as any other new ES feature, like say the dynamic import() keyword, you'd just have to target your bundler/transpiler emit to your runtime support level -- maybe this will be "ES2023" or something.
In that case, though, .ts should always mean TypeScript, specifically, and .js should always mean ES / JS (at some level of the standard, which must be defined externally).  Of course, in a given case, a .ts file could contain only the subset of the TS that also happens to be supported by ES20XX, but it would be a bad idea for x.js to ever import y.ts, much the same as it would make no sense to import y.vue or y.coffee.

@thw0rted deno also imports ts with .ts specifiers there it is wide used and standard

OK, so, TS is a special case, in that Deno can have a JS entry point that imports from a TS file (with a .ts extension), and it will transparently load and run that file without an intermediate transpile step -- and it could not do that with, say, .vue or .coffee or anything else that can, hypothetically, be transpiled to JS.
That still doesn't change the fact that ES20XX Totally Vanilla JS That Happens To Support Some Type Declarations, should carry a .js (or maybe .cjs / .mjs) extension, and .ts should always mean Microsoft (TM) TypeScript (R).  Right?

Deno would be a case of a runtime that sees a .ts specifier, in a .js file, and really does mean to refer to a .ts file on disk that it intends to execute directly.

My comment above had a critical typo in it. What doesn't happen (to my understanding) is that Deno sees an import of .ts, doesn't find a file with that name, and then instead loads the .js file corresponding to that .ts file.
Resolution to extant files on disk is (or files which will eventually exist), I think at this point, uncontroversial. Resolution from build outputs back to build inputs is the thing that I am asserting doesn't exist (and shouldn't).

lets all vote and push Tc39 type annotations and drop the .ts exstension and call it a day.

One minor note; internally we’ve been kicking around hybrid (resurrected from #29353) as an alternative name for conventional, lest anyone read a value judgment into the name, or—perish the thought—conventions change.

Sorry for the late post but i got a good idea including aux resource import Maps as source of truth for resolve.
I guess this is why they got designed. At the end it is the smallest Engine Wide convention that is Resolver Loader Implementation interdependent.  At the end thats what we do with Chromium Fuchsia and other big Projects. Google uses GN to create import maps for the diffrent systems and then hands thats over to the build system (Runtime) Yes in this case the build system also got a on runtime evaluation to create custom ASM based on build values.
And ECMAScript got a standard for that kind of Build/Runtime Maps. Typescript should adopt the importMaps standard as any other ECMAScript Engine Implementer would do it TypeScript can be rated as Language (Runtime Distribution) that Embeddeds the ECMAScript Engine. So its Role is Host so it should use importMaps no matter what the internals are to create that.

, lest anyone read a value judgment into the name, or—perish the thought—conventions change.

Conventions don't change! We're all still manually rendering script tags and using global scope for imports. Right? 😉 window.jQuery forever!

i guess i got something that is acceptable a configureable tsconfig.json option that references a resolve hook that runing in the targeted runtime it self.
thats the only way. as typescript this way does not know when to rerun the hook a additional helper extension with support for the given runtime needs to touch the tsconfig.json to restart the hook
this way we would have light whight interop between a external watch resolve implementation that can send signals via file touch to the language server that watches the tsconfig.json files which are else static.

One minor note; internally we’ve been kicking around hybrid (resurrected from #29353) as an alternative name for conventional, lest anyone read a value judgment into the name, or—perish the thought—conventions change.

Node itself calls the extensionless resolution support mode in ESM "node" which might not be a big deal since I imagine anyone getting deep enough to be setting --experimental-specifier-resolution=node is probably an advanced user, but it still feels like "node" and "not-node" are not the only framing for this since Node also supports custom loaders that can resolve anything, as I'm sure you're aware - we do file-by-file esbuild transpilation of extensionless imports in certain contexts.
I don't know why I would want to / should have to use conventional/hybrid over node16 if it supported extensionless resolution.  It seems like even if a future major verison of node changed module resolution behaviour (such that node16 and, say, node20 needed to exist), it might be the case that both of those systems support extensionless resolution depending on node process arguments, but their differences might make conventional/hybird start to get messy?
Would something like
"moduleResolution": "node16",
"moduleResolutionOptions": { "relativeExtensionsRequired": false }

be reasonable?

I'm inclined to agree, but I would like to separate the desire to make module resolution perfect/best from the more tactical value of just supporting extensionless imports and package.json exports in node mode.

That's why I'm saying that just "extensionless node16" might be good enough and make it so hybrid doesn't even need to exist?

No, there are other significant differences between typical bundler resolution and Node 16; namely CJS vs ESM detection and the interop between them.

Okay - so if someone is using Node with --experimental-specifier-resolution=node they will have to use hybrid/conventional/whatever it gets called?

@andrewbranch this experimental specifier resolution node only means that the same hazard gets added to imports which already exists for require so 5 diffrent resolution algos get joined into import
hope that makes sense to you before this flag import was relativ strict :)

Please forgive me if this was covered and I missed it, but instead of trying to replicate all/most possible consumers' resolution needs could TS delegate resolution to the consumer?
Establish a well defined spec for a plugin defining resolution then collaborate with popular ecosystem projects to provide resolvers as packages that are designed to meet their needs. Couldn't that both relieve a lot of burden from TS and provide room for whatever directions the community goes in terms of module resolution? Let TS focus on type checking, leave module resolution to various platforms.
Again sorry if that's already been covered and isn't a possibility for one reason or another.

Yes, I did mention it:

At the same time, we have resisted allowing resolver plugins for many reasons:

security and performance concerns
implementing correct module specifier generation for a given resolution mode for declaration emit, auto-imports, and path completions is extremely non-trivial
concerns about reliability under file-watching modes (a resolution must return not just what it found, but where it looked and found nothing, and where it looked up auxiliary information used in resolution such as package.json files)
we would like to avoid allowing bespoke per-project resolution simply to reduce chaos

I’m 100% sympathetic to the frustration with the complexity. What we’re hoping to do is allow for some sort of delegation of configuration rather than delegation of implementation. We’re prioritizing https://github.com/microsoft/TypeScript/issues/29118 alongside this work so bundlers can ship a base tsconfig specifying their out-of-the-box semantics, so you’ll be able to do something like
{
  "extends": [
    "../tsconfig.base.json", // local settings, strict, target, lib, etc.
    "webpack"                // webpack's module resolution defaults
  ],
  "compilerOptions": {
    // whatever you need to override
  }
}

assuming Webpack is on board with the idea.
I think typically, you don’t need to do both (1) and (2) (if you bundle then you are removing the need for your platform to do most module resolution—perhaps a few externals aside). For (3) you should be able to find presets available for either your platform or your bundler. If you find a way to fix (4) (the JavaScript ecosystem on the whole), let me know.

Thanks for the clarification 😃 I definitely see how it could open up a can of worms to allow arbitrary resolution plugins.
What about the possiblity of going the other direction -- separate module resolution out into a library so that other tooling could plug in exactly the same resolution that TS is using?
That could reduce the minor incompatibilities that are sure to crop up and perhaps help get more tooling standardized on a single implementation of module resolution. Sorry if that's a naive idea based on a lack of full understanding of problem space, I'm just excited about any ideas that might reduce all the frustrating fragmentation in the larger ecosystem.

I don’t recall off the top of my head what stuff is marked @internal (probably a lot of it), but typescript is a library and in fact I know the TS plugin for rollup does exactly that. But I’m skeptical it would really be broadly usable, because while we do our best to mirror the resolution of existing platforms, over here in Mirror Land, a lot of the time we’re ultimately resolving to types, not JS. So this is not just a matter of file extension, but also node_modules/@types, typesVersions, and the types condition in package.json exports. It would take a good bit of work to abstract these things so they’re useful for anyone but us. I like the brainstorming, and it’s something we’d be philosophically open to, but these other platforms haven’t exactly been begging us to share our version of module resolution with them.

What about adding a new typescript keyword? At the feature-level along the lines of the enum keyword, not a type-level extension.
So import and require are passed through as JavaScript with minimal or no change, with the new thing (eg load) handling any higher-level functionality, specifically anything involving rewrites, aliases, extension mappings, etc.

@ferbs your correct that is exactly the other way that works implementing a own Loader that is share able like any other typescript helper in fact typescript already creates such import loader helpers when the settings like inter Op are enabled for the star imports and syntactic imports we can expand on them engine. that closes the circle of resolution.

[In --moduleResolution conventional/hybrid] if we wanted to be a bit prescriptive, I would be tempted to prohibit .cts and .cjs files, disallow import m = require(...) syntax in TypeScript files, and disable resolution of require calls in JS files.... As in TS files today, users will still be free to write require calls, but they will not have special resolution behavior.

I need to make a decision on this soon, and am still leaning toward doing what I described above. I feel pretty confident about disallowing import ... = require and ignoring const ... = require, but slightly less confident about disallowing .cts and .cjs files. In many existing bundlers and non-Node runtimes, these extensions just have no special meaning, so there’s no technical reason we couldn’t allow them and treat them the same as .ts or .js files, which could perhaps be useful for importing a file that also gets consumed by another system where the extension is meaningful. The strong downside is it just doesn’t make a lot of sense to have a CJS file where you’re only allowed to write import and module resolution from the file isn’t impacted by the extension. Webpack is also a weird case where .cts files are allowed to have imports and requires co-mingling, but .cjs files have special meaning and are only allowed to use require, as in Node.
If anyone makes heavy use of require or .cts or .cjs files in bundlers, now would be a good time to speak up!

I have a couple of monorepos which combine cts and mts files (though normally each in their own package). Another use-case is config files that are required to be commonjs in a project with type: module. And if you throw react into the mix that also includes tsx files (which I would really rather be mts files, but I haven't found a way to get JSX in mts...).
I personally prefer the explicitness of mts/cts, so I basically never use bare ts files anymore.

@Alxandr and you’re importing these into an app with a bundler? Why do some packages need to be primarily cts? Do you have any examples you can share?

@andrewbranch
the usecases for CTS
are mainly JS Integrations in other Software for Example near any Importent Software Supports ES3-ES5
On Linux / Mac the whole desktop is script able for example.
In NGINX and 100th's of Databases u can use ES3
The most Prominent cases as of time of Writing for CJS are NWJS and ELECTRON main fild files.
As also everything that needs to catch events from the Engine it runs in needs to be CJS as this is Sync and MJS is always Async so MJS always adds 1 extra engine thick before it runs for the instantiation phase.

Got it. If you’re using Node, you can continue using --moduleResolution nodenext and everything should work great. I’m only concerned about use cases in --moduleResolution hybrid, which is for bundlers and non-Node runtimes that don’t have the same flavor of CJS/ESM dichotomy that Node has.

That's what I'm talking about though. All application sources are written in ESM, and built with a bundler. The configuration for the build-system/bundler however, is written in CJS.

Heads up that Bun is about to make some changes to how it does module resolution, so this part is no longer the case

Running in Bun
The situation is exactly the same as the above, since Bun’s module resolver is a port of esbuild’s, and it consumes TS files directly

Starting in Bun v0.2.3, Bun will resolve modules from a global shared cache and automatically install dependencies from npm on import. Bun's builtin package manager automatically creates a lockfile while installing & resolving dependencies, and now Bun's module resolver will use the lockfile to determine what version of a package to load from the global shared cache. The node_modules folder is now optional.
package.json, if present, works the same as before (along with "paths" support in tsconfig.json resolution, "exports" & "imports" conditions in package.json, consuming TypeScript files directly, etc). The default version of an imported dependency is inferred from the package.json
Before:
import * as lodash from 'lodash';
// ./node_modules/lodash/index.js

After:
import * as lodash from 'lodash';
//<EMAIL_ADDRESS>
For dependencies of dependencies, the module resolver again reads from the lockfile to determine what the resolved version is and loads it from the shared global cache.
What's the best way to make this work well with TypeScript's editor experience?
Note that, for compatibility, this feature is turned off if a node_modules folder is present. When a node_modules folder exists, bun resolves modules the same as before (like esbuild, using node's module resolution algorithm, with tsconfig.json support etc)

@Jarred-Sumner move the types into<EMAIL_ADDRESS>and call it a day

The Typescript classic resolve which is inhired also by node moduleResolution is a 1:1 mapping file.js file.d.ts

@Jarred-Sumner move the types into<EMAIL_ADDRESS>and call it a day

How would TypeScript know that the lodash specifier is in<EMAIL_ADDRESS>

Typescript does not know that but as soon as you<EMAIL_ADDRESS>or open it Typescript will know that it should look into the index.d.ts

@Jarred-Sumner and you directory layout is not the best you should look into existing implementations like npm and pnpm for a cache you should not choose such a big file structure like flat module names with versions that does not scale well on most systems.
thats why most aggree on
/cache/sha256/xx/xx/bbbbbbbbbbbbbbbbbbbb <= xxxxbbbbbbbbbbbbbbbbbbbb

that is easyer to scan and lookup

Starting in Bun v0.2.3, Bun will resolve modules from a global shared cache and automatically install dependencies from npm on import.

Just FYI this is effectively what the Bloomberg runtime & tools do. Auto-download dependencies & types into a shared global cache directory where each folder is named with an explicit name+version.
We made this work using vanilla tsc resolution (no LSP needed) by having the tools automatically write a file into each project called ambient-modules.d.ts that contains ambient module declarations that act as symlinks from the specifier "lodash" to the location in the global cache.
// ambient-modules.d.ts
declare module "lodash" {
  export * from "/global-cache/lodash-1.2.3";
  export default from "/global-cache/lodash-1.2.3";
}

I think we have since switched that to injecting entries into tsconfig "paths" as it felt more central/explicit. More details here.

@robpalme do you know maybe anyway how we can push the adoption of tc39 type annotation proposal https://github.com/tc39/proposal-type-annotations when we would get that landed even if it would be only babel
that would open up the end of the resolve story at all. as we can simply author then in .js

Thank you @robpalme, we will try that.

I would like to emphasize what @robpalme brought up earlier, explicitness.
Including resources WITH a file extension SHOULD be the standard. Please work towards making this possible.
Hiding the file extension adds a new layer of complexity and adds ambiguity. Code is much easier to reason about when there is a 1 to 1 mapping, e.g. #include "something.h" -> there is a file named exactly something.h and you can easily search for it.
All such hiding of parts of the resource name that the user interacts with hurts more than it helps (and don't get me started on Windows file explorer hiding file extensions).
The same applies for instance to Angular cli's unhelpful "fiendly" behaviour of automatically adding src/app in front of the argument you give when using the tool to generate new things. Until you're burned enough times your first attempt to run the cli tool in the most intuitive and natural way, e.g. ng generate component src/app/feature/todo always fails because you then end up with src/app/src/app/feature/todo/todo.component.* files which obviously was not what was intended.
Or LaTeX.
# Is the logo image used anymore?
ack -l logo.png *.tex
# Well the above query will not tell because maybe there still is an \includegraphics{logo} somewhere...

Or similarly the related lack of command line argument consistency for npm
npm start     # Runs "start" entry in scripts in package.json
npm mystart   # Fails to run "mystart" entry in scripts in package.json

If npm developers have had any decency in their command line API design the start sub command should newer have existed and instead everything should run through the generic run sub command, e.g. consistently
npm run start
npm run mystart

If typing npm run start is too much work then you can write your own npmstart.sh, start.rex or whatever shortcut in your favourite scripting language.
From Joshua Bloch's excellent talk How to design a good API and why it matters (slides):

User of API should not be surprised by behavior

It's worth extra implementation effort
It's even worth reduced performance

Hiding file extensions will cause surprises and is a bad idea.
In C++ they switched from #include <iostream.h> to #include <iostream> but at least the corresponding file was also correspondingly renamed so that #include <iostream> maps to just /usr/include/c++/11/iostream and NOT /usr/include/c++/11/iostream.h. So that is a removal, not hiding; there is no file name translation magic involved here.
Explicitness and consistency is vastly more important than laziness and convenience.

@hlovdal first of all your correct my hot fix for all the years was always using correct specifiers to workaround all resolve related issues and then remove them for typescript processing your observations are correct out of user view
but out of implementer view there where no extensions as this got designed not in ECMAScript and not in JS so not in Typescript.
While we the developers started to compose big JS projects we found out that file extensions are beneficial when working with many sources even none JS once.
today we want to workaround that via 2 concepts that can on the long run again solve the extenson problem
asserts
import me from 'my-ts' assert { typescript }

i personal do not like that but ok at last it is specific
this is what i like custom url specifiers used for custom resolve algos specification
import me from 'ts:my-module'

Importent
@andrewbranch importent upgrade!
@DanielRosenwasser
NodeJS 19 gets Loader Hook Support before frozzen intrinsics!
this closes this and many other issues still collecting and writing up migration path. i guess as you run in NodeJS Mainly and other Platforms also support load resolve hooks this solves the root cause of the moduleResolution Problem even allows to Algin CJS and ESM Context Resolutions and adding Instrumentations,...... it is a to big list at present to post here but saw your hybrid branch you should stop that and look into --import --require flags of nodejs 19 and related issues. sorry for the ping but i guess it saves you and all others time.

Ok i got it some thing action able i guess at last the world got nothing better prove me wrong

[ ] https://github.com/lemanschik/why-you-should-switch-away-from-npm-nodejs

Wanted to give a few updates here.

The module resolution mode for bundlers, called conventional in this proposal and then subsequently hybrid, is implemented at #51669 but will likely not be called hybrid after all. A discussion is ongoing in #51714. I hope to settle on a name and merge the PR this week, which means it will ship in 5.0.
We decided in a design meeting a few weeks ago to hold off on minimal for right now, because with its intentional lack of support for node_modules, the story around resolving typings for dependencies that you put in your app somewhere, whether in a folder called node_modules or vendor or anything else, felt very incomplete. A bit of discussion on this is at #50600. I’m still very interested in having a module resolution mode that’s appropriate for browser-native ESM, but I’m struggling to get much real-world feedback on how folks are approaching this. If you have a frontend app, especially written in TypeScript, especially with dependencies, that uses ESM in the browser without going through a bundler first (I know Vite emits ES modules, but it handles module resolution internally so it’s irrelevant to this scenario), I definitely want to hear from you! But for now, hybrid was the higher priority and minimal will not land in 5.0.
I plan to add to and update our module documentation, which is pretty scattered/incomplete/stale right now. If you have any current questions or points of confusion about modules and TypeScript, or if you used to be confused but something in particular made it click for you, please tell me about it in #51876.

I was keeping it open because we never shipped anything for minimal, which is blocked by the design problem of #50600. We’ve received very little feedback in the meantime that people need it though, so there are no immediate plans to make anything new.

If you have a frontend app, especially written in TypeScript, especially with dependencies, that uses ESM in the browser without going through a bundler first (I know Vite emits ES modules, but it handles module resolution internally so it’s irrelevant to this scenario), I definitely want to hear from you!

I don't know how I missed this request, but I build all of my frontend apps with ES modules and no bundler with TypeScript.
Here is the template I use for all of my apps:
https://github.com/Zoltu/preact-es2015-template


Fix publicsuffixes in native image

   ?? SampleTest ?
      ?? testExternalSite() ?
      ?? failingTest() ?
      ?        Expecting:
      ?         <"hello">
      ?        to be equal to:
      ?         <"goodbye">
      ?        but was not.
      ?? testMockWebServer(MockWebServer) ?
      ?? testPublicSuffixes() ?

Test run finished after 3931 ms
[         9 containers found      ]
[         0 containers skipped    ]
[         9 containers started    ]
[         0 containers aborted    ]
[         9 containers successful ]
[         0 containers failed     ]
[        60 tests found           ]
[         1 tests skipped         ]
[        59 tests started         ]
[         0 tests aborted         ]
[        54 tests successful      ]
[         5 tests failed          ]


Doesn't seem to work on Android < 5

I am trying to use this on Android. I have a number of devices and all seem to work except those running 4... I am using React Native 0.40.0

First of all sorry for the late response.
Can you elaborate and give more information?


DA-3200: Active community leaderboard is displaying 8 records on a page

To fix pagination issues -

Removing wild card from index pattern (as in case of git, it was also querying github-issue)
Allowing where condition for single data source requested & sortField=cnt in the main query.

Deployed.


Checks Run Update: start_column and end_column are required fields in output.annotations

endpoint scope: Checks Run
endpoint name: Update a check run
route: PATCH /repos/:owner/:repo/check-runs/:check_run_id
documentation URL: https://developer.github.com/v3/checks/runs/#update-a-check-run

Current specification
https://github.com/octokit/routes/blob/38f1312c7967ef1da1219a27fb77d2e0b8180aff/routes/checks/runs/update-a-check-run.json#L153
and
https://github.com/octokit/routes/blob/38f1312c7967ef1da1219a27fb77d2e0b8180aff/routes/checks/runs/update-a-check-run.json#L160
I expected
Weirdly it took me weeks to realise this was a problem, the actual documentation https://developer.github.com/v3/checks/runs/#update-a-check-run doesn't describe these as required however I've concluded this is the case.
Will raise a PR to fix those routes in this repo

Hey @eknowles are you sure that start_column and end_column are required? If they really are, ideally you would create a script that reproduces the problem and let the docs team know so that they can investigate. If they add the **Required**:  prefix, octokit/routes will update automagically

The docs have been updated, the responses show start_line / end_line as integers now

Hey @eknowles did you reach out to the support team? I don’t think start_column and end_column are required, they surely are not when creating a check run. I use that API in the WIP app: https://github.com/wip/app/blob/4388cd39ce06baeb2bf646af180b02422368c9bf/lib/pro/set-status.js


bug(header) inconsistent and absent focus and hover states

page header links have some issues (some were changing the order the scss was loaded in the webpack bundle)
application name
+ missing hover state
+ missing focus state
language dropdown
+ not even focusable. this is a problem addressed by #176
login / out
+ missing focus state
+ no labels of any kind (text or aria), nor any identifying mark as to what it should do
general
+ when you actually hover things, the text color is the element but the background color is it's parent - the <li> - which also has a hover state.  this doesnt seem like a good pattern to follow

i will fix these, and use the fixes in lumberjack.  should i keep these fixes in lumberjack's fork, or should it be applied to this one as well? (and in the latter case, what colors etc should i use?)

The last screenshot is probably only on your application. In my applications when I hover a menu item it's background color fade to a darker one and the foreground stays the same.


Assertion Failure when presenting UIImagePickerController

I'm trying to present a UIImagePickerController, but the app crashes with the following error:
*** Assertion failure in +[PXUITableViewHeaderFooterView subclassInstance:], /Users/nicolasvalenzuela/Git Projects/pixate-freestyle-ios/src/Kernel/Categories/NSObject+PXSubclass.m:64
Please, I need help on this, since it's crashing in a major functionality I'm working on.

Can you please try to use StylingKit instead of pixate-freestyle-ios.
It is derivative of pixate-freestyle-ios and includes a lot of stability and performance issues and iOS 9 compatibility.

@nico-val did you ever find a workaround?

@Zbeyer I couldn't fix this using pixate-freestyle-ios. It hasn't been maintained for 3 years now, so I followed @adc-amatosov suggestion and migrated to StylingKit. I suggest you doing the same.


NOTICK: Update dependency versions for detekt, hibernate, jackson, javalin and junit

This PR updates the following dependency versions:

Detekt 1.20 -> 1.21
Hibernate 5.6.9.FINAL -> 5.6.10.FINAL
Jackson 2.13.2 -> 2.13.3
Javalin 4.6.3 -> 4.6.4
JUnit 5.8.2 -> 5.9.0

Notes:

The detekt update has resulted in a lot of extra errors being flagged in the AMQP code. I re-baselined it but I still had to set some new suppressions. The rest of the codebase was unaffected.
The detekt update also resulted in errors being reported in the detekt configuration. These have been addressed.
The JUnit update ended up being more impactful than might be expected from a minor version bump, as a bugfix introduced in this version is actually a breaking change. Previously test lifecycle methods (e.g. a method marked as @BeforeEach) were incorrectly allowed to be private. I've updated instances where this was causing a problem. See https://junit.org/junit5/docs/snapshot/release-notes/ for details.

These are not the only dependencies that need updating, but this PR is large enough without fixing further dependencies.

@chrisr3 Updating JUnit causes an osgi resolution problem in the sandbox test code. I've had a go at resolving by adding some extra dependencies into the test code, but it's very possible there's a better approach here. I'm not totally familiar with what's going on so you might want to have a quick look.

Don't merge just yet in the event this succeeds - establishing whether a forced quasar downgrade helps with the e2e tests.

Should be ok now, we're back on snapshot versions of quasar.


Upgrate to yocto morty version

We are currently basing our linux host OS on yocto jethro. This task should bring all the boards to yocto morty.
Ref.: https://wiki.yoctoproject.org/wiki/Releases

As a first step, I will update them to krogoth. Just so we are up to date. And then inspect which boards have the BSPs on morty, and which are problematic

@agherzan meta-ti [http://git.yoctoproject.org/git/meta-ti] has no morty branch

@floion I know ti uses branch late strategy for releasing new branches. I would suspect that master would work just fine.

Should include https://github.com/resin-os/resinos/issues/154

fixed by https://github.com/resin-os/meta-resin/pull/543 and https://github.com/resin-os/meta-resin/pull/523

All the boards board release 2.0 are on morty (or closest to morty possible based on BSP support). Moving this out of the milestone as there are still old boards not updated yet.

Closing this issue as all the boards we had in plan for 2.0 are on morty or the latest supported version by the BSP.

Reopening as qemu device types are planed for morty update too,

resin-qemu was upgraded by https://github.com/resin-os/resin-qemu/pull/50


Arithmetic overflow found on wrap_columns()

I executed fuzzing with textwrap-0.16.0. Proper panic message should be added by assert! or need to use checked operations.
Thread '<unnamed>' panicked at 'attempt to multiply with overflow', textwrap-0.16.0/src/lib.rs:1173

pub fn wrap_columns<'a, Opt>(
    text: &str,
    columns: usize,
    total_width_or_options: Opt,
    left_gap: &str,
    middle_gap: &str,
    right_gap: &str,
) -> Vec<String>
where
    Opt: Into<Options<'a>>,
{
    assert!(columns > 0);

    let mut options: Options = total_width_or_options.into();

    let inner_width = options
        .width
        .saturating_sub(core::display_width(left_gap))
        .saturating_sub(core::display_width(right_gap))
        .saturating_sub(core::display_width(middle_gap) * (columns - 1));   // overflow!

reproduce with :
let mut fuzz_arg0: &str = "J";
let mut fuzz_arg1: &str = "\u{8}\n\0?@";
let mut fuzz_arg2: &str = "";
let mut fuzz_arg3: usize =<PHONE_NUMBER>2109585368;
let mut fuzz_arg4: &str = "";
let mut fuzz_arg5: usize = 0;
wrap_columns(fuzz_arg0, fuzz_arg3, fuzz_arg5, fuzz_arg2, fuzz_arg1, fuzz_arg4);

Hi @HeeillWang, thanks for reporting this!

Proper panic message should be added by assert! or need to use checked operations.

Could you either add an asset that forbids using a width of zero of add the checked arithmetic? I think it's okay to document that the width should be non-zero and just reject such input since it doesn't make sense.

It seems like the overflow occurs because size of column is too big, not because width is zero.

It seems like the overflow occurs because size of column is too big, not because width is zero.

Hey @HeeillWang, I haven't debugged this further — if you make a PR for this, then I would be very happy to include it.


Multi gpu fix

jenkins: retest Ubuntu-CPU please

jenkins: retest Ubuntu-GPU-single please

jenkins: retest Ubuntu-GPU-multi please

jenkins: retest Ubuntu-CPU please

jenkins: retest Ubuntu-GPU-single please

jenkins: retest Ubuntu-GPU-multi please

jenkins: retest Ubuntu-GPU-multi please

LGTM, can you upstream these changes? Thanks!

LGTM, can you upstream these changes? Thanks!

I have started a pr here, https://github.com/tensorflow/tensorflow/pull/51470. I will add reviewer when it is ready.


Update remark related validation

Closes #248
Summary:
This pull request addresses the validation logic for remarks in the address book application. The changes aim to relax the constraints on remarks, allowing for a wider range of characters and improving the handling of certain inputs.

Key Changes:

ParserUtil.java:

Updated the parseRemark method to process escaped forward slashes (\/) by replacing them with normal slashes (/).

Though rare, this should support cases when the user wants to add a remark that collides with our existing prefixes

Adjusted validation to use the processed remark for checking validity.

Person.java:

Updated the isSamePerson method's comment to include remark in the list of attributes considered for equality.

Remark.java:

Removed the alphanumeric constraint from the VALIDATION_REGEX.
Updated MESSAGE_CONSTRAINTS to reflect the new validation rules, focusing on non-blank and length constraints.
Modified isValidRemark to only check the length of the remark.

JsonAdaptedPerson.java:

Trimmed remarks before validation and creation of Remark objects to ensure consistency.

Test Updates:

ParserUtilTest.java and JsonAdaptedPersonTest.java: Updated invalid remark tests to reflect the new length-based validation.
RemarkTest.java: Expanded valid remark tests to include various characters such as punctuation and symbols.

Rationale:
The changes aim to make the remark field more flexible by allowing a broader range of characters, which can be useful for users who need to include punctuation or special characters in their remarks. This update also ensures that remarks are not blank (when inputting in the add command) and do not exceed the maximum length, maintaining data integrity.

Impact:
These changes will allow users to input more descriptive and varied remarks without encountering validation errors due to character restrictions. The updates to the test cases ensure that the new validation logic is thoroughly tested and reliable.

I try that your escaped / feature in the remark field, but it is not working as intended.
input:

output:

I try your escaped / feature in the remark field, but i think is not working as intended as it includes the escaped slash in the output.

Nice catch, missed the replacement of return new Remark(trimmedRemark);  to return new Remark(processedRemark);


dubbo 2.7.13 not work with spring boot 2.5.3

[x] I have searched the issues of this repository and believe that this is not a duplicate.
[x] I have checked the FAQ of this repository and believe that this is not a duplicate.

Environment

Dubbo version: 2.7.13
Operating System version: Windows 10
Java version: 11

Steps to reproduce this issue

use spring boot 2.5.3
use dubbo 2.7.13
enable dubbo
start the application
spring boot will not start

Pls. provide [GitHub address] to reproduce this issue.
Expected Result
What do you expected from the above steps？
Actual Result
What actually happens?
If there is an exception, please attach the exception trace:
2021-08-09 16:44:09.624 ERROR 4388 --- [           main] o.s.b.d.LoggingFailureAnalysisReporter   :

***************************
APPLICATION FAILED TO START
***************************

Description:

An attempt was made to call a method that does not exist. The attempt was made from the following location:

    org.apache.dubbo.config.spring.beans.factory.annotation.ReferenceAnnotationBeanPostProcessor.<init>(ReferenceAnnotationBeanPostProcessor.java:106)

The following method did not exist:

    org.apache.dubbo.config.spring.beans.factory.annotation.ReferenceAnnotationBeanPostProcessor.setClassValuesAsString(Z)V

The method's class, org.apache.dubbo.config.spring.beans.factory.annotation.ReferenceAnnotationBeanPostProcessor, is available from the following locations:

    jar:file:/C:/Users/Sheiy/.m2/repository/org/apache/dubbo/dubbo/2.7.13/dubbo-2.7.13.jar!/org/apache/dubbo/config/spring/beans/factory/annotation/ReferenceAnnotationBeanPostProcessor.class

The class hierarchy was loaded from the following locations:

    org.apache.dubbo.config.spring.beans.factory.annotation.ReferenceAnnotationBeanPostProcessor: file:/C:/Users/Sheiy/.m2/repository/org/apache/dubbo/dubbo/2.7.13/dubbo-2.7.13.jar
    com.alibaba.spring.beans.factory.annotation.AbstractAnnotationBeanPostProcessor: file:/C:/Users/Sheiy/.m2/repository/com/alibaba/spring/spring-context-support/1.0.9/spring-context-support-1.0.9.jar
    org.springframework.beans.factory.config.InstantiationAwareBeanPostProcessorAdapter: file:/C:/Users/Sheiy/.m2/repository/org/springframework/spring-beans/5.3.9/spring-beans-5.3.9.jar

Action:

Correct the classpath of your application so that it contains a single, compatible version of org.apache.dubbo.config.spring.beans.factory.annotation.ReferenceAnnotationBeanPostProcessor

2021-08-09 16:44:09.627  INFO 4388 --- [bboShutdownHook] o.apache.dubbo.config.DubboShutdownHook  :  [DUBBO] Run shutdown hook now., dubbo version: 2.7.13, current host: <IP_ADDRESS>
Disconnected from the target VM, address: '<IP_ADDRESS>:5196', transport: 'socket'

Process finished with exit code 1

![image](https://user-images.githubusercontent.com/12210038/128680116-3ea18377-ce6e-4034-9304-26664b971d98.png)

Please check the dependency named "com.alibaba.spring:spring-context-support" whose version should be 1.0.11 and above.

@mercyblitz ok  thanks for your reply


Multiple NavMeshAgents of the same size share one NavMesh

It would be nice if there was a way to have the navmesh surface allow more than a single agent type on its navmesh.
I am guessing it mostly has to do with the edges of the navmesh, and the radius of the agent?  For example, if an agent has a 0.5 unit radius, the navmesh generated is roughly 0.5 units away from everything.
Couldn't the distance from edges be a separate value, and then the radius of the navmesh agent be part of the tunneling/distance from edge calculation?
For example, I could set the navmesh 'edge radius' to 0.25, meaning that when generated, it would be roughly 0.25 units away from all the geometry, etc.
Next, when calculating positioning, I would use something like...  agentRadius - navmeshEdgeRadius to determine how close to the edge of the navmesh I would allow.
In this way, if my edges are at 0.25, and my agents radius is 0.25, it would work like it normally does now.  If my agents radius is 0.125, it would overlap the edge a bit, but still look correct, as far as my agent goes.
If my agent is 1.0 radius I would really be 0.75 units away from the edge...
There may need to be some finagling for areas where the agents radius is larger than the navmesh path radius, for example, if the width of the path is only 0.5, but my agent has a radius of 1.0, the path should return 'blocked' to me, since my agent couldn't fit.
In any case, this is just an example, I'm sure the geniuses at Unity can come up with something that works.
TL;DR: Please come up with a way to have one navmesh surface, shared between more than one agent type, especially if those agent types share things like radius, height, etc.   Is this possible?

Couldn't the distance from edges be a separate value, and then the radius of the navmesh agent be part of the tunneling/distance from edge calculation?

It could, and i tried this approach a couple of years ago -  but there are several drawbacks:

Robustness is difficult in highly tessellated regions.
Performance is worse - because the agent can no longer be considered a point.
No longer possible to see directly from the debug graphics if an agent is able to pass through a corridor or not.

so we gave up on that approach

Ahh ok, glad to know it was at least considered :D I'm just having a hard time justifying creating three navmesh surfaces, one for 0.25 unit radius, one for 0.5 unit radius, and one for 1.0 unit radius.
It does seem to be very fast though, I guess it'll be a necessary evil, if I want to have a broad range of sizes.
Thank you!

Yes - it's the way to go for now. Be assured the runtime cost is lower this way (compared to dealing with over/under-sized agents on one navmesh).


Add isPRFile404

Does not follow https://github.com/sindresorhus/refined-github/pull/3314#discussion_r449908749 😄
isPRCommit404 exists. I need the file 404

test is timing out...

I don't know what's wrong with the tests, they're throwing here too: https://github.com/fregante/github-url-detection/pull/21

Try writing the tests again, copy the existing one and make small changes until it works/break. That's how I fixed https://github.com/fregante/github-url-detection/pull/21

Weird...
Ok will try

@fregante passed...

🚀 ?


Feat: calculate systemInfo everyday periodically and store them in datastore

calculate systemInfo everyday periodically and store them in datastore
Description of your changes

Fixes #
I have:

[ ] Read and followed KubeVela's contribution process.
[ ] Related Docs updated properly. In a new feature or configuration option, an update to the documentation is necessary.
[ ] Run make reviewable to ensure this PR is ready for review.
[ ] Added backport release-x.y labels to auto-backport this PR if necessary.

How has this code been tested

Special notes for your reviewer

@wangyikewxgm Please provide a remote server to save data.


[BUG]: Gemini API Error

How are you running AnythingLLM?
AnythingLLM desktop app
What happened?
Using Anything LLM 1.6.1 on OSX.  Setup Gemini API and getting the error: [GoogleGenerativeAI Error]: Error fetching from https://generativelanguage.googleapis.com/v1/models/gemini-1.5-pro-exp-0801:streamGenerateContent?alt=sse: [404 Not Found] models/gemini-1.5-pro-exp-0801 is not found for API version v1, or is not supported for generateContent. Call ListModels to see the list of available models and their supported methods.
The gemini-1.5-pro-latest version works as expected.
Are there known steps to reproduce?
No response

This is still failing in Anything LLM desktop 1.6.2


Bugfix/Do not bail if --clean is passed and no consultation exists

Context
The generate_themes command threw an exception when we passed --clean and there was no consultation to clean.
As this can't be a destructive action, swallow the exception and log a line that we're not cleaning anything, for information.
Changes proposed in this pull request
See commit
Link to JIRA ticket
N/A
Things to check

[X] I have added any new ENV vars in all deployed environments and updated the .env.example and .env.test files in the repo

Closing in favour of #301


Databox Installer

Installing the 0.1 release should be quick and easy.

SD card images? Should work for Pis and also for NUCs.

SD images would be nice if we have time.

I'm taking a look at converting my databox-vagrant repo into something more like an installer (as I could do with a working version to be developing against). Which branch do you think would currently be the best to work against which incorporates the most recent changes?

Hi Dom,
I would hang fire. The latest versions of the CM will launch a containerised app store and docker registry. It is working quite smoothly. The next step is to put the CM into a container and the install process will be as simple as install docker and pull this image ;-).
If you still want to play then the hypercat branch is probably the best.
Cheers
Tosh

Okay, I'll try to be more chill :)
One thing I thought of was that instead of modifying the hosts file, maybe use something like https://github.com/gliderlabs/resolvable which provides a (Dockerised) nameserver that dynamically creates entries for running containers?

It now runs in a container!! see https://github.com/me-box/databox-container-manager/pull/23


Fix memory leak problem when using slua-standalone dll

Fix memory leak problem when using slua-standalone dll

Had fixed


Refactor

This is a refactor of the entire codebase and is not directly backwards compatible.

Rewrite to TypeScript

Use classes rather than one file...

Update Rcon to use promises

Also update from the old Buffer to the newer version

Print the command to run on the server running the Minecraft server when starting the tool, with an option to disable

SHOW_INIT_MESSAGE: boolean

Breaking Changes

REGEX_MATCH_CHAT_MC is now split up into two parts, REGEX_MATCH_CHAT_MC and REGEX_SERVER_PREFIX.

Bug fixes / Updates

Implements code from #40
Support for @ mentioning (was here before, but putting for completeness) ( #9 )

Format username#id (ex: @destruc7i0n#7070)
ALLOW_USER_MENTIONS: boolean

Everything from #20 ( with code from #40 )

SHOW_PLAYER_CONN_STAT: boolean
SHOW_PLAYER_ADVANCEMENT: boolean
SHOW_PLAYER_DEATH: boolean
SHOW_PLAYER_ME: boolean

Support for slash commands, with roles dictating who can do them ( #30 )

ALLOW_SLASH_COMMANDS: boolean
SLASH_COMMAND_ROLES: string[] - list of the names of roles (case-sensitive) that can run commands

Finally fixes @ everyone and @ here mentions, with setting to enable them ( #34 )

ALLOW_HERE_EVERYONE_MENTIONS: boolean

Allow specifying the channel name rather than the id

DISCORD_CHANNEL_NAME: string

@destruc7i0n Hey man sorry I ghosted my pull request from so long ago. Life got a little crazy.  I hope my features gave you some ideas for how to make this a little cooler and hopefully i can contribute in the future.

@TheZackCodec All good! Gave me a reason to rewrite and improve this


PiHut Status HAT - PiSquare no longer connects

I've just plugged my first hat on top of the PiSquare having got a Pi and PS communicating. However, when I run the same PiSquare software example (pisquare_client.py) it no longer connects:
I2C(1, freq=200000, scl=7, sda=6)
Traceback (most recent call last):
File "", line 20, in
IndexError: list index out of range
(the line is print("I2C Address      : " + hex(i2c.scan()[0]).upper())  # Display device address)
I've not yet worked out why this is happening, but works perfectly when no HAT is attached.
Any thoughts?

@GeekyTim This issue occurs because your PiHut status HAT pins match the onboard OLED display pins, which is why the display works normally after removing the hat. If you wish to run this HAT, please do not utilize the onboard OLED display pins; instead, remark below lines and lines relating to OLED.-
i2c = I2C(1,freq=200000,sda=Pin(6),scl=Pin(7))
print(i2c)
print("I2C Address      : "+hex(i2c.scan()[0]).upper()) # Display device address
print("I2C Configuration: "+str(i2c))
oled = SSD1306_I2C(WIDTH, HEIGHT, i2c)                  # Init oled display
oled.fill(0)
oled.text("Client",30,0)
oled.show()


Mismatch in constructing an SFrame from file

Saving and then loading a single column Sframe (containing None), yields a different output depending on the save format
sf0 = tc.SFrame({'A':[1.0, float('NaN'), float('NaN'), None, None, None, 2.0]})
print('ref. SFrame:')
sf0.print_rows(10)

sf0.save('foo')
sf0.save('foo.csv', format='csv')

sf1 = tc.SFrame('foo')
sf2 = tc.SFrame('foo.csv')

print('load from binary image:')
sf1.print_rows(10)
print('load from csv:')
sf2.print_rows(10)

ref. SFrame:
+------+
|  A   |
+------+
| 1.0  |
| nan  |
| nan  |
| None |
| None |
| None |
| 2.0  |
+------+
[7 rows x 1 columns]

...

load from binary image:
+------+
|  A   |
+------+
| 1.0  |
| nan  |
| nan  |
| None |
| None |
| None |
| 2.0  |
+------+
[7 rows x 1 columns]

load from csv:
+-----+
|  A  |
+-----+
| 1.0 |
| nan |
| nan |
| 2.0 |
+-----+
[4 rows x 1 columns]

@guyinbar - thanks for the detailed issues.
Here is the result of cat ./foo.csv:
A
1
nan
nan

2

It looks like when we read a CSV, we're not interpreting blank lines as None.

I'm honestly not sure what the correct behavior should be here.  None is not a supported CSV value, so it must be inferred by other means.  it's not clear to me that blank lines should be parsed into None.


docs: update runners list on README.md

Add more runners to README and a link to all dependants of this awesome package.
Big thanks @NiklasPor!

Thanks for contributing! 👏👏


removeUnusedVertices Removing Used Vertices

Testing the gltfPipeline with more involved models than the CesiumBox, I identified removeUnusedVertices seems to be removing all primitives except the main primitive of the model (is my terminology correct here? Still getting a feel for glTF).
See attached images
gltfPipeline with removeUnusedVertices step

gltfPipeline without removeUnusedVertices

@JoshuaStorm If you could attach the model files in a zip, it'll be easier for me to take a look at this.

@lasalvavida
Yep, sorry about that. See attached.
removeUnusedIssue.zip

It looks like this is actually caused by combinePrimitives, still looking for a fix.

It is not, the root cause of this needs to be found and corrected. The stages should be able to be run in any order (with dependencies of course) with no issue.


New package: Finch v0.1.0

Registering package: Finch
Repository: https://github.com/willow-ahrens/Finch.jl
Created by: @willow-ahrens
Version: v0.1.0
Commit:<PHONE_NUMBER>ec1c2e29da299eec48cc0afc62f71a
Reviewed by: @willow-ahrens
Reference: https://github.com/willow-ahrens/Finch.jl/commit/7952269550ec1c2e29da299eec48cc0afc62f71a#commitcomment-83856047
Description: Sparse, Compressed, and Structured Tensor Processing

The package is already named Finch in my Arxiv manuscript, so I'd like to keep the name, if that's alright with everyone. I don't think there's a significant worry that people will confuse Finch and Fetch.


Decrease inbound queue test gas & update snapshot

Using the headProof.pos field in the mock parachain client to avoid comparing whole proof structs and copying them from storage to calldata.
Gas consumption is generally lower since the last time we committed a snapshot, with these exceptions:

InboundChannelTest:testSubmitShouldNotFailOnHandlerOOG (211741 -> 643392)
NativeTokensTest:testCreateSuccessful (38508 -> 75682)
NativeTokensTest:testHandleUnlockMessageSuccessful (125590 -> 151017)
NativeTokensTest:testLockRevertsZeroAmount (11261 -> 11366)
NativeTokensTest:testLockSuccessful (130192 -> 156773)
TokenVaultTest:testInsufficientBalance (17117 -> 17294)
TokenVaultTest:testNonOwnerCannotWithdraw (20672 -> 48935)
TokenVaultTest:testWithdrawSuccessful (124729 -> 124731)

Let's hold off on merging this, there will be a merge conflict with some other work I have in the pipeline

This depends on the Solmate proof change (the inbound queue accepting a proof struct in calldata instead of bytes), but the Solmate change has been reverted because there were missing changes in the relayer. I'll create a separate PR with the Solmate proofs, relayer changes and these changes, and add links in the description of each PR (original Solmate PR, this one & the new one).


Subtitle upload field missing

The "Create Event" dialog in the old ui (in OC 15) offers an option to upload subtitles alongside the presenter and presentation track. We should also offer that option.

We already have that, I just missed it.


safety check is deprecated, replace with scan

+===========================================================================================================================================================================================+

DEPRECATED: this command (`check`) has been DEPRECATED, and will be unsupported beyond 01 June 2024.

We highly encourage switching to the new `scan` command which is easier to use, more powerful, and can be set up to mimic the deprecated command if required.

+===========================================================================================================================================================================================+

added new safety scan GitHub action
using default safety policy, I tried a custom policy file but kept getting errors parsing this file, I think this may be due to bugs in the plugin itself.
62044 pip<23.3 and 70612 Jinga2>0 vulnerabilities are no longer failing or are ignored by the default policy.
removed safety check from ADO script
new action is passing, says 12 vulnerabilities are ignored due to default policy

Safety Action version: 1.0.0
Safety CLI version: safety, version 3.2.3
Safety 3.2.3 scanning /github/workspace
2024-12-11 12:15:07 UTC

Account: API key used
 Git branch: [8](https://github.com/cjkrolak/ThermostatSupervisor/actions/runs/12276140343/job/34252705770?pr=851#step:4:9)50_safety_check_action
 Environment: cicd
 Scan policy: None, using Safety CLI default policies

Python detected. Found 1 Python requirement file

✅ requirements.txt: No issues found.

Tested 20 dependencies for known security issues using default Safety CLI policies
0 security issues found, 0 fixes suggested
([12](https://github.com/cjkrolak/ThermostatSupervisor/actions/runs/12276140343/job/34252705770?pr=851#step:4:13) vulnerabilities were ignored due to project policy)


pythonPackages.tmdb3: disable on python3

Motivation for this change
Wanted to use this library, and noticed it didn't work on python3.
Not sure if we should keep the library around at all, as python2 has been deprecated for a while now. For now I've simply disabled it on python3.
Things done

[X] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[X] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[X] Fits CONTRIBUTING.md.

@FRidh I'm fine with removing it.

Result of nixpkgs-review pr 101951 1

1 package built:

python27Packages.tmdb3

Result of nixpkgs-review pr 101951 run on x86_64-linux 1

1 package built:

python27Packages.tmdb3


Repair Options actually Sabotage them

How to replicate: try to repair anything with the cheat menu
result: it will actually sabotage it

I can confirm this.


JSON Agent

This PR implements a fully-fledged JSON Agent deployment with HTTP and MQTT transports as #185 requested. Specifically, the following got implemented.

Manifests to run JSON Agent in a K8s cluster. The agent comes with an internal endpoint for provisioning and two external endpoints to get JSON device data in, one over HTTP, the other over MQTT---see #181 for our current MQTT setup.
Routing. We expose the HTTP endpoint at <kitt4sme-base-url>/jsonagent/ so devices can POST JSON data at <kitt4sme-base-url>/jsonagent/iot/json?k=<api-key>&i=<device-id>. To send JSON data over MQTT, devices use the MQTT WebSocket implemented in #181 and the topic: json/<api-key>/<device-id>/attrs.
Security. Istio handles TLS termination for the HTTP endpoint and delegates security decisions to the existing FIWARE OPA policy. Security over MQTT works as explained in #181.
In-memory configuration. The service pre-loads device mappings and other config in memory to speed up JSON to FIWARE data translation---see config.js.
Persistence. No Mongo DB backend. Data sits in memory (it isn't much actually) but it's loaded from K8s configmaps, so that's the persistence backend. (If the pod restarts, config map data gets loaded again in memory; all service pods share the same config map data.)
IaC management. The manifests of all agent-related resources are tied to an Argo CD app in the mesh infra project so we can easily manage deployments through a GUI too.
Simplified config model. All the agent config sits in our repo, including devices and mappings. Argo CD automatically deploys this data to the configmaps the service uses as a persistence backend---see point above. No need to fiddle with awkward service calls to figure out what devices have been defined, what their mappings are, or to add new devices. It's all defined in config.js  in our repo. Look at that file to figure out the lay of the land at a glance. Edit it to add or modify device defs, mappings etc. Argo CD takes care of propagating your changes to the cluster.

Demo
Here's a sum up of how to demo most of what this PR implemented.
First off, build your own KITT4SME cluster in a Multipass VM as explained in the bootstrap procedure. Wait a bit until all services are ready. The commands below use the <IP_ADDRESS> IP address to reach the Multipass VM; replace <IP_ADDRESS> in each command with your VM's IP.
IaC
Log into Argo CD. There's a json-agent app among the platform infra services. Check out all the resources deployed and have a look at the logs. Open the config map, you should be able to see the exact same content as in the config.js included in this PR.
Provisioning
Provision a service with two devices, one sending data over MQTT and the other over HTTP. Notice it's best to do that in config.js, but we'll do that manually here to speed things up. So as a rule, there would be no need to expose JSON Agent's provisioning API. Since we're bending the rules here, port-forward the agent's provisioning port.
$ kubectl port-forward svc/jsonagent 4041:4041

On to the provisioning business. Create a service with an API key of gr33t, entity type of Greeting and HTTP endpoint of /iot/d.
$ curl -iX POST 'http://localhost:4041/iot/services' \
  -H 'Content-Type: application/json' \
  -H 'fiware-service: greeting' \
  -H 'fiware-servicepath: /' \
  -d '{
 "services": [
   {
     "apikey":      "gr33t",
     "entity_type": "Greeting",
     "resource":    "/iot/json"
   }
 ]
}'

Create two devices to send a greeting message. The message is in the JSON format: { "w": data }, where data is a greeting string the device sends. The corresponding NGSI entity has type Greeting and a words attribute holding the actual greeting. The first device has an ID of greeter001 and sends its data over MQTT, whereas the second has ID greeter002 and sends data over HTTP.
$ curl -iX POST \
  'http://localhost:4041/iot/devices' \
  -H 'Content-Type: application/json' \
  -H 'fiware-service: greeting' \
  -H 'fiware-servicepath: /' \
  -d '{
 "devices": [
   {
     "device_id":   "greeter001",
     "entity_name": "urn:ngsi-ld:Greeting:001",
     "entity_type": "Greeting",
     "protocol":    "PDI-IoTA-JSON",
     "transport":   "MQTT",
     "attributes": [
       { "object_id": "w", "name": "words", "type": "Text" }
     ]
   }
 ]
}
'

$ curl -iX POST \
  'http://localhost:4041/iot/devices' \
  -H 'Content-Type: application/json' \
  -H 'fiware-service: greeting' \
  -H 'fiware-servicepath: /' \
  -d '{
 "devices": [
   {
     "device_id":   "greeter002",
     "entity_name": "urn:ngsi-ld:Greeting:002",
     "entity_type": "Greeting",
     "protocol":    "PDI-IoTA-JSON",
     "transport":   "HTTP",
     "attributes": [
       { "object_id": "w", "name": "words", "type": "Text" }
     ]
   }
 ]
}
'

With this setup, greeter001 is expected to send its UL payload to the MQTT topic

json/gr33t/greeter001/attrs

whereas greeter002 is supposed to POST its JSON payload to the URL

http://<IP_ADDRESS>/jsonagent/iot/d?k=gr33t&i=greeter002

since our Istio config routes /jsonagent/<rest> to /<rest> on port 7896 of the jsonagent service.
Finally, check you can retrieve the service and devices you've just created.
$ curl 'http://localhost:4041/iot/services' \
  -H 'fiware-service: greeting' \
  -H 'fiware-servicepath: /'

{"count":1,"services":[{"apikey":"gr33t","resource":"/iot/json","service":"greeting","subservice":"/","_id":1,"creationDate":1668087518563,"entity_type":"Greeting"}]}

$ curl 'http://localhost:4041/iot/devices' \
  -H 'fiware-service: greeting' \
  -H 'fiware-servicepath: /'

{"count":2,"devices":[{"device_id":"greeter001","service":"greeting","service_path":"/","entity_name":"urn:ngsi-ld:Greeting:001","entity_type":"Greeting","transport":"MQTT","attributes":[{"object_id":"w","name":"words","type":"Text"}],"commands":[],"static_attributes":[],"protocol":"PDI-IoTA-JSON","explicitAttrs":false},{"device_id":"greeter002","service":"greeting","service_path":"/","entity_name":"urn:ngsi-ld:Greeting:002","entity_type":"Greeting","polling":true,"transport":"HTTP","attributes":[{"object_id":"w","name":"words","type":"Text"}],"commands":[],"static_attributes":[],"protocol":"PDI-IoTA-JSON","explicitAttrs":false}]}

Sending device data over MQTT
We're going to use an external WebSocket client to simulate device data coming in over MQTT.
Browse to http://www.emqx.io/online-mqtt-client. Hit the "New Connection" button and enter the following data: name=kitt4sme, client-id=tasty, host=ws://<IP_ADDRESS>, path=/mqtt/, port=80, username=iot. You also need to enter the "iot" user's password, which I can't type here obviously. Hit connect, then send the following JSON message to the json/gr33t/greeter001/attrs topic: { "w": "howzit!" }.
Check the "howzit!" greeting trekked all the way to Orion. It should be stored in the "Greeting" entity having an ID of urn:ngsi-ld:Greeting:001.
$ curl \
  'http://<IP_ADDRESS>/orion/v2/entities/urn:ngsi-ld:Greeting:001/attrs/words/value' \
  -H 'fiware-service: greeting' \
  -H 'fiware-servicepath: /'

"howzit!"

Sending device data over HTTP
Let's also send a greeting from greeter002. This device sends its data over HTTP
$ curl -iX POST \
  'http://<IP_ADDRESS>/jsonagent/iot/json?k=gr33t&i=greeter002' \
  -H 'Content-Type: application/json' \
  -d '{ "w": "ahoy, matey!" }'

HTTP/1.1 403 Forbidden
date: Thu, 10 Nov 2022 14:00:52 GMT
server: istio-envoy
content-length: 0

What?! Yep, that's right. Our FIWARE OPA policy checks you've got a valid JWT token, since we didn't have one, we got shown the door. How rude.  }'

This time the POST goes through and Orion gets our friendly greeting
$ curl \
  'http://<IP_ADDRESS>/orion/v2/entities/urn:ngsi-ld:Greeting:002/attrs/words/value' \
  -H 'fiware-service: greeting' \
  -H 'fiware-servicepath: /'

"ahoy, matey!"

As expected, the greeting got stored in the "Greeting" entity having an ID of urn:ngsi-ld:Greeting:002.
Cool bananas!


Exec tachikoma update<PHONE_NUMBER>4709

:hamster::hamster::hamster:   Strategy none
Powered by Interval Pull Request as a Service - Tachikoma.io Announcement: 2015-06-28 Updated permissions - Tachikoma.io

Coverage remained the same at 67.647% when pulling 8ec7b892fca82268d70e4d01164f05f8cd387e36 on pazjacket:tachikoma/update-20180918214709 into ec251e61adc687ae0589e5283f1773ece07521c0 on sanemat:master.