Path mappings based module resolution

Proposed module resolution strategy
UPDATE: proposal below is updated based on the results of the design meeting.
Initial version can be found here.
Primary differences:

instead of having baseUrl as a separate module resolution strategy we are introducing a set of properties
that will allow to customize resoluton process in existing resolution strategies but base strategy still is used as a fallback.
rootDirs are decoupled from the baseUrl and can be used without it.

Currently TypeScript supports two ways of resolving module names: classic (module name always resolves to a file, module are searched using a folder walk)
and node (uses rules similar to node module loader, was introduced in TypeScript 1.6).
These approaches worked reasonably well but they were not able to model baseUrl based mechanics used by
RequireJS or SystemJS.
We could introduce third type of module resolution that will fill this gap but this will mean that once user has started to use this new type then support to
discover typings embedded in node modules (and distributed via npm) is lost. Effectively user that wanted both to use baseUrl to refer to modules defined inside the project
and rely on npm to obtain modules with typings will have to choose what part of the system will be broken.
Instead of doing this we'll allow to declare a set of properties that will augment existing module resolution strategies. These properties are:
baseUrl, paths and rootDirs (paths can only be used if baseUrl is set). If at least one of these properties is defined then compiler will try to
use it to resolve module name and if it fail - will fallback to a default behavior for a current resolution strategy.
Also choice of resolution strategy determines what does it mean to load a module from a given path. To be more concrete given some module name /a/b/c:

classic resolver will check for the presense of files /a/b/c.ts, /a/b/c.tsx and /a/b/c.d.ts.
node resolver will first try to load module as file by probing the same files as classic and then try to load module from directory
(check /a/b/c/index with supported extension, then peek into package.json etc. More details can be found in this issue)

Properties

baseUrl
path mappings
rootDirs

BaseUrl
All non-rooted paths are computed relative to baseUrl.
Value of baseUrl is determined as either:

value of baseUrl command line argument (if given path is relative it is computed based on current directory)
value of baseUrl propery in 'tsconfig.json' (if given path is relative it is computed based on then location of 'tsconfig.json')

Path mappings
Sometimes modules are not directly located under baseUrl. It is possible to control how locations are computed in such cases
using path mappings. Path mappings are specified using the following JSON structure:
{
    "paths": {
        "pattern-1": ["list of substitutions"],
        "pattern-2": ["list of substitutions"],
        ...
        "pattern-N": ["list of substitutions"]
    }
}

Patterns and substitutions are strings that can have zero or one asteriks ('*').
Interpretation of both patterns and substitutions will be described in Resolution process section.
Resolution process
Non-relative module names are resolved slightly differently comparing
to relative (start with "./" or "../") and rooted module names (start with "/", drive name or schema).
Resolution of non-relative module names (mostly matches SystemJS)

// mimics path mappings in SystemJS
// NOTE: moduleExists checks if file with any supported extension exists on disk
function resolveNonRelativeModuleName(moduleName: string): string {
    // check if module name should be used as-is or it should be mapped to different value
    let longestMatchedPrefixLength = 0;
    let matchedPattern: string;
    let matchedWildcard: string;

    for (let pattern in config.paths) {
        assert(pattern.countOf('*') <= 1);
        let indexOfWildcard = pattern.indexOf('*');
        if (indexOfWildcard !== -1) {
            // if pattern contains asterisk then asterisk acts as a capture group with a greedy matching
            // i.e. for the string 'abbb' pattern 'a*b' will get 'bb' as '*'

            // check if module name starts with prefix, ends with suffix and these two don't overlap
            let prefix = pattern.substr(0, indexOfWildcard);
            let suffix = pattern.substr(indexOfWildcard + 1);
            if (moduleName.length >= prefix.length + suffix.length &&
                moduleName.startsWith(prefix) &&
                moduleName.endsWith(suffix)) {

                // use length of matched prefix as betterness criteria
                if (longestMatchedPrefixLength < prefix.length) {
                    // save length of the prefix
                    longestMatchedPrefixLength = prefix.length;
                    // save matched pattern
                    matchedPattern = pattern;
                    // save matched wildcard content
                    matchedWildcard = moduleName.substr(prefix.length, moduleName.length - suffix.length);
                }
            }
        }
        else {
            // pattern does not contain asterisk - module name should exactly match pattern to succeed
            if (pattern === moduleName) {
                // save pattern
                matchedPattern = pattern;
                // drop saved wildcard match
                matchedWildcard = undefined;
                // exact match is found - can exit early
                break;
            }
        }
    }

    if (!matchedPattern) {
        // no pattern was matched so module name can be used as-is
        let path = combine(baseUrl, moduleName);
        return moduleExists(path) ? path : undefined;
    }

    // some pattern was matched - module name needs to be substituted
    let substitutions = config.paths[matchedPattern].asArray();
    for (let subst of substitutions) {
        assert(substs.countOf('*') <= 1);
        // replace * in substitution with matched wildcard
        let path = matchedWildcard ? subst.replace("*", matchedWildcard) : subst;
        // if substituion is a relative path - combine it with baseUrl
        path = isRelative(path) ? combine(baseUrl, path) : path;
        if (moduleExists(path)) {
            return path;
        }
    }

    return undefined;
}

Resolution of relative module names
Default resolution logic (matches SystemJS)
Relative module names are computed treating location of source file that contains the import as base folder.
Path mappings are not applied.
function resolveRelativeModuleName(moduleName: string, containingFile: string): string {
    let path = combine(getDirectoryName(containingFile), moduleName);
    return moduleExists(path) ? path : undefined;
}

Using rootDirs
'rootDirs' allows the project to be spreaded across multiple locations and resolve modules with relative names as if multiple project roots were merged together in one folder. For example project contains source files that are located in different directories on then file system (not under the same root) but user still still prefers to use relative module names because in runtime such names can be successfully resolved due to bundling.
For example consider this project structure:
 shared
 └── projects
     └── project
         └── src
             ├── viewManager.ts (imports './views/view1')
             └── views
                 └── view2.ts (imports './view1')
 userFiles
 └── project
     └── src
         └── views
             └── view1.ts (imports './view2')

Logically files in userFiles/project and shared/projects/project belong to the same project and
after build they indeed will be bundled together.
In order to support this we'll add configuration property "rootDirs":
{
    "rootDirs": [
        "rootDir-1/",
        "rootDir-2/",
        ...
        "rootDir-n/"
    ]
}

This property stores list of base folders, every folder name can be either absolute or relative.
Elements in rootDirs that represent non-absolute paths will be converted to absolute using location of tsconfig.json as a base folder - this is the common approach for all paths defined in tsconfig.json
///Algorithm for resolving relative module name
function resolveRelativeModuleName(moduleName: string, containingFile: string): string {
    // convert relative module name to absolute using location of containing file
    // this step is exactly the same as when doing resolution without path mapping
    let path = combine(getDirectoryName(containingFile), moduleName);

    // convert absolute module name to non-relative
    // try to find element in 'rootDirs' that is the longest prefix for "path' and return path.substr(prefix.length) as non-relative name
    let { matchingRootDir, nonRelativeName } = tryFindLongestPrefixAndReturnSuffix(rootDirs, path);
    if (!matchingRootDir) {
        // cannot extract non relative name
        return undefined;
    }
    // first try to load module from initial location
    if (moduleExists(path)) {
        return path;
    }
    // then try other entries in rootDirs
    for (const rootDir of rootDirs) {
        if (rootDir === matchingRootDir) {
            continue;
        }
        const candidate = combine(rootDir, nonRelativeName);
        if (moduleExists(candidate)) {
            return candidate;
        }
    }
    // failure case
    return undefined;
}

Configuration for the example above:
{
    "rootDirs": [
        "userFiles/project/",
        "/shared/projects/project/"
    ]
}

Example 1
projectRoot
├── folder1
│   └── file1.ts (imports 'folder2/file2')
├── folder2
│   ├── file2.ts (imports './file3')
│   └── file3.ts
└── tsconfig.json

// configuration in tsconfig.json
{
    "baseUrl": "."
}

import 'folder2/file2'

baseUrl is specified in configuration and has value '.' -> baseUrl is computed relative to
the location of tsconfig.json -> projectRoot
path mappings are not available -> path = moduleName
resolved module file name = combine(baseUrl, path) -> projectRoot/folder2/file2.ts

import './file3'

moduleName is relative and rootDirs are not specified in configuration - compute module name
relative to the location of containing file: resolved module file name = projectRoot/folder2/file3.ts

Example 2
projectRoot
├── folder1
│   ├── file1.ts (imports 'folder1/file2' and 'folder2/file3')
│   └── file2.ts
├── generated
│   ├── folder1
│   └── folder2
│       └── file3.ts
└── tsconfig.json

// configuration in tsconfig.json
{
    "baseUrl": ".",
    "paths": {
    "*": [
            "*",
            "generated/*"
        ]
    }
}

import 'folder1/file2'

baseUrl is specified in configuration and has value '.' -> baseUrl is computed relative to
the location of tsconfig.json -> projectRoot
configuration contains path mappings.
pattern '*' is matched and wildcard captures the whole module name
try first substitution in the list: '*' -> folder1/file2
result of substitution is relative name - combine it with baseUrl -> projectRoot/folder1/file2.ts.
This file exists.

import 'folder2/file2'

baseUrl is specified in configuration and has value '.' -> baseUrl is computed relative to
the location of tsconfig.json and will be folder that contains tsconfig.json
configuration contains path mappings.
pattern '*' is matched and wildcard captures the whole module name
try first substitution in the list: '*' -> folder2/file3
result of substitution is relative name - combine it with baseUrl -> projectRoot/folder2/file3.ts.
File does not exists, move to the second substitution
second substitution 'generated/*' -> generated/folder2/file3
result of substitution is relative name - combine it with baseUrl -> projectRoot/generated/folder2/file3.ts.
File exists

Example 3
rootDir
├── folder1
│   └── file1.ts (imports './file2')
├── generated
│   ├── folder1
│   │   ├── file2.ts
│   │   └── file3.ts (imports '../folder1/file1')
│   └── folder2
└── tsconfig.json

// configuration in tsconfig.json
{
    "rootDirs": [
        "./",
        "./generated/"
    ],
}

All non-rooted entries in rootDirs are expanded using location of tsconfig.json as base location so after expansion rootDirs will
look like this:
    "rootDirs": [
        "rootDir/",
        "rootDir/generated/"
    ],

import './file2'

name is relative, first make it absolute using location of containing file as base location - rootDir/folder1/file2
for this string find the longest prefix in rootDirs - rootDir/ and for this prefix compute as suffix - folder1/file2
since matching entry in rootDirs was found try to resolve module using rootDir - first check if rootDir/folder1/file2
can be resolved as module - such module does not exist
try remaining entries in rootDirs - check if module rootDir/generated/folder1/file2 exists - yes.

import '../folder1/file1'

name is relative, first make it absolute using location of containing file as base location - rootDir/generated/folder1/file1
for this string find the longest prefix in rootDirs - rootDir/generated and for this prefix compute as suffix - folder1/file1
since matching entry in rootDirs was found try to resolve module using rootDir - first check if rootDir/generated/folder1/file1
can be resolved as module - such module does not exist
try remaining entries in rootDirs - check if module rootDir/folder1/file1 exists - yes.

The proposal seems sound :+1:
I think the references to data should not be considered for now: their added value is marginal but the added configuration complexity would not be (eg: either refs are mandatory or there would have to be a good way to distinguish them from directory names). They seem like a nice improvement to consider after this is implemented and usage data about the mappings becomes available.

The biggest concern with this is that it sounds like a lot of complexity.  I can understand the desire to align to SystemJS in hopes that it aligns to whatwg and eventually the loaders that are implemented in the runtime environments.  Maybe it is the me being "myopic" but having lived with AMD for nearly a decade now, there are few use cases where my loader configuration is overly complex.  For the most part "it just works".  Up to this point with TypeScript, largely as well "it just worked", now I fear that I will have to spend a day trying to get TypeScript to understand where my modules are.
To be more specific, relative module resolution... Is there not a 95% happy path for relative module resolution?  The AMD spec specifies:

Relative identifiers are resolved relative to the identifier of the module in which "require" is written and called.

That is about as straight forward of a resolution logic as you can get.
And when that use case does not work, there is baseUrl, path, packages and the sledge hammer map.  Each of those are simple and straight forward, without a significant amount of options.  All the tools are there and the thing is 95% of the time, there is minimal configuration to get going, often with a single built layer for production, no configuration.
I am likely tilting at windmills, but sometimes it does seem like we are creating 32 varieties of Colgate here...

we have a request for this specific scenario, but I do agree that in majority of cases it is not necessary. That  is why it should be opt-in thing: if rootDirs are not specified then all relative module names are resolved using location of containing module which is the behavior most of people would expect. I've tried to emphasize the intent by increasing complexity in examples :

simplest scenario - possible - baseUrl is enough (and sometimes it can be inferred)
more complicated case that needs complicated path manipulations (but no specific behavior for relative names ) - possible - path mappings and baseUrl should be enough
specific case than requires applying path mappings to relative names as well - possible - use path mappings, baseUrl and rootDirs.

I guess I am just missing how the last two points can't be addressed by a simple paths that only takes a path, either or absolute to the baseUrl, and then some recursion that keeps relative MIDs relative to the importer.  Even with the other scenarios, I suspect you would also still need some sort of sledgehammer map.  The specific use case I run into now that makes this challenging (and therefore map would work) is when I am using MIDs that contain AMD plugins.  Right now I have to create an ambient module that imports and then exports what I am trying to do.

@kitsonk I have the use case @vladima is talking about. For me, just a set of --includeDirs would be sufficient (but I do need the canonicalization-of-relative-paths!). Once you add that, you might as well try and be compatible with what SystemJS does, I don't think it substantially increases complexity over that.

@mprobst Could you provide a concrete example of this need and how you would solve it with SystemJS? I'm having a hard time coming up with a real-world example of merging two code trees together that couldn't be solved by creating a package for one tree and importing it in the other.

@bryanforbes I think @vladima's example is good. Imagine you have something that generates TypeScript source. E.g. Angular 2 will have a build tool that transforms templates to TypeScript code that you can directly call in your app, for various reasons (mostly app startup time).
So you have your Angular controllers together with your templates, src/some/widget/foo_controller.ts and src/some/widget/foo_template.html. But you don't want to generate files in your regular source folder, as that creates a mess with version control, so you follow best practice and have a src folder and a build folder. The Angular template compiler generates build/some/widget/foo_template.ts. In your foo_controller.ts, you import FooTemplate from './foo_template';.
This works if you pass as rootDirs src and build, as ./foo_template would get canonicalized to some/widget/foo_template, then looked up in src and build in order, and found in build/some/widget/foo_template.ts. In SystemJS, I believe you would have a mapping of src/* and build/*.

@mprobst Thanks for clarifying! I didn't understand the generated directory containing TypeScript files, but your explanation helps. What still confuses me is why merging two trees together necessitates another configuration flag with potentially duplicated settings (rootDirs) instead of just using path mapping to solve everything.

My concern about using path mapping for both canonicalization of relative module names and remapping non-relative module names is that this configuration settings become overloaded:

it becomes more difficult for the end-user in understanding of the concept since it is no longer just path mapping
it may lead interesting bugs when some value of path mapping  that was not intended to be used for canonicalization will be picked for this purpose.

To deal with duplication I'd rather prefer to have something for data sharing (i.e. references) instead of re-purposing field whose meaning is a already well-defined

:+1:

Can this go in 1.7? Pretty please with a cherry on top? And is there some way I can play with this now?

@vladima Sorry for the delay in replying. I think my concern now is that you're giving new names to already defined concepts. As I see it (and aside from the obvious difference that these property names take arrays), paths is basically SystemJS's map, and rootDirs is basically like SystemJS's paths. Why not just use those names instead? The ideas are similar to what we already know from AMD and SystemJS, so why invent new names?

@bryanforbes I'm not sure how well these map with SystemJS, but for what it's worth, rootDirs is a much more specific and understandable name than just paths. I'd take rootDirs over paths any time.

It looks like a lot of projects in the wild have already been using babel for ES6 in combination with webpack as a pretty standard configuration. It might be worth looking at how the module path resolution works in webpack.
There is no need to introduce other concepts/naming, it could be taken from https://webpack.github.io/docs/resolving.html and its configuration https://webpack.github.io/docs/configuration.html#resolve.

@opichals I might be missing something, but by my reading webpack's resolve does not meet the requirements above for resolving files relative to a set of directories.

@mprobst The resolve.root configuration variable makes it possible for a module to be searched for in folders similarly to the rootDirs from the proposal.
The webpack's resolve.root can be an array of absolute paths (could not directly confirm this just by reading the docs, so I checked the sources.

@opichals yes, but relative imports are not canonicalized against the list of roots, are they? I read the docs as saying if I load ./foo from a file physically located at /bar/baz, I'll always end up at /bar/foo instead of searching my rootDirs.

@mprobst True. I would find that extremely confusing if any loader attempted to resolve relative path against anything else than just the folder it is physically located at (something require.js tries to do and became a nuisance to configure because of). As stated above such resolution logic seems to add complexity which reflects in complicated use and debugging.

I think we collected a couple of compelling use cases above.
Standa Opichal<EMAIL_ADDRESS>schrieb am Mi., 21. Okt. 2015,
16:15:

@mprobst https://github.com/mprobst True. I would find that extremely
confusing if any loader attempted to resolve relative path against anything
else than just the folder it is physically located at (something require.js
tries to do and became a nuisance to configure because of). As stated above
such resolution logic seems to add complexity which reflects in complicated
use and debugging.
—
Reply to this email directly or view it on GitHub
https://github.com/Microsoft/TypeScript/issues/5039#issuecomment-149928200
.

I would find that extremely confusing if any loader attempted to resolve relative path against anything else than just the folder it is physically located at (something require.js tries to do and became a nuisance to configure because of).

When does require.js do this?

You can achieve this with RequireJS because it applies the path mappings to any path segment of any require no matter whether the require module path is absolute or not. e.g. for { path: { 'pkg': '../folder1' } }:

require('pkg/file1') resolves to ../folder1/file1,
require('./local/folder2/pkg/folder1') resolves to ./local/folder2/../folder1/file1.

Also it when two require() calls resolve to a single file using different arguments the module gets loaded twice.
For the proposal I would rather like to see an API-based extensibility approach to support the likes of 'Example 3' (let the user replace the resolution function somehow through a configuration or a plugin).

A benefit of been able to resolve the import without requiring a path configuration, is that it will allow  you to import the .ts files of a third party library completely written in ts,  instead of the usual  master.js + .d.ts that has to be imported, very useful for widgets libraries like angular material, where you are only interested in a few files not the entire lib, not sure if you ever though on supporting that kind of scenario

All files being .ts,
import {foo} from '../node_modules/my-lib/module'  // OK
import {foo} from 'my-lib/module'                  // FAILS (Cannot find module)

…while for .js + .d.ts both imports work.
Will the second import with the proposal?

@gabrielguerrero the current behavior will not change. If you do specify path mappings it will be used, if you do not the current resolution logic will be used. i think this should work in your case.

Hey @mhegazy yeah the path/map will cover my case so cool if it gets implemented, I was just thinking if you wanted a no config required way, the resolve of parent dir in the import could provably work for most of the cases people need, so just giving ideas

Man, I'm feeling great about moving this feature to 1.7...
Someone tell Anders I said it was a go. :)

+1 million! The ability to do this will greatly improve support for different builds, assets organization, etc. For example with JSPM now I'm forced to duplicated my dependencies using npm just to get tsc happy.

:+1: I'm currently manually copying d.ts files to a faux node_modules folder in order to get tsc to compile. This looks like it would clean up my use case!

I haven't heard a yes or no to my question of whether or not this can be moved from 1.8 to 1.7. If there was a beta of 1.7 with this in it, I'd test it today!

1.7 is in stabilization mode right now, so I'm pretty sure the answer is no.

this is currently work in progress. I'll post on this thread once I have something to try

Just to confirm - would I be correct in thinking that the resolution of .d.ts files are within the scope of this issue?
I've got a scenario where Project A depends on the build output of Project B. I want to bundle the auto-generated d.ts files from Project B alongside the artefact for use in Project A.
In Project A I have: import {X} from 'projectb', where 'projectb' is the name of my exported module (paths set up in RequireJS). However, I've completely struggled to get Project A's compile step to understand any type information relating to X (the types of any exports from Project B).

Another use case for path mappings involves working with (git) submodules. We do this on large projects to separate repositories (work domains). But is has a undesirable side effect when the source of a module exists multiple times in a project due to the use of submodules (the TypeScript compiler treats those duplicate sources as separate modules which is technically correct). Assume the following simplified example:
Project git
|-- A
|   |-- a.ts
|   |-- C (git submodule)
|       |-- c.ts
|-- B
    |-- b.ts
    |-- C (git submodule)
        |-- c.ts

Submodule C in a separate git
|-- c.ts

Let's assume the submodule C contains a class which is used in module A and module B. Module C is a stand alone module (has it's own repository) which is embedded in module A and B using a git submodule (so we can maintain C in a single place and easily update modules who depend on it using a git submodule update). This results in duplicate sources for module C in the project repository.
Now let's say class A extends C:
// a.ts
import { C } from "./C/c.ts"; // Import C which is a submodule rep under A

class A extends C {
}

And b.ts contains a function which accepts any class which is derived from class C:
// b.ts
import { C } from "./C/c.ts"; // Import C which is a submodule rep under B
import { A } from "../A/a.ts";

function DoSomethingNice(c: C): void {
}

DoSomethingNice(new A()); // Oh noes, error!!!

The above example generates a compiler error. Class A is not derived from the same class C as used in module B. TypeScript returns something like Types have separate declarations bla bla. This makes sense, since the compiler doesn't know that the submodule C in module A (implemented in ./A/C/c.ts) is exactly the same as C in module B (implemented in ./B/C/c.ts). It's a duplicate source.
Now the world would be perfect when the TypeScript compiler recognizes duplicate/identical implementations (by generating some sort of hash for each module), knowing which files are identical and implement the same module (please develop TypeScript team :wink:). That would really pave the road for working with projects spanned over multiple repositories and using git submodules to chain it all together (sometimes resulting in duplicate sources for the same module used multiple times in a project)! It would even strengthen the use of git modules, since the compiler would fire errors when different versions (branches) of a submodule are used within the same project.
Though with path mappings it would be possible to achieve sort of the same setup. Mapping duplicate module sources to a single location in the project.
Looking forward to TypeScript 1.8!

What module targets does this proposal affect? Specifically, will this bring baseUrl support to the commonjs and es6 module targets?

currently module resolution strategy is decoupled from module targets so you can use whatever combination you like by specifying moduleResolution option. If this option is omitted then compiler will try to use strategy it thinks will be the most appropriate: i.e. use node if target is commonjs

Another work of art for Babel - https://github.com/tleunen/babel-plugin-module-alias

This is pretty essential to developing any real application with Angular 2.
Looks like using: "moduleResolution": "node", in tsconfig.json is required for Angular 2.
So the only way to include custom components is to use relative paths... yuck.
This is a nightmare when you start to scale and move around.
At least allow "moduleResolution": "node" to also resolve based on an absolute path from the root dir.

+1 can't wait :)

+1 want to use jspm with tsc

+1 want to use jspm with tsc

+1
Same here. JSPM rocks, but TypeScript integration can be a lot better with this feature.

+1

Somehow I got it in my head that this was slated for 1.8, which clearly didn't happen.
I realize that the in the grand scheme, there's a "real" path mapping strategy that needs to be developed, and that it somehow has to work with JSPM, etc.
But (a) is there any scenario in which /someModule would not be interpreted as referring to someModule.ts at the project root? and (b) if /someModule is to refer to the project root, could that portion of the problem be solved on the fast track while the bigger mapping strategy and technology are still being developed?
This would at least enable requirejs/ES6-friendly path references and would allow developers to avoid creating full blown node modules just to be able to reference commonly used modules from a known location without having to go relative (in other words, '../../../someModule' is a real pain in the butt).

+1 for @laurelnaiad's suggestion.  This would be a quick win for that part of the problems discussed above.

Now that this is merged (🎉) how would one set the path mappings so that the compiler picks up the modules that were installed via jspm?

Now that this is merged

@frederikschubert Is it? I don't think this is merged :rose:

@basarat looks like #5728 is indeed merged

Sweet :candy: Can this issue be closed :rose:

https://github.com/Microsoft/TypeScript/pull/5728 is merged. should be in typescript@next later tonight. please try it out and let us know if there are any issues.

How do you install vnext in Visual Studio?

@alexdresko see- http://www.screencast.com/t/ogvAf2sQ

@amcdnl That's for VSCode, which is helpful certainly, but I'm mainly interested in installing vNext into Visual Studio 2015. I apologize for not being more specific.

@mhegazy By adding this to the 2.0 milestone, does that mean it won't be going into 1.8?

@alexdresko no, this will be in 1.8 ... see the roadmap page

I have the same question as @frederikschubert, how can we use this awesome new functionality to make tsc aware of jspm modules?

I'm testing this out and got Example 1 above working (only using baseUrl). But having trouble with Example 2. Probably I am doing something wrong but cannot see what. Here is what I have:
$ tree
.
├── folder1
│   ├── file1.ts
│   └── file2.ts
├── generated
│   └── folder2
│       └── file3.ts
└── tsconfig.json

3 directories, 4 files

$ cat tsconfig.json
{
    "compilerOptions": {
        "module": "system",
        "sourceMap": true,
        "outDir": "js_out",
        "noEmitOnError": true
    },
    "files": [
        "folder1/file1.ts"
    ],
    "baseUrl": ".",
    "paths": {
        "*": ["*", "generated/*"]
    }
}

$ cat folder1/file1.ts
import {hello} from 'folder1/file2';
import {world} from 'folder2/file3';
console.log(hello + world);

$ cat folder1/file2.ts
export const hello = "Hello";

$ cat generated/folder2/file3.ts
export const world = " World!";

$ tsc --version
Version 1.9.0-dev.20160128

$ tsc
folder1/file1.ts(2,21): error TS2307: Cannot find module 'folder2/file3'.
folder1/file1.ts(2,21): error TS2307: Cannot find module 'folder2/file3'.

@mhegazy By adding this to the 2.0 milestone, does that mean it won't be going into 1.8?

that is correct. We branched for 1.8 already, and we just need to stabilize it. path mapping, and readonly property support are now available in typescript@next, and in TS 2.0 but not in 1.8.

@alexdresko no, this will be in 1.8 ... see the roadmap page

sorry about that. road map should be updated now.

@janakerman, baseURL and Paths are part of the compilerOptions, your tsconfig.json file should look like:
{
    "compilerOptions": {
        "module": "system",
        "sourceMap": true,
        "outDir": "js_out",
        "noEmitOnError": true,
        "baseUrl": ".",
                "paths": {
                        "*": ["*", "generated/*"]
                }
    },
    "files": [
        "folder1/file1.ts"
    ]
}

@mhegazy Thanks, I know I was missing something obvious :-). Now it works!

@mhegazy if it is done, why did it get pushed to 2.0? :(

@mhegazy if it is done, why did it get pushed to 2.0? :(

Sorry about that, but we need to stabilize the release to ship it, and adding more code, adds more bugs, and delays the whole process.
It should be in the nightly today, npm install typescript@next and now on NuGet (https://www.myget.org/gallery/typescript-preview)

Also, is there any versioned docs? like that show these nightly updates?

i am not sure i understand the question

@alexdresko I had commented on the PR about support for proj files in vs and seems like it isn't supported.
https://github.com/Microsoft/TypeScript/pull/5728#issuecomment-173732744

Regarding specifying path mappings in .csproj - this won't work, paths and rootDirs will be options that can be specified only via tsconfig.json

@prabirshrestha I want tsconfig file support. I was referring to tooling and compiler support for this new stuff.
@mhegazy So vnext is 2.0? Does it also contain everything in 1.8 already?

So vnext is 2.0? Does it also contain everything in 1.8 already?

The plan is the next release is labeled 2.0. we try to keep them around 8 weeks apart. typescript@next has all 1.8 features/bug fixes + any post 1.8 work.

From what I understand the nuget packages provide you with support for Visual Studio builds (and msbuild) but not for intellisense as that would require changes in the compiler host of the VS TypeScript plugin.
Also, from what I understand the updates to the compiler host will only be available by the time 2.0 is released, so until then Visual Studio builds can be used with this new feature but you'll have to resort to some other strategy to get intellisense working.

Also, from what I understand the updates to the compiler host will only be available by the time 2.0 is released, so until then Visual Studio builds can be used with this new feature but you'll have to resort to some other strategy to get intellisense working.

Most changes do not require you to install a new version of the VS TypeScript plugin in. The nightly build currently does not include the full plugin setup, we are working on getting an installer published nightly along with the npm package
to use the nighlies in VS, please chek out the steps in https://github.com/Microsoft/TypeScript/wiki/Nightly-drops#visual-studio-2013-and-2015

So is support for JSPM available now in 1.8 nightly builds? or do we need to wait for 2.0?
JSPM is awesome and would love to have support for it with TS... tx for doing this...
regards
Sean

@born2net this change does not support JSPM directly, you can add a path mapping for every package you use.
see https://github.com/Microsoft/TypeScript/issues/6012 for JSPM support.

@mhegazy If I want to use the new 'path' and 'baseUrl' mapping functionality with Visual Studio 2015 now, can I install the nightly build and expect that to work with Visual Studio, or are there Visual Studio components that would also need to be updated (that aren't available yet)?  I currently have 1.8 installed and working with VS 2015 (after successfully working through some install issues documented in #6958).  Is the new path resolution logic solely the responsibility of tsc, or would intellisense need updates for that as well?

Using the 1.9.x nightly build, I'm unable to get the new path mappings based module resolution to work with the command-line compiler (tsc). Should it be working? Obviously, this is a bleeding edge feature, but it would be a huge simplifier for our development process, so we're trying to start using it now (since it already works for us with SystemJS for runtime compilation).

@mko works for me using<EMAIL_ADDRESS>

@mpseidel Can you post an example usage or take a glance at my usage below? It might be in my configuration.
File System:
client/
-- app/
---- components/
------ app.component.ts
---- jspm_packages/
------ npm/
--------<EMAIL_ADDRESS>---------- core.d.ts
---------- core.js
-- tsconfig.json

Compiler Config in tsconfig.json
{
    "compilerOptions": {
        "target": "ES6",
        "module": "commonjs",
        "sourceMap": true,
        "emitDecoratorMetadata": true,
        "experimentalDecorators": true,
        "removeComments": false,
        "noImplicitAny": false,
        "moduleResolution": "baseUrl",
        "baseUrl": "",
        "paths": {
            "angular2/*"<EMAIL_ADDRESS>        }
    }
}

Module Resolution in app.component.ts
import { Component } from 'angular2/core';

Error Messages
Upon running tsc on any files in our project, we get the following error:
error TS6063: Argument for '--moduleResolution' option must be 'node' or 'classic'.

If we run the TypeScript compiler via grunt-typescript or atom-typescript, we get the following error:
error TS2307: Cannot find module 'angular2/core'.

@mko
As far as I understand baseUrl is not a valid option value for moduleResolution. I use the baseUrl option in combination with moduleResolution:node
This is my tsconfig.json. I use it for both client side (webpack) and for server side node code.
{   "compilerOptions": {     "sourceMap": true,     "target": "es6",     "module": "commonjs",     "jsx": "react",     "experimentalDecorators": true,     "baseUrl": ".",     "paths": {       "*": [         "*",         "app/*",         "app/client/*"       ]     }   } }

@mko try to omit the moduleResolution option or specify "classic" or "node"

@mko also take a look at tryLoadModuleUsingOptionalResolutionSettings in typescript.js. It is being called both from classicNameResolver and nodeModuleNameResolver

Interesting. The way I read it in the original PR #5728 it looked like baseUrl was meant to be a moduleResolution value. I guess I got my signals crossed there.
With any of the three options (moduleResolution: "node", moduleResolution: "classic", and without any moduleResolution value, I continue to get error TS2307: Cannot find module 'angular2/core'..
Your paths value uses an interesting combination of wildcards that I saw in another one of the threads that said they had it working. We really need this to work for importing our jspm_packages since their folder names include versions (which is really odd and makes referencing those packages in imports a pain if not using SystemJS).

Okay. So, I was able to get it to work with moduleResolution: "classic" by changing baseUrl to . and restructuring to move our jspm_packages under that base URL instead of using the ../ previous directory prefix.
I'll be really glad when this feature is fully documented. ;)

Have you tried baseUrl: "." And ... : "./jspm_packages..."?

Yep. That's what I changed it to to get it to work.

I'm trying to get this feature to work in Visual Studio 2015.
I've followed the instructions in https://github.com/Microsoft/TypeScript/wiki/Nightly-drops#visual-studio-2013-and-2015 and have Version 1.9.0-dev.20160211 configured for Visual Studio. All of the syntax errors went away so it seems to be working when I save, but when I do a build visual studio gives the error Build: Unknown compiler option 'baseUrl' and the same for 'paths'.
Regardless of save or build, none of my .js files ever get created when working in visual studio but they work just fine when I just run tsc in the directory at the command line.
Any ideas?

Unfortunately this is not 'script-side-only' change and it needs updates for the managed side (that were not yet officially released) to work correctly.

Thanks for the clarification. Is there a timeline on when it will be released and work correctly?

Is there any way to make this work now in VS (hacks, code, pre-release, etc)?

@vladima did this make it to 1.8.0? I cannot use next as tslint has a peer dep of 1.7.x and npm doesn't install either versions.

if you use typescript@next, you should use tslint@next as well.

@mhegazy yep, figured that out after one of the maintainers pointed it out :) thanks

+1

What's the reason for not using option rootDirs for resolving non-relative module names?

To explain my comment above:
I've a project which is spreaded across multiple locations. Instead of using relative names for resolving modules I prefer to use also non-relative module names:
 shared
 └── projects
     └── project
         └── src
             ├── commons
                 └──defs.ts
             ├── viewManager.ts (imports 'views/users/view1', imports 'commons/defs')
             └── views
                 ├── users
                     ├── view2.ts (imports './view1', imports 'commons/defs')
                     └── view3.ts (imports 'commons/defs')
                 └── customers
 userFiles
 └── project
     └── src
         └── views
                 ├── users
                     └── view1.ts (imports './view3', imports 'commons/defs')

apologies, did not notice your question. Yes, your understanding is correct, baseUrl / paths are used to resolve non-relative module names and rootDirs are applied only for relative names

Well, that works fine, but looks at bit done twice:
  "compilerOptions": {
    "baseUrl": "src/main",
    "paths": {
      "app/*": [
        "../branding/lh/app/*",
        "../dev/app/*"
      ]
    },
    "rootDirs": [
      "src/main/",
      "src/branding/lh/",
      "src/dev/"
    ],
    "outDir": "build/ts2js",
    ...

this is great, it's what we need for jspm_modules (instead of node_modules)...

Can this implementation rewrite rooted paths to relative paths for a CommonJS target? E.g.
├── folder1
│   ├── file1.ts
├── folder2
│   └── file2.ts
└── tsconfig.json

with moduleResolution node and baseUrl "." and module commonjs, and this TypeScript:
file1.ts
import two from "folder2/file2";

Can it emit this JS such that Node can actually execute the require?
file1.js
var two = require("../folder2/file2");

@jwbay this feature, along with the rest of the module resolution capabilities, are only to help the compiler find the module source given a module name. no changes to the output js code. if you require "folder2/file1" it will always be emitted this way. you might get errors if the compiler could not find a folder2/file1.ts, but no change to the output.
You can find more documentation at https://github.com/Microsoft/TypeScript-Handbook/blob/release-2.0/pages/Module Resolution.md#additional-module-resolution-flags

This might be silly, but you have considered simply providing a hook into module resolution logic where users of the Typescript compiler are simply able to provide an async resolution function that returns the actual path on the filesystem? Instead of trying to find a "one size fits all" solution, this would allow the community to figure out ways to make it work with different packaging systems, file system layouts, etc.

Typescript statically checked. Async is pretty far from static, so this sounds like it would be a stretch.

@guncha It has been considered but the anwer was it was too much work. See here.

anyone knows if path mapping via tsconfig already landed in ts@next as does not seem to work,
tx
Sean

Path mapping support was checked in a while back. Can you please elaborate what are you trying to do and why do you think it does not work?

tx for the quick reply...
sure I am trying to move my project which is based on jspm 100% so I can get rid of my node_moduls dir. This is the angular2 project:
https://github.com/born2net/ng2Boilerplate
and so I configured WebStorm to use latest TS 1.9@next
and setup my tsconfig as:
{
  "compilerOptions": {
    "target": "ES5",
    "inlineSources": true,
    "module": "system",
    "moduleResolution": "node",
    "removeComments": true,
    "emitDecoratorMetadata": true,
    "experimentalDecorators": true,
    "noEmitHelpers": false,
    "sourceMap": true,
    "noResolve": false,
    "rootDir": ".",
    "baseUrl": ".",
    "paths": {
      "@angular/core": [
        "jspm_packages/npm/<EMAIL_ADDRESS>      ],
      "@angular/platform-browser-dynamic": [
        "jspm_packages/npm/<EMAIL_ADDRESS>      ],
      "rxjs/*": [
<EMAIL_ADDRESS>      ]
    }
  },
  "filesGlob": [
    "./**/*.ts",
    "!./node_modules",
    "/src/typings/app.d.ts"
  ],
  "exclude": [
    "src/public/world_data.ts",
    "temp",
    "node_modules",
    "node_modules/*",
    "jspm_packages",
    "typings",
    "dist",
    "typings/*",
    "typings/main.d.ts",
    "typings/main"
  ],
  "compileOnSave": false,
  "buildOnSave": true
}

but noticed I put a fake XXXX to see if I get a TS error in WebStorm IDE (which again is configured to use TS 1.9) and yet no TS path errors popup.
tx again,
Sean.

as I was typing the question I had an idea, remove the
 "jspm_packages",

in the exclude and good news, I am starting to get some errors, which leads me to another question, is there a way to just map eveyrhing that tried to access via node_modules to redirect to jspm_packages?

Well, you probably aren't referencing any modules named @angular/XXXXXX* in your code.
I think you are mis-understanding the purpose of paths.  It basically says to the compiler, when you have a module that you need to resolve, and it matches this pattern, look here to see if it exists.  Because you still have module resolution set to node, you have informed the compiler to also look in node_modules.  It is doing what you told it to do.
If you want to know exactly how the compiler is trying to resolve the modules, you can use --traceResolution as a compiler option.

I understand, what value should I put for module resolution instead of node,
thanks

or just remove that line

Or read the documentation.
TypeScript defaults to moduleResolution of node.  The other mode is classic.

ok thx again, will give it a shot

so the good news is that moving to 'classic' fixed it so it's no longer looking at the node_modules directory. the bad news is that it is still not looking in the jspm_packages folder.
current config is:
{
  "compilerOptions": {
    "target": "ES5",
    "inlineSources": true,
    "module": "system",
    "moduleResolution": "classic",
    "removeComments": true,
    "emitDecoratorMetadata": true,
    "experimentalDecorators": true,
    "noEmitHelpers": false,
    "sourceMap": true,
    "noResolve": false,
    "rootDir": ".",
    "baseUrl": ".",
    "paths": {
      "@angular/core": [
        "jspm_packages/npm/<EMAIL_ADDRESS>      ],
      "@angular/platform-browser-dynamic": [
        "jspm_packages/npm/<EMAIL_ADDRESS>      ],
      "rxjs/*": [
<EMAIL_ADDRESS>      ]
    }
  },
  "filesGlob": [
    "./**/*.ts",
    "!./node_modules",
    "/src/typings/app.d.ts"
  ],
  "exclude": [
    "src/public/world_data.ts",
    "temp",
    "node_modules",
    "node_modules/*",
    "jspm_packages",
    "typings",
    "dist",
    "typings/*",
    "typings/main.d.ts",
    "typings/main"
  ],
  "compileOnSave": false,
  "buildOnSave": true
}

here is a dump of: tsc --traceResolution
trying just to fix "@angular/core": at this point ( I can do the rest once that works)
======== Resolving module '@angular/core' from 'C:/msweb/studioDashboard/src/App.ts'. ========
'baseUrl' option is set to 'C:/msweb/studioDashboard/jspm_packages', using this value to resolve non-relative module name '@angular/core'
'paths' option is specified, looking for a pattern to match module name '@angular/core'.
Module name '@angular/core', matched pattern '@angular/core'.
Trying substitution 'npm/<EMAIL_ADDRESS>candidate module location: 'npm/<EMAIL_ADDRESS>File 'C:/msweb/studioDashboard/jspm_packages/npm/<EMAIL_ADDRESS>does not exist.
File 'C:/msweb/studioDashboard/jspm_packages/npm/<EMAIL_ADDRESS>does not exist.
File<EMAIL_ADDRESS>does not exist.
File<EMAIL_ADDRESS>does not exist.
File<EMAIL_ADDRESS>does not exist.
File<EMAIL_ADDRESS>does not exist.
File<EMAIL_ADDRESS>does not exist.
File<EMAIL_ADDRESS>does not exist.
File<EMAIL_ADDRESS>does not exist.
File<EMAIL_ADDRESS>does not exist.
======== Module name '@angular/core' was not resolved. ========

this is configuration that works for me:
{
  "compilerOptions": {
    "target": "ES5",
    "inlineSources": true,
    "module": "system",
    "moduleResolution": "node",
    "removeComments": true,
    "emitDecoratorMetadata": true,
    "experimentalDecorators": true,
    "noEmitHelpers": false,
    "sourceMap": true,
    "noResolve": false,
    "baseUrl": "./jspm_packages",
    "paths": {
      "@angular/*": [
        "npm/<EMAIL_ADDRESS>      ],
      "@angular/platform-browser/src/browser/browser_adapter": [
        "npm/<EMAIL_ADDRESS>      ],
      "rxjs/*": [
<EMAIL_ADDRESS>      ]
    }
  },
  "filesGlob": [
    "./**/*.ts",
    "!./node_modules",
    "/src/typings/app.d.ts"
  ],
  "exclude": [
    "src/public/world_data.ts",
    "temp",
    "node_modules",
    "node_modules/*",
    "jspm_packages",
    "typings",
    "dist",
    "typings/*",
    "typings/main.d.ts",
    "typings/main"
  ],
  "compileOnSave": false,
  "buildOnSave": true
}

a few notes:

classic module resolution only looks for files that match requested module name. It won't work in case of Angular so you'll need to set moduleResolution to node
browser_adapter needs a special entry in configuration since BrowserDomAdapter does not appear in the root public surface of platform-browser module and thus import peeks inside the module (I don't know if this is a well accepted practice to work with this component - just trying to keep original code as-is)

thank you for that, it seems to be working, I  am gonna play with it some more to make sure!

So did this get implemented?

yes and it's AWESOME ...
I am going to add it to the Angular 2 Kitchen sink: http://ng2.javascriptninja.io
and source@ https://github.com/born2net/ng2Boilerplate
Regards,
Sean

tx to @vladima for his support!

Is there any way to get typescript to compile with it, then? I get an error
that paths isn't recognized. I'm on 1.8.10.
On Jun 1, 2016 2:02 PM, "JavaScriptNinja"<EMAIL_ADDRESS>wrote:

tx to @vladima https://github.com/vladima for his support!
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/TypeScript/issues/5039#issuecomment-223092367,
or mute the thread
https://github.com/notifications/unsubscribe/AAavtOTHbFrSDfyEvrSW4dfkkixjE7Vbks5qHdddgaJpZM4GGr-Q
.

u need to install 1.9 via npm install typescript@next

Any reason why the substitution pattern in paths config has to have only one asterisk? Allowing two would nicely support a directory structure where modules live in separate directories named the same as the module, e.g., a lib/foobar/foobar.ts could be imported as lib/foobar.

We did not see that as a common scenario supported by module loaders. but it could be added in the future.

This feature appears to work brilliantly, thanks. I wonder if there is any way to output the paths as relative – so for example import whatever from 'app/whatever' would be emitted by tsc as import whatever from '../../app/whatever' (or whatever the correct path is)? The reason this is of interest is that it appears requireing modules with "absolute" paths is slow when running with node, which means it takes a decent amount of time for my Mocha test suite to start up while it processes the requires. If not, I might be able to tackle this with Babel...

I wonder if there is any way to output the paths as relative –

The compiler does not rewrite module names. module names are considered resource identifiers, and are mapped to the output as they appear in the source

Is there a way to recreate this directory structure using TS: https://gist.github.com/ryanflorence/daafb1e3cb8ad740b346
In short:
Webpack allows you to do this:
  resolve: {
    extensions: ['', '.js', '.json', '.ts', '.tsx'],
    modulesDirectories: ['shared', 'node_modules']
  },

Which means that Webpack will recursively look up the directory tree for the shared dir (the same way it does with the node_modules dir).
This allows us to import shared components even from deeply nested directories without referencing the relative path:
import Button from 'Button';

This will work as long as Button lives in any shared folder that is higher up in the directory tree.
Is there a way to tell tsc to look for modules in this fashion?

Is it intentional that the "paths" values are computed relative to "baseUrl" when baseUrl is set?

Is it intentional that the "paths" values are computed relative to "baseUrl" when baseUrl is set?

yes. this is the same behavior other module loaders like require follow as well.

Should outDir prevent some usage of paths?
Thsi was working:
{
    "compilerOptions": {
        "module": "commonjs",
        "target": "es5",
        "noImplicitAny": false,
        "sourceMap": false,
        "rootDir": ".",
        "baseUrl": ".",
        "paths": {
            "util/*": [
                "../../../../Util/src/main/ts/*"
            ]
        }
    }
}

Then when I add outDir:
{
    "compilerOptions": {
        "module": "commonjs",
        "target": "es5",
        "noImplicitAny": false,
        "sourceMap": false,
        "rootDir": ".",
        "baseUrl": ".",
        "outDir": "../../target/ts",
        "paths": {
            "util/*": [
                "../../../../Util/src/main/ts/*"
            ]
        }
    }
}

The compiler says: "error TS6059: File 'C:/tmp/ts/paths/Util/src/main/ts/Util.ts' is not under 'rootDir' 'C:/tmp/ts/paths/SisModules/src/main/ts'. 'rootDir' is expected to contain all source files."

if you are using outDir, the compiler needs to mirror the input folder structure to the output directory. the root of the input is defined by rootDir. if there are files in the input, regardless if you are using paths or not, are not under rootDir, the compiler does not know how to emit them, and thus the error.

@mhegazy but "the compiler needs to mirror the input folder structure to the output directory" unfortunately creates the original paths in module names instead of the ones mapped to the virtual paths. It's weird at least.

unfortunately creates the original paths in module names instead of the ones mapped to the virtual paths. It's weird at least.

that is how it is meant to work. the compiler needs the paths to find the declaration of your module. module names are resource identifiers and should be emitted as is and not altered. please see https://github.com/Microsoft/TypeScript/issues/5039#issuecomment-232470330.

I'm not experienced in TypeScript nor modules, but I think the vpaths names would also be good for resource identifiers. In the browser they make much more sense than the real paths.

This is not meant to be a substitute of your require.config files, or system.config. this is meant to be a mirror of them to tell the compiler what your loader already knows.

@mhegazy Why not add a compile option,  so people can decide whether to rewrite module names or not?
I'm using react-native with Typescript, it's hard to resolve this problem.

+1

@rochapablo
I'm using https://github.com/ds300/react-native-typescript-transformer  now,
Use absolute paths is works for me.
And it gives another advantage,  we no need to use tsc -watch to compile our code before react-native package.

Damn! it worked!
Thank you @sapjax!

@fforres, follow the instructions it will work.


Variable for a mutation needs to be a string?

Using a variable like this:
{
  "id": 1
}

for a mutation gives this error:
error="cannot convert 1 to Int8".
Using it as a string works good.
I don't know if it's wanted, I'll open a PR for fixing the example.

I'm talking about the example here: https://supergraph.dev/docs/graphql#using-variables.
But I think it works because of `JSON.stringify()", right?

Works if you make it a string. It's a bug i'll look into it.
{
  "id": "2"
}

@dosco
Original error throws from  pgtype
,"error":"cannot convert 1 to Int8","errorVerbose":"cannot convert 1 to Int8:\n    github.com/jackc/pgtype.(*Int8).Set\n        /go/pkg/mod/github.com/jackc/pgtype@v1.4.0/int8.go:141"}

I tried to execute query generated by super graph with same variable but pgtype and pgx doesn't throw an error. Now I don't know why the independent program works and why super graph throws an error
my sample program
package main

import (
	"database/sql"
	"log"

	_ "github.com/jackc/pgx/v4/stdlib"
)

func main() {
	db, err := sql.Open("pgx", "postgres://postgres:postgres@localhost:5432/app_development")
	if err != nil {
		log.Fatal(err)
	}
       // query generated by super graph
	sql := `SELECT jsonb_build_object('products', "__sj_0"."json") as "__root" FROM (VALUES(true)) as "__root_x" LEFT OUTER JOIN LATERAL (SELECT coalesce(jsonb_agg("__sj_0"."json"), '[]') as "json" FROM (SELECT to_jsonb("__sr_0".*) AS "json" FROM (SELECT "products_0"."name" AS "name", "products_0"."created_at" AS "created_at" FROM (SELECT "products"."name", "products"."created_at" FROM "products" WHERE (((("products"."user_id") = $1 :: bigint) AND (("products"."id") = $2 :: bigint))) LIMIT ('50') :: integer) AS "products_0") AS "__sr_0") AS "__sj_0") AS "__sj_0" ON true`
     // variable pass by super graph
      vars := []interface{}{2, 1}

	log.Println(vars)
	rows, err1 := db.Query(sql, vars...)
	if err1 != nil {
		log.Println(err1.Error())
	} else {
		var data []byte
		rows.Next()
		e := rows.Scan(&data)
		if e != nil {
			log.Println(e.Error())
		} else {
			log.Println(string(data))
		}

	}
}

Output
2020/07/09 17:40:17 {"products": [{"name": "Hop Rod Rye", "created_at": "2020-07-03T09:19:43.961361"}]}

I observed that my sample program use the latest version of pgx (v4.7.1) and super-graph is using the previous version of pgs.
To verify I update pgx and pgtype version in super-graph but after updating latest pgx I am having the same error.

I have not had time to look into this. Try upgrading pgx in super graph. I feel the fix might be to convert ints into strings in args.go since in the end all values are casted in the generated sql.

I am able to reproduce this issue in my sample program. the data type of 1 is float64 and pgx is not able to convert float64 to int
package main

import (
	"database/sql"
	"log"

	_ "github.com/jackc/pgx/v4/stdlib"
)

func main() {
	db, err := sql.Open("pgx", "postgres://postgres:postgres@localhost:5432/app_development")
	if err != nil {
		log.Fatal(err)
	}
	// define 1 as float64 to reproduce error
	var val float64 = 1
	sql := `SELECT jsonb_build_object('products', "__sj_0"."json") as "__root" FROM (VALUES(true)) as "__root_x" LEFT OUTER JOIN LATERAL (SELECT coalesce(jsonb_agg("__sj_0"."json"), '[]') as "json" FROM (SELECT to_jsonb("__sr_0".*) AS "json" FROM (SELECT "products_0"."name" AS "name", "products_0"."created_at" AS "created_at" FROM (SELECT "products"."name", "products"."created_at" FROM "products" WHERE (((("products"."user_id") = $1 :: bigint) AND (("products"."id") = $2 :: bigint))) LIMIT ('50') :: integer) AS "products_0") AS "__sr_0") AS "__sj_0") AS "__sj_0" ON true`
	vars := []interface{}{2, val}

	log.Println(vars)
	rows, err1 := db.Query(sql, vars...)
	if err1 != nil {
		log.Println(err1.Error())
	} else {
		var data []byte
		rows.Next()
		e := rows.Scan(&data)
		if e != nil {
			log.Println(e.Error())
		} else {
			log.Println(string(data))
		}

	}
}

Output
2020/07/11 14:15:03 cannot convert 1 to Int8


Hyrax Migration

Not sure if we're going to do this yet, but placing some links here for reference assuming we do. The discussion, and migration of the Hydra in a box application from Sufia to Hyrax(the consolidated Sufia/CC project).
https://github.com/projecthydra-labs/hyrax/issues/4
https://github.com/projecthydra-labs/hyku/commit/a8b25cdf6630b425c8eed34fabb58b9a14523677

@mcritchlow Unfortunately we may need to work on ticket https://github.com/ucsdlib/horton/issues/21 to upgrade horton to use rails <IP_ADDRESS> first before migrating to hyrax. I got the following dependency error:
Resolving dependencies....
Bundler could not find compatible versions for gem "railties":
In Gemfile:
hyrax (= 0.0.1.alpha) was resolved to 0.0.1.alpha, which depends on
railties (~> 5.0)
rails (= <IP_ADDRESS>) was resolved to <IP_ADDRESS>, which depends on
  railties (= <IP_ADDRESS>)
generate    hyrax:install

The git source git://github.com/projecthydra-labs/hyrax.git is not yet checked out. Please run bundle install before trying to start your application
What do you think?

@lsitu - Yep, looks like they've pinned to Rails 5. Ok, Rails 5 first then.

@mcritchlow I'm trying migrate the feature/rails5 branch and get Object ingest working now. With the structure changes, the migration of collection codes seems to have a little trouble and I am looking into it now.

@mcritchlow I've created PR https://github.com/ucsdlib/horton/pull/23 for Hyrax migration. As noted in the PR, only the CSV Ingest/import support issue haven't been fixed, which is derived from the underlying support with batch ingest in Hyrax.


CLI arguments

I added python's built-in argparser library to your code and did some changes in code
I added two arguments as I only saw 2 input prompts in the code
Example:
$ python3 winLabeler.py --path "C:\Users\User\summer_photos\example.png" large

I currently don't use windows so I couldn't test it properly. Please test and share your thoughts with me so we can improve this
Issue #1

Hi, and thanks for your contribution.
I have tested this and works great! Although I should have been more specific in the first issue, my bad.
I have added a comment in #1 with more detail.

I added all arguments you mentioned on issue #1
Path argument doesn't require a flag and -ns, -o flags are optional with default values "False" and "out.ico".
Please test it and let's fix if there are any problems with it


consistency on the naming of Overlay / Overlays

I think the official name is Overlay but there are quite a few references to Overlays (plural) in the specification and readme.md
If they are references to an Overlay description, I believe they should be lowercase or reworded.
https://github.com/OAI/Overlay-Specification/blob/d029f0d7d2c8861e00de4467ab795e86d9e35824/versions/1.0.0.md?plain=1#L137
https://github.com/OAI/Overlay-Specification/blob/d029f0d7d2c8861e00de4467ab795e86d9e35824/versions/1.0.0.md?plain=1#L165
https://github.com/OAI/Overlay-Specification/blob/d029f0d7d2c8861e00de4467ab795e86d9e35824/versions/1.0.0.md?plain=1#L188
https://github.com/OAI/Overlay-Specification/blob/d029f0d7d2c8861e00de4467ab795e86d9e35824/README.md?plain=1#L20
https://github.com/OAI/Overlay-Specification/blob/d029f0d7d2c8861e00de4467ab795e86d9e35824/README.md?plain=1#L22

Hi @jeremyfiel,
Thanks for reporting this!
Would it be sufficient to address this in the next release?
Thanks in advance!

Since voting is already open after many rounds of feedback, the specification itself is closed for changes. We can definitely pick it up for a future patch release though.

Would it be sufficient to address this in the next release?

Sure!

@jeremyfiel Since we have som other pending changes I tried to address your concerns in

#78

Rationale: the specification defines

An Overlay is a JSON or YAML structure containing an ordered list of Action Objects that are to be applied to the target document.

Thus the use of "Overlay" in the section headlines and the README text seems justified.

@ralfhandl  this should be updated to 1.0.0 release milestone
thanks for your follow up!


How can we fetch the participants on active meeting 0r attendeId's on the meeting?

React Native Chime Demo is using externalUserId for handling the users information that is  joining to the meeting. I want to get all the references of attendees on the meeting.

Hello,
The demo's event emitter is sending the attendeeId and externalUserId (Android iOS). You can modify the handler to do what you need with each participant's attendeeId. Let us know if you have further questions.


Rewards as Interest Multipliers

6.1.23. This is an old ticket and may be stale.  Review the current proposed implementation of Zug-compatible rewards before doing any work with this.
storage optimization.

@RitaMAllenCA, I somehow missed this being migrated into the Epic. This is auction reward accrual accounting changes, right? That's, strictly speaking, independent of Zug-compatible rewards (would apply to any scheme we come up with)

Implemented a different reward mechanism.

@devendran-m should be noted that this is orthogonal to the reward mechanism, this ticket is about making system auction internal accounting more efficient, but this can be re-opened if it becomes an issue once again (I believe we made a few optimizations over time that made this less relevant)


[FEATURE] Transform complex dynamic imports

The problem that you want to solve.
Issue reported here: https://github.com/snowpackjs/snowpack/discussions/2840
We should be able to transform:
import(`../locales/${langCode}.json`);

to:

import(`../locales/${langCode}.json.proxy.js`);

Your take on the correct solution to problem.
We could add some advanced handling to our internal transform utility to parse the expression inside of a dynamic import into an AST (with something like acorn) or even something simpler like Regex. Then, we could transform the template string based on its raw parts.
Are you willing to submit a pull request to implement this change?
This is a little tricky, but fairly self contained / a good place to start if you're looking to contribute to Snowpack!

I wonder if we could combine #2900 and #2881, expand the import.meta.glob lexer to collect all import statements, then handle import.meta.glob and proxy import transforms all in one go? That sounds like it would be the most performant solution.
We could also speed up the lexer by only scanning forward from import tokens rather than the whole file.

Also this particular use case might be solved by our upcoming import.meta.glob feature? I'll have to test this one!
// writing the following
const langs = import.meta.glob(`../locales/*.json`);

// should produce
const langs = {
  '../locales/en.json': () => import('../locales/es.json.proxy.js'),
  '../locales/es.json': () => import('../locales/es.json.proxy.js')
}

Oh interesting! What if we just added a warning when we saw a complex dynamic import, telling you to use glob instead?

import.meta.glob can't be found in the document.


Duplicate Effects dispatched for Lazily-loaded modules

Angular version: 2.2.1 (or 2.2.3)
Bug description If a lazily-loaded module uses EffectsModule.run(SomeEffects), actions that are dispatched from SomeEffects will be dispatched twice. If EffectsModule.run(SomeEffects) is imported in the root app module, those actions will not be dispatched twice.
Side Note: it seems the replace-reducer action is not getting logged correctly in store dev tools logger (ngrx/store/init should not have the lazily-loaded state but you can see here

that it does have the "lazy" state)
Reproduced here:
https://github.com/exk0730/angular-webpack2-starter/tree/no-universal-support

@exk0730 this is router bug: https://github.com/angular/angular/issues/12869

@qdouble thanks.

@qdouble is there a known workaround to this (even if its hackish)? should be just not user lazy loaded modules? the guys at the angular team don't even seem to care.


fix(web-components): updated select searchable to update emitted events

Summary of the changes
Updated Searchable so the input value is not updated when highlighting options within the ic-menu. icOptionSelect and icChange are only emitted once the option and been selected. This update also allows the input value to be updated outside of input or menu selection.
Added onBlur to host close menu
PLEASE NOTE: The UX for Searchable select has slightly changed:

Before: When highlighting menu options, the value in the input would reflect the highlighted option and trigger icChange.
After: When highlighting menu options, the value in the input is no longer updated until Enter is pressed. icChange is emitted once the value is updated.

Related issue
#525
Checklist

[x] I have added relevant unit and visual regression tests.
[ ] I have manually accessibility tested any changes, if relevant.
[ ] I have ensured any changes match the Figma component library.

Can we give each select in the stories a unique id to save having to reload the page each time you go to a new story? It seems to fail setting the options

After selecting an option in a searchable select, then moving focus elsewhere, then come back, there is no longer a tick to show the selected option  - is that correct?

Another from the external filtering story:

Type a phrase so that it times out
Tab to the retry button
Click the retry button whilst it is focused
The menu closes


Issues with path to blast executable?

I've pasted the error output below. It appears that when the program is trying to use blast for the plastome finishing step, it is missing a "/" in the path to the blast executable. It looks like it is trying to call:
/apps/eb/BLAST+/2.10.1-gompi-2019b/binblastn
But the correct path is:
/apps/eb/BLAST+/2.10.1-gompi-2019b/bin/blastn
Is this an issue with installation?
TrimmomaticPE: Started with arguments:
 -threads 4 /scratch/eab77806/trimmed_reads/m017_R1_P.fastq.gz /scratch/eab77806/trimmed_reads/m017_R2_P.fastq.gz m017_0.trimmed_P1.fq m017_0.trimmed_U1.fq m017_0.trimmed_P2.fq m017_0.trimmed_U2.fq ILLUMINACLIP:/apps/eb/Fast-Plast/1.2.8-foss-2019b-Perl-5.30.0/bin/adapters/NEB-PE.fa:1:30:10 SLIDINGWINDOW:10:20 MINLEN:40
Using Long Clipping Sequence: 'ACACTCTTTCCCTACACGACGCTCTTCCGATC'
Using Long Clipping Sequence: 'GATCGGAAGAGCACACGTCTGAACTCCAGTC'
ILLUMINACLIP: Using 0 prefix pairs, 2 forward/reverse sequences, 0 forward only sequences, 0 reverse only sequences
Quality encoding detected as phred33
Input Read Pairs: 4178953 Both Surviving: 4098605 (98.08%) Forward Only Surviving: 63267 (1.51%) Reverse Only Surviving: 15619 (0.37%) Dropped: 1462 (0.03%)
TrimmomaticPE: Completed successfully
Use of uninitialized value $bt2_args[5] in join or string at /apps/eb/Bowtie2/2.4.1-GCC-8.3.0/bin/bowtie2 line 423.
Use of uninitialized value $bt2_args[6] in join or string at /apps/eb/Bowtie2/2.4.1-GCC-8.3.0/bin/bowtie2 line 423.
Use of uninitialized value in exists at /apps/eb/Bowtie2/2.4.1-GCC-8.3.0/bin/bowtie2 line 81.
Use of uninitialized value in exists at /apps/eb/Bowtie2/2.4.1-GCC-8.3.0/bin/bowtie2 line 81.
Use of uninitialized value $bt2_args[5] in join or string at /apps/eb/Bowtie2/2.4.1-GCC-8.3.0/bin/bowtie2 line 459.
Use of uninitialized value $bt2_args[6] in join or string at /apps/eb/Bowtie2/2.4.1-GCC-8.3.0/bin/bowtie2 line 459.
4177491 reads; of these:
  4098605 (98.11%) were paired; of these:
    4030841 (98.35%) aligned concordantly 0 times
    35542 (0.87%) aligned concordantly exactly 1 time
    32222 (0.79%) aligned concordantly >1 times
    ----
    4030841 pairs aligned concordantly 0 times; of these:
      7037 (0.17%) aligned discordantly 1 time
    ----
    4023804 pairs aligned 0 times concordantly or discordantly; of these:
      8047608 mates make up the pairs; of these:
        8029608 (99.78%) aligned 0 times
        2838 (0.04%) aligned exactly 1 time
        15162 (0.19%) aligned >1 times
  78886 (1.89%) were unpaired; of these:
    77424 (98.15%) aligned 0 times
    773 (0.98%) aligned exactly 1 time
    689 (0.87%) aligned >1 times
2.04% overall alignment rate
sh: /apps/eb/BLAST+/2.10.1-gompi-2019b/binblastn: No such file or directory
sh: /apps/eb/BLAST+/2.10.1-gompi-2019b/binblastn: No such file or directory
sh: /apps/eb/BLAST+/2.10.1-gompi-2019b/binblastn: No such file or directory
sh: /apps/eb/BLAST+/2.10.1-gompi-2019b/binblastn: No such file or directory
sh: /apps/eb/BLAST+/2.10.1-gompi-2019b/binblastn: No such file or directory
sh: /apps/eb/BLAST+/2.10.1-gompi-2019b/binblastn: No such file or directory
Died at /apps/eb/Fast-Plast/1.2.8-foss-2019b-Perl-5.30.0/fast-plast.pl line 876.

Yes, that is an installation error. Did you give Fast-Plast the path to something pre-installed or have FP install blast for you?

I believe the former. I had university staff install this on our HPCC (the GACRC at UGA, which you might be familiar with!), so I will have to reach out to them about this. Is blast involved in identifying the IRs by chance?
Also, I am having issues using the built-in bowtie indices. If I try to use Ericales for example, FP will create an empty file called Ericales.fsa in the Bowtie_Mapping directory, and no bowtie index files or mapped reads files, and obviously the spades assembly can't run. I can grab the error log for those runs and start another issue for that if that would help.

Blast is used to ID genes to SC or IR. SC and IR and identified using a perl script that looks for reverse complement regions of a sliding window.
Can you post the command you used to call Fast-Plast? Also your other error files? You should be getting an Ericales.fsa file.

Sure, the command is:
fast-plast.pl -1 /scratch/eab77806/trimmed_reads/m017_R1_P.fastq.gz -2 /scratch/eab77806/trimmed_reads/m017_R2_P.fastq.gz --name m017 --bowtie_index Ericales --coverage_analysis
I attached the error file. Again, there is an Ericales.fsa in the mapping directory, but it is empty.
m017_results_error.log

Hi -after installing the last updated version of Fast-Plast (everything went OK, no single error during installation) I'm getting an error related to Trimmomatic.
Any clue what could be wrong? thanks
2022	Starting Fast-Plast v.1.2.9.
Assemblying plastome with 0 single end libraries and 1 paired end libraries.
2022	Determining best kmer sizes.
K-mer sizes for SPAdes set at 31,37,43.
2022	Starting read trimming with Trimmomatic.
Using /home/jorge/Fast-Plast/bin/Trimmomatic-0.39//trimmomatic-0.39.jar.
2022	Starting read mapping with bowtie2.
Using /home/jorge/Fast-Plast/bin/bowtie2-<IP_ADDRESS>-linux-x86_64/bowtie2.
Samples for Poales used to make bowtie2 indices.
ERROR: No trimmed read files were identified to run SPAdes.  Please check 1_Trimmed_Reads.


chore(master): release 1.1.2

:robot: I have created a release beep boop
1.1.2 (2024-11-29)
Bug Fixes

schema & argument types (1689922)

This PR was generated with Release Please. See documentation.

:robot: Created releases:

v1.1.2
:sunflower:


Docker --link flag is about to be removed

现存什么问题没有很好被解决
In the docker-compose documentation:
https://docs.docker.com/compose/compose-file/#links

Warning: The --link flag is a legacy feature of Docker. It may eventually be removed. Unless you absolutely need to continue using it, we recommend that you use user-defined networks to facilitate communication between two containers instead of using --link. One feature that user-defined networks do not support that you can do with --link is sharing environmental variables between containers. However, you can use other mechanisms such as volumes to share environment variables between containers in a more controlled way.

But that somehow is still in walle's installation guide:
https://walle-web.io/docs/installation_docker.html#Prepare-Before-Deploy
    links:
      - python

@wi1dcard 这个还没更新


Issues with select (validate and labelling)

I am having several issues with the Select component in the following code (using the latest v5.0 of redux-form-material-ui)
https://pastebin.com/m2fiqqs0
1.) I am trying to label the Select field. I have tried with the following label, FormControl, and FormControlLabel. How do I go about this?
2.) I am trying to use a validate with the select. It does work; however, required is not displayed.
Any information on these issues would be greatly appreciated.

I am also having this issue and believe it is because of hasHelperText: false overwrite in the Select component. It looks like this is stopping errors from being displayed


Version idents: Posix -> POSIX, linux -> Linux.

This is in accordance with the recent discussions on the NG.
This patch changes Posix -> POSIX and linux -> Linux to be consistent in casing. The old identifiers are still kept around for compatibility.

I don't see any gain to changing the existing identifiers. It's an illusion of progress, yet making disruptive and arbitrary amounts of work for others as they have to go edit their code. We've got much more important things that need doing.


What about Rust?

Sorry if I write in the golang package but this is the only for the backend I think.
What do you think about starting a connect-rust?
Rust is a very powerful and trending language.
Are you interested?

Sorry if I write in the golang package but this is the only for the backend I think.

That's fine! There's also connect-es for backends, but discussing this is okay. We're fully open to a Connect implementation in Rust, but we'd need strong community involvement - we're not the right people to develop a Rust project fully in-house. There's already https://github.com/AThilenius/axum-connect, which is headed in exactly the direction that makes sense to me - I'd suggest taking a look at that, filing issues, and contributing!
We'll have a better home for these discussions, and a better proposal process, in the next few weeks.

I'll wait for the the "better home" to subscribe and follow and I'll close this. Thanks!

@frederikhors probably good to discuss here: https://github.com/orgs/connectrpc/discussions/7


Migrate core package 'incompatible-packages' into Atom repo

This issue tracks the steps necessary to migrate the incompatible-packages package into the core atom/atom repository under the packages path:
Migration Checklist
Phase 1 - Prepare repo for migration

[x] Merge or close existing pull requests on the atom/incompatible-packages repository

Some guidelines for merging vs closing PRs:

Merge when the PR is small, useful, has no merge conflicts with master, and has passing tests
Close when the changes are non-trivial, the PR has conflicts with master, or there are failing tests in CI

If you close a PR, let the user know that we are currently moving the repository over to atom/atom so they should feel free to reopen the pull request there once the package code has been migrated.
Phase 2 - Migrate the package code

[x] Create a fresh branch off master in your local clone of atom/atom with the name migrate-incompatible-packages-package
[x] Delete Atom's node_modules folder if you've built there before and make sure Atom's package-lock.json file is unmodified
[x] Clone the  atom/incompatible-packages repo into the packages path of your local Atom repository and then delete its .git subfolder
[x] Delete any unnecessary repo configuration files in the packages/incompatible-packages folder like appveyor.yml, .travis.yml, ISSUE_TEMPLATE.md, and PULL_REQUEST_TEMPLATE.md
[x] Remove any information from the package's README.md which will be inaccurate after the migration is complete (e.g. CI status badges or contribution instructions)
[x] Edit Atom's package.json file to change the following:

[x] Update the incompatible-packages entry in packageDependencies to reflect the local path of the package:"incompatible-packages": "file:./packages/incompatible-packages",

[x] Delete the incompatible-packages entry in the dependencies section since it may contain a hardcoded tarball link.  This will help avoid issues in the next step
[x] Change the repository field to point to the main repo URL https://github.com/atom/atom

[x] Run a full Atom build using script/build and verify that there are no unexpected build errors
[x] Launch the Atom binary in the ./out folder and manually verify that the migrated package is working without any obvious issues.  For example, if you're migrating a color theme, turn on that theme to make sure that it changes the editor colors as expected.
[x] Open the Chrome Developer Tools console (Ctrl+Shift+I or Cmd+Option+I) and ensure that there are no errors written there.  If there are, add them to this issue and investigate.
[x] Update the second column of the incompatible-packages row in packages/README.md to reflect the new local path for the package:[`./packages/incompatible-packages`](./incompatible-packages)

[x] Commit the migrated package code and any changes to Atom's package-lock.json files to your branch using this commit message::arrow_right: Migrate core package 'incompatible-packages' into ./packages

Phase 3 - Make sure the package works and send a PR

[x] Run the full suite of Atom tests with script/test and verify that they pass locally
[x] If any tests fail, investigate and fix them then commit the fixes to your branch
[x] Create a PR against atom/atom to verify that CI passes
[x] Once CI is clean and the PR is approved, merge it

Phase 4 - Archive the incompatible-packages repo

[x] Move all existing issues from the  incompatible-packages repo to atom/atom using github-issue-mover or a script
[x] Add a label titled packages/incompatible-packages to the migrated issues so that they can be found easily
[x] Edit the original package repo's README.md (don't commit directly) to point contributors to the code's new home in atom/atom.  You can use the following text:### This package is now a part of the [core Atom repository](https://github.com/atom/atom/tree/master/packages/incompatible-packages), please direct all issues and pull requests there in the future!

---

[x] Send a PR to the incompatible-packages repository with the updated README.md so that watchers will be notified of the change.  You do not need to wait for reviews, it can be merged immediately.
[x] Archive the package repository by going to its Settings page, scroll down to the "Danger Zone" section and click "Archive this Repository"

Migration is now complete! :tada:

Migration is complete :tada:

https://github.com/atom/atom/pull/18104
https://github.com/atom/incompatible-packages/pull/26


[Trait] Add an example of ServiceMonitor Trait

Description of your changes
Add ServiceMonitor Trait
Add ServiceMonitor Trait example yamls and README
How has this code been tested?
Refer to README
Checklist
I have:

[x] Title of the PR starts with OAM type (e.g. [Workload] or [Trait] or [Scope]).
[x] Updated/Added any relevant [documentation] and [examples].
[x] Unit/E2E Tests added.

closing as we have a better monitoring trait


flutter-devicelab-mac-1 is not available over ssh

flutter-devicelab-mac-1

rebooted, mac-1 is now ssh'able.


backendError 500 when using the  webmaster.searchanalytics api

I'm getting the following error when trying to use the webmaster.searchanalytics api:
{ Error: Backend Error
    at Request._callback (/home/tom/Programming/MagpieMicroService/node_modules/google-auth-library/lib/transporters.js:85:15)
    at Request.self.callback (/home/tom/Programming/MagpieMicroService/node_modules/request/request.js:186:22)
    at emitTwo (events.js:125:13)
    at Request.emit (events.js:213:7)
    at Request.<anonymous> (/home/tom/Programming/MagpieMicroService/node_modules/request/request.js:1163:10)
    at emitOne (events.js:115:13)
    at Request.emit (events.js:210:7)
    at IncomingMessage.<anonymous> (/home/tom/Programming/MagpieMicroService/node_modules/request/request.js:1085:12)
    at Object.onceWrapper (events.js:314:30)
    at emitNone (events.js:110:20)
  code: 500,
  errors:
   [ { domain: 'global',
       reason: 'backendError',
       message: 'Backend Error' } ] }

Code is the following:
'use strict';

var google = require('googleapis');
var fs = require('fs');
var key = require('./client_secret.json');
var prompt = require('prompt');

var tokens_file = './tokens/tokens.json';
var scopes = [
    'https://www.googleapis.com/auth/webmasters'
];

var OAuth2 = google.auth.OAuth2;
var oauth2Client = new OAuth2(
    key.installed.client_id,
    key.installed.client_secret,
    key.installed.redirect_uris[0],
);

fs.readFile(tokens_file, 'utf8', (err, data)=>{
    if (err) {
        console.log('No token file: ',err);
        var url = oauth2Client.generateAuthUrl({
            access_type: 'offline',
            scope: scopes,
        });
        console.log(
            'Head to the URL ',
            url,
            ' and paste the authorization code in:'
        );
        prompt.start();
        prompt.get(['authorizationCode'],(err, result)=>{
            if (err) {return console.log(err);}
            var code = result.authorizationCode;
            oauth2Client.getToken(code, (err, tokens)=>{
                if (err) {return console.log(err);}
                console.log(tokens)
                fs.writeFile(
                    tokens_file,
                    JSON.stringify(tokens),
                    (err)=>{
                        if (err) {return console.log('Error ', err);}
                        console.log('Tokens Saved')
                    }
                );
                oauth2Client.setCredentials({
                    access_token: tokens.access_token,
                    refresh_token: tokens.refresh_token
                });
                //Make the authorised call here
                const webmasters = google.webmasters('v3');
                let date = new Date();
                let startDate = new Date(date-7);
                let endDate = new Date(date-7);
                let params = {
                    'startDate': "2017-09-25",//startDate.toISOString(),
                    'endDate': "2017-09-26",//endDate.toISOString(),
                    'dimensions': ['query', 'device'],
                    'rowLimit': 5000,
                    'startRow': 0,
                    'siteUrl': encodeURIComponent('http://www.url.com/de/'),
                    'auth': oauth2Client,
                };
                console.log(params.siteUrl);
                console.log(params);
                webmasters.searchanalytics.query(params,function(err, data){
                    if (err) return console.log('There was an error: ', err);
                    console.log('This is the data: ', data);
                });
            });

        });
        return;
    };
    console.log('onfile')
    var tokens = JSON.parse(data);
    console.log('tokens: ', tokens);
    oauth2Client.setCredentials({
        access_token: tokens.access_token,
        refresh_token: tokens.refresh_token
    });
    console.log(oauth2Client)
    //Make the authorised call here
    // google.options({auth:oauth2Client});
    const webmasters = google.webmasters('v3');
    let date = new Date();
    let startDate = new Date(date-7);
    let endDate = new Date(date-7);
    let params = {
        'startDate': "2017-09-25",//startDate.toISOString(),
        'endDate': "2017-09-26",//endDate.toISOString(),
        'dimensions': ['query', 'device'],
        'rowLimit': 5000,
        'startRow': 0,
        'siteUrl': encodeURIComponent('http://www.url.com/de/'),
        'auth': oauth2Client,
    };
    console.log(params.siteUrl);
    console.log(params);
    webmasters.searchanalytics.query(params,function(err, data){
        if (err) return console.log('There was an error: ', err);
        console.log('This is the data: ', data);
    });
});

Hi @ajudi
From Google's documentation (See Error Code: 500), this backEnd Error happens when:

...the client periodically retries a failed request over an increasing amount of time.

Can you wrap this:
    webmasters.searchanalytics.query(params,function(err, data){         if (err) return console.log('There was an error: ', err);         console.log('This is the data: ', data);     });
in a try/catch, log the error and post is here for review?

It doesn't actually produce an error sadly. I tried the following:
try{
            webmasters.searchanalytics.query(params, {}, function(err, data){
                if (err) return console.log('There was an error: ', err);
                console.log('This is the data: ', data);
            });
        }catch(e){
            console.log('caught error')
            console.log(e)
        }

I guess because it's being passed to the callback?

It also seems to be writing a number of files in my directory. Not anything that I can unencode or recognise. All seem to be in the same format.
<IP_ADDRESS>.00443-<IP_ADDRESS>.39358

Any ideas why that might be.

I looked in to this further as our Python implementation works. It seems as though it's an error in the javascript libraries. Can't really tell where the error is but if you try and use the API explorer on the google website it's also broken. Works fine if you just use POST from the request library with the key I get from auth. It's not a rate limiting issue the API is broken.

@ajudi It's curious that the Javascript and API explorer fails while the python code succeeds. Can you share the python code that works? Might help us debug the issue.

Sure.
import argparse
import csv
import datetime
import hashlib
import httplib2

from apiclient import discovery
from oauth2client import client
from oauth2client.file import Storage
from oauth2client import tools

import settings

def execute_request(service, client_url, request):
    """
    Execute a searchAnalytics.query request.
    :param service:       The webmasters service to use when executing the
                          query.
    :param client_url:    The site or app URI to request data for.
    :param request:       The request to be executed.
    :returns:             An array of response rows.
    """
    return service.searchanalytics().query(siteUrl=client_url,
                                           body=request).execute()

def get_credentials():
    """
    Check for credentials, in the storage data file.
    If no credentials are stored of if the stored credentials
    are invalid, new credentials are return by the function run_flow().
    The new credentials are also stored in the storage argument, which
    updates the file associated with the Storage object.
    :returns credentials:    credentials object
    """
    flow = client.flow_from_clientsecrets(settings.CLIENT_SECRET,
                                          scope=settings.SCOPE,
                                          redirect_uri=settings.REDIRECT_URI)

    storage = Storage(settings.STORAGE_FILE)
    credentials = storage.get()
    if credentials is None or credentials.invalid:
        parser = argparse.ArgumentParser(parents=[tools.argparser])
        flags = parser.parse_args(['--noauth_local_webserver'])
        credentials = tools.run_flow(flow=flow, storage=storage, flags=flags)

    return credentials

def build_service_object():
    """
    Build the service object for google drive.
    The service object is constructed with methods specific
    to the given api, in this case, google drive.
    :returns service:    service object for google drive api
    :example:
    >>> from apiclient import discovery
    >>> service = discovery.build('drive', 'v2'...)
    """
    credentials = get_credentials()
    http_auth = credentials.authorize(httplib2.Http())
    service = discovery.build('webmasters', 'v3', http_auth)

    return service

def set_date_range():
    """
    Set the date range available by querying the Search Console API with a
    simple query.
    This will return the list of available dates, which should
    always be 90 days in length. There is an offset of between 3 and 6 days
    from todays date that needs to be accounted for, hence starting the query
    on 2015-01-01.
    This function is designed to gather the data by 30 day period, i.e. day 0
    to day 29, day 30 to day 59, etc.
    :returns query_dates:    List of tuples, containing string formatted
                             dates.
    """
    list_of_dates = []
    end_date = datetime.date.today().strftime('%Y-%m-%d')
    request_body = {'startDate': '2015-01-01',
                    'endDate': end_date,
                    'dimensions': ['date']}
    service = build_service_object()
    response = execute_request(service, settings.CLIENT_URL, request_body)

    for row in response['rows']:
        list_of_dates.append(row['keys'][0])

    query_dates = [(list_of_dates[0], list_of_dates[29]),
                   (list_of_dates[30], list_of_dates[59]),
                   (list_of_dates[60], list_of_dates[-1])]
    return query_dates

def query_search_console_data(dates, service, device):
    """
    Query the Search Console using ignore characters, and date ranges.
    Returns raw unencoded data.
    :param dates:          dates in tuple to query
    :param service:        service object
    """
    list_data = []

    print 'Requesting {} data'.format(device)
    for character in settings.IGNORE:
        request_body = {
            'startDate': '{}'.format(dates[0]),
            'endDate': '{}'.format(dates[1]),
            'dimensions': ['query',
                           'device'],
            'dimensionFilterGroups': [
                {'groupType': 'and',
                    'filters': [
                        {'dimension': 'query',
                         'operator': 'notContains',
                         'expression': character},
                        {'dimension': 'device',
                         'operater': 'equals',
                         'expression': device}]}],
            'rowLimit': 5000}

        response = execute_request(service,
                                   settings.CLIENT_URL,
                                   request_body)
        encoded_response = encode_response(response)
        metrics_list = row_builder(encoded_response, dates, list_data)
    print 'Returning metrics_list'
    return metrics_list


Plugin Stopped working with. Code 500: Internal Server Error / TypeError...

Recently Obsidian Rest Api Plugin stopped working  after a restart after a power outage (not sure if related)
Since then i tried uninstalling and reinstalling the plugin, but nothing changed.
In the console i see this message:
TypeError: argument entity must be string, Buffer, or fs.Stats
    at etag (plugin:obsidian-local-rest-api:39419:15)
    at generateETag (plugin:obsidian-local-rest-api:40374:16)
    at ServerResponse.send2 [as send] (plugin:obsidian-local-rest-api:43551:20)
    at ServerResponse.json (plugin:obsidian-local-rest-api:43579:19)
    at RequestHandler.returnCannedResponse (plugin:obsidian-local-rest-api:45578:63)
    at RequestHandler.eval (plugin:obsidian-local-rest-api:46188:12)
    at Generator.next (<anonymous>)
    at eval (plugin:obsidian-local-rest-api:79:61)
    at new Promise (<anonymous>)
    at __async (plugin:obsidian-local-rest-api:63:10)

In the client side (https://coddingtonbear.github.io/obsidian-local-rest-api/) I get this response:
Code 500 Undocumented
Error: Internal Server Error
along with the same error message as above as the 'Response body'

Here is an issue on a different project that seems to be related. It also has a fix, but I have no idea how to apply it (or can it even be applied) here.
https://github.com/ionic-team/ionic-cli/issues/598

+1, I also meet this

Can't believe I missed this issue. I'm also seeing this, starting a few days ago (I initially thought it was something wrong with a tag or metadata in one of my notes, which has happened to me before). Anyway, I took some notes into discussion here https://github.com/coddingtonbear/obsidian-local-rest-api/discussions/78
I (incidentally) tested to comment out line 39419, and then "note recall" started working again, and no "ERR" in Obsidian Web anymore. Does not solve the issue, but at least I can use note recall again 😄

I'm afraid this isn't a problem in obsidian-web, but in the API it interacts with -- could I convince you all into moving this discussion here: https://github.com/coddingtonbear/obsidian-local-rest-api/discussions/78?


request: change default port to something other than 8081

Hi, can we change the default port to something a little more unique than 8081?  This port is also used by default by eg. Quarkus unit tests so every time run a Quarkus build when VSCode is running it fails unless I specifically change the port in my application.properties which isn't ideal.

there is work in progress to pick a free port https://github.com/KaotoIO/vscode-kaoto/issues/60

Great!  Unless of course 8081 is free at the time vscode starts up and I'm not yet running my unit test :)

For information, choosing a free port at runtime is not as easy as planned.
The plan is to continue searching in this direction until end of week; if no solution found, we can relatively easily - if no other surprises - implements to use a different port. It will be less powerful but will mitigate port conflict with other applications.
From top of my head, it would consists of changing code at 2 places:

change port here: https://github.com/KaotoIO/vscode-kaoto/blob/ce67f86971f354475ab1d192718356f6be168a1c/webpack.config.js#L138
specify port when launching Kaoto backend https://github.com/KaotoIO/vscode-kaoto/blob/ce67f86971f354475ab1d192718356f6be168a1c/src/extension/extension.ts#L74 using something like child_process.spawn(nativeExecutable, ['-Dquarkus.http.port=8096']);