Decide on a consistent pattern for macro augmentation uris

We need to choose what the macro augmentation URI will look like for a given library, so the CFE/Analyzer/IDE can all be on the same page when it comes to stack traces, debugging, and navigate to definition.
Proposal
A custom scheme
The way VsCode and possibly LSP in the future handle custom file content providers is through registering a custom scheme associated with a specific provider of file contents for those URIs.
This seems like a logical/consistent way to represent these URIs to me, and has a high chance of mapping well to IDEs and LSP in the future, so I propose we use this strategy.
The CFE currently already does this, I think using the  org-dartlang-augmentation scheme. I would like to be a bit more specific about this being for macro augmentations, maybe org-dartlang-macro-augmentation although the scheme does get quite large. Please provide your suggestions for this :).
Specifics of mapping library URIs to their macro URIs
This is a little bit tricky because the original library URIs might have different schemes (package, file, a mulit-root scheme, or possibly even something different). Theoretically, there could be multiple library URIs with different schemes but the same paths, so we can't just swap the scheme.
We also don't want to use the true absolute file URI, as those might not even be the same, across the analyzer/CFE.
It should also be noted that these URIs will be user visible, through stack traces etc, so we want them to look somewhat reasonable.
Option A: encode the original library URI as the first path segment
This is a very simple transformation to get correct, but pretty ugly for users:

package:a/a.dart => org-dartlang-macro-augmentation:package%3Aa%2Fa.dart
file:///a/b.dart => org-dartlang-macro-augmentation:file%3A%2F%2F%2Fa%2Fb.dart

Option B: encode just the original library scheme & authority as the first path segment
Sort of similar to what windows does for drive letters, we could URI encode the original scheme and authority (everything preceding the path) into the first segment of the new path. This would be somewhat less ugly than encoding the entire URI, but gets weird with relative versus absolute URI paths. Likely, if the path was relative, it would end up encoded with the scheme in the first segment:

package:a/a.dart => org-dartlang-macro-augmentation:package%3Aa/a.dart (note the first part of the relative path here is not preceded by a / because it is relative).
file:///a/b.dart => org-dartlang-macro-augmentation:file%3A%2F%2F/a/b.dart

Option C: special rules for known schemes, with a fallback for other schemes
We could also try to specify special rules for known schemes, and fall back on one of the above options for other schemes. This could potentially make things look nicer for users.
Option D: Encode the original scheme into the special scheme
I would probably just append it to the end, with a - prefix, so:

package:a/a.dart => org-dartlang-macro-augmentation-package:a/a.dart
file:///a/b.dart => org-dartlang-macro-augmentation-file:///a/b.dart

This would be pretty readable and also easy to translate. It also handles absolute and relative URIs, as well as any other lesser used components of URIs with ease.
We would have to register multiple file content providers, one for each scheme, but we already know the actual schemes we support so this likely wouldn't be an issue.
Multi-root schemes
These are especially difficult, although we already have to handle them very specially. They do appear in stack traces and so IDEs (or web devtools) already have to do some sort of custom mapping. They probably just don't work for a lot of scenarios today, I am not exactly sure.
cc @scheglov @johnniwinther

Note that it would probably be good to resolve this sooner than later, before we get too many places building in assumptions about how either tool does it today (already some stuff internally relies on the current analyzer strategy of replacing .dart with .macro.dart)

In the analyzer everything is built around files, which have path and URI. Everything that gets analyzed must be a file, at least nominally, such as have a path. But we already "fake" the content of macro generated augmentations (intermediate and combined). We always know inside the analyzer when the file is a macro generated one.
In DAS we also have an expectation that everything is a file, e.g. AnalysisNavigationParams is
  List<NavigationRegion> regions;
  List<NavigationTarget> targets;

  /// The files containing navigation targets referenced in the file. They are
  /// referenced by NavigationTargets by their index in this array.
  List<String> files;

@pq @bwilkerson who plan to work on macros support in the analysis server.

In the analyzer everything is built around files, which have path and URI

Can the URI be a non-file: uri? Does it have to correspond directly to the path?

The question is what good it will do to us.
If the protocol that we use to talk to IntelliJ is all file path based.

It would still allow the analyzer/CFE to use the same URI to identify these library augmentations, without both the CFE/Analyzer actually trying to write to the same file. For debugging for instance, it should help. For example the Intellij plugin, when it sees the user pause at a breakpoint in one of these URIs, it would ask the analyzer for the file path for that URI, which would probably actually be some path under .dart_tool/analzer or some other analyzer cache. This would be the same underlying file the user would navigate to when going to definition etc, so any breakpoints they set, etc, would be there.
It also avoids any issues with a real (non-generated) file existing at whatever path we choose to represent macro APIs.

@DanTup

... which would probably actually be some path under .dart_tool/analzer or some other analyzer cache.

The analyzer team has a goal of attempting to find a solution that doesn't require writing these files to disk. If that proves to be impossible then, of course, we'll do what we have to do. But it would be better to work from the assumption that the files aren't actually written to disk until proven otherwise.

maybe org-dartlang-macro-augmentation although the scheme does get quite large. Please provide your suggestions for this :).

Is there a reason we need to use org-dartlang and not just dart? I don't think there should be much competition for these schemes (and I've often thought it'd be nice if the SDK sources came via dart: documents into the IDEs instead of paths from the SDK.. we have to do weird mapping in the debugger to make them map to the same SDK-based filepaths so breakpoints match up... we wouldn't need this if both server and debugger used these same URIs).
FWIW, I think I prefer option D from the above because it's less noisy/has less encoding (or, with file/package moved to the first path segment without additional encoded colons). I question if they need to be so verbose though... Is there anything wrong with dart-macro-package:a/a.dart or dart-macro:package/a/a.dart? Do we need augmentation if macro URIs are always augmentations? These URIs show up a lot in stack traces which are already quite verbose and individual lines already often wrap.

To prevent conflicts we could say that it is an error to have a non-macro generated files named xyz.macro.dart.
I like the option D, with the variant that the first path segment encodes the original URI scheme, e.g. dart-macro:package/a/a.dart. Also, I think these URIs are always absolute, there is nothing to resolve them relative to.
For debugging this would mean that IntelliJ would convert dart-macro:package/a/a.dart into the file path using execution.mapUri request, just as this happens for package URIs currently.

Is there a reason we need to use org-dartlang and not just dart?

I don't think so other than just trying to be very careful about not having conflicts. I tend to agree we could replace that portion with just dart. Historically we have used org-dartlang-* though for other custom external URI schemes.

FWIW, I think I prefer option D from the above

Same, I do think it is the nicest to look at and also very general purpose.

I question if they need to be so verbose though... Is there anything wrong with dart-macro-package:a/a.dart or dart-macro:package/a/a.dart?

Not with the former, but dart-macro:package/a/a.dart is problematic because unless we are specializing package: uris, then that would translate to package:/a/a.dart. This is because we do also see absolute uris (in file uris, http uris, etc), and for those we do want to retain the leading slash.
Also, for uris which have the // (usually file and http uris), we need to retain that. So that is why I specified that we URI encode everything before the path (so the scheme, the optional //, and the optional authority).
Basically, we have to be able to reliably, with no context, go back and forth between these URIs.

Also, I think these URIs are always absolute, there is nothing to resolve them relative to.

Package URIs are "relative", they don't have a leading slash. Originally they were relative to the "packages" dir. Now they are just very special in that the first segment defines the package, and the rest of the path is relative to the package lib root. We could treat them specially, and say we know they are always actually relative, but I generally prefer something that is fully generalized across all schemes.

Note also that for Option D, the // can be retained in the new URI easily. All you do is change the scheme and keep the rest, which is why in the example above file:///a/b.dart becomes org-dartlang-macro-augmentation-file:///a/b.dart. I like the simplicity of that a lot.

Ah, I think I get it. Prepending with dart-macro- is a better way to think about it.

Not with the former, but dart-macro:package/a/a.dart is problematic because unless we are specializing package: uris, then that would translate to package:/a/a.dart. This is because we do also see absolute uris (in file uris, http uris, etc), and for those we do want to retain the leading slash.

I'm not sure I understand. My idea was that anything starting with package/ or file/ or http/ would just be treated as if that was before the colon in the other option, except it wouldn't require registering many schemes with the client. It seems like we already need to special case every supported scheme to register with the client anyway, so saying the first segment after the : must be package/, file/ etc doesn't seem like a problem?

Note also that for Option D, the // can be retained in the new URI easily.

Yeah, I did like that. One possible consideration is whether the user/client will be able to browse/list these schemes. The VS Code API linked above was for document content, but there's a similar one for registerFileSystemProvider. I don't know the specific differences, but it might be that one of them shows up in the explorer tree (so the user could just browse these files directly, rather than only reaching them from Go-to-Definition etc.) so the difference between one scheme and multiple might impact how that appears. (I'm not sure we should design this based on VS Code's UI, but often other clients mirror VS Code's behaviour so it might be indicative of how things might appear across multiple clients).

Do macros only introduce one augmentation file per library?
(Otherwise I'm sure we can give them sequential numbers).
What if the generated augmentation file itself contains macro-annotations?
(Likely to happen for a higher level macro which generates built-value declarations.)
Dart files need a URI. It shouldn't need to have a "file name", as long as there is a way to go from URI to content. A need for a file name is an artifact either of our own implementation, or of integration with something else that insists on file names. (Relative to some file system, presumably.)
Some files do have file names, and we keep referring to them by their file name from the command line and IDE, and need to go back and forth between file name and URI for those, but that doesn't mean there cannot be files that only have a URI. As long as every Dart tool can access the file content, it should work.

Do macros only introduce one augmentation file per library?

Yes, just one per library (at the end at least). There may however be multiple that exist while macros are being ran (which then all get merged later). These could end up being visible to users if we aren't careful to hide them, if there are diagnostics reported by macros (including crashes), during this intermediate state where multiple files exist.
If we end up not being able to hide them, we would have to figure out a sequential numbering scheme or something along those lines in addition to the custom scheme.

What if the generated augmentation file itself contains macro-annotations?

One could argue that this should result in an additional augmentation library but it isn't specified to right now (although it may be left a bit ambiguous). I am not sure there would be a lot of value in it. The ordering rules wouldn't match up perfectly either with an equivalent user-generated augmentation which makes me nervous.

Dart files need a URI. It shouldn't need to have a "file name", as long as there is a way to go from URI to content. A need for a file name is an artifact either of our own implementation, or of integration with something else that insists on file names. (Relative to some file system, presumably.)
Some files do have file names, and we keep referring to them by their file name from the command line and IDE, and need to go back and forth between file name and URI for those, but that doesn't mean there cannot be files that only have a URI. As long as every Dart tool can access the file content, it should work.

Yes I definitely agree that we should keep everything in terms of URIs until the last moment possible. Any communication across tools should be in terms of URIs. Was there anything specific that is concerning to you, from in particular Option D?

What if the generated augmentation file itself contains macro-annotations? (Likely to happen for a higher level macro which generates built-value declarations.)

If code generated by a macro contains new macro annotations, these are added to the set of macro annotations to run, and all results are merged.

Code is here
import 'dart:async';
import 'package:_fe_analyzer_shared/src/macros/api.dart';

macro class AddClassA implements ClassTypesMacro {
  const AddClassA();

  FutureOr<void> buildTypesForClass(clazz, builder) async {
    final identifier = await builder.resolveIdentifier(
      Uri.parse('package:test/a.dart'),
      'AddClassB',
    );
    builder.declareType(
      'MyClass',
      DeclarationCode.fromParts([
        '@',
        identifier,
        '()\nclass A {}\n',
      ]),
    );
  }
}

macro class AddClassB implements ClassTypesMacro {
  const AddClassB();

  FutureOr<void> buildTypesForClass(clazz, builder) async {
    builder.declareType(
      'B',
      DeclarationCode.fromString('class B {}\n'),
    );
  }
}

@AddClassA()
class X {}

library augment 'test.dart';

import 'package:test/a.dart' as prefix0;

@prefix0.AddClassB()
class A {}

class B {}

@DanTup brought up a good point in discord that package: URIs are actually ambiguous, in particular when there are multiple projects open in one window (which results in multiple package configs and analysis contexts).
We could resolve that issue by saying that the scheme replacement is actually done after resolving any package: uris to their root uri.
So, in practice most of the time almost everything would end up as a dart-macro-file: uri.
As a concrete example, the macro URI for the package URI package:foo/foo.dart would be created by first resolving that uri using the package config, so given a package config like this:
{
  "configVersion": 2,
  "packages": [
    {
      "name": "foo",
      "rootUri": "file:///my/pub/cache/foo",
      "packageUri": "lib/",
      "languageVersion": "3.0"
    }
  ]
}

We would first convert the package URI to file:///my/pub/cache/foo/lib/foo.dart and then convert that to the custom macro URI scheme dart-macro-file:///my/pub/cache/foo/lib/foo.dart.

@DanTup do you know what the IDE does today if you click a package: URI in a stack trace?

It varies depending on where it shows up. Generally, VS Code knows nothing about package: URIs - we have never registered a handler for this scheme. However:

In the Debug Console we will often detect these (in the debug adapter) and attach some location metadata to the line (which appears as a clickable link on the right side of the debug console). This allows us to provide a file system path (and we use the VM Service's resolve methods to resolve the package URI to a file path).
In the Terminal, we register a TerminalLinkProvider which regexes package: links from output and uses its own lookup (some code inside the Dart-Code extension that locates and parses package_config files and just takes the first match when there are multiple matching packages because it has no context to know which one (if any) maps it has are correct).

BTW, I had a quick test of the two VS Code APIs that seemed like they might be useful here (FileSystemProvider vs TextDocumentContentProvider):
FileSystemProvider
A FileSystemProvider can provide a full fs implementation (including directory listings) and can be read-only or writable. It’s possible to include a virtual filesystem in the workspace to appear in the explorer:

Showing the filesystem in the explorer like this requires adding it as a “workspace folder”. While it’s possible to do this automatically, it creates an unsaved workspace that the user would be prompted to save when existing, so it’s not an ideal experience.
It’s also possible for a user to “open” folders from filesystem provider directly (code –file-uri dart-macro-package:foo/foo.dart) which doesn’t make a lot of sense with no actual Dart project to tie it to.
This API seems designed more around just providing remote filesystems that aren't tied to what else is open in the workspace, so probably isn't a good choice.
TextDocumentContentProvider
Using a TextDocumentProvider provides the same ability to provide text document (and notify VS Code when the content changes) and appears similarly in the editor (although tab tooltips do not show the scheme).

It’s not possible for these documents to be editable nor is it possible to browse a list of them. They can be returned in results from calls like Go-to-Definition and Sources for debug stepping.
This one would probably a better choice if we go down the route of not having real files on disk. I presume not being able to browse is probably not a big deal.
Note: Neither of these APIs are currently implemented in LSP and we might need to contribute one if we want to use it. TextDocumentContentProvider would likely be much simpler than a whole virtual fs.

We could still provide a custom way of navigating to the macro augmentation library for any given open file presumably?

Yep! Although whether it'll be standard LSP or not depends on the specifics (a "source" Code Action that jumps to the macro library for the current file might be possible within standard LSP).

Sounds good.

The definition of belonging to a package is to have a package URI for that package, or to be located in a subdirectory of the package root.

Is this written down somewhere? In package:test we had to recently do a fix to support somebody writing tests under lib (the merits of which are a separate discussion). But the compilers don't auto-convert file uris to package uris today, only the top level dart tool will do that. I wonder if we should file an issue.

But the compilers don't auto-convert file uris to package uris today

The analyzer does to this, because for it every file is potentially an entry point. Any file with path /home/my/lib/a.dart in a Pub package will be get (using package_config.json file) the corresponding package:my/a.dart.

Ok, we decided at the meeting to day to go with pretending macro- to the original library URI. This is short, easy to recognize, and easy to translate back and forth.
So, package:foo/foo.dart becomes macro-package:foo/foo.dart, and file:///C:/Foo/Bar.dart becomes macro-file:///C:/Foo/Bar.dart etc.
These URIs should be used whenever talking across tools or to users, etc. Internally tools may do translate that to whatever they choose (for instance analyzer will likely convert that to a .macro.dart file path, since it only deals in file paths).
I fill file issues on the implementations here and close this issue.

I'm sorry to be late here, but I think using macro-package:/macro-file:/etc as schemes is a mistake.
URI schemes are there to define the interpretation of the rest of the URI, everyting after :. The scheme itself should not carry "data" which can vary. If the goal is to carry the existing URI along, while stating that this is a macro introduced URI which must be resolved by asking the "macro service" for the source, then it should:

Use just one unique scheme.
Put the entire original URI into the data part of the URI, after the first :.

There is no requirement on the data part of the URI, its format is entirely defined by the scheme.
The Dart Uri class is written to assume that schemes are hierarchical, HTTP-like, with host/authority, path, query and fragment parts, duely separated. That sometimes makes it parse non-standard schems into different hierarchical parts, when all the user needs is the entire data as one. (We special-case package and data schemes for that reason.)
I suggest using, fx, dart-macro-generated:$originalUri, and think it will work significanly better than the proposed multi-scheme (with a technically open-ended number of possible schemes) approach.
Anyone can easily check whether the scheme is one of those (uri.isScheme("dart-macro-generated")), without having to go through dart-package, dart-file, dart-http, dart-https, dart-dart, dart-data, and whatever other URI we might support today or tomorrow.
Any text which can go into the scheme and authority part of a URI, can also go into the path, so conversion would be:
Uri toMacroGeneratedUri(Uri original) => Uri(
      scheme: "dart-macro-generated",
      path: "${original.scheme}:"
          "${(original.hasAuthority) ? "//${original.authority}" : ""}"
          "${original.path}",
      query: original.hasQuery ? original.query : null,
      fragment: original.hasFragment ? original.fragment : null,
    );

(We can probably do something more efficient internally if we know it's a _Uri object, so we don't need to reparse the already validated and normalized parts. Sadly can't use .replace since we cannot remove an authority.)

@lrhn To convert back then can you just do Uri.parse(original.path) consistently?

If there are no query and fragment parts, then I expect just parsing the path string should give the original back.
Will have to count the number of encoding+decoding that might be applied to weird characters to be absolutely sure.

To me the scheme conversion just seems a lot simpler and less error prone. You don't have to handle query params/fragments/authority specially, or worry about any encoding/decoding.
The macro-* scheme does in a way encode some "data", and yes it does lead to theoretically an unbounded number of schemes, however withing a given context the schemes are always well defined ahead of time. They are directly derived from the existing schemes, and can be explicitly enumerated for any given context.
When looking for a concrete definition of how schemes should be used, I couldn't find anything forbidding this. I did however find the web+ scheme prefix which is essentially the same as we are proposing here.
What if we went with macro+ instead of macro- to be more consistent with that?

Was a consensus reached about this?
(I do still think it'd be better if we had a dart prefix to avoid potential for collisions in the IDEs where we need to register ourselves as the provider for a scheme)

I personally still find the simplicity of using the scheme to be most compelling, maybe with dart-macro+ as the prefix?

I personally still find the simplicity of using the scheme prefix to be most compelling, maybe with dart-macro+ as the prefix?

Do you mean like dart-macro+package:foo/foo.dart? If so, these can be ambiguous and won't work for the URIs that go to the IDE (because when the IDE asks for the source of dart-macro+package:foo/foo.dart the server won't have any other context to decide which package it resolves under).
But perhaps that's ok. We have the same issue with package:foo/foo.dart today. The communication between LSP and the client (and DAP and the client) is currently always file:/// URIs, so it could be that we'll just now always be file:/ or dart-macro+file:/ in those places. If we need to resolve dart-macro+package:foo/foo.dart in the debugger (to send to the client as dart-macro+file:), we could first do what we'd do for package:foo/foo.dart to get to a file path, and then send it ask dart-macro+file:?

Do you mean like dart-macro+package:foo/foo.dart?

Yes, this is essentially the same as the macro- scheme prefix, just dart-macro+ as the prefix instead.

But perhaps that's ok. We have the same issue with package:foo/foo.dart today.

Right, I assume this has all the same edge cases as that does? If internally the protocol needs to speak in terms of file uris, that is fine, but users should see these dart-macro+<original-library-uri> uris in stack traces etc.
Presumably stack traces are also already ambiguous, but I don't think it makes the issue worse.

Presumably stack traces are also already ambiguous ...

Stack traces are unambiguous because there can't be multiple translations of a given URI within the running app. You do have to know which package_config.json was used by the compiler / runtime to do the translation in order to apply the same translation, but that's usually not too hard.
The problem is in the IDE where you could easily have two or more packages open, each of which could have their own interpretation of a given URI. The communication protocol has to disambiguate the URIs for the client, which is why we use file: URIs today. If a given URI resolves to two different locations on disk, then we can send those locations with minimal risk of confusion. (Yes, confusion is still possible, but we seem to be getting away with it in practice.)
But if we start sending URI's for macro generated code (because there are no files on disk), then I think we open ourselves to the same problem all over again. That's true even if they're dart-macro+ URIs.

I do think for users, this would be less useful to see, as they will become very long (pub cache paths).

Should we say that dart-macro+ uris should actually correspond to the resolved URI, for package uris?

Yes, I think we might need to do that in the communications between the analysis server and the IDE.
I agree that they would be less nice in stack traces (not sure where else they'd be visible to users), but we might also need to use the resolved paths in the debugger protocols because if we don't we might end up with the IDE asking server for information about a URI that server can't translate to a specific file. (That's conjecture based on a lack of knowledge about how this is all handled, so I'll let someone else confirm or deny.)

When we run a Dart file, we know its analysis context, so we know unambiguously how to resolve package: URIs.
So, we have API to resolve URI in execution domain of DAS.

(I'll just note that the API you're referring to is currently only available in the legacy protocol.)
That's definitely a possibility. We could use the server to disambiguate the URIs returned by the debug protocol.
The one downside to that that I can think of is that it puts the onus on the client to know that it needs to perform this translation, so it won't work in cases where the client is using LSP in a generic way. That seems less than ideal, but might be the best we can do. It would certainly be nice if we could have a common form of URI that didn't have that drawback.

I think we should be able to use dart-macro+file:/ and dart-macro+package:/ in the same way that we have file:/ and package:/ today. The analyzer would already know how to map between them, and everything out of (and back in to) the LSP server would be the file version (in the same way that today it's a file:/ URI and never package).
For the debugger, when a library is a package: (or dart:) URI, we use the VM Service's lookupResolvedPackageUris to convert it to a file: URI (or org-dartlang-sdk: which we then map onto the local SDK location) URI. I presume we could make this also work for dart-macro+package: (or dart-macro+dart:? *) and convert to dart-macro+file: (or dart-macro+org-dartlang-sdk:?) so this is symmetrical (@bkonyi does this seem feasible?).
I don't think we should have a dependency on the analyzer for debugging though.. That would take us way off the DAP spec and I'm still hopeful at the end of this that we can have completely standard LSP and DAP implementations so that all of this could work in other editors (assuming the implement the necessary features of the specs).
* - I didn't think about this before, but if you Go-to-Definition on an SDK source you currently end up in source files in the local SDK folder. If these files may use macros (I don't know?) and you Go-to-Definition to something generated by a macro, where do you end up? Will we have generated macro files we could send over LSP as dart-macro+org-dartlang-sdk:` or similar so we can map the debugger onto the same URIs? (we need Go-to-Definition and URIs from the debugger to match up to avoid users being able to see "two copies" of the same file, which have their own sets of breakpoints/etc.).

For now at least, we won't be using macros in dart: libraries. There may be a desire to though at some point, and we can try to answer some of these questions then? I think that probably though, the analyzer would have to actually run the macros on those SDK libraries, and we would not ship them somewhere in the SDK, although we could in theory.

For the debugger, when a library is a package: (or dart:) URI, we use the VM Service's lookupResolvedPackageUris to convert it to a file: URI ...

That ought to work. I assume we'd also use DAP to fetch the content of the file.
The only potential hole there is if the VM and the server have different ideas about the content of the same macro generated file. As long as they both use the same code to generate the content we should be safe there.

I don't think we should have a dependency on the analyzer for debugging though.

I agree.

That ought to work. I assume we'd also use DAP to fetch the content of the file.

Actually no, we need to return a URI from the debug adapter that VS Code then recognises as one that we're providing content for via the LSP server and it'll be fetched from the server the same as if you did Go-to-Definition via the analyzer to get to the same URI. This is the only way we can avoid the user having "two copies" of what the user considers the same file.
It's true that this relies on the sources in the debugger and server matching, but we already have that concern today. If you modify a file in the editor and don't hot reload it (or old versions of the code hang in the running app), what you see in the editor doesn't match what you're debugging. Unfortunately VS Code/DAP doesn't currently have a good way for us to handle this (the ideal IMO would be if the source match the one on disk is shown, and otherwise the user sees a note like "the source being debugged doesn't match on disk, download from the VM?" which is what Visual Studio did for .NET in that situation) but it's something I hope might improve in future.

For the debugger, when a library is a package: (or dart:) URI, we use the VM Service's lookupResolvedPackageUris to convert it to a file: URI (or org-dartlang-sdk: which we then map onto the local SDK location) URI. I presume we could make this also work for dart-macro+package: (or dart-macro+dart:? *) and convert to dart-macro+file: (or dart-macro+org-dartlang-sdk:?) so this is symmetrical (@bkonyi does this seem feasible?).

lookupResolvedPackageUris should work for mapping between any URI format we choose. In particular, we can map between the URIs defined by sourceUtf8Bytes (AKA resolved_url() on the VM's Script type) and importUriUtf8Bytes (AKA url() on the VM's Script type) in the SourceInfo type in the kernel binary specification.

Ok, for now I'm working on the following assumptions then...

When the analyzer users an internal path like /a/b/c.macro.dart (which is the macro file for /a/b/c.dart) I will translate it to dart-macro-file:///a/b/c.dart with something like Uri.file().replace(scheme: 'dart-macro-file', path: pathWithDotMacroRemoved). And when I get an incoming dart-macro-file: URI, I swap the scheme to file: and re-insert the .macro before .dart. All of these paths (between analyzer and IDE) will be file versions (because communication between IDE and server are always absolute file paths today, even for package files).

I'm assuming that in the debug adapter where I can map package:foo/foo.dart to the local file today (using lookupResolvedPackageUris) I will be able to do the same thing here to arrive at the same dart-macro-file:/// version of a macro-generated script (again, it will always be dart-macro-file: in the same way that today the debug adapter always arrives at a file URI even for packages).

Some kinds of changes (like the specific scheme or translation of .dart<->.macro.dart) are trivial. Others (for example if we can't arrive at a dart-macro-file URI in the debug adapter) may be more complicated.

Ok, in the meeting today we discussed this again and decided to go with the dart-macro+ scheme prefix. As an example, package:a/a.dart would become dart-macro+package:a/a.dart and file:///b/b.dart would become dart-macro+file:///b/b.dart.
We still could change this in the future if we run into issues with the actual implementation.
The specific schemes we definitely need to support are:

dart-macro+file
dart-macro+package

With possible support as well for the following:

dart-macro+dart (only if we end up supporting macros for dart: libraries which we probably won't)
dart-macro+data
dart-macro+http

We also have these multi-root schemes which we might someday want to support:

dart-macro+org-dartlang-app ?
dart-macro+org-dartlang-sdk ?
dart-macro+google3 ?
others?

Whoops, I forgot we said we'd use +. I'll update my stuff to be that :-)

I was just chatting with @jacob314 and we were discussing the idea of having a different way of displaying the URIs to users than the URI we need to use here. Right now, the URIs show up in places editor tabs like this:

But VS Code does give us some flexibility in how this is displayed (but not much - we can't run code, it's all declarative). For example for the screenshot above the label is set to "${scheme}://${path}" but if I change it to ${path}#macro then it becomes clearer from the editor tab that this is the macro generated content versus the user-owned file:

And the tooltip will look more like the one on the original file (not sure if we can remove the leading backslash on Windows, will file an issue):

One one hand, it would be nice if the URIs used by LSP/DAP are consistent with what the user sees. On the other hand, having something in the editor tabs like the #macro suffix makes the distinction between the user-generated and macro-generated files much clearer.

I like the editor tab without the #macro better; it looks less cluttered. I'm not sure about the tooltip, I kind of don't like either, but I don't know whether there are other options.
One question is whether it's important for us to denote for the user that the file is macro generated.
Can/are we marking these files as being read-only? If the user isn't able to edit the file then that might be enough of a hint without additional clues.
Do these files show up in the Explorer > Workspace view? If so what does that look like?

One question is whether it's important for us to denote for the user that the file is macro generated.

I'm not sure if it is, but the main advantage I see in the #macro suffix above was that you can tell the difference between the real danny.dart and the macro-generated file. Otherwise they both just have danny.dart in their tabs (and if you have a macro-generated file open, there's probably a reasonable chance you also have the user-editable file open too).

Can/are we marking these files as being read-only? If the user isn't able to edit the file then that might be enough of a hint without additional clues.

Yeah, these files are read-only - VS Code won't let you edit them (and the cursor is static and does not flash... although to me that makes it look a bit like the editor has frozen 🙃).

Do these files show up in the Explorer > Workspace view? If so what does that look like?

They show up in "Open Editors" but not in the file tree (since they're not in that tree). Looks like VS Code doesn't show the folder name next to them here either (I can't find any option to control this).
Without the suffix:

With the suffix:

Also slightly confusing there is that VS Code does not "unselect" the last file in the explorer tree when you switch to an editor that is not in the tree. So in both of the screenshots above, danny.dart inside bin is highlighted (in the bottom part of the left pane) but that is not the active file (the macro generated file is). I believe this is by-design, but it seems unfortunate so might be worth raising with VS Code.

Otherwise they both just have danny.dart in their tabs ...

Good point.
Could we have danny.macro.dart? That still looks cleaner to me than the pound sign. It does run the risk of colliding with a real file, but I think we've decided that very few users will create files with a .macro.dart suffix.
If we did that, could we associate a different icon with .macro.dart files?

... and the cursor is static and does not flash...

That's a very (overly?) subtle way of indicating that the file is read-only.

One question is whether it's important for us to denote for the user that the file is macro generated.

It occurred to me after clicking "Comment" that this is actually a broader question. Do users want to know which portions of the code were macro generated? For example, if the user hovers over a the name of a macro generated method, do they want the hover to identify the method as having been generated?

Could we have danny.macro.dart?

We can do that only if we include it in the actual URI (the formatting we can apply to the label is very limited). If we do that, we'd need to ensure that the URIs generated in the debugger also contain .macro.dart.
(It's possible that was the plan, but I had assumed until now that the use of .macro.dart in the analyzer was analyzer-specific and wouldn't exist in the runtime URIs)

If we did that, could we associate a different icon with .macro.dart files?

It might be possible to do this, but it may be a more complicated change. Icons appear to be tied to languages and not file extensions. It might be possible to define a new language for .macro.dart but it's not clear if files wouldn't match the original Dart one, and it might impact anything that is tied to the existing Dart language (both in our extension, and other extensions that might check the editor language).

For example, if the user hovers over a the name of a macro generated method, do they want the hover to identify the method as having been generated?

I hadn't thought about this, but I think we should. We already show URIs in some hovers, for ex:

So probably it's a bug if we don't show the "correct" thing here too :-)

I definitely see the advantage of showing it as thing.dart#macro in the editor tab/file tree. My only concern would be that users will see the dart-macro+ scheme URIs as well (in stack traces at least?). So it would be slightly inconsistent, but I think overall the benefits of the #macro style would be worth it from the examples you have shown.

My only concern would be that users will see the dart-macro+ scheme URIs as well (in stack traces at least?).

Yeah, I think if we should only do this if we can get it everywhere including those stack traces. I don't think how feasible that is (I think we're basically talking about removing the dart-macro+ prefix and adding a #macro suffix any time we display the URI to a user, and reversing it if we get it as input (for example handling clicking on links in terminal output).
Alternatively, if we don't think we could do this everywhere, we could just additionally tag #macro on the end for the editor only (as a slightly improved display on the tabs), but still have the original scheme at the start. We could teach users that the #macro is only for some convenience in displays like this and not really part of the identifier.

... adding a #macro suffix any time we display the URI to a user ...

If I understand what's being proposed, I think that might be confusing to users, given that # is a normal part of URI syntax. The URI isn't referring to a location within the file that contains the macro invocations, it's referring to a different file.

I was just discussing this with @jacob314 and we came to the conclusion that adding #macro at the end universally would likely be the most user friendly, especially if that is how editors are going to show it. At that point I think the editor doesn't even really have to do any special handling at all?
Basically, we prefix the scheme with dart-macro+ for tools, because having a custom scheme is helpful for registering handlers of these files, and we also add the #macro suffix for users because it is easier to see at a glance, in particular in file trees and editor tabs, that this is a macro generated file.

If I understand what's being proposed, I think that might be confusing to users, given that # is a normal part of URI syntax. The URI isn't referring to a location within the file that contains the macro invocations, it's referring to a different file.

Fwiw, TIL that # is an allowed character in file paths, at least on Linux but I think also windows (@DanTup can possibly confirm?). So, while it is a bit weird, it is also valid as a file path as well as a URI. I would be concerned though about tools parsing paths as uris, which converts that to a fragment, and then dropping the fragments when they go back to a path.

TIL that # is an allowed character in file paths ...

I didn't know that either.
I wonder whether something like danny.dart.macro might read better than danny.dart#macro. It wouldn't cause problems if tools parsed it as a URI, and it seems less likely to collide with a user-generated file name.

I wonder whether something like danny.dart.macro might read better than danny.dart#macro. It wouldn't cause problems if tools parsed it as a URI, and it seems less likely to collide with a user-generated file name.

I would be fine with that too, but I actually like that the #macro stands out as different. It is after all different, so standing out is I think a good thing.

If I understand what's being proposed, I think that might be confusing to users, given that # is a normal part of URI syntax. The URI isn't referring to a location within the file that contains the macro invocations, it's referring to a different file.

The URI would still have a different scheme, so it's not that we have two matching URIs except for the fragment meaning two different files, we would just have two different URIs and one happens to also have a fragment.
That said, we may have conflated two things here and it's worth being clear that there are two distinct (but potentially the same) variables here:

The URI that is used to identify the document (this will be what goes over-the-wire for LSP and DAP)
The label that is displayed to the user in the editor tabs/tooltips (this is some function of the URI, but we can only manipulate it in some specific ways)

(there is also a "name" provided by the debug adapter that we completely control, but I think we should try to ensure that that exactly matches (2), even though it's not quite what happens with non-macro files... this "name" can be package:foo/foo.dart in the debugger, but would be a real file path in the editor tabs/tooltips)
The URI and label could be exactly the same, but we can also have a difference (for example, we can append #macro (or  (macro)) in the label without it being part of the URI. I don't think we necessarily have to have the label match the URI used everywhere.
I do like #macro in the IDE labels, but I'm less sure about using it in the underlying URIs (in part because I don't know whether LSP/VS Code would really like fragments on URIs that are supposed to identify documents). If we're not sure we want the URI and label to be different, I agree maybe we should use .macro.dart instead (we already have that in the server, and currently strip it off when changing the scheme - but there's no reason we can't keep it).

IIRC, the only two characters you cannot put into a Unix file name, if you really try, is / and \x00.

I would be concerned though about tools parsing paths as uris, which converts that to a fragment, and then dropping the fragments when they go back to a path.

That's always a problem, one we have on the command line too.
File paths and URIs are so significantly different in the details, that accepting a string input which can be either, will almost certainly lead to some inputs being misinterpreted.
When the same input can be both and mean different things, it's just a good old-fashioned ambiguity. Either choice will sometimes be the wrong one.
If a tool does known-to-be-URI things using string operations, they are setting themselves up for surprises.
If you do Uri.file("/foo/bar/baz.dart#macro"), the URI text becomes file:///foo/bar/baz.dart%23macro.
If you just concatenate with file://, you'll have a bad time.
If we make sure to say what is a URI and what is a file path (and that there are resources that have no file path), then it's up to the tools to not mess up.
If we don't say which is which, they don't really have a chance.

+1 to what Danny said about 2 conversations happening here:

One is on the URI that will be used in the LSP/DAS protocol
The second is how much of this URI will be shown to users in various surface areas in the IDEs, i.e. the editor tab, debugger, hovers, completion, etc.


Synonyms in Dispatch

Tool
Name: Dispatch
Describe the problem
I'm using word alterations in QnA Maker. To keep the QnA KBs clean I type questions only containing one of these synonyms. I want Dispatch to refer to the correct QnA KB even when the user enters a synonym of the word that is used as direct question in QnA Maker
Describe the solution you'd like
Have the QnA Maker word alterations as phrase lists in Dispatch
Describe alternatives you've considered
Putting more different questions in QnA maker (but this defeats the purpose of uploading synonyms / word alterations
[enhancement]

Dispatch will be pulling synonyms from QnA maker.  Thanks for reporting this issue, Hans!

Merged with master, will be in npm end of April.  Thanks!


[BUILD] nanovdb viewer compile error with Optix

Environment
Operating System: Manjaro Linux
Version / Commit SHA: 4a186c2ed4fe3ea925b9c1c6251ac8d0ad3772f0
CMake Version: 3.22
Compiler: gcc 11.1, cuda 11.5, Optix 7.4
Describe the problem
/tmp/openvdb-nanovdb/nanovdb/nanovdb/cmd/viewer/RenderLauncherOptix.cpp:406:41: error: ‘OPTIX_COMPILE_DEBUG_LEVEL_LINEINFO’ was not declared in this scope; did you mean ‘OPTIX_COMPILE_DEBUG_LEVEL_MINIMAL’?
406 |     module_compile_options.debugLevel = OPTIX_COMPILE_DEBUG_LEVEL_LINEINFO;
In Optix 7.4, 'OPTIX_COMPILE_DEBUG_LEVEL_LINEINFO' doesn't exist anymore

There is a fix in #955


Notificatin UI heavily flickers if creating notifications in a tight loop

Testing https://github.com/Microsoft/vscode/issues/44319
        for (let i = 0; i < 50; i++) {
            let base = 100 * i;
            setTimeout(() => {
                vscode.window.showInformationMessage('Hello World One');
            }, base);
            setTimeout(() => {
                vscode.window.showInformationMessage('Hello World Two');
            }, base + 50);
        }

Yeah a bit ugly, the old solution had the same issue and so far was not reported:

I would like to stay away from introducing some odd debouncing logic.

java script
On Tue, Feb 27, 2018 at 3:06 PM, Dirk Bäumer<EMAIL_ADDRESS>wrote:

Testing #44319 https://github.com/Microsoft/vscode/issues/44319
    for (let i = 0; i < 50; i++) {
        let base = 100 * i;
        setTimeout(() => {
            vscode.window.showInformationMessage('Hello World One');
        }, base);
        setTimeout(() => {
            vscode.window.showInformationMessage('Hello World Two');
        }, base + 50);
    }

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/vscode/issues/44512, or mute the thread
https://github.com/notifications/unsubscribe-auth/Ain6PWUnW4POlkTqXNSbhI2ftcXakHvQks5tY9q7gaJpZM4SUtC_
.


Avoid using pyplot

We currently abuse pyplot a little. This PR avoids using pyplot, but it also breaks interactive plotting in Jupyter. Until that is addressed, it shouldn't be merged.
A better solution might just be to avoid using matplotlib in Juptyer altogether.

The fundamental issue here, identified by @tacaswell, is that the kernel cannot process messages while executing a cell. This is the best workaround available without significant changes to ipykernel.


.Net: Strongly Typed Agents

Some users may prefer to DI agents based on type.
Since our concete agents are all sealed, customer are unable to creat strong-typed agents:
class ProbeAgent : ChatCompletionAgent
class SupervisorAgent : ChatCompletionAgent

Other DI pattern exist:

Keyed Services
Agent Provider
Adapater Agent (unsealed)

Was going to log an issue on this but found this. IMO unsealing them and also adding a constructor option to pass Name and such in would help this pattern.
I recently have been using the agent types and my natural inclination was to create a class that represented my agent that encapsulates the configuration and functionality of my agent. For example what I would like to do is something like this:
internal class MyAgent : ChatCompletionAgent
{
    private const string INSTRUCTIONS = """
            you are a helpful assistant.
        """;
    public MyAgent(Kernel kernel) :
        base(name: "MyAgent", kernel: kernel, instructions: INSTRUCTIONS)
    {
        kernel.ImportPluginFromObject(this);
    }

    [Description("usefull tool")]
    [KernelFunction()]
    public async Task<string> MyFunction()
    {
        return "Hello World!";
    }
}

I could now inject other things that my plugins might use (a HTTPClient instance with pre-configured tokens to be used for example). I feel good because all the code for this specific agent is in one place and I can refer to it by its type. I can register it as MyAgent and refer to it that way or even as IEnumerable<ChatCompletionAgent> to get all agents if I want to write the code that consumes the agents in that way.
The way it is now almost forces me to use keyed services. Which is a fine pattern, just not the only pattern I can think of and wasn't the way I naturally started with this. My natural inclination was to create a class that represents my thing not create a bundle of config in my Program.cs that represents the thing. Is there something wrong with what I am describing that means it shouldn't be possible?

I think it makes sense to derive from Agent (or KernelAgent), but it seems better to not derive from something like ChatCompletionAgent. That bakes into your public surface area what's actually an implementation detail, that you're using ChatCompletionAgent as the implementation. If tomorrow you decided you instead wanted to use an OpenAIAssistantAgent, it would be a breaking change, and also publicly visible, to change the base type.
I think better would instead to use a decorator pattern, both deriving from Agent and containing / delegating to a ChatCompletionAgent, e.g.
internal sealed class MyAgent : Agent
{
    private readonly ChatCompletionAgent _impl = new ChatCompletionAgent(...);

    ... override ...
}

delegating wherever relevant to _impl.

@glennc - I've worked with some customers who are subclassing ChatHistoryKernelAgent and using composition (an internal ChatCompletionAgent) to address custom requirements around telemetry and instruction management.  This is simliar to what @stephentoub has described, but requires much less code to implement as the "channel" contracts are already addressed.


Can I add base64 as image type in image plugin

"Kung Fu Hero": {
_type: "image",
source: base64 string,
format: MimeType.base64,
width: 200,
height: 200
}
like this

Yes, you can add base64 images. See here how: https://github.com/alonrbar/easy-template-x/issues/40


responds to both horizontal and vertical scrolls

Describe the bug
I have a horizontal carousel in a vertical ListView and swiping vertically on the carousel usually blocks scrolling. The carousel should only consume gestures in the horizontal direction.
To Reproduce
Steps to reproduce the behavior:

<View style={{height: 1000}} />

<View style={{height: 1000}} />

Expected behavior
When scrolling vertically and the carousel should bubble vertical gestures to the parent
Screenshots
If applicable, add screenshots to help explain your problem.
Versions (please complete the following information):

react: v17.0.2
react-native: v0.66.4
react-native-reanimated: v2.9.1
react-native-reanimated-carousel: v3.0.3
react-native-gesture-handler: v2.5.0

Smartphone (please complete the following information):

Device: Samsung S20
OS: Android
Browser stock
Version 12

Additional context
Add any other context about the problem here.

There's a workaround for this (mentioned in the read me btw)
<Carousel
  {...}
  panGestureHandlerProps={{
    activeOffsetX: [-10, 10],
  }}
/>

Works for me, should work for you as well

The fix works, thanks @szymonwrobel


PTY detection issues.

Hello,
Thanks for your hard work! This is really cool.
We are integrating it to castor and be able to ship task runners as standalone lib.
But we face a strange bug with PTY:
With the static binary:
proc_open('echo 1 >/dev/null', [['pty'], ['pty'], ['pty']], $pipes)

Warning: proc_open(): PTY (pseudoterminal) not supported on this system

Actually, my term DO supports PTY.
>…om/jolicode/castor(add-compile-command *%) php -a
Interactive shell

php > var_dump(proc_open('echo 1 >/dev/null', [['pty'], ['pty'], ['pty']], $pipes));
resource(7) of type (process)

Thanks for reporting this bug.
I just looked for the source code and found that this line of patch code was added when the event extension was added ten months ago. https://github.com/crazywhalecc/static-php-cli/blob/cac3ffc7d7fb34bc05603571d2c2da099efeed6b/src/SPC/store/SourcePatcher.php#L214
This issue is related to event extension support on macOS: it will encounter openpty errors when compiling event extensions on macOS.
After removing this line, openpty just works. I will fix this after my vacation.

Warning: proc_open(): PTY (pseudoterminal) not supported on this system

The problem with PTY not being supported has been fixed, but I'm not 100% sure that solved your problem. I've merged the code into the main branch, you can test it with spc binary (nightly build) or test it with git clone.


Bug: Generated templates doesn't include mock for react-inline-svg for @grafana/ui

Package Name
create-plugin
What happened?
when using create-plugin it doesn't include the mock for react-inline-svg component defined here.
https://github.com/grafana/plugin-tools/blob/648d226cd34cecfaf840b26afe86166d29b9bf57/packages/create-plugin/templates/common/.config/jest.config.js#L11-L14

What you expected to happen
mock for react-inline-svg should be included in the config generated
How to reproduce it (as minimally and precisely as possible)
npx @grafana/create-plugin
? What is going to be the name of your plugin? something
? What is the organization name of your plugin? yesoreyeram
? How would you describe your plugin? some nice thing
? What kind of plugin would you like?  datasource
? Do you want a backend part of your plugin? Yes
? Do you want to add Github CI and Release workflows? No

cat yesoreyeram-something-datasource/.config/jest.config.js

Environment
System:
    OS: macOS 13.0.1
    CPU: (12) x64 Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
    Memory: 645.56 MB / 16.00 GB
    Shell: 5.8.1 - /bin/zsh
  Binaries:
    Node: 16.16.0 - /usr/local/bin/node
    Yarn: 1.22.19 - ~/.yarn/bin/yarn
    npm: 9.3.1 - /usr/local/bin/npm
  Browsers:
    Chrome: 109.0.5414.87
    Firefox: 108.0.2
    Safari: 16.1

Additional context
No response

Try as I might I cannot reproduce this issue locally. Looking at the prompts and screenshots suggests you're using an older version of create-plugin. Is it possible you installed it globally and npx is picking it up from there? Alternatively I think npx  caches things in ls $(npm config get cache)/_npx.
A quick way to test if it's an environment issue or a bug in the code would be to do npx<EMAIL_ADDRESS>and see if it continues to happen.
Here's the output from my runs locally. The react-inlinesvg.tsx mock is mentioned in the list of scaffolded files:
jackwestbrook@Jack's Macbook > ~/Sandbox/plugin-tools-tests > npx @grafana/create-plugin
? What is going to be the name of your plugin? something
? What is the organization name of your plugin? yesoreyeram
? How would you describe your plugin? some nice thing
? What type of plugin would you like? datasource
? Do you want a backend part of your plugin? Yes
? Do you want to add Github CI and Release workflows? No
? Do you want to add a Github workflow for automatically checking "Grafana API compatibility" on PRs?

Run the following commands to get started:

    * cd ./yesoreyeram-something-datasource
    * yarn install to install frontend dependencies.
    * yarn dev to build (and watch) the plugin frontend code.
    * mage -v build:linux to build the plugin backend code. Rerun this command every time you edit your backend files.
    * docker-compose up to start a grafana development server. Restart this command after each time you run mage to run your new backend code.
    * Open http://localhost:3000 in your browser to create a dashboard to begin developing your plugin.

Note: We strongly recommend creating a new Git repository by running git init in ./yesoreyeram-something-datasource before continuing.

    * View create-plugin documentation at https://grafana.github.io/plugin-tools/
    * Learn more about Grafana Plugins at https://grafana.com/docs/grafana/latest/plugins/developing/development/