[WIP] Entrypoints

This is very early, not even done in terms of base POC requirements, but functional enough to send out and get initial feedback. Don't take anything as finalized, lots of the details were optimized for speed and re-using existing code as much as possible.
Right now I've tested w/ Go and Python. NodeJS is a TODO.
Features/Demos
CLI for CI tasks implemented in Go

Expand
To help iterate on entrypoint features, I've begun migrating Dagger's CI automation to use entrypoints implemented in Go. The progress so far is in the new ci/ directory.
The targets are organized in a hierarchy of commands+subcommands, which can be explored from the CLI interface via the --help flags.
Top-level entrypoint ci:
$ dagger do --project ci/dagger.json ci --help
Loading+installing project (use --debug to track progress)...
Connected to engine fe69f84ad50b
Running command "ci"...
Usage:
  dagger do ci [flags]
  dagger do ci [command]

Available Commands:
  engine      Dagger Engine targets
  sdk         Dagger SDK targets

Flags:
      --branch string
  -h, --help            help for ci
      --repo string

Global Flags:
      --cpuprofile string   collect CPU profile to path, and trace at path.trace
      --debug               show buildkit debug logs
      --pprof string        serve HTTP pprof at this address
  -p, --project string
      --workdir string      The host workdir loaded into dagger (default ".")

Use "dagger do ci [command] --help" for more information about a command.

Available Commands shows the available targets (engine and sdk so far)
ci also takes some top level arguments for --branch and --repo which the entrypoint implementation uses to pass some global context down to the subcommands
Improvements Coming Soon™

Once default targets are implemented, ci won't even need to be explicitly typed in the CLI. It will just be the default target called when invoking dagger do
The --branch and --repo args are only needed because entrypoints don't yet have access to the caller's filesystem. They can be removed once that's added (though they are a good example of flags in the meantime)
Support for doc strings on arguments needs to be added (e.g. --branch string the name of the branch from the git repo)
Including all the dagger CLI global flags in the output is too noisy, they should probably be removed from this output
Having to explicitly pass dagger.json in the path should be replaced with optionally just passing the parent dir (e.g. --project ci/)

Subcommands:
There's multiple available and can also be explored with the --help flag on any subcommand.
Here's an example of executing one:
$ dagger do --project ci/dagger.json ci --repo github.com/dagger/dagger --branch main sdk go lint
Loading+installing project (use --debug to track progress)...
Connected to engine af5c7630141e
Running command "lint"...
level=info msg="[config_reader] Config search paths: [./ /app/sdk/go /app/sdk /app / /root]"
level=info msg="[config_reader] Used config file ../../.golangci.yml"
level=info msg="[lintersdb] Active 27 linters: [bodyclose depguard dogsled dupl exportloopref goconst gocritic gocyclo gofmt goimports goprintffuncname gosec gosimple govet ineffassign misspell nakedret prealloc revive rowserrcheck staticcheck stylec
heck typecheck unconvert unparam unused whitespace]"
level=info msg="[loader] Go packages loading at mode 575 (compiled_files|deps|exports_file|files|imports|name|types_sizes) took 6.012650031s"
level=info msg="[runner/filename_unadjuster] Pre-built 0 adjustments in 3.092193ms"
level=info msg="[linters_context/goanalysis] analyzers took 6.246671394s with top 10 stages: buildir: 3.220857137s, the_only_name: 1.504856241s, inspect: 249.091197ms, printf: 139.582094ms, fact_deprecated: 131.045815ms, ctrlflow: 130.746273ms, buildssa: 102.183487ms, nilness: 89.524337ms, fact_purity: 75.107843ms, typedness: 57.541827ms"
level=warning msg="[linters_context] rowserrcheck is disabled because of generics. You can track the evolution of the generics support by following the https://github.com/golangci/golangci-lint/issues/2649."
level=info msg="[runner] Issues before processing: 63, after processing: 0"
level=info msg="[runner] Processors filtering stat (out/in): path_prettifier: 63/63, cgo: 63/63, autogenerated_exclude: 57/63, identifier_marker: 57/57, exclude-rules: 5/57, filename_unadjuster: 63/63, exclude: 57/57, nolint: 0/5, skip_files: 63/63, skip_dirs: 63/63"
level=info msg="[runner] processing took 2.151097ms with stages: exclude-rules: 859.257µs, identifier_marker: 562.88µs, nolint: 393.962µs, autogenerated_exclude: 171.501µs, path_prettifier: 131.709µs, skip_dirs: 24.417µs, cgo: 3.291µs, filename_unadjuster: 1.375µs, max_same_issues: 500ns, uniq_by_line: 417ns, skip_files: 334ns, diff: 292ns, exclude: 250ns, source_code: 249ns, severity-rules: 207ns, max_from_linter: 124ns, max_per_file_from_linter: 84ns, sort_results: 83ns, path_prefixer: 83ns, path_shortener: 82ns"
level=info msg="[runner] linters took 3.22365722s with stages: goanalysis_metalinter: 3.221435368s, rowserrcheck: 6.833µs"
level=info msg="File cache stats: 16 entries of total size 121.5KiB"
level=info msg="Memory: 94 samples, avg is 145.9MB, max is 448.5MB"
level=info msg="Execution took 9.245905833s"

Improvements Coming Soon™

By default, no progress logs are shown, which can result in a long awkward silence as the command executes. You can enable them with the --debug flag, but they are the plain progress output and thus hard to follow. The real solution here should be to by default display them with our new fancy TUIs (cc @vito)

Entrypoint Implementation
All of the targets are implemented in the ci/ directory, which is really just a Go main package plus a dagger.json with some minimal metadata.
In the same way that Dagger pipelines are "just code", so are entrypoints. You just write some functions and the Dagger SDK automatically converts that into the CLI interface under dagger do.
Basic Example (no subcommands)
Before going into the ci command hierachy, here's a starter example where each entrypoint is just a function:
func main() {
	// provide all the entrypoints you want here
	dagger.Serve(
		Build,
		Test,
	)
}

// Doc string for my build
func Build(ctx dagger.Context, foo string) (string, error) {
	baseImage := ctx.Client().Container().From("golang:1.20-alpine")
	// ...
}

// Doc string for my test
func Test(ctx dagger.Context, bar string) (string, error) {
	// ...
}

Each function will become a command available to dagger do. Build becomes a build command, Test becomes test, etc.
The function's first arg is required to be dagger.Context. This implements the standard context.Context interface but also has a .Client() method with a pre-created dagger client to use.
Arguments become flags, e.g. build --foo $MY_FOO_VAL. The first ctx dagger.Context is excluded from that rule. It's also valid to take no args.
Improvements Coming Soon™:

Support for more input types besides strings. Will just be primitive types like integers and possibly lists/maps of those (for now...)
Support for just returning error rather than always having to return something else
The dagger Client should be pre-populated with Pipeline labels that match your entrypoint's name

dagger do ci implementation (with subcommands)
For Dagger's CI, we want a hierarchy of commands and the ability pass configuration+context from parent commands to subcommands. This doesn't end up complicating the code very much though.
In ci/main.go we define the top level ci entrypoint and the subtargets available under it, sdk and engine:
func main() {
	dagger.Serve(CI)
}

func CI(ctx dagger.Context, repo, branch string) (CITargets, error) {
	return CITargets{
		SrcDir: ctx.Client().Git(repo).Branch(branch).Tree(),
	}, nil
}

type CITargets struct {
	SrcDir *dagger.Directory
}

// Dagger SDK targets
func (t CITargets) SDK(ctx dagger.Context) (SDKTargets, error) {
	return SDKTargets(t), nil
}

type SDKTargets struct {
	SrcDir *dagger.Directory
}

// Dagger Engine targets
func (t CITargets) Engine(ctx dagger.Context) (EngineTargets, error) {
	return EngineTargets(t), nil
}

type EngineTargets struct {
	SrcDir *dagger.Directory
}

The CI func accepts the --repo and --branch flags we saw before in the --help output. It uses those with the dagger client to create a Directory object, same as any existing pipeline code would.
It needs to pass that Directory down to subcommands, so it returns a struct CITargets which holds configuration that should be propagated.

The subcommands, sdk and engine and defined by just adding methods to CITargets.
Rules for how methods are converted to commands in the CLI are the same as the plain funcs in the previous starter example
Due to the fact that they are methods on CITargets, sdk and engine have access to the Directory created by CI (in the SrcDir field).

In this case, the subcommands just pass that configuration down to their own subcommands; splitting them up here is more for organizational purposes.

The structs passed from parent->child commands don't show up in the CLI interface. They are purely used in the implementation.
These structs can be named arbitrarily and the only requirement for their fields is that they are json serializable.

The implementation of the rest of the targets are split up into separate files in the package. Here's the implementation of Python SDK targets (with some truncation for better readability, full file is in PR):
type PythonTargets struct {
        // configuration for targets
	RepoSrcDir *dagger.Directory
	SDKSrcDir  *dagger.Directory
	Base       *dagger.Container
}

func (s SDKTargets) Python(ctx dagger.Context) (PythonTargets, error) {
	const (
		path          = "/root/.local/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
		venv          = "/opt/venv"
		appDir        = "sdk/python"
		pythonVersion = "3.11"
	)
	sdkSrcDir := s.SrcDir.Directory(appDir)

	// We mirror the same dir structure from the repo because of the
	// relative paths in ruff (for docs linting).
	mountPath := fmt.Sprintf("/%s", appDir)

	base := ctx.Client().Container().
		From(fmt.Sprintf("python:%s-alpine", pythonVersion)).
		WithEnvVariable("PATH", path).
		WithExec([]string{"apk", "add", "-U", "--no-cache", "gcc", "musl-dev", "libffi-dev"}).
		WithExec([]string{"pip", "install", "--user", "poetry==1.3.1", "poetry-dynamic-versioning"}).
		WithExec([]string{"python", "-m", "venv", venv}).
		WithEnvVariable("VIRTUAL_ENV", venv).
		WithEnvVariable("PATH", fmt.Sprintf("%s/bin:%s", venv, path)).
		WithEnvVariable("POETRY_VIRTUALENVS_CREATE", "false").

       <rest of code...>

	return Python{
		SDKSrcDir:  sdkSrcDir,
		RepoSrcDir: s.SrcDir,
		Base:       deps,
	}, nil
}

// Lint the Dagger Python SDK
func (p PythonTargets) Lint(ctx dagger.Context) (string, error) {
	// TODO: pipeline should be automatically set
	c := ctx.Client().Pipeline("sdk").Pipeline("python").Pipeline("lint")

	eg, gctx := errgroup.WithContext(ctx)

	var poeLintOut string
	eg.Go(func() error {
		var err error
		poeLintOut, err = p.Base.
			WithExec([]string{"poe", "lint"}).
			Stdout(gctx)
		return err
	})

	<rest of code...>

	return strings.Join([]string{
		poeLintOut,
		poeLintDocsOut,
	}, "\n"), eg.Wait()
}

The (s SDKTargets) Python method defines the python subcommand on sdk.

In this case, it doesn't take any args (though it could if desired)
But it does do some work to take the SrcDir received from its parent command and create a common base image  that will be used across the rest of its subcommands, passed once again just through a struct (PythonTargets)

Then the final Lint target is defined as a method on PythonTargets, which reuses config from its parents and actually does the linting.

"Hello World" Python entrypoint

Expand
Entrypoints will be supported in all of our official Dagger SDKs. Similar to our clients, there is also a common API underneath them all which enables 3rd party implementations of entrypoints, both for different languages and even for languages already supported if different structures/approaches are desired.
You can see a very basic but working python example in demos/python-entrypoints. The targets can be explored with --help flags and then invoked in the exact same way as the previous examples with Go showed, e.g.
dagger do --project demos/python-entrypoints/dagger.json hello say --msg friendo
Loading+installing project (use --debug to track progress)...
Connected to engine af5c7630141e
Running command "say"...
Hello friendo!

The implementation uses the 3rd party strawberry framework for defining targets and command hierarchies (whereas Go uses some homegrown code).
While there are obviously differences between Python and Go, the general idea of how you structure your entrypoints is quite similar between the two and the low level API underneath them both (graphql) is the exact same, making it easy for the dagger CLI to handle both languages equivalently. The caller of entrypoint doesn't need any knowledge of what language they are calling.

Remote Projects with Git URLs

Expand
The previous examples were specifying projects that existed locally (i.e. `--project ci/dagger.json`).
There is also support for running projects that exist in a remote Git repository without the need to explicitly pull them down first, e.g.
dagger do --workdir 'git://github.com/sipsma/dagger.git#entrypoints' --project ci/dagger.json ci sdk go lint

There's a lot of potentially interesting use cases here. For one simple example, users could define builds for their binaries as Dagger entrypoints and then anyone who wanted to install it could just run:
dagger do --workdir 'git://github.com/org/repo.git#v0.1' install --output-path ~/bin

We obviously need a pretty locked-down-by-default security model here, though the fact that entrypoints run in containers help a lot. Access to the caller's host (import/export files, env vars, etc.) will need some form of explicit opt-in, perhaps similar to docker run actually.

Improvements Coming Soon™

The command is too verbose right now, need some sugar to enable more compactness in common cases
Using the --workdir flag here is probably not right as it is already used for something else, it will most likely change

External CI integration

Expand
I've done some very preliminary investigation on how this can integrate with external CI systems. Focus was on how much yaml we can convert to code inside and entrypoint and how we can integrate with CI events. There still some POC work left to do here, including converting more of our CI yamls over to entrypoints which will require actual handling of event payloads.

CI Event payload handling

It's already possible to invoke an entrypoint with an event payload, e.g. dagger do ci --event ${{ toJson(github.event) }} ...
That technically unblocks a lot of use cases, but leaves a lot of onus on the user parse the payload
This may be okay for now, but we could also add a lot of sugar here in follow ups. We could go all the way up to defining a common format for CI events that we automagically pass to your entrypoint in the dagger.Context (and similar in other languages).
There are many possible intermediate options too. Will save any of those implementations for follow-ups

What can't be converted to an entrypoint (yet)

Automatic distribution of an entrypoints execution to separate machines (i.e. still need multiple GHA jobs if you want that)
Direct execution on Windows/MacOS hosts ("direct execution" as is not cheating by executing inside docker/colima on macos, which is just a Linux VM)

Sugar via the dagger-for-github action

In the spirit of converting as much yaml->dagger as possible, I did a very quick trial of what an updated version would look like here (not published to marketplace, just used in my tests)

It has to do a checkout and install go in order to build the CLI, but the final prod version would just download a released CLI

So far, the only real magic this has added (besides installing the dagger CLI) is automatically setting up your project to use a git:// URL based on the env vars set by GHA (here)

This does let you eliminate use of the checkout action in your yaml. Besides the benefit of "less yaml" this is also more performant in that the engine directly pulls from Git (including using remote caching) rather than the repo being pulled and then synced as a local dir into the engine. It's rare for that to be a bottleneck, for particularly large repos it can make a meaningful difference.

Implementation Notes

Expand
I mostly re-used the pre-existing project+extension code from the previous prototypes. We should take this time to reassess it and update it as needed, but in general I do like being able to use the more generic extension plumbing as the internal, non-user facing foundation for this. It should set us up to quickly add new features like artifacts and eventually full-blown extensions.
One example of something to consider is the idea from a while back where invoking extensions could be implemented entirely client side as Container.WithExecs, which eliminates the need to stitch graphql schemas in the session. This has tradeoffs of course, but worth consideration now before we commit to one implementation or another.
The high-level way dagger do works is:

The code-to-schema implementation takes your code and converts it into a graphql schema. This schema is totally internal, users do not need to be aware of it.

For Go, this is the custom code we already wrote (with some new tweaks+refactoring)
For Python, this is just the 3rd party strawberry library

dagger do reads dagger.json, uses the project API to query the schema of the project and converts the schema into cobra commands.

Graphql Description metadata becomes the cobra usage string, the Run of the command just invokes the extension, etc.

TODOs
A few major TODOs (amongst many, many other smaller ones):

[x] Test with Python
[x] Think more about how this interacts with "event inputs" from various CI systems (i.e. pull request, manual workflow run, etc.), whether/how that can be used to automatically execute certain pipelines, whether we can use this to optimize caching, etc.

Broad topic, will break down as I think of more ideas around it. Would like to at least have a plan even if it's not fully implemented in this PR

[ ] Ability to have entrypoint interact with caller's Host (i.e. export/import files).

Right now if an entrypoint uses the Host API it refers to files inside its container. We want to have the ability to reference both the in-container Host and the caller Host.

[ ] Support for "default target" (comparable to e.g. all in make)

This likely ties into both integration with "event inputs" above. I.e. you may want your default target to be code that determines what even should run and then calls those.

[x] Subcommands
[ ] Think more about some monorepo-related issues (and break down into finer sub-todos)

E.g. hierarchies of dagger.json, entrypoints defined with different SDK languages in the same unified interface, etc.

[ ] Support for help docs on arguments
[ ] Support more primitives besides string such as int. Consider also supporting list and map of those primitives

Support for core types (e.g. Container, Directory as inputs/outputs) is out of scope for this entrypoint effort, will be tackled in follow-up projects.

[ ] Support for required vs. optional arguments

Minor TODOs (incomplete list):

[x] Pass dagger client direct to entrypoint funcs to reduce boilerplate
[ ] Automatically set pipeline labels on dagger clients (both Go and Python)
[ ] Flag for being able to see internally generated graphql schema to assist debugging
[ ] Cleanup code

There are some basic egregious things noted with TODOs in the comments throughout which make sense to clean up ASAP
There are also some slightly larger scale refactorizations that would also help a ton, particularly in sdk/go/server.go, which has grown "organically" for a bit too long now. I will save this for a follow-up most likely though.

Follow ups:

[ ] Integration with our TUI

Interactive lists of entrypoints, docs, etc. You can invoke them and see the output
Ask users for values when they don't provide a required arg

[ ] NodeJS
[ ] Reassess and flesh out dagger.json

I don't think the code for reading/writing it even exists anymore, if we continue going down that path need to re-add and keep iterating on it.

[ ] Reassess current project API

I was able to just re-use as is, but this is a good time to consider what shape we want this to ultimately take.

[ ] Optimize project load as much as possible. There's a very noticeable delay (~2-3 seconds) even when only calling --help on a subcommand and everything is cached.

suggestion from @robertu94 to cache schema in dagger.json may help a lot here

What is the UX for creating the JSON file here? From the examples it looks like it was pre generated. Additionally if the schéma is truly internal, then the use facing commands should either use executables or source files, and generate and cache the JSON on the fly.
Additionally the issue of the default target doesn't seem to be addressed. What is the thought there?

@robertu94 Thanks for the comments! I was planning on going point-by-point through the preceding issue today to pick up everything I missed in terms of requirements and suggestions, so apologies if any of my responses here overlap with what you've already suggested. Really appreciate all your feedback on this topic, it's genuinely incredibly helpful to have those overviews of the landscape and pros+cons of each approach!

What is the UX for creating the JSON file here? From the examples it looks like it was pre generated.

Yeah dagger.json is leftover from our prototypes a while back. The idea is for it to be primarily read+written through a CLI interface while remaining human readable if you want to inspect it directly. There are some examples of usage in the original PR, but take that all as very preliminary and up in the air.
Also, I need to revive the code in that PR as it got moved around and got deleted at some point as we shifted priorities over the last few months. So if you have any suggestions around this approach or ideas on totally different approaches now is the perfect time!

Additionally if the schéma is truly internal, then the use facing commands should either use executables or source files, and generate and cache the JSON on the fly.

dagger.json actually doesn't hold the schema (at least currently, maybe it should). It holds basic metadata like the name of the project, which language the project is using, etc. It also can hold dependencies this project has on others, but that whole aspect isn't yet needed for just entrypoints; it'll become relevant again when we expand out to support the more generic "extensions" idea.
The schema is generated internally (it's a graphql schema file) and actually is cached thanks to the fact that it's generated with a buildkit ExecOp (same mechanism behind WithExec in our core APIs).
That being said, we should make the schema viewable to assist power users and debugging. I added a TODO for supporting that behind a flag.
I do also like the idea that dagger.json could hold the generated schema... There is a problem right now in that even trying to list available entrypoints (dagger do --help) takes a few seconds even when its cached in buildkit. I'd like to see if we can reduce that latency, but caching in dagger.json rather than just buildkit actually may even simpler + more effective. Added a note about this to the description

Additionally the issue of the default target doesn't seem to be addressed. What is the thought there?

Great question, added addressing that to my TODO list in the description.
In general I'm seeing two possible routes:

Configure this in dagger.json
Configure this in your entrypoint code itself

The exact form this would take depends heavily on the language.
If the SDK language has some built-in notion of default methods or functions, that's the preferred route
After that, features like @annotations, decorators, etc. seem like the next best approach
And for languages like Go that don't have anything like that, we'll just have to use whatever we have left.

For Go specifically, there are conventions around "pragmas" that can annotate functions, but they are essentially just comments with a special syntax. That's probably the best route here unfortunately.

I'm undecided, but if I had to choose at this point I'd probably go with just using dagger.json for this since it's so simple and language agnostic. There'd probably be a subcommand along the lines of dagger project get-default-entrypoint, dagger project set-default-entrypoint <name of entrypoint>, etc.
Again, very open to any suggestions here! I recall you mentioning this in the other issue too, so I'll pick up your ideas there too when I go through that later today.

@robertu94 @sipsma on the topic of default command (equivalent of the default rule in a Makefile, or default target in Dockerfile): I don’t think we need that feature. neither npm nor mage have it. I don’t think it adds a whole lot in the context of commands; it makes more sense for build tools.
Just my 2c.

I love that this brings back the intuitive command-oriented UX of dagger do. Seamless support for subcommands is fantastic. On the DX, having simple functions with minimal boilerplate is perfect.
A few notes and questions:

I am now convinced that “command” is the right noun to describe what is invoked by dagger do. I forgot who suggested it but: it’s better than “script”, “task” and “action”.

Terminology: I propose that to designate the program that implements a project’s commands, we try using the term “controller” rather  than “entrypoint”.

Can I enumerate and invoke commands from the GraphQL client? For example could someone in the community easily develop a web UI or that does this? What would the graphql queries look like?

Related to the previous question: I think maybe when loading a controller, SDKs should not generate a dynamic schema, but rather dynamic resolvers for a common static schema. So the CLI would populate its cobra commands by sending simple queries like { project { commands(recursive: true) { name; description; flags { name; description; type } } } }. Note that type is an enum of possible cobra flag types, not graphql type.

Then to invoke a command: { project { command(name: “sdk”) { withStringFlag(name: “repo”, value: “github.com/…”) { command(name: “go”) { command(name: “lint”) { withBoolFlag(name: “strict”, value: true) { error; output; duration; } } } } } } }

Can controllers be stacked? I think they should. So for example each SDK could have a controller written in its own language (using an older stable release of itself to avoid bricking 😁), and dagger.json loads them all. The dagger do shows the combined output of the enumeration query for all controllers.

I think we should adopt the mage (and others) convention for subcommands: sdk:go:test rather than sdk go test. This frees up the grammar for other things:

Batch commands with flags only:dagger do sdk:go:test —branch main sdk:go:lint —strict
Batch commands with mandatory positional args only (like mage): dagger do sdk:go:test main sdk:go:lint true
Only command, with both flags and positional args. dagger do —branch main sdk:go:lint —strict ./src

I think access to host filesystem for each controller should be scoped to the directory that loaded that controller.

read access: project { directory } ?
write access: directory { export } with all paths relative to project dir? leading slashes ignored.

I think dagger do with no argument should show the list of available commands.

@robertu94 @sipsma on the topic of default command (equivalent of the default rule in a Makefile, or default target in Dockerfile): I don’t think we need that feature. neither npm nor mage have it. I don’t think it adds a whole lot in the context of commands; it makes more sense for build tools.
Just my 2c.

I've not used either mage or npm extensively, but here is where I think there is some value of providing a default: automated tooling and lower initial learning curve.
If there is a default, then there is a something consistent that an IDE, script,  or CI "action"/"stage"/"step" to call when integrating with Dagger instead of requiring tooling to ask what are all the available pipelines, and what their arguments may be. This means that integrations with 3rd party tools can be smaller at least initially.
As for a what the default is, I would suggest run all defined pipelines. This is semantically equivalent to what the make files generated by meson, CMake, and Autotools do [^1]. It's also what testing libraries like GTest, Junit, pytest, etc do by default, and I'd argue that dagger's primary intended use case is more like a testing library than Mage or npm. Also I believe npm test runs all tests by default.
[^1]: Technically make/ninja run the first defined target by default so that may be reasonable by default too, but for other reasons I think all is the better default

Yeah dagger.json is leftover from our prototypes a while back. The idea is for it to be primarily read+written through a CLI interface while remaining human readable if you want to inspect it directly.

I would hide it from the cli args, and instead just pass the project directory (it's implicit that a dagger project dir must have a dagger.json file at its root).
I love the simplicity of this proposal, it's a language agnostic mage and it even feels simpler to scaffold.

I would hide it from the cli args, and instead just pass the project directory (it's implicit that a dagger project dir must have a dagger.json file at its root).

@samalba Yes totally agree, should be a trivial update to make, will do

I am now convinced that “command” is the right noun to describe what is invoked by dagger do. I forgot who suggested it but: it’s better than “script”, “task” and “action”.
Terminology: I propose that to designate the program that implements a project’s commands, we try using the term “controller” rather than “entrypoint”.

@shykes Yes agree that "command" seems like the best match now that we have something concrete to look at.
In terms of "controller" vs. "entrypoint", I am more ambivalent between the two. I am actually wondering if we even need separate terminology here. Maybe we can just refer to the implementation of the command as a command too? Most of the time I think that would be unambiguous, and in the cases where it isn't, it's easy to say "command implementation" to be clear you are referring to the code implementing the command as opposed to the more abstract "command" you type after dagger do ....
As we start generalizing to concepts like artifacts, I feel like we could still just continue that pattern (i.e. you can refer to the "artifact implementation" if the need to disambiguate is strong).
That feels like what I would naturally tend to do anyways, but let me know what you think.

Can I enumerate and invoke commands from the GraphQL client? For example could someone in the community easily develop a web UI or that does this? What would the graphql queries look like?

Related to the previous question: I think maybe when loading a controller, SDKs should not generate a dynamic schema, but rather dynamic resolvers for a common static schema. So the CLI would populate its cobra commands by sending simple queries like { project { commands(recursive: true) { name; description; flags { name; description; type } } } }. Note that type is an enum of possible cobra flag types, not graphql type.

Right now dagger do is implemented by loading the project and performing a graphql introspection query to get the schema for it, which is used to create the cobra command structure. This was primarily because it required the least number of changes since it's essentially what our client codegen does already (just against the core schema rather than project schemas).
However, graphql introspection gives you very low-level information. It has everything you need technically but is not as easy to work with as something more high-level and static.
I think giving a higher-level static schema for commands specifically is a great idea. It not only saves work for anyone trying to build interfaces on top of this, it also formalizes the contract (i.e. what is a command, what properties does it have, etc.) in the API itself rather than trying to document the map between graphql introspection and commands, which sounds like a nightmare, both for the person writing the documentation and the people reading the documentation.
In terms of the implementation, I actually don't think we even need language-specific SDKs to do extra work for this. They can continue to just convert the code into a more generic graphql schema. Then, we can implement the higher-level static "command schema" in our project API.
The advantage there is that we only have to implement it once in dagger's internal project code rather than asking each SDK to generate the command schema. Then, as we expand out to follow ups like artifacts, environments, tests, and eventually full-blown extensions, the SDKs don't need any more updates. They continue to just output generic graphql schemas and we do the conversion to more the more well-defined static schemas in our project API.
I'll see how long the rest of the pre-merge cleanups+updates take and how much effort it would be to move to this model (it honestly might implicitly do some of the code cleanup I wanted anyways as a side-effect). Depending on that I'll either update this PR with it or do it in a follow-up ASAP.

Can controllers be stacked?

Yeah totally agree this needs to be possible, I was putting this in the bucket of "monorepo-related issues" to tackle.
One possible model is to use project dependencies as specified in dagger.json. But it also could be nicer DX to just recursively load all dagger.jsons found under a given filesystem tree. Maybe that's modeled as an "implicit dependency", or maybe that's just it's own thing.
This one is a big enough can of worms in terms of design that I think we should save it for a follow-up issue/discussion after this merging this PR (happy to hear more thoughts in the meantime though of course!)

I think access to host filesystem for each controller should be scoped to the directory that loaded that controller.

Also agree, at least in terms of defaults. I think there are cases where you may want to explicitly grant access to something outside that directory (one big one being host unix sockets, which typically aren't going to be in your cwd), but that would need to be a very explicit opt-in on behalf of the caller.
There's a very related discussion to be had around getting the env vars of the caller. Should be a way to for callers to explicitly pass them, but giving the command access to them all implicitly is gonna be too big a security hole.
I think there's a few possible ways of modeling the explicit passing of other host resources. We could put it in the project API such that command implementations can request them in code:
func MyCommand(ctx dagger.Context) error {
   // placeholder api
   yourSocket, err := ctx.Client().Project().CallerHostSock("/please/give/me/this.sock")
   // err is non-nil if it isn't allowed
}

Or we could even just make these be required args to the command, e.g.:
// We'd need to update our implementation to handle types like dagger.Socket as inputs.
func MyCommand(ctx dagger.Context, yourSocket *dagger.Socket) error {
   // use yourSocket...
}

Which would mean the caller has to run e.g. dagger do my-command --your-socket /i/grant/you/this.sock.
Same idea would apply to other host dirs outside the caller's cwd, env vars, etc.
I'm not sure which is best yet, but this is another thing which is probably best for a follow-up design discussion to this PR. As it stands, commands just can't access any of this, so we can loosen that up after figuring this all out.

(Also going to get back on the rest of the comments around more mage-like syntax and default commands, just sending this batch out first)

I think we should adopt the mage (and others) convention for subcommands: sdk:go:test rather than sdk go test. This frees up the grammar for other things:
* Batch commands with flags only:dagger do sdk:go:test —branch main sdk:go:lint —strict
* Batch commands with mandatory positional args only (like mage): dagger do sdk:go:test main sdk:go:lint true
* Only command, with both flags and positional args. dagger do —branch main sdk:go:lint —strict ./src

@shykes Have mixed feelings on this. I totally agree on the need for batching commands and positional args.
My concerns with the mage-syntax are:

I think that splitting by: may just be less intuitive to someone not familiar with mage relative to splitting out by space, which IME is the standard approach.

Is there any other convention for the target:subtarget syntax? I'm not aware of it besides mage but may be missing something
This is highly subjective too of course, just my opinion

Right now it's possible for each subcommand to take its own flags separately, but that would get a little less clean with a mage syntax

e.g. it's currently possible to implement something like this: dagger do sdk --foo=fooval go --bar=barval lint --baz=bazval, which feels it could be useful in a lot of situations
We could group all flags together like dagger do sdk:go:lint --foo=fooval --bar=barval --bar=bazval, which creates a gotcha in that you can't ever create any flags with the same name on commands+subcommands. Maybe fine? But nice to avoid gotchas when possible of course.
Similar to above, if subcommands each accept positional args, I'm not sure how you could specify them for each subcommand in an unambiguous way

Once we mix in positional args too, I think we might hit some weird parsing corner cases

e.g. dagger a:b c:d, if a:b accepts optional positional args, we need some way to differentiate between c:d being a separate command and a:b taking c:d as a positional argument
Maybe there's ways of doing that, would need to think more, but if not then I think we'd maybe end up having to do something like using -- to separate out commands, e.g. dagger a:b -- c:d, or something like that
This parsing ambiguity will actually also be true if we separate by spaces, so I don't see that either approach has an advantage over the other on this point

If we keep the space-separated syntax, we could support batching commands by separating by some special delimiter, e.g. -- as mentioned above

e.g. dagger do  -- sdk go lint -- sdk go test
Not 100% sold on -- specifically, this is slightly different than the convention it's normally used in, but feel like it's close enough to not be too weird
Maybe we could even support something fancier like splitting at subcommands, e.g. dagger do sdk --foo=fooval go -- lint -- test? Not convinced by that, kind of looks weird, but could work.

We need to figure out positional args for either syntax we land on. The idea in the back of my head has been to reserve a name args that has to be a list of strings, e.g.:
func Lint(ctx dagger.Context, args []string, someFlag string)

Commands wouldn't have to accept args, but if they do it will be set to any positional args provided by the caller.
Don't love reserving names, but it feels like an okay tradeoff as long it's kept to a small set (we also currently reserve the name help).

RE: default commands @robertu94 @shykes
A lot of considerations here (my own are shifting as I write this all out and think more).
I'll also throw in an additional motivation for something akin to this which is that in the current state of this PR you always have to explicitly type a top level command out even if it only exists to accept args and then branch out to subcommands

e.g. you have to type dagger do ci --branch main sdk go lint even though ci may be the only "top-level" command.
When there's only one top-level command, this feels like boilerplate and that you should be able to just type dagger do --branch main sdk go lint

There's definitely ways of addressing this that don't involve creating the concept of "default command" (and maybe they are preferable) but it feels like if such as concept were to exist it could be the model for solving the above problem, so it's bouncing around in my head with all this too.

As for a what the default is, I would suggest run all defined pipelines.

There are definitely a lot of cases where this would make perfect sense, but I think there's also a lot where it may not be desired. For our own CI in dagger, it could be nice to run all the tests+lints as the default command, but we will also have commands that push official release images, publish SDK versions, etc. which we don't want to run by default

We should of course put safety guards around the more "dangerous" targets such that they are skipped unless there's some explicit opt-in in one form or another (or the presence of secrets that developer machines don't have)
But my paranoia tells me that could become a footgun since it relies on the safeguard being totally correct; e.g. someone runs dagger do and due to a bug in their command implementation they accidentally publish a new prod image.
I guess this gets at what you said @shykes around this sort of convention making more sense in tools that only do builds, which aren't usually dangerous in the same way a tool that can could theoretically do deployments is.

If there is a default, then there is a something consistent that an IDE, script, or CI "action"/"stage"/"step" to call when integrating with Dagger instead of requiring tooling to ask what are all the available pipelines, and what their arguments may be. This means that integrations with 3rd party tools can be smaller at least initially.

I do totally see the point here, a default command could enable "config-less" integration with other tools. But, similar to above, the fact that what makes sense as a default command could vary widely from one project to another throws a pretty big wrench into that. I think it may end up being too unstable to rely on just always running the default and instead being more safe to just have a little bit of extra explicitness, at least for now.

Another thing that might help clarify the thinking here @robertu94, what we're implementing in this PR is just the first step in a series of related ideas (currently mostly just in discussions amongst the team, should be in writing in the near future).
For example, one feature we will likely follow-up with are what we are currently calling "artifacts", which would be the standard Dagger interface for specifying a DAG of containers, files, etc. that a project can build.
In the current imagination, a lot of the design may end up very similar to what we're doing with commands, e.g. it may just be that we also support returning Containers/Directories/Files from code (whereas commands only support primitive types):
func MyBinaryArtifact(ctx dagger.Context, debugSymbols bool) (*dagger.File, error) {
...
}

TBD what the CLI for this would look like, but as a placeholder you could imagine maybe dagger artifacts --export ~/bin/ my-binary-artifact.
I think at that time we would definitely look more closely at re-using conventions from build-specific tools. We will have a lot more leeway to enforce opinions, e.g. maybe we block internet access during the execution of artifact implementations in order to enforce hermicity of builds and eliminate some of the safety concerns around "I accidentally ran a command that published to prod".
There's also talk of formalizing interfaces to deployments, running tests, etc. What we're doing here initially is more of a low-opinion starting place.

In terms of "controller" vs. "entrypoint", I am more ambivalent between the two. I am actually wondering if we even need separate terminology here. Maybe we can just refer to the implementation of the command as a command too? Most of the time I think that would be unambiguous, and in the cases where it isn't, it's easy to say "command implementation" to be clear you are referring to the code implementing the command as opposed to the more abstract "command" you type after dagger do ....
As we start generalizing to concepts like artifacts, I feel like we could still just continue that pattern (i.e. you can refer to the "artifact implementation" if the need to disambiguate is strong).
That feels like what I would naturally tend to do anyways, but let me know what you think.

My main concern here is one of "technical marketing". When we explain Dagger, we quickly get to an important feature: it's programmable. To add awesome CI/CD capabilities to your project, you don't write a YAML file or other weird DSL: you write code. My problem is: what word to I use to designate the code to write?  My point is that, when explaining how Dagger works and how to use it in very broad strokes, before even getting into the specifics of command vs. artifact vs. test etc... I need a word to fill in the sentence: "To start using Dagger in your project, just choose an SDK and write your project's ".
In that specific context ☝ I think "controller" is a better word than "entrypoint", because it sounds more like a special kind of program that you created with an SDK. But, of course I'm open to discussing it.

I think giving a higher-level static schema for commands specifically is a great idea. It not only saves work for anyone trying to build interfaces on top of this, it also formalizes the contract (i.e. what is a command, what properties does it have, etc.) in the API itself rather than trying to document the map between graphql introspection and commands, which sounds like a nightmare, both for the person writing the documentation and the people reading the documentation.

Yay! Yes that's exactly right. It should be trivial for any tool to discover and invoke a project's commands, not just the official CLI. That means it should be easy to do those things via the API.

Can controllers be stacked?

Yeah totally agree this needs to be possible, I was putting this in the bucket of "monorepo-related issues" to tackle.
One possible model is to use project dependencies as specified in dagger.json. But it also could be nicer DX to just recursively load all dagger.jsons found under a given filesystem tree. Maybe that's modeled as an "implicit dependency", or maybe that's just it's own thing.
This one is a big enough can of worms in terms of design that I think we should save it for a follow-up issue/discussion after this merging this PR (happy to hear more thoughts in the meantime though of course!)

I agree that how exactly to make use of the stacking is a can of worms. Plenty of fun bikeshedding ahead :).

I was just thinking about "implicit dependency" over the week-end, and how that could be a useful tool in the toolbox - that is, if we can decide in what direction to do the walking :) I remember that CUE actually supports both directions depending on the context: up towards the root for package unification; and down "recursively loading under a given tree" for another type of evaluation, the details of which elude me.
I also think stackable controllers might be a way to ship extensions sooner than anticipated - ie. before supporting codegen. Think about it: if I can write a controller that, say, scans my project's package.json and dynamically exposes each npm script as a dagger command... What if I can just package that controller into an extension, so that you can do the same thing in your project? This would be "extensions lite", because you wouldn't get arbitrary pipeline composition. But it could still be quite useful. Something to think about :)

In short: lots of ways we could use stackable controllers, which we can figure out later. But, I think we should design them to be stackable from the get-go, to keep our options open.

My concerns with the mage-syntax are:

I think that splitting by: may just be less intuitive to someone not familiar with mage relative to splitting out by space, which IME is the standard approach.

Is there any other convention for the target:subtarget syntax? I'm not aware of it besides mage but may be missing something

That's a good question. I know of the Heroku and Mage CLIs. But I don't know how widespread or controversial it is beyond those two. I just asked the Internet, to help figure it out :)
If it turns out that using colons is controversial, then that would be a plus for using spaces.

Right now it's possible for each subcommand to take its own flags separately, but that would get a little less clean with a mage syntax

Not necessarily a dealbreaker, but I agree it's a plus for using spaces.

If we keep the space-separated syntax, we could support batching commands by separating by some special delimiter, e.g. -- as mentioned above

If we stick to spaces (probably the safer choice, unless colons turn out to be incredibly popular), the main benefit is familiarity; so I would be wary of even a little deviation like allowing multiple uses of -- instead of just one. That's a subtle but potentially super confusing change to anyone familiar with the usual behavior.
Looking at the examples, I feel like the answer to batching (assuming we stick to spaces) might be to rely on the shell's job control features: &, wait etc. It's also consistent with our approach to concurrency in SDKs :)

We need to figure out positional args for either syntax we land on.

Skipping the topic of positional args for now. I think it deserves its own issue actually. It's a small but crucial part of our CLI design, and is pretty decoupled from other topics discussed here, and even this particular PR.
Awesome conversations! As planned, having a real POC to look at has really unblocked us on so many dimensions of the design. Thanks for getting the ball rolling!

That's a good question. I know of the Heroku and Mage CLIs. But I don't know how widespread or controversial it is beyond those two. I just asked the Internet, to help figure it out :)
If it turns out that using colons is controversial, then that would be a plus for using spaces.

OK, I am getting good replies to that tweet, and a clearer picture is emerging:

There is in fact a strong convention of using colon as separators in build and build-adjacent tools. Originally started by Rake (ruby), and followed by NPM (javascript), Maven (java), Laravel Artisan (php), and Mage (go).
Original purpose was to simplify batching
I guess cascading flags are just not a thing in that model
Heroku is the only example of using this convention outside of user-defined scripts or build targets.

One reply says:

system” like Rake in which you’d want to specify multiple tasks to run in order, then go for it. otherwise i don’t think it’s a good idea, and Heroku probably just did it so it “looked more Railsy” since that’s their origins.

@sipsma @shykes

Is there any other convention for the target:subtarget syntax? I'm not aware of it besides mage but may be missing something

I believe that bazel, buck, and pants all use /.  My own dev-workflow  tool uses :.  My one piece of advice is don't use spaces, extra spaces sometimes get "eaten" by parsers and they can't be seen easily on slides or in talks.

But my paranoia tells me that could become a footgun since it relies on the safeguard being totally correct; e.g. someone runs dagger do and due to a bug in their command implementation they accidentally publish a new prod image.

This is a really good point.   I guess I didn't consider this because I assumed that the principle of least privilege (i.e. only CI or a few senior ops people have privileges to push to prod) would prevent the dev from accidentally pushing to prod, but that wouldn't be true in small teams.  In this case, the pipeline would "fail" on the image push, and otherwise do all of the "safe things".  Similar things would also apply to database migrations, website updates, etc...

But, similar to above, the fact that what makes sense as a default command could vary widely from one project to another throws a pretty big wrench into that. I think it may end up being too unstable to rely on just always running the default and instead being more safe to just have a little bit of extra explicitness, at least for now.

I think that is a feature, not a bug.  Different projects have different needs.  The other default that other tools (make, ninja mentioned before is first definition.  The things in these tools that have side effects (install, clean) are not defaults and would be analogous to what you describe)  If you want that to do multiple things you can do something like
async def default_pipeline(cxt):
     with TaskGroup(...) as tg:
            tg.start_soon(fedora_pipeline)
            tg.start_soon(centos_pipeline)

And then you have a config-less default, and a way to ensure that only "safe" things get run as defined by the project.

maybe we block internet access during the execution of artifact implementations in order to enforce hermeticity of builds

this is an appealing feature at first blush, but really hard to work around in environments that weren't designed to have this from  the beginning.  A shocking number of packages need internet access during their configure stage.  That said, it is a feature that I look forward to 😄

Maybe there's ways of doing that, would need to think more, but if not then I think we'd maybe end up having to do something like using -- to separate out commands, e.g. dagger a:b -- c:d, or something like that

I've seen several tools do this, and I think it is a reasonable idea.

In the current imagination, a lot of the design may end up very similar to what we're doing with commands, e.g. it may just be that we also support returning Containers/Directories/Files from code (whereas commands only support primitive types)

I like where this is going, but I would need to think about possible footguns and issues.  My first thought is getting the error-handling story right, but I really haven't thought more about this.

@sipsma @shykes

Is there any other convention for the target:subtarget syntax? I'm not aware of it besides mage but may be missing something

I believe that bazel, buck, and pants all use /.  My own dev-workflow  tool uses :.  My one piece of advice is don't use spaces, extra spaces sometimes get "eaten" by parsers and they can't be seen easily on slides or in talks.

But my paranoia tells me that could become a footgun since it relies on the safeguard being totally correct; e.g. someone runs dagger do and due to a bug in their command implementation they accidentally publish a new prod image.

This is a really good point.   I guess I didn't consider this because I assumed that the principle of least privilege (i.e. only CI or a few senior ops people have privileges to push to prod) would prevent the dev from accidentally pushing to prod, but that wouldn't be true in small teams.  In this case, the pipeline would "fail" on the image push, and otherwise do all of the "safe things".  Similar things would also apply to database migrations, website updates, etc...

But, similar to above, the fact that what makes sense as a default command could vary widely from one project to another throws a pretty big wrench into that. I think it may end up being too unstable to rely on just always running the default and instead being more safe to just have a little bit of extra explicitness, at least for now.

I think that is a feature, not a bug.  Different projects have different needs.  The other default that other tools (make, ninja mentioned before is first definition.  The things in these tools that have side effects (install, clean) are not defaults and would be analogous to what you describe)  If you want that to do multiple things you can do something like
async def default_pipeline(cxt):
     with TaskGroup(...) as tg:
            tg.start_soon(fedora_pipeline)
            tg.start_soon(centos_pipeline)

And then you have a config-less default, and a way to ensure that only "safe" things get run as defined by the project.

maybe we block internet access during the execution of artifact implementations in order to enforce hermeticity of builds

this is an appealing feature at first blush, but really hard to work around in environments that weren't designed to have this from  the beginning.  A shocking number of packages need internet access during their configure stage.  That said, it is a feature that I look forward to 😄

Maybe there's ways of doing that, would need to think more, but if not then I think we'd maybe end up having to do something like using -- to separate out commands, e.g. dagger a:b -- c:d, or something like that

I've seen several tools do this, and I think it is a reasonable idea.

In the current imagination, a lot of the design may end up very similar to what we're doing with commands, e.g. it may just be that we also support returning Containers/Directories/Files from code (whereas commands only support primitive types)

I like where this is going, but I would need to think about possible footguns and issues.  My first thought is getting the error-handling story right, but I really haven't thought more about this.

I need a word to fill in the sentence: "To start using Dagger in your project, just choose an SDK and write your project's (blank)".

In that specific context ☝ I think "controller" is a better word than "entrypoint", because it sounds more like a special kind of program that you created with an SDK. But, of course I'm open to discussing it.

Ah sure, that makes total sense... In that context yeah I agree it's helpful to have separate terminology and that "controller" is more descriptive than "entrypoint".
I'll think about it some more, but right now I don't have a better suggestion than controller.
I don't know entirely why I have this word association with "controller", but some reason I feel like if I was reading documentation and it said e.g. "Now you just need to write the controller code for your project" I would start getting mentally prepared for writing a bunch of complicated boilerplate-y code to integrate with a framework.
 But this is highly likely to be very particular to me and also isn't a big deal in the first place since our docs can just have examples right away showing how simple it actually is to eliminate that concern.

That's a good question. I know of the Heroku and Mage CLIs. But I don't know how widespread or controversial it is beyond those two. I just asked the Internet, to help figure it out :)

Awesome, thank you, incredibly helpful! Mixed opinions of course, but by eye there is a lot more opposition for : than support and the majority of the support is fairly lukewarm/caveated (with some exceptions).

Looking at the examples, I feel like the answer to batching (assuming we stick to spaces) might be to rely on the shell's job control features: &, wait etc. It's also consistent with our approach to concurrency in SDKs :)

Yeah that comparison is spot on I think. Basically, instead of fighting the language, our baseline should be to accept the conventions that exist and only layer on extra niceties later after a lot more consideration and feedback. Shell job concurrency and tools like xargs don't result in what I'd call beautiful code, but that's just shell code being shell code.

This is complicated by the fact that there are indeed mixed conventions in this case, : is not a one-off as it turns out and has advantages. But just based on the feedback so far the ~averaged sentiment probably leans more towards POSIX-y space separate commands and --flags being the "standard".

The cool thing too is that if a user ends up with a big pile of shell, they should also be able to just refactor to add a new command that implements the same thing, but it will be in the language of their choice with cleaner syntax and better type safety. Then they can just call that from the CLI instead.

This will currently be complicated by the fact that this is only really easy if you are the person controlling the command's controller code. But in our path to extensions, we will need to pick up features that let you import/wrap controllers across any SDK language, which should make this easier? Worth thinking more about this point.

Either way, I think that last point about enabling rapid iteration and easy paths to cleaner refactorization ("incremental revolution" 🙂) will be crucial to get right. Once we get there, should probably be a guide for doing exactly that.

I believe that bazel, buck, and pants all use /. My own dev-workflow tool uses :. My one piece of advice is don't use spaces, extra spaces sometimes get "eaten" by parsers and they can't be seen easily on slides or in talks.

All the above being said, these concerns are totally valid too. If we stick with spaces, users will have to contend more often with the rules around wrapping "space separated values" in "" or '' (definitely a common gotcha, myself included) and the readability tradeoffs.

My current synthesis of all of the above is that I'm inclined to continue with spaces for this PR and skip built-in batching support for now.
dagger do will still be hidden so we'll have time to pivot to : or similar if desired. As we start prototyping with this more, especially on more and more complicated use cases, it will be easy to see if using spaces results in a mess that could be cleaned up with : or other approaches.
And if we do end up sticking with spaces after officially releasing this, I think there will still be the option to layer on batching via -- or similar in the future in a backwards compatible way (or at least in a way that only breaks a tiny set of corner cases).

Here’s my summary of the “ask the internet about colon-separators” thread

Super excited for this! Also this is a really fun discussion. Finally found time to catch up.
@sipsma I'm keen to integrate Progrock into this now that it's merged. I can submit a PR to your branch so it doesn't get in the way or get push -fed.

re: "controller" cc @shykes

I don't know entirely why I have this word association with "controller", but some reason I feel like if I was reading documentation and it said e.g. "Now you just need to write the controller code for your project" I would start getting mentally prepared for writing a bunch of complicated boilerplate-y code to integrate with a framework.
 But this is highly likely to be very particular to me and also isn't a big deal in the first place since our docs can just have examples right away showing how simple it actually is to eliminate that concern.

FWIW I was about to comment the exact same thing. My mind jumps to Kubernetes CRDs and custom controllers which makes it seem more complicated than it really is.
I also don't have a better suggestion yet, other than maybe not having a special word for it strengthens the point that "it's just code," even if it results in clunkier language. But ideally we can find a word that's easy to guess.

re: batching & positional args
I think it would be really nice if it was easy to mimick this pattern in Bash:
#!/usr/bin/env bash

go test "$@"

For me this is an important ingredient for really embracing Dagger as your portable dev toolchain, so it can abstract away any complicated setup without getting in the way of running more high fidelity commands just as you might with go test.
There have been a lot of times where I want to do e.g. ./hack/make engine:test ./core/integration -run MyTest but can't because Mage doesn't support it.
For me, batching is a lower priority than positional args, because as @aluzzardi pointed out:

Or maybe, a simpler/more naive approach is to do this in userland (e.g. define "virtual" targets that just invoke the actual targets)

Personally I'm unlikely to write dagger run foo:bar baz:buzz more than once without immediately making a virtual alias, since they're so easy to add, they're much more flexible, and they help you codify your development practices across the team.
So my vote would be to support positional args in the common way, where any extra args get passed along, and any flags passed as positional args require -- for disambiguation from the entrypoint flags. It would be extra nice to not need to use -- somehow, but I don't think that's a dealbreaker; it's a little bit of friction but not an unfamiliar one.

re: colon vs. spaces
When used in combination with positional args, space-separated command "paths" can make it hard to tell where the command path ends and its args begin:
dagger do sdk go test ./foo # am I running 'sdk go' 'test ./foo' or 'sdk go test' './foo'?
dagger do sdk:go:test ./foo # pretty clear that `./foo` is the only argument

Of course, using -- makes things very clear, but it'd be nice to not have to type it in simple cases. We could probably avoid it by seeing that we're at a "leaf" in the command hierarchy.
You could also argue that this is a feature, not a bug. It's kind of nifty that sdk go test ./foo looks like go test ./foo! Maybe we would end up with a convention where your command name matches the primary commandline that it runs, so it naturally flows into its arguments?
Another option could be / since it matches how we've been displaying pipeline paths in a few places (sdk / go / test). Just throwing that out there to see if there are strong reactions in favor or against. :)
dagger do sdk/go/test ./foo

One drawback is it could look like it's running a ./sdk/go/test script. Looks like Bazel avoids this by putting // at the front, making it just funky enough to raise suspicion.

Pushed another batch of updates, think it's mergeable at this point with plenty of follow-ups to come soon after.

Biggest change is a total refactor of the project API (main commit). Now includes schema for commands, which is used by dagger do to create its command structure (cc @shykes)

Main schema file here
I ran into some bugs in how generated go code handles returned lists of objects. Fixed one and just worked around another for now, can fix separately

Another significant change is to how we handle various paths related to workdir and projects.
Previously I was abusing workdir to be what's really the project root, which totally breaks down in meaning with projects specified by git. Now, concerns are more cleanly separated and defined, but it's too complicated and needs improvements in follow-ups. It's at least reflecting reality now.
Turns out there's really 3 paths in play, at least right now:

The caller workdir

It's possible we may stop caring about this in the near-ish future, but it's still a thing right now

The project "root dir"

This is what will be loaded as the root source dir for the command code, for both local projects and git projects

This is fairly often not going to be the directory that holds dagger.json. For instance, if you want to define your Go commands in a subdirectory of your repo, you need this project root dir to be set to the directory that actually holds your go.mod. Otherwise your command code won't be buildable.

This is not the same thing as workdir in the case of projects specified by git url, in which case the workdir is still local to the caller but the project root is a git repo

This distinction matters in cases where the project might want to interact with the caller's host but the project is pulled from git

This is now specified with the flag --project (or -p)

The project "config path"

This is the path to the directory that holds dagger.json. As mentioned above, it's very common for this to not be the root dir (if not the majority of the time)
This is now specified with the flag --config (or -c)

In general, I really don't like the complication here, but it's a current reflection of reality until we make larger changes that would expand the scope of the PR too much. Some possibilities:

If we require a dagger.json to always be present in the project root dir and update it to allow specifying subpaths to the actual command code, then we can get rid of the distinction between 2 and 3.

I don't know if we can really always force users to put a file in the root of their repo though (maybe it's a big monorepo they don't have full authority over)

We could get rid of the concept of workdir.

We still need to figure out how commands can get access to the caller's host, but if we went with the idea I mentioned at the bottom of this comment, then I wonder if the concept of "." needs to be embedded in our api anymore. It would just be a value passed through CLI and normalized to be absolute before it reaches our engine/core code.

For me, batching is a lower priority than positional args

If we supported flags of type "list of string", then you could skirt around the need for positional args by just accepting a flag with that type. The only difference would be you need to type dagger do foo --bar 1 2 3 instead of just dagger do foo 1 2 3.
What do you think of that? How painful is it? Asking because positional args feel like they throw a wrench in a lot of other stuff, so I'm looking for ways that we could just avoid them without making the UX way worse.

Also, will get back on colons vs. spaces tomorrow. Have more thoughts and may give a shot at trying to switch to colon just to see how hard it is (or cut that short if it gets too complicated)

Did some more thinking and incorporated all the feedback above on the topic of spaces vs. colons (thank you everyone for your input!) and wrote out some side-by-side comparisons to try to get a more concrete idea on the tradeoffs.
Flags
Spaces

dagger do ci --repo abc sdk go lint --strict
dagger do ci --foo=123 engine test --foo=456

You can use flags with the same name on different commands

Colons

dagger do ci:sdk:go:lint --repo abc --strict
Not possible to use flags with same name on different commands in the hierarchy.

I tried thinking of a realistic case where this would actually matter, but I couldn't think of one. So maybe a good indication it's not big deal in practice unless anyone else thinks of something

Positional Args
This can also be broken down into required and optional positional args, which matters in terms of parsing ambiguity.
Right now, it's pretty fuzzy how we could allow users to specify whether an input is a required flag, an optional flag, a required positional arg, an optional positional arg, etc.

In Python it would probably be type annotations
In Go, it may end up being wrapper types around all the inputs we want to accept, e.g. type RequiredPositionalString string, which is ok I guess but feels unpleasant

There is also a fallback option here, which is to not support positional args. If we add support for flags of type "list of string" then you can get something close to the same end effect as positional args, with the tradeoff of a little extra verbosity (--foo a b c instead of just a b c) and less flexibility in terms of required vs. optional.
Spaces

dagger do ci build ./cmd/mybin

./cmd/mybin is a required positional arg to the build command

dagger do ci test ./pkg/a ./pkg/b

./pkg/a and ./pkg/b are optional positional args (maybe default of no args is test all packages or something like that)

dagger do ci ./pkg/b lint rule123

positional arg on both ci (scope all subcommands to ./pkg/b) and lint (only test for linting rule 123)
Note that if the positional arg on ci was optional, it's not clear how to do unambiguous parsing in a clean way since we'd have to figure out whether lint was a positional arg or a subcommand

Colons

dagger do ci:build ./cmd/mybin
dagger do ci:test ./pkg/a ./pkg/b
I don't think it's possible to accept positional args for anything but "leaf" commands. I can't think of a sane way to let them propagate to subcommands. This may be fine though.

Batching
For both options, there's the fallback of just not supporting this in the CLI syntax. Users can instead:

just write command code that does the batching they want
use shell job control and tools like xargs to parallelize

Spaces
Hypothetical example of using -- to split separate commands:
dagger do -- ci --repo abc sdk go lint --strict -- ci --repo abc sdk python lint --disable-rule=pep123
Colons
dagger do ci:sdk:go:lint --repo abc --strict ci:sdk:python:lint --repo abc --disable-rule=pep123

Now that I've written this out, I'm seeing pretty clearly how the colon approach does in fact often result in much better readability. This side by side comparison from above is pretty telling:

dagger do ci --repo abc sdk go lint --strict
dagger do ci:sdk:go:lint --repo abc --strict

The fact that all the flags get combined at the end is just way easier for my brain to parse.
And the tradeoff of commands/subcommands not being able to have overlap in terms of flag names is probably a very obscure issue with straightforward workarounds (just use different names for the flags).

So, I'm going to reserve my right to flip-flop and now be in favor of using : 😄. We should also consider other separators like @vito mentioned (e.g. /), but that's presumably something we can play around with very easily after migrating from spaces.
Please let me know if you have any more thoughts on this! I'm going to try out the implementation, but it's not a concluded topic yet by any means.

@sipsma Switching to : seems good to me. It's a lot easier to tell at a glance what value is going where. :+1:
re: passing args as a single flag, it seems like you would have trouble passing flags through it:
dagger do engine:test --args ./core/integration -run Foo

You could think to quote it, but then you're really just passing one big arg:
dagger do engine:test --args './core/integration -run Foo'

IMO -- is still the clearest way to handle this.

Right now, it's pretty fuzzy how we could allow users to specify whether an input is a required flag, an optional flag, a required positional arg, an optional positional arg, etc.

I wonder if we could use generics as type hints for this. Would something like this be feasible?
func Cmd(foo string, barOpt dagger.Optional[string], baz ...string) error {
  bar := barOpt.Or("default")
  // ...
}

Allows running like this:

dagger do cmd --foo x --bar y a b c

calls Cmd("x", dagger.Optional[string]{&"y"}, "a", "b", "c")

dagger do cmd x a b c

calls Cmd("x", dagger.Optional[string]{nil}, "a", "b", "c")

dagger do cmd x --bar y a b c

calls Cmd("x", dagger.Optional[string]{&"y"}, "a", "b", "c")

dagger do cmd x

calls Cmd("x", dagger.Optional[string]{nil})

dagger do cmd x -- a b c

calls Cmd("x", dagger.Optional[string]{nil}, "a", "b", "c")

Basically:

Any non-optional function arg may either be a required arg or a required flag.
A splat function arg takes any remaining unparsed arguments.
A dagger.Optional function arg can only be passed as a flag. This is a compromise to prevent ambiguity with splat args.

I don't think this is perfect - it's hard to have something really clear that demonstrates all of this - but maybe some of it us useful.

re: passing args as a single flag, it seems like you would have trouble passing flags through it:

@vito Ah okay, in retrospect I didn't fully grok your previous comment about this at the time but I totally see what you're saying now; when you want to just wrap other 3rd party CLIs like go test you need to pass in values to the command which are actually just flags, and you want to make sure they are not interpreted by the dagger do parser.
I guess the only other option is to write your command signature such that it accepts all the flags that the command you are calling accepts (i.e. every flag for go test), which is sometimes a valid option if you want to enforce what can and can't be called. But if you really just want to make a transparent wrapper, that would become a nightmare. Using -- to disambiguate those cases makes total sense.

I wonder if we could use generics as type hints for this. Would something like this be feasible?

Yeah that's another good option for Go. I guess the difference would be whether we codegen pre-made wrapper types for every permutation of type:

dagger.OptionalString
dagger.RequiredInt
etc...

Or just provide some handmade generics that cover all of it.
In most "normal" cases, generics would definitely be the way to go here since no one would want to go through the overhead of setting up codegen just for something like this. But in our particular case, we already rely on codegen for our clients so it wouldn't be a big deal to just add some more to that.
So I'm not sure yet which I'd prefer. I do wonder if using codegen could allow us to more easily enforce consistency across our different SDKs, but hard to say until we tackle all this. Right now, this PR only knows of strings and assumes they are all required, gonna save this all for a follow up. Good to have both options on the table for that time though.

Updated with the switch to using : separator, involved a little more fighting with cobra but nothing too bad. And the Project API remains constant, so there's at least that anchor no matter the frontend's choice how to expose it.
Given the size of the PR I'm going to cutoff adding any new features, next steps for me are:

Clean up the TODO list, add more follow ups. Once this is merged I'll split out into issues.
If time while others are reviewing, I'll work on a few other misc bare-bones cleanups like in sdk/go/server.go

So, reserve positional args to define sub-command hierarchy and use flags to be explicit on arguments to the command, all at the end.

Yeah that's probably possible too but I think if we're gonna do that we may as well just also have a different separator to make it more readable and simplify the parsing of subcommands vs. positional args.
In the updated code it's a one-line change to switch to a different separator, so we can play around with this more too as desired.

First: I am so proud of the high-quality bikeshedding going on in this thread.
I wanted to interject a few notes besides the colon-versus-spaces discussion:

I love the sugar for making commands as seamless as possible (just a top-level function, maybe a decorator in python etc) BUT I think we should define the layer slightly below that: something like a canonical interface or struct that can 1) implement all future entrypoints, not just commands, and 2) can remain stable as we iterate on the most convenient sugar in each language.

Example:

func main() {
 	ctrl := dagger.NewController()
           .WithCommand("build", Build)
           .WithCommand("test", Test)
           .WithArtifact(// TODO)
           .WithTestSuite(// TODO)
       ctrl.Serve()
 }

You get the idea.

I love the sugar for making commands as seamless as possible (just a top-level function, maybe a decorator in python etc) BUT I think we should define the layer slightly below that: something like a canonical interface or struct that can 1) implement all future entrypoints, not just commands, and 2) can remain stable as we iterate on the most convenient sugar in each language.

@shykes  Yeah my working theory has been that all the different types resources (commands, artifacts, etc.) would not actually need differentiation on this level. E.g.
func main() {
  dagger.Serve(MyCommand, MyArtifact, MyTestSuite)
}

// we know it's a command because it returns a string
func MyCommand(ctx dagger.Context) (string, error) {
   ...
}

// we know it's an artifact because it returns a container (or directory, file, etc.)
func MyArtifact(ctx dagger.Context) (*dagger.Container, error) {
   ...
}

// we know it's a test because it returns a (hypothetical) TestResult object
func MyTestSuite(ctx dagger.Context) (*dagger.TestResult, error) {
}

Basically, what determines a resource's type is literally just the types in your code. And in theory, there doesn't even need to be a restriction on a function mapping to only one resource type. Anything that can be construed as resource type X will show up in the project API under the calls related to X.
I really like that as a goal since it's so flexible and low boilerplate for the user.
However, I don't know yet whether this theory will work out in practice. Certainly possible it has some corner cases or ends up being confusing somehow.
So I'm thinking for now we could do this:

Change dagger.Serve to dagger.ServeCommands (or something like that)
If we end up not needing to differentiate different resource types, we can deprecate dagger.ServeCommands (leave it in for back compat tho) and add dagger.Serve back.

Also, before I forget, one of the follow ups here has to be experimenting with updating the Project API to add support for actually invoking commands too.
The new Project API currently gives you metadata about commands/flags/etc. for a given project, but you still need construct the final graphql query yourself to actually invoke it.
We could probably make the invocation also part of the static project API. It would be "dynamic" in that clients won't get type safety at the time they are writing their pipeline code (i.e. in their IDE), but there will be checking at runtime.
e.g.
output, err := c.Project().Load(c.Host().Directory(".")).Command("lint").WithFlagValue("strict", "true").Invoke(ctx)

The interesting thing here is that this is, if you squint, a very primitive form of more generalized extensions (just with no codegen). But it's worth a shot right after this PR because it really would be a very small lift from the current code.
cc @shykes

Re: Project API. I agree that we should start sketching out the desired GraphQL API for entrypoints right away. There are several reasons for this, one of them is that we’re probably missing ambiguities in our ongoing design discussion here (what will the DX be, which parts of the DX will be easy to change window dressing vs. bearing walls; what will require SDK-specific work and what won’t). The sooner we bikeshed over a complete graphql spec the better IMO.

@shykes In case you missed it, there is already a graphql API here: https://github.com/dagger/dagger/pull/5060/files#diff-04e962ef3f48b9ca4d82210744cf26a5b558d93cea08e3996a71ca3ae79cf2d1
dagger do already uses that instead of graphql instrospection to create commands, made that update a few days ago. The API just doesn't have support for invocation yet, the API is read-only metadata still.
So the bikeshedding can commence! I do think it is probably better to merge something before the bikeshedding has finished it course just because trying to maintain this PR in terms of rebasing is going to become a nightmare. But happy to start the discussion here and continue it post-merge (could just make a dummy PR to enable commenting on it?)

@shykes In case you missed it, there is already a graphql API here: https://github.com/dagger/dagger/pull/5060/files#diff-04e962ef3f48b9ca4d82210744cf26a5b558d93cea08e3996a71ca3ae79cf2d1
dagger do already uses that instead of graphql instrospection to create commands, made that update a few days ago. The API just doesn't have support for invocation yet, the API is read-only metadata still.
So the bikeshedding can commence! I do think it is probably better to merge something before the bikeshedding has finished it course just because trying to maintain this PR in terms of rebasing is going to become a nightmare. But happy to start the discussion here and continue it post-merge (could just make a dummy PR to enable commenting on it?)

Nice! Yes I had missed it.
And yes to merging something and continuing the discussion elsewhere. I think actually an issue might be better than a PR.
What I propose:

We checkpoint the overall discussion and continue it on #4414 - which has the benefit of already having context and an audience.
We create new issues for certain specific sub-topics. For example "colons vs spaces" probably deserves its own issue. Perhaps the new GraphQL project API deserves one too?

@shykes Yeah my working theory has been that all the different resource types (commands, artifacts, etc.) would not actually need differentiation on this level. E.g.
[...]

Basically, what determines a resource's type is literally just the types in your code. And in theory, there doesn't even need to be a restriction on a function mapping to only one resource type. Anything that can be construed as resource type X will show up in the project API under the calls related to X.
I really like that as a goal since it's so flexible and low boilerplate for the user.

This is where we need to discuss the other end of this - what does the GraphQL API look like for artifacts, tests etc.
Your example has an implicit assumption that all other entrypoints have roughly the same querying model: list the things; lookup one thing by name. But I think that may only hold true for commands - and that other entrypoints may need a richer querying model, and as a result a richer backend API for implementing them.
For example, artifacts: getting the contents (ie. a container or directory) is the easy part - but how do we organize the artifacts themselves - by name, version, architecture, arbitrary labels? all of the above? Do we want to support search? I think implementing an artifact entrypoint might be more data-driven, more like developing an actual API.
Also, for each kind of entrypoints, there is a different set of existing products and APIs out there, which we should look at for inspiration. For example, the artifacts entrypoint API might be compared to Artifactory; Docker Hub; NPM; Github's release and package registry API; etc.

We checkpoint the overall discussion and continue it on https://github.com/dagger/dagger/issues/4414 - which has the benefit of already having context and an audience.
We create new issues for certain specific sub-topics. For example "colons vs spaces" probably deserves its own issue. Perhaps the new GraphQL project API deserves one too?

Yeah that sounds great, I just updated the TODO section of this PR with all the different areas of follow ups needed. Once this is merged I'll organize them into issues (may combine some if they are related enough).

This is where we need to discuss the other end of this - what does the GraphQL API look like for artifacts, tests etc.
Your example has an implicit assumption that all other entrypoints have roughly the same querying model: list the things; lookup one thing by name. But I think that may only hold true for commands - and that other entrypoints may need a richer querying model, and as a result a richer backend API for implementing them.
For example, artifacts: getting the contents (ie. a container or directory) is the easy part - but how do we organize the artifacts themselves - by name, version, architecture, arbitrary labels? all of the above? Do we want to support search? I think implementing an artifact entrypoint might be more data-driven, more like developing an actual API.

Totally agree that the starting point should be a proposal for an artifact API (and even better a POC implementation along with it).
I actually don't think support for all the richness in the API necessarily imposes any extra special requirements for the controller code (i.e. different WithCommand, WithArtifact, etc.).
The implementation of dagger.Serve doesn't actually have any knowledge of what a Command is at all right now. All it knows are function/method signatures, which it communicates back to our project API (in the form of graphql schema). Then the project API does the work to present that as the command API.

BTW I did change the name of dagger.Serve to dagger.ServeCommands as mentioned previously, just as a precaution for now. All of the above is still true though.

Totally agree that things are going to get a lot more complicated with Artifacts and the rest of the different kinds of entrypoints, but I would love it if as much as possible of that complication stays centralized in the project API as opposed to imposing special requirements on the controller code.
There's going to be some requirements of course, but my hope is that they can be constrained to be the types in function/methods signatures, which allows dagger.Serve and the equivalents in other SDKs to remain mostly agnostic and generic, leaving the project api to take care of the harder parts.
I'm just setting this as an ideal goal in my head, not a requirement. Once we have requirements for the artifact api, if it turns out the best way to implement it requires more specialization in the SDKs, then that's fine, we should do whatever we need to in order to present the best API and keep the DX clean.

I ran into an issue - if you use the --debug flag and don't specify the full target, the CLI doesn't catch the error until a bunch of tasks complete. Can you throw an error earlier if someone doesn't include a sub selection?
I ran ./bin/dagger do --config ./ci ci and forgot to include the :sdk:go:lint

@d3rp3tt3
This is the output I currently get when running ./bin/dagger do --config ./ci ci
$ ./bin/dagger do --config ./ci ci
Loading+installing project (use --debug to track progress)...
Running command "ci"...

Command ci - Dagger CI targets

Available Subcommands:
  ci:engine           Dagger Engine targets
  ci:engine:lint      Lint the Dagger engine code
  ci:sdk              Dagger SDK targets
  ci:sdk:go           Dagger Go SDK targets
  ci:sdk:go:lint      Lint the Dagger Go SDK
  ci:sdk:nodejs       Dagger Nodejs SDK targets
  ci:sdk:nodejs:lint  Lint the Nodejs SDK
  ci:sdk:python       Dagger Python SDK targets
  ci:sdk:python:lint  Lint the Dagger Python SDK

Flags:
  --branch
  --repo

Which doesn't have the Error: Field "ci" of type "CITargets!" must have a sub selection. you have. Were you running with the latest changes (and rebuilt with ./hack/dev)? I don't remember ever seeing that particular error before, but it's possible it's in some commit somewhere and I fixed it at some point.

if you use the --debug flag and don't specify the full target, the CLI doesn't catch the error until a bunch of tasks complete.

All those same tasks run both when debug is on and off; they just don't show up in the output unless you turn debug on.
That being said, the underlying problem is that it takes a while to even get to the error part, whether debug is on or off. I have a follow-up TODO in the PR description for minimizing this latency, but it's only going to be possible to make it truly minimal when the project you are using is cached.
If you are running a project you've never ran before, it needs to be built before we can even show you the --help docs because we don't know the commands/flags/etc. until it has been built. When you include --debug, you're seeing all the progress of building the project.
I added that line Loading+installing project (use --debug to track progress)... as a very small bandaid, but the real solution here is to just integrate all this with the TUIs. That way we can actually show the project loading process in a nice way. Then, as long as we optimize the latency when the whole project is cached, this shouldn't be an issue anymore.

@d3rp3tt3
This is the output I currently get when running ./bin/dagger do --config ./ci ci
$ ./bin/dagger do --config ./ci ci
Loading+installing project (use --debug to track progress)...
Running command "ci"...

Command ci - Dagger CI targets

Available Subcommands:
  ci:engine           Dagger Engine targets
  ci:engine:lint      Lint the Dagger engine code
  ci:sdk              Dagger SDK targets
  ci:sdk:go           Dagger Go SDK targets
  ci:sdk:go:lint      Lint the Dagger Go SDK
  ci:sdk:nodejs       Dagger Nodejs SDK targets
  ci:sdk:nodejs:lint  Lint the Nodejs SDK
  ci:sdk:python       Dagger Python SDK targets
  ci:sdk:python:lint  Lint the Dagger Python SDK

Flags:
  --branch
  --repo

Which doesn't have the Error: Field "ci" of type "CITargets!" must have a sub selection. you have. Were you running with the latest changes (and rebuilt with ./hack/dev)? I don't remember ever seeing that particular error before, but it's possible it's in some commit somewhere and I fixed it at some point.

I was running off of an older commit. I updated, rebuilt, and now I see the same help that you pasted.

That being said, the underlying problem is that it takes a while to even get to the error part, whether debug is on or off. I have a follow-up TODO in the PR description for minimizing this latency, but it's only going to be possible to make it truly minimal when the project you are using is cached.

That makes sense! We can minimize the user-perceived duration of this by providing some output in the terminal (not full debug, but some detail about what's happening).


running replayer in integration tests is flaky

In zkapps-integration-test and zkapp-nonce-integration-test, running replayer step sometimes fails because the archive-node pod can not be found. When looking at the log, there's no error message says the archive hit any specific exceptions or crashed explicitly.

Fixed in https://github.com/MinaProtocol/mina/pull/15854.


bootloader: recognize stm8 chips with 64Kb flash

Add a detection for stm8 chips with 64Kb flash, such as the STM8L151R8. Their flash starts at 0x8000, the highest address is 0x17FFF.
ST's documentation refers to such chips as "high density", see e.g. the DS6948 datasheet.

thanks for pointing this out. I missed this because the STM8S 64kB variant internally is actually a 128kB chip, so this check would not work. However, for STM8L it does make sense  :-)


CONTRIBUTING.md: Remove contridiction, second PRs

This statement kind of contradicted that rebasing was optional...
Also, I guess we never mentioned anything about duplicate pull requests specifically, and we probably should since this has happened a lot in the past.

It still needs to be kept. We support both squash+merge and rebase+merge.

Should it be rephrased, then?  Or do you think it's fine as-is?

Mostly leaving as-is works, actually.
Like, relying on squash-merging for cleanup isn't something we actually want, it's just something we begrudgingly accept because it's easier than walking new contributors through rebasing.

Updated


Would be cool to detect Zoom call status

Is your feature request related to a problem? Please describe.
Detect if the agent host is on a zoom call
Describe the solution you'd like
Using something like xdotool you can detect if a zoom meeting is current or not,etc.
xdotool search --class zoom
xdotool search --class zoom getwindowname 92274836

or some simple example:
#!/bin/bash

while true; do
  # search --sync waits for the window to show up.
  #if xdotool search --sync --name "Zoom Meeting ID:" > /dev/null; then
  if xdotool search --sync --name "Zoom Meeting" > /dev/null; then
    echo "In zoom :)"
  else
    echo "No zoom"
  fi
  sleep 1
done

Things you might use this status for:

turn on the "On Air" light outside your home office door to let people know you're on a call
dim/set lights a certain way for calls
mute noisy things in your environment to avoid distractions in Zoom calls

Hey @geekpete thanks for trying out go-hass-agent. Yeah, I've been mulling how I can get a better application name which might help ease making automations in Home Assistant when certain apps have focus or are running.
In my primary use-case, with playing games on Steam, I just have to match on steam_app_* in the Active App sensor. I made a template binary sensor "Playing Games" which is true when the Active App matches the former. But I could imagine it would be nicer and more useful to have a human-friendly application name.
Does the "Active App" sensor work for you at the moment and what does it report when you have a Zoom window focused (likely need to look at the sensor state history in Home Assistant)?

Another way to solve this would be to allow custom scripts to be added to this tool.
These scripts then report their output as a certain sensor, using some refresh interval.
This is how https://github.com/LAB02-Research/HASS.Agent works, and this seems very powerful. Especially with the Linux command-line ecosystem.

Hey @fhp I like that idea! I'll mull over how to implement it. Look out for this feature in an upcoming release ;)

Hey @fhp, script sensors can now be created with release 5.2.0.  See the docs for how to configure them.
This may also meet your feature request @geekpete?
I am still thinking about how I can get the application name/title as a native sensor and there does seem to be some development in the major desktops towards exposing this in an easier way.

Awesome, thanks for your work. I surely will take a look!

I'm going to close this issue out on the assumption that the script sensor feature can fulfil the original request. If it is missing functionality, or there is a bug in using the script sensor functionality, please open an issue to report. Thanks for trying out go-hass-agent!

Finally got around to setting this up on my laptop, and the scripts work great!
Added the webcam script from #71 and a variant of that script to detect if my laptop is docked (by checking if lsusb | grep Dock returns anything).
So now I know if I'm in a call, and if my laptop is in my office, and using the screen_locked property, if I am near my laptop :grinning: That's a good start for a lot of automations.

Awesome!


[aes/dv] Rename masking/no_masking to masked/unmasked

Addressed Pirmin's comment in #16958
so that this matches to Kmac's naming scheme.
Signed-off-by: Weicai Yang<EMAIL_ADDRESS>

Thanks @weicaiyang for doing this!
There are still two occurrences of aes_masking/no_masking (not to be confused with aes_masking_off which is a chip-level test):
- hw/top_earlgrey/formal/top_earlgrey_fpv_sec_cm_cfgs.hjson
- hw/top_earlgrey/dv/top_earlgrey_sim_cfgs.hjson

oh, I missed them. Fixed, thanks @vogelpi