[RFC #5] Integration with external data and tools

This RFC discusses mechanism to access data and interact with tools outside the Typst document and its direct file system environment. The goal is to make it possible to access data from an external source, run code in another programming language or produce plots with external tools.
Movation

Use external tooling, e.g. to generate a plot
Execute code blocks in another language, e.g. Python or Julia (Juypter-notebook-style)
Integrate experiment data from remote source or database

Available options
Preprocessing
The simplest approach and the only approach that is possible right now is preprocessing your Typst files externally, as done in typst_pyimage. This approach is rather flaky and does not integrate well with content that is created programmatically.
Shell Escape
The most direct (and widely requested) approach is allowing Typst to invoke commands on the local system shell. This is useful to run code in another programming language or use third-party tools to create things like plots.
Advantages:

Makes Typst more flexible and capable
We can reuse tools written in arbitrary other programming languages

Disadvantages:

Compilation heavily depends on the environment in question (Different shell syntax on Linux vs Windows, not to speak of the Web App)
Arbitrary unwanted side effects on the system are possible, which is a security issue
Caching and memoization become much harder as we wouldn't know when to re-run the shell command

A direct shell escape would solve some problem, but would introduce a whole bunch of new ones.
Plugins
Define a clear plugin API to extend Typst. Typically proposed in the form of WebAssembly plugins. These could be well-encapsulated, mitigating concerns about the compilation environment, side effects and memoization. Only tools that can be compiled to WebAssembly could be integrated. WebAssembly plugins might be a good addition to Typst, but cannot cater to all use cases from above.
HTTP Requests
Provide a way to make a GET HTTP request to a local or remote service. The service may call to the shell internally, but shall expose a clear OS-independent interface. The service can itself decide when to re-run the command and communicate this to Typst through cache control headers. Aside from integration with external tools, this also allows access to data sources, for example with experiment data.
Integrating third-party tools in this way is not particularly convenient, but possible. Through this proposal, we can support data access and external tool usage with a single subsystem while mitigating a whole bunch of problems with shell escape. In the CLI, this feature would need to be enabled with a flag, possibly also with the option to only allow requests to particular hosts. For reproducibility, we could introduce some mechanism for storing the request results locally.
Advantages:

Makes Typst more flexible and capable
Allows access to non file system data sources (e.g. experiment data)
Allows us to reuse tools written in arbitrary other programming languages, through a well-defined interface that is not OS dependent
Can be integrated into the Web App
Caching and memoization can be handled through standard HTTP cache control
System side effects are restricted to what the service exposes

Disadvantages & Risks:

Service needs to be managed and started separately
The service itself can be implemented in a problematic way (i.e. a service that simply runs an arbitrary shell process based on the request, with no regard for security or cache control). However, the user then would need to start that bad service in addition to Typst and toggle the CLI flag.

Don't get me wrong: Whenever possible, WebAssembly plugins are a better option. This system increases Typst's flexibility to things that can not be easily encapsulated into a plugin, in a way that is amenable to caching and possible to integrate into the Web App. This is not intended as the primary backing system for plotting or packages, but more as a last-resort option for complex requirements.
Final Remarks
Please consider this as a request for comments in the truest sense of the word. I am not certain whether we want something like this and if so how we want to implement, but I'd like to see meaningful discussion on how we can cater to these more complex use cases.

What about a method for fetching external data? The method could have the following signature:
get(url: string, always: bool) -> path

Typst fetches the data and returns a path to the resource. Then it is up to the user to load the file appropriately:
let path = get("https://arbitrary_data.com?pandas=yes%20please", always: true)

let image_content = image(path, width: 50%)

let csv_data = csv(path)
let json_data = json(path)
let text_data = read(path)
let toml_data = toml(path)
let xml_data = xml(path)
let yaml_data = yaml(path)

Fetching resources is constrained to happening once before the document compilation begins. The compilation process looks like so:

If always is true, fetch the data and overwrite the previously requested files. Store the files in memory or a temporary folder like /tmp/typst/<sanitized_url>/file.
If always is false, only fetch the data if the file does not exist. Store files permanently in a folder like fetched/<sanitized_url>/file.
Continue remaining document processing as normal.

Selected paraphrased messages of mine from the extensive Discord discord relating to the topic:

Journals and conferences that require typst source
You don’t see people writing preprocessors for typst’s alternatives, because typst’s alternatives currently fill the gaps where people would want a preprocessor (eg. plotting)
The correct option for embedding code results is now (and, I hope, will forever be) having your code (which is run by an external interpreter) export some kind of typst-readable format (csv/json) and embedding results those using typst. This is not an ad-hoc solution, rather, it allows the user the most flexibility while retaining security and usability and also being fairly obvious/straightforward and requiring negligible maintenance from the typst team other than supporting readers for common data formats.
typst_pyimage, which implements a python preprocessor to generate plots and other images, best reflects not a user want to run python code but rather a user want to have good plots. Running python code is just the best avenue to this result right now. I strongly doubt that there are many (as in, more than three) other possible reasons to require a preprocessor that do not reflect clearly necessary improvements to typst that can be better done within typst than without.
Users are lazy, as programmers and security-conscious individuals we are not representative users, and we cannot rely on mere compile-time warnings or CLI flags to protect people from potentially devastating issues.

As for executing arbitrary code, the only wieldy approach is requiring external tools to return plain text or store data in documents whose paths are accessible to Typst. Typst can read the plain text directly, or if processing is required, Typist is able to open images and files in serialization formats like yaml, toml, xml, etc.
For executing external scripts in arbitrary programming languages, I would suggest a function like this:
run(code: string, runner: string) -> string

An usage example would be the following:
let sum = run("print(1 + 9)", runner: "/usr/bin/python3")

let image_path =  run(
   "
   image = some.library.create_image()
   path = '/tmp/example'
   image.save_at(path)
   print(path)
   ",
   runner: "/usr/bin/python3"
)

As stated in the Discord discussion, I believe it is much better we add a custom plugin system to Typst (or other alternative to run arbitrary code), which would let us sandbox those plugins the way we want and have a "recommended way" to do this sort of thing, instead of forcing the user who wishes to do that to use alternatives such as typst_pyimage which wouldn't necessarily be sandboxed and/or audited by us.
Sure, it is much better to increase Typst's capabilities (e.g. to provide plot natively). But it's impossible that typst will cover every single use case, and the compiler will always have more power than the language itself. Users will always want to customize things every way they can, and if there's a way to do so without having to use LittleJoe01's custom untrusted compiler which allows for plugins, but rather using Typst's built-in way with permission control, I think this would be much safer.
Always keep this in mind: If the user really wants to do something, they will find a way around it. Of course, this doesn't necessarily mean that we need to adapt ourselves for everything - but since Typst is a language which has programming capabilities, users will eventually need to use plugins for whichever complex processing they're doing in their typst programs. E.g. they might want to use pandas, they might want to use advanced linear algebra functions (which wouldn't be efficient if implemented in Typst), and other things like that.

Typst fetches the data and returns a path to the resource. Then it is up to the user to load the file appropriately:
let path = get("https://arbitrary_data.com?pandas=yes%20please", always: true)

let image_content = image(path, width: 50%)

Can you provide some examples where this is needed? Why not downloads the resources yourself and save them in your project's folder structure? What does it bring that cannot be achieved in pure typst? How do you make this secure? Do you simply trust code implicitly? Do you only allow "trusted packages" do use this? How do you deal with permissions of dependencies? And more precisely, when you set dependency permission, is it "infectious", i.e propagating to all of its subdependencies?
Image the case where we have a malicious package, call it package_a that depends on a very common package, say tablex. As it happens, tablex uses this feature to automatically render tables from scraped data, amazing you say! Now, in your project you use tablex so you give it permissions(network: true) 🎉. Now package_a depends on tablex and uses it to circumvent the fact you didn't give it permissions, therefore my question is: are permissions transitive? How do you deal with a malicious package creating elements that do have permission to do web queries?

let sum = run("print(1 + 9)", runner: "/usr/bin/python3")

Okay, let's say I am working on this document with you, I run windows or some other distribution that doesn't use /usr/bin/python3 as the default binary for python, how do you solve this? How is the WebApp supposed to be running this code? And all of my previous questions regarding permissions still stand for this as well, except that now it's even worse from a security point of view. How is this an acceptable solution?
Opening the door to these kind of issues and hand-waving the problem with "we'll just use permission" or "just use a CLI flag" are not solutions. I have yet to see, either here or on the long Discord thread any solution for these problem. Truth be told, I have yet to see an example of somewhere where you even need these features.
Regarding my opinion:

If you need to run external code to get results for your documents, you should do it manually (or using a bash script/make/whatever) beforehand and simply import it in your document
If you have very specific need, such as automatic assembling of documents for automated documentation generation, you should probably does it with an external tools, since you'll probably need settings, a database of documents, etc. And this goes far beyond what you can do in a typesetting software.
 Can't wait to see the first CVEs for typst.
There are features missing in typst, namely plotting, but these can be added to typst, and people can use matplotlib/pandas/etc separately and import them as images (SVGs support text now 🎉) until they are ready.
If you are using typst in a Jupyter-like fashion, which is imo a very good use case, you should wait and work on HTML export and write a Jupyter plugin once it's ready that allows replacing markdown with typst.
Typst must not scope creep to cover the use cases of every piece of software under the sun, covering the use case of Jupyter is, imo, out of scope. But as previously mentioned, I'd like to see nice support for typst in Jupyter, not the other way around.
HTTP requests would not be a good idea for the WebApp as it breaks [https://developer.mozilla.org/en-US/docs/Web/HTTP/Cross-Origin_Resource_Policy](cross-origin policies) which, imo, should be enabled everywhere.
HTTP requests require the user to setup a web server, I can already see someone making a "shell as a service" and people just using this, which, again, circumvents any security advantages it might have.
A proper API for HTTP requests is big, like really big, if you can control headers, handle redirection, etc.
I still haven't seen a good use case of shell-escape.
I still haven't seen a good use case of http-escape.
It breaks memoization, saying that your rely on HTTP caching is great, but what if there is no HTTP caching? Does typst take latency * processing_time * number_of_requests time more for each incremental compilation?
"Temporary solutions often become permanent problems"
Typst is, at the end of the day, a typesetting software, why does it need shell access at all? I should be able to run it in the most locked down docker container I can make, and currently I can and that's how it should be.
A plugin system however, keeps some degree of safety, using WASM:

It would be encapsulated and therefore safe, it is contained within your document
Be far more powerful by having access to internal APIs
Likely be faster (at least than slow HTTP requests)
Run on every platform under the sun, including the webapp with no compatibility issues, if it compiles to WASM, it runs anywhere (feel like I've heard that before ☕)
Could handle memoization explicitly and/or implicitly
However, it would take a ton more work, that being said, I believe that's good work

WASM plugins are the ideal implementation of extensions in my opinion. I'm also a bit unsure regarding HTTP, but plugins would theoretically make that possible either way.

are permissions transitive?
Logically, yes. Wouldn't make sense for them not to be, precisely because of what you describe.

If I might be so frank, this is a terrible idea, and I would elucidate as much with two main points.
Firstly, as @rpitasky pointed out, one of LaTeX's biggest use cases is providing a raw source to journals and conferences that can then be easily re-formatted into the correct style, incorporated into a larger document, or post-processed for citation and reference analysis. It is critical to the safety and efficiency of journal editors and conference organizers that the source be entirely self-contained.
Now, for the purposes of a journal or conference submission, one could simply omit use of external tools, but the issue here lies in that establishing this system would motivate using GET requests or external program calls for resources at compile-time. A journal specifying that documents cannot access external resources would seem like a primitive restriction rather than a practical measure.
Secondly, external program calls are entirely unnecessary for what Typst does and needs to do, and only introduce unnecessary risk. Compare this proposal to Jupyter notebooks, a common analogy thus far. Jupyter notebooks allow one to embed Python blocks into text and run them asynchronously to generate data/images/whatever else.
But, Jupyter has a clear difference from Typst: Jupyter is a Python interpreter. This means that, for example, Google Colab (an online notebook service) doesn't have to worry too much about what a notebook might do; simply provide a container, put in your standard anti-RCE measures, futz with the builtins a bit, and leave it be.
Meanwhile, Typst is given the power to call... absolutely anything on the user's machine. Sure, you could restrict access in all kinds of obvious ways, but that doesn't prevent a document from coming zip'd with a malicious Bash script that a lazy user (or, say, a journal editor) would easily overlook. And this does not even include what you could do with arbitrary Internet access atop it.
The correct procedure for providing the tooling capabilities desired here is to simply require data to be imported from common formats. Typst should handle absolutely 0% of the execution of tools that are not already part of itself and vetted to be safe. Although LaTeX handles this suboptimally in many cases (see any and every thread on StackExchange about importing SVGs), it handles it with the correct intent. Need a better plot? Export to an image. Need a big table? Export to JSON or TOML.
If Typst needs to improve what formats it can take in to better make use of existing tooling, we can, nay should, do that. A plugin system might be a fair solution to this since we can restrict calls to a vetted API, though at some point it becomes a question as to what benefit it offers over a combination of regular packages and entirely external tools. All in all, though, security flaws are far too likely for the currently proposal to be acceptable, and attention should instead be directed toward improving Typst's native tools and import capabilities.

WASM plugins are the ideal implementation of extensions in my opinion. I'm also a bit unsure regarding HTTP, but plugins would theoretically make that possible either way.

are permissions transitive?

Logically, yes. Wouldn't make sense for them not to be, precisely because of what you describe, in the sense that, for plugin A to use plugin B, plugin A will have to ask for, at least, the permissions of plugin B (and that must be made explicit) to function.

One of the big advantages with the plugins, is that you don't really need permission except maybe file system access. But as previously mentioned, you'd want something that restricts access to the current "project" directory either way.

Yeah, although, if we really want the system to be as flexible as possible, then plugins will have to be able to ask to the user for broader permissions. In fact, to be concise, I think we should use something similar to the Flatpak model: by default, the app just specifies what it needs to run (e.g. filesystem access, environment variables, ...), and the user chooses what it can and can't use in the end (and can extend permissions or restrict them - if the app is broken, ok, the user chose that). See the Flatseal app for some inspiration here.
Of course, that means that some plugins won't work in the web app - e.g. plugins that require executing certain specific binaries in the PATH. But that's usually ok, as you'd simply avoid those plugins then (or deny those specific permissions). The main point is that you'll be able to know, beforehand, everything that a plugin needs to use from your system.
And, of course, using WASM plugins is better to be able to use the compiler's API.

Why is a plugin system even necessary? There are significant downsides to any extension system, including but not limited to user fragmentation, stability, needing to maintain a plugin-facing API, and complexity. I emphatically believe such a system is worse, for many of the same reasons @kg583 mentioned. Emphasis is mine; I strongly believe there is no benefit.

A plugin system might be a fair solution to this since we can restrict calls to a vetted API, though at some point it becomes a question as to what benefit it offers over a combination of regular packages and entirely external tools. All in all, though, security flaws are far too likely for the currently proposal to be acceptable, and attention should instead be directed toward improving Typst's native tools and import capabilities.

Why is a plugin system even necessary?

Just think about the hundreds of feature requests that Typst gets. There is no way we will be able to implement everything in the standard library. Sure, you could use typst packages - but there are many things which Typst can't do without the help of Rust code. I mentioned a few examples, like pandas or linear algebra, but there are certainly hundreds of other use-cases.

I firmly believe that using an external tool to generate typst-readable files is not a workaround that users have to find, again for the reasons that kg mentioned.

It is a workaround, because now they have to move everything to typst-readable files, in order to be able to use their data generated in typst with their workflow. All data that they used to manage in their typst file will have to be managed in other formats, and the typst file will be just a ton of calls to read, json, ..., and this isn't even considering how data in Typst can change based on additions to the document. At this point, they will reconsider even doing anything in Typst. After all, Typst is supposed to allow markup but more, including complicated calculations. A mathematician might very well wish to generate things for their lecture on matrices and vectors using a proper linear algebra library, for example. I'm mentioning very basic examples, but truth is that there are TONS of specific cases that Typst itself won't handle. And that's where plugins conveniently come in: using a safe and controlled sandbox system, they will be able to implement new elements which the users can use and bring custom Rust functionality. After all, we can't expect Typst to cover every single possible case under the sun. And, when the user really needs such functionality (e.g. for their thesis), they will just monkey-patch the compiler for it, which is undesirable.

Just think about the hundreds of feature requests that Typst gets. There is no way we will be able to implement everything in the standard library. Sure, you could use typst packages - but there are many things which Typst can't do without the help of Rust code.

 If you have the technical ability and inclination to learn the proposed typst plugin api and implement a rust plugin, you have the technical ability and motivation to learn the internal typst API and make a pull request. No other piece of software I have ever heard of does anything like this. This might make sense for closed-source software but regardless it additionally has the same fundamental security and trust problems as executing arbitrary code in any language.

I mentioned a few examples, like pandas or linear algebra, but there are certainly hundreds of other use-cases.

Typst is not for general computation! If you’re working with the sorts of data that pandas implies, you should be outputting your results in some CSV and using typst scripting to format your results. KG’s response drives this home very well, so I’m not going to repeat it here.
As for your example of a lecturer, I believe this linear algebra beyond basic support for vectors/matrices absolutely is the responsibility of a typst library. If you need something heavy-duty, just save the results to a text file. The alternative of allowing plugins to run anything is, in KG’s words, frankly a terrible idea.

If you have the technical ability and inclination to learn the proposed typst plugin api and implement a rust plugin, you have the technical ability and motivation to learn the internal typst API and make a pull request.

Oh, please don't get me wrong - you can make a pull request! Problem: now Laurenz is responsible for the code you added. Laurenz will have to maintain your code. Sure, you can make more PRs later - but nothing stops you from just never coming back. So, if you implement something catering to a very specific use-case, it will likely remain outdated until the end of times. In the end, either this repository will be flooded with PRs like "Updates specific use-case X to consider Y", giving more work for the Typst maintainers, or there will be a bunch of issues from people saying things like "Hello, I tried to use the Russian-converter library module, but it doesn't seem to support the language from the Kirov russian oblast, please help!". See the problem? In the end, it's better that specific things are handled and maintained by people who know how to maintain that thing. And the best part? If the plugin goes unmaintained, others will be able to fork, or create alternatives. Plugins are what enable the community to participate more deeply!

No other piece of software I have ever heard of does anything like this.

You haven't? Here are a few examples: Programming languages in general; the Linux kernel; OBS; LuaTeX; ...

This might make sense for closed-source software but regardless it additionally has the same fundamental security and trust problems as executing arbitrary code in any language.

Of course. The burden then goes on the user to give the appropriate permissions to the plugin, and also to download them from reputable authors.

Typst is not for general computation! If you’re working with the sorts of data that pandas implies, you should be outputting your results in some CSV and using typst scripting to format your results. KG’s response drives this home very well, so I’m not going to repeat it here.

But you can use it for that! And maybe what you want to do isn't too complex - maybe you just want to multiply a few matrices here and there... and using a native library would make compilation much, much faster.

As for your example of a lecturer, I believe this linear algebra beyond basic support for vectors/matrices absolutely is the responsibility of a typst library. If you need something heavy-duty, just save the results to a text file.

Problem: Your matrix uses some data that you define in your typst file. Now you have to move all your data to a file, and all your logic to a Python script. Why do that when you can do everything in typst?

The alternative of allowing plugins to run anything is, in KG’s words, frankly a terrible idea.

Run anything that you explicitly allow to be run. Important to mention!

I firmly believe that using an external tool to generate typst-readable files is not a workaround that users have to find, again for the reasons that kg mentioned.

It is a workaround, because now they have to move everything to typst-readable files, in order to be able to use their data generated in typst with their workflow. All data that they used to manage in their typst file will have to be managed in other formats, and the typst file will be just a ton of calls to read, json, ..., and this isn't even considering how data in Typst can change based on additions to the document. At this point, they will reconsider even doing anything in Typst.

Moving everything to typst-readable files is really not a big ask if we do our due diligence. Take your favorite programming language and look at what formats it handles in its standard library. Whatever the answer is, you now know a) these formats are (approximately) easy to parse and b) that enough people use them to warrant inclusion in the standard library. I could tell you the top 5 right now that cover the vast majority of text interchanged between software: JSON, TOML, YAML, CSV, and XML. Tack on DSLs like GraphML and your top 5 image formats and you're golden.

A mathematician might very well wish to generate things for their lecture on matrices and vectors using a proper linear algebra library, for example.

As a mathematician, I use SageMath very often for visualization, and SageMath lets me do two wonderful things: export plots to PNG and generate a LaTeX representation of an object. Making this just as utilitarian for typst as it is for LaTeX is straightforward. Since PNG support is already squared away, one would just need a method to generate a typst representation; this is something that could eventually be merged into SageMath once typst is robust and popular enough, but until that point one could write a package in SageMath that does the task.
from typst import Typst
from sage.graphs.digraph import DiGraph

G = DiGraph()
G.add_edge(1, 2); G.add_edge(2, 3); G.add_edge(3, 1)
T = Typst(G)
T.repr()

The above example is very minimal, as such a Typst object could presumably do plenty of other useful things. And indeed, a SageMath package would be entirely more capable of generating a representation than a typst plugin because SageMath is a proper programming language (literally Python with some bells and whistles) and the plugin API would have to be rather bloated to match. The only potential utility is rendering an object directly, but this feels like far too little gain.

The only potential utility is rendering an object directly, but this feels like far too little gain.

How is that far too little gain when it's the core of typst as a tool?
Additionally, it could allow support of non-trivial formats or proprietary formats, which there are tons. I've used this example before, but a QR code generate makes perfect sense to have as a plugin, it does something that you cannot easily do in typst: implement a full QR code system (there are rust crates for that), it is self contained, it can be very effectively be memoized, and it makes sense to be part of your document. Same with plotting.

No other piece of software I have ever heard of does anything like this.

You haven't? Here are a few examples: Programming languages in general; the Linux kernel; OBS; LuaTeX; ...

You are forgetting the scope of this project in an attempt to be snarky. Typst is designed to render documents. Half of the examples above are designed to execute arbitrary code. We don't need to be doing that and I have no idea why this is a contentious take.

Of course. The burden then goes on the user to give the appropriate permissions to the plugin, and also to download them from reputable authors.

Yes! This is an absolutely huge and dangerous ask for the average user! Every other security exploit in existence is the consequence of somebody running something they shouldn't have or have been able to in the first place.

Typst is not for general computation! If you’re working with the sorts of data that pandas implies, you should be outputting your results in some CSV and using typst scripting to format your results. KG’s response drives this home very well, so I’m not going to repeat it here.

But you can use it for that! And maybe what you want to do isn't too complex - maybe you just want to multiply a few matrices here and there... and using a native library would make compilation much, much faster.

Yes, you can, but that doesn't mean we need to be able to do everything. Scope is again an important thing to keep in mind, as I'd rather do some things well than everything half-decently.

As for your example of a lecturer, I believe this linear algebra beyond basic support for vectors/matrices absolutely is the responsibility of a typst library. If you need something heavy-duty, just save the results to a text file.

Problem: Your matrix uses some data that you define in your typst file. Now you have to move all your data to a file, and all your logic to a Python script. Why do that when you can do everything in typst?

For the same reason I don't do generate all my diagrams in TikZ even though I can; it's simply easier and more expressive to draw them with other software.

The alternative of allowing plugins to run anything is, in KG’s words, frankly a terrible idea.

Run anything that you explicitly allow to be run. Important to mention!

That the author of the document explicitly allows to be run. Again, take the view of a journal editor: you've just received a manuscript that claims to generate its figures using an outside repository of raw data. There's a script contained in the submission that typst runs (via, per the quote I'm replying to, a runner rather than a plugin API) to make the figures... except it doesn't do that, at all, and instead crawls the editor's PC to view unpublished manuscripts and communications and whisk them off to the author.
Alright, so what's the solution? Make execution a per-user configuration? Fine, now the editor won't get figures at all, and so would have to request that all sources be self-contained, entirely defeating the point of the integration. These worries should not ever have to cross anyone's mind when compiling a typst document, and I reiterate how stunned I am that this is even a question.

You are forgetting the scope of this project in an attempt to be snarky. Typst is designed to render documents. Half of the examples above are designed to execute arbitrary code. We don't need to be doing that and I have no idea why this is a contentious take.

Sorry if I didn't write this the right way, I didn't mean to pass off as snarky. What I was replying to was their assertion (or, at least, the one that I interpreted from their message) that they haven't seen other projects which enable the use of plugins to extend what is possible. Regarding being a markup lang, LuaTeX was included to ensure I wasn't deviating too much from the point here.

Yes! This is an absolutely huge and dangerous ask for the average user! Every other security exploit in existence is the consequence of somebody running something they shouldn't have or have been able to in the first place.

Sure, but then you wouldn't download anything in your computer. If the user is willing to do something, and is aware of the consequences, then let them!

Yes, you can, but that doesn't mean we need to be able to do everything. Scope is again an important thing to keep in mind, as I'd rather do some things well than everything half-decently.

We don't need to, and that's precisely why we shouldn't bloat the standard library with tons of use-cases, instead delegating them to compiler extensions.

For the same reason I don't generate all my diagrams in TikZ even though I can; it's simply easier and more expressive to draw them with other software.

Of course, you're free to do that. But many people prefer to generate it in the document.

That the author of the document explicitly allows to be run. Again, take the view of a journal editor: you've just received a manuscript that claims to generate its figures using an outside repository of raw data. There's a script contained in the submission that typst runs (via, per the quote I'm replying to, a runner rather than a plugin API) to make the figures... except it doesn't do that, at all, and instead crawls the editor's PC to view unpublished manuscripts and communications and whisk them off to the author.
Alright, so what's the solution? Make execution a per-user configuration? Fine, now the editor won't get figures at all, and so would have to request that all sources be self-contained, entirely defeating the point of the integration. These worries should not ever have to cross anyone's mind when compiling a typst document, and I reiterate how stunned I am that this is even a question.

Usually, journals will set rules and/or style guides to avoid this kind of thing. E.g.: we don't allow this or that extension (if any at all). Papers will be rejected if you don't provide your figures as svg. Etc. That's just how things work in real life. The journal editor won't run malicious code by accident because either Typst will warn them that the plugin is missing, or ask them to give permissions to plugin X or Y (then it will be up to them to allow said plugins to do anything to their computer).

I think it is much, much more likely that a journal would go “we don’t allow typst plugins at all” or (perhaps even more likely) “we don’t allow typst at all” than single out individual plugins in a constantly updating ecosystem.

I think it is much, much more likely that a journal would go “we don’t allow typst plugins at all” or (perhaps even more likely) “we don’t allow typst at all” than single out individual plugins in a constantly updating ecosystem.

And that's a perfectly valid ask. People writing to that journal would then be aware of the rules and avoid using them (or use LaTeX instead, in the latter case), assuming the journal uses proper communication.

Yes! This is an absolutely huge and dangerous ask for the average user! Every other security exploit in existence is the consequence of somebody running something they shouldn't have or have been able to in the first place.

Sure, but then you wouldn't download anything in your computer. If the user is willing to do something, and is aware of the consequences, then let them!

You are far more optimistic about the average user's competency and attention to consequences than I, but that isn't even the main concern. The fact of the matter is that I simply do not worry about what gets run when I compile a TeX document. If typst cannot offer the same security, I'm not touching it anymore.

That the author of the document explicitly allows to be run. Again, take the view of a journal editor: you've just received a manuscript that claims to generate its figures using an outside repository of raw data. There's a script contained in the submission that typst runs (via, per the quote I'm replying to, a runner rather than a plugin API) to make the figures... except it doesn't do that, at all, and instead crawls the editor's PC to view unpublished manuscripts and communications and whisk them off to the author.
Alright, so what's the solution? Make execution a per-user configuration? Fine, now the editor won't get figures at all, and so would have to request that all sources be self-contained, entirely defeating the point of the integration. These worries should not ever have to cross anyone's mind when compiling a typst document, and I reiterate how stunned I am that this is even a question.

Usually, journals will set rules and/or style guides to avoid this kind of thing. E.g.: we don't allow this or that extension (if any at all). Papers will be rejected if you don't provide your figures as svg. Etc. That's just how things work in real life. The journal editor won't run malicious code by accident because either Typst will warn them that the plugin is missing, or ask them to give permissions to plugin X or Y (then it will be up to them to allow said plugins to do anything to their computer).

I'm well aware of how things work in real life; I've got multiple publications, which is exactly why I voiced my concerns as I did. Journals would rather block everything than some things, at which point we've lost all utility.

You are far more optimistic about the average user's competency and attention to consequences than I, but that isn't even the main concern.

Hence why Typst should make very explicit warnings about permission control.

The fact of the matter is that I simply do not worry about what gets run when I compile a TeX document.

I don't get it - TeX has shell-escape (which effectively allows arbitrary plugins), so how can you be so sure of that?
I'm assuming you don't enable shell-escape - and Typst would give you that exact same permission control, but with much more granularity. Journals will be able to completely control the sandbox if they want to, compared to just enabling or disabling shell-escape. I think we can agree that this is better than TeX's system, right? And therefore, if journals are using TeX, which has a much worse permission control system than the one proposed here, then they can use Typst as well.

If typst cannot offer the same security, I'm not touching it anymore.

As stated above, it'd offer even better security.

I'm well aware of how things work in real life; I've got multiple publications, which is exactly why I voiced my concerns as I did. Journals would rather block everything than some things, at which point we've lost all utility.

I mean, they probably don't allow TeX shell-escape already (instead of giving up on TeX), so I can't see how they would be disappointed having much better permission control at their disposal.

(Not to mention that journals would be able to provide their own plugins for their own customization, so this system can be even more of a plus.)

If typst cannot offer the same security, I'm not touching it anymore.

That comes off as a bit abrasive, I know that we all want the best for typst and we all have strong opinions, but we need to converse in a friendly manner as to avoid partisanship. Additionally, this just sounds like those people going "Well, I'll just fork it".

As stated above, it'd offer even better security.

To me, that statement can only really be true with self-contained plugins that have no system access outside of the project itself.

Journals will be able to completely control the sandbox if they want to

The only use for LaTeX shell-escape I ever encountered was syntax highlighting with some library (not listings, but another one whose name I couldn't be bothered to remember). That library uses Python to do syntax highlighting, could it be used in a published paper? (note that I have never (yet) published a paper so I am a bit less experienced on that topic)

Anything requiring shell-escape in LaTeX would never be accepted by any reputable journal.

What about an option to the typst compiler to export certain elements, e.g. raw blocks?
A user wanting to process code from the typst document could invoke the typst compiler, collect all code blocks of interest and do what ever he wants with them (generate some diagrams etc.). Those elements (raw blocks) would have a unique assignable name that can then be used to reimport the generated data.
That would generate no security problems but exporting files insides the current root. What is done with those files is up to the user or up to his/her build environment.
TLDR: Feature to export raw blocks (or maybe a special block) from typst to files or pipes, the user can then process them.

TLDR: Feature to export raw blocks (or maybe a special block) from typst to files or pipes, the user can then process them.

While that does fix the security issue, I have to ask, how is that different from the following:
#let my_python_code = read("my_python_code.py")

#raw(my_python_code, lang: "python")

Why does the code need to be inside of typst?
Don't get me wrong, I think this is a feature that could 100% be in typst and has its uses, for example to build a testing framework that checks that all of your code snippets compile/work. But it feels more tangential to the issue we're discussing here.
I have previously said that I am in favour of people "pre-processing" their data, which, is already what we're all doing by exporting figures, etc. and importing them as images in typst code.

As stated above, it'd offer even better security.

To me, that statement can only really be true with self-contained plugins that have no system access outside of the project itself.

I don't think so, given that TeX shell-escape already gives unrestricted access to the system. So, with Typst user being able to arbitrarily restrict access to the system, I believe it'd be objectively better than TeX's current external tool system.

The only use for LaTeX shell-escape I ever encountered was syntax highlighting with some library (not listings, but another one whose name I couldn't be bothered to remember). That library uses Python to do syntax highlighting, could it be used in a published paper? (note that I have never (yet) published a paper so I am a bit less experienced on that topic).

I've used shell-escape before with a tool that caches TikZ figures in the project directory to speed up compilations. That seems like a legitimate use of plugins for me, actually. However, using shell-escape was bad not because of the plugin, but because of the unrestricted access to the system. A proper permissions system (where we'd just say "sure, you can write .png files to the project directory") would work fine here.

That seems like a legitimate use of plugins for me, actually.

Well caching should be handled by memoization, unless we are talking about cold compile times in which case you are right.

A proper permissions system (where we'd just say "sure, you can write .png files to the project directory") would work fine here.

That is very true, but let's be honest, most people are lazy (again, me included) and will just go like permission: '*' to give all permissions instead of spending a ton of time setting field by field permissions. I think that the easiest solution is also the safest:

you want external tools? use external tools (i.e instantiated outside of typst)
you want more typesetting features? use packages and plugins

Anything requiring shell-escape in LaTeX would never be accepted by any reputable journal.

I don't know enough about TeX journals to confirm or deny this, but what I can say is that this would be likely due to the unrestricted system access that shell-escape gives, without any warning (which is obviously a security hole). A proper permission control system, which makes the user aware of every action a plug-in tries to take or would try to take, and lets the user allow or forbid such actions (e.g. through control of the relevant World), would fix this.

without any warning (which is obviously a security hole)

Warnings don't improve security, when was the last type you read what apt or pacman or nix told you before just pressing 'Y' on your keyboard and letting it rip 😂

TLDR: Feature to export raw blocks (or maybe a special block) from typst to files or pipes, the user can then process them.

While that does fix the security issue, I have to ask, how is that different from the following:
#let my_python_code = read("my_python_code.py")

#raw(my_python_code, lang: "python")

Why does the code need to be inside of typst?
Don't get me wrong, I think this is a feature that could 100% be in typst and has its uses, for example to build a testing framework that checks that all of your code snippets compile/work. But it feels more tangential to the issue we're discussing here.
I have previously said that I am in favour of people "pre-processing" their data, which, is already what we're all doing by exporting figures, etc. and importing them as images in typst code (e.G. gnuplot, ...).

Because I want to have all my data and code for generating figures in my document and like to have all of it compiled at once. In my oppinion this is a use-case. Today I use org-mode heavily and the feature to be able to execute code, generate figures, etc. from within my documents is great.

If typst cannot offer the same security, I'm not touching it anymore.

That comes off as a bit abrasive, I know that we all want the best for typst and we all have strong opinions, but we need to converse in a friendly manner as to avoid partisanship.

I'm not exactly sure what this comment is supposed to get at. Partisanship is exactly the point here, because I think this proposal is an awful idea (and with good reason). I made a claim about my own priorities, which could very easily apply to other users.

Additionally, this just sounds like those people going "Well, I'll just fork it".

Not sure what this is about either. People can do what they want. You can care about my opinion or not. But if this is a reason for being dismissive of my concerns, I don't particularly appreciate it.

The only use for LaTeX shell-escape I ever encountered was syntax highlighting with some library (not listings, but another one whose name I couldn't be bothered to remember). That library uses Python to do syntax highlighting, could it be used in a published paper? (note that I have never (yet) published a paper so I am a bit less experienced on that topic).
Because if it can, then publishers already allow a pretty big security hole imo.

Supposing that they do (I have no experience with this particular tool), why does it matter? If they allow a security hole out of ignorance, we shouldn't give them more opportunities. We have the ability to do the due diligence at the source that would otherwise be delayed to every end user.

Well caching should be handled by memoization, unless we are talking about cold compile times in which case you are right.

Yes, I mean cold compile, which can be relevant if it takes too long (e.g. you can cause Typst to take over 30 minutes on the cold compile depending on how heavy the stuff you use is).

That is very true, but let's be honest, most people are lazy (again, me included) and will just go like permission: '*' to give all permissions instead of spending a ton of time setting field by field permissions.

Not necessarily. Flatpak does it the right way in my opinion: it enables the permissions the plugin requests for by default (and tells you that before install), and then you can choose to take a look and disable them if you wish (there is no '*' option - you'd have to explicitly open the config app for that).
There can be a summary like flatpak does:

filesystem access 1
shell access 2

Which would allow the user to be aware of things.
And, of course, if they allow it anyway... then they're allowing it! If you're being lazy with all that info given to you, then you can't really complain if the plugin uses exactly what you allowed it to...
Of course, doesn't mean plugins can't misbehave when given permissions, but if you don't give them permissions, then nothing bad will happen.

I think that the easiest solution is also the safest:

you want external tools? use external tools (i.e instantiated outside of typst)

you want more typesetting features? use packages and plugins

That is the status quo basically (bar plugins for typesetting), and people have shown to want more than that.
To be clear - if you disable plugins (e.g. the journal editor), then everything will remain as it is. But if you really want to use plugins, then nobody should stop you! That's all.

(e.G. gnuplot, ...).

Why did you edit my quote?

Because I want to have all my data and code for generating figures in my document and like to have all of it compiled at once. In my oppinion this is a use-case.

What you are proposing, beyond the "exporting of code", is basically shell-escape and therefore does not solve the security issues it causes (as has been discussed ad nauseam). Permission systems, while in theory, solve security holes, in practice I have serious doubts that they'd fix anything judging by most people (and mine) carelessness and laziness.
Additionally, nothing prohibits you from having your code in separate files and import them as is, executing them with any tool of your liking. I suggested on the Discord that we could have a kind of typst-make tool that helps make this process easier, but my point is always the same, these are completely external tools with external files.
If you export code from a typst document, and run commands on them you have a huge security hole: a malicious package could add a hidden code block, that gets exported with the rest of your code and then ran. Therefore, this is not a secure solution.

Warnings don't improve security, when was the last time you read what apt or pacman or nix told you before just pressing 'Y' on your keyboard and letting it rip 😂

Of course, I didn't mean that warnings would make it shell-escape safer - just that the absence of warnings or, really, any information regarding what plugins are doing really sucks and makes it even worse. With a permission system, you can not only be aware of what plugins are doing, but also actually restrict what they can do (it'd have an actual security impact).

(e.G. gnuplot, ...).

Why did you edit my quote?

By accident, sorry.

What you are proposing, beyond the "exporting of code", is basically shell-escape and therefore does not solve the security issues it causes (as has been discussed ad nauseam).

Nope. I as a user can decide what to do on my own. I do not need any software that tries to prevent my use-cases on purpose. My proposal of being able to write files or data from typst is not at “shell-escape” at all.

If typst cannot offer the same security, I'm not touching it anymore.

That comes off as a bit abrasive, I know that we all want the best for typst and we all have strong opinions, but we need to converse in a friendly manner as to avoid partisanship.

I'm not exactly sure what this comment is supposed to get at. Partisanship is exactly the point here, because I think this proposal is an awful idea (and with good reason). I made a claim about my own priorities, which could very easily apply to other users.

Additionally, this just sounds like those people going "Well, I'll just fork it".

Not sure what this is about either. People can do what they want. You can care about my opinion or not. But if this is a reason for being dismissive of my concerns, I don't particularly appreciate it.

How have I been dismissive? And if that's the feeling I gave you then I am sorry, I didn't mean to come off as dismissive. I just meant that the phrasing/form of the comment was abrasive, without wanting to comment on the concern you were showing. I agree that security is a must, hence why I have been arguing against shell-escape, against HTTP request and in favour of a self-contained plugin API with no system access. While I don't agree with your arguments against a plugin API, overall, I agree with you. I'd rather see neither shell-escape nor plugin API rather than just shell-escape.
However, going in more details, if you wouldn't use typst because of shell-escape, what would you use? LaTeX has this same issue which is one of the reasons why I love typst, whether I am on my Windows boot or Linux, that code-highlighting package isn't causing me headaches (on Windows I have had nothing but issues with it). Which was one of my original arguments against any form of shell-escape, it keeps typst platform-independent which I find to be a very important feature.
Note: the comment was deleted as I was writing this one

How have I been dismissive? And if that's the feeling I gave you then I am sorry, I didn't mean to come off as dismissive. I just meant that the phrasing/form of the comment was abrasive, without wanting to comment on the concern you were showing. I agree that security is a must, hence why I have been arguing against shell-escape, against HTTP request and in favour of a self-contained plugin API with no system access. While I don't agree with your arguments against a plugin API, overall, I agree with you. I'd rather see neither shell-escape nor plugin API rather than just shell-escape.

Yes, I think we agree on those lines. I am becoming more inclined to the idea of a plugin API, though I still have reservations about implementation given the current state of development and the amount of disparity in intentions and opinions already present here. I am, at the end of the day, an advocate for extreme care in this situation, given how little it would take for something to go massively wrong for a user or our reputation as a budding LaTeX alternative.

Permission systems, while in theory, solve security holes, in practice I have serious doubts that they'd fix anything judging by most people (and mine) carelessness and laziness.

If done right, that won't be an issue (or, if it is, then the user is at fault). Regardless, you won't be forced to install any plugin should you not want to.

I suggested on the Discord that we could have a kind of typst-make tool that helps make this process easier, but my point is always the same, these are completely external tools with external files.

As I see it, that's basically just shifting the problem at hand: the user who wants to use plugins will instead have to rely on a third-party compiler then, with no sandboxing or permission system.
See what I mean? This user (who wants plugins) will always exist. What changes is: do we want to be able to control this process, giving them the choice of safer plugins? Or do we want to force said user to follow random guides on the internet which instruct them to use a third-party compiler that might be completely unsafe? That is the question.
Anyone possibly worried about plugins being "the end of Typst" or similar can rest assured: disabling plugins and any kind of external tool access will be very simple (and likely the default). This change is focused on users who actually want to use external tools or plugins. We're simply making their lives easier, in a safer way. If you don't want to use that, then that's entirely fine - when you try to compile their document, the compiler will be very explicit about either unresolved dependency or say what they're trying to do with plugins.

Permission systems, while in theory, solve security holes, in practice I have serious doubts that they'd fix anything judging by most people (and mine) carelessness and laziness.

If done right, that won't be an issue (or, if it is, then the user is at fault). Regardless, you won't be forced to install any plugin should you not want to.

I again really don't like the undertone of pinning this on the user. Programmers may have the necessary latent caution (sometimes), but the average user probably won't. I don't mean to be in any way dismissive of any person or category of people here; just cognizant of the fact that cybersecurity training is usually an ongoing and not-always-diligent process in any workplace or academic environment. "If done right" sweeps a lot of attention under the rug.

I again really don't like the undertone of pinning this on the user. Programmers may have the necessary latent caution (sometimes), but the average user probably won't. I don't mean to be in any way dismissive of any person or category of people here; just cognizant of the fact that cybersecurity training is usually an ongoing and not-always-diligent process in any workplace or academic environment. "If done right" sweeps a lot of attention under the rug.

Of course, that's a valid concern, but, as I mention in my comment, this is only for users who are actually willing to use plugins, and I think having them use a third-party compiler (which would likely have much less information regarding security) instead of a built-in solution which warns them about possible consequences of what they're doing is much worse. If they don't know enough about the subject, they should not look into installing such plugins (and the warning would make that clear, maybe asking for a confirmation). There are many different possible ways to go about doing this, of course.

What about an option to the typst compiler to export certain elements, e.g. raw blocks?

I've just had this fun idea where you can use the CLI to perform queries on your document, e.g. extract all raw blocks with a certain lang tag, even those that are generated programmatically. Not sure how feasible it is or how useful it would be, but it would be fun at least.

I again really don't like the undertone of pinning this on the user. Programmers may have the necessary latent caution (sometimes), but the average user probably won't. I don't mean to be in any way dismissive of any person or category of people here; just cognizant of the fact that cybersecurity training is usually an ongoing and not-always-diligent process in any workplace or academic environment. "If done right" sweeps a lot of attention under the rug.

Of course, that's a valid concern, but, as I mention in my comment, this is only for users who are actually willing to use plugins, and I think having them use a third-party compiler (which would likely have much less information regarding security) instead of a built-in solution which warns them about possible consequences of what they're doing is much worse.

Except it's not only for users willing to user plugins, but authors. Any collaborative effort requires agreement between all involved, and any unidirectional effort (e.g. journals) requires nothing short of a flat-out ban to maintain practicality and safety. I'm also not sure why a third-party compiler is the dichotomy here; external tools that output something that typst can read are by far the more likely option for most use cases.

What about an option to the typst compiler to export certain elements, e.g. raw blocks?

I've just had this fun idea where you can use the CLI to perform queries on your document, e.g. extract all raw blocks with a certain lang tag, even those that are generated programmatically. Not sure how feasible it is or how useful it would be, but it would be fun at least.

Also was thinking about that. I think it would be very useful, because it would allow processing data from typst => ... my toolchain ... => typst and therefore allow for all possible use-cases.

Except it's not only for users willing to user plugins, but authors. Any collaborative effort requires agreement between all involved, and any unidirectional effort (e.g. journals) requires nothing short of a flat-out ban to maintain practicality and safety.

But isn't that precisely my point? If you don't wish for plugins, you don't have to use them. If you do, and your co-authors agree, you should be able to. That's all! After all, if someone has a different workflow, authors will still have to agree on that. For example, if they use a Python script to generate an external file for typst (as suggested), authors would still have to agree on using said script. And running typst (as I emphasize again) doesn't mean instantly running a script - you will always have to be explicit on your authorization.

I'm also not sure why a third-party compiler is the dichotomy here; external tools that output something that typst can read are by far the more likely option for most use cases.

It's not necessarily a dichotomy, but it is needed most of the time to be able to not only parse Typst, but also output Typst elements, or custom Typst elements (with e.g. custom Layout and Show implementations). And the tool will likely want to access things from other crates, which might access the network or the filesystem, and then we're back to square one.

This change is focused on users who actually want to use external tools or plugins. We're simply making their lives easier, in a safer way. If you don't want to use that, then that's entirely fine - when you try to compile their document, the compiler will either be very explicit about unresolved dependencies or say what they're trying to do with plugins (and not let it go forward).

I think the issue here is that the "users" mentioned here are not always going to be the authors of the document. People will want to share Typst projects with each other. Using an external tool to process data and convert it into a format that Typst can read allows a user A who trusts the tool to share the Typst code and formatted data with user B, even if B does not trust the external tool. Whereas with plugins/shell escape/external tools that run while the document is being compiled, both A and B would need to trust the plugin in order for B to compile the document at all.
User B's options are effectively "spend time convincing themselves that the plugin is trustworthy and being used in a safe way," "run this plugin despite not actually trusting it," "ask A to rewrite it to not use plugins," or "don't compile this document at all." The first option is going to be far too much work for someone who needs to compile many Typst documents from different authors on a regular basis, like the journals that @kg583 mentioned. The second is obviously terrible for security, but likely to end up occurring in practice, and the last two defeat the point of adding plugins in the first place.
Overall I feel that being able to grab a random Typst document and modify and compile it yourself without having to worry about security or other tools being installed on your system is an excellent thing. It greatly outweighs the benefits that being able to run arbitrary code directly during the compilation process would give over just running the code externally prior to compiling the document.

And running typst (as I emphasize again) doesn't mean instantly running a script - you will always have to be explicit on your authorization.

How often have you seen someone click "OK" without reading anything? It could very well end up another case of warning fatigue.

Yes, I think we agree on those lines. I am becoming more inclined to the idea of a plugin API, though I still have reservations about implementation given the current state of development and the amount of disparity in intentions and opinions already present here. I am, at the end of the day, an advocate for extreme care in this situation, given how little it would take for something to go massively wrong for a user or our reputation as a budding LaTeX alternative.

This is something I 100% agree on, at this stage, a plugin API would be, at best, unwise. I think that we need to reach a more mature state where APIs are more stable. If we look at #712, we can see that lots of internal API changes are, at least, being thought about. Therefore, we cannot rely on the internal API at this stage.

How often have you seen someone click "OK" without reading anything? It could very well end up another case of warning fatigue.

This is exactly what I meant about another comment regarding people pressy Y while installing a program. We get so bombarded by warning and requests that are always the same that we grow complacent and just ignore them. The solution to this for typst?
Enforced safety by default, if people go out of their way to do things in an unsafe manner, it's on them. If typst is the gun in the "shooting yourself in the foot" analogy, then making the workarounds are the action of pressing on the trigger and looking for ways to circumvent security features are the bullets. You only have yourself to blame. Making it as difficult as possible to do the wrong thing is likely the best alternative (locking the gun in a safe), that and offering safe, easy to use, ways of doing things (just use a knife, duh! 😂).

Whereas with plugins/shell escape/external tools that run while the document is being compiled

I don't think that self-contained plugins would belong to that list, as long as it's fully encapsulated, it is safe.

Even now it's quite straightforward to extend the typst standard library with custom functions. Adding a bridge to wasm to define such functions is clearly the next logical step.

With regards to the updated "main" RFC post, here are my thoughts:

Queries are a good midterm solution that fills in a real gap in typst
Plugins are the best long term solution

Therefore my suggestion would be to implement the query system as a long term solution that maintains platform independence and safety and let users use it for whichever purpose they want to use it. It may be far more useful than executing code! Then, later on, implement the plugin API with encapsulated WASM plugins for maximum safety, exposing as much of the internal APIs as can safely and reasonable be done. This gives us the best of both worlds.

Before comment
I prefer Loadable Plugin + HTTP Escape (Or other protocol-based RPC)
About Plugin System
Plugins pose a huge security challenge, as evidenced by issues arising from their use in Greasemonkey scripts (permission granting challenge), liburing (permission inheritance challenge), and VSCode extensions (plugin ecosystem and management). Even with good security practices in place, it's possible to unintentionally introduce flaws by plugin system, meaning plugins with high permissions may be responsible for early waves of CVEs in Typst.
To mitigate this, I believe the following points are key:

Run Plugins in complete sandboxes, such as wasmtime, even if it may reduce their performance.
Plugins may only have access to a read-only filesystem and serveral output directory, like typst::World.
Plugins must be one by one approved by users and can't be implicitly loaded and executed by other plugins, like vscode marketplace.

I also suggest designing and export typst query apis before implementing plugin systems. We would prioritize plugins that offer significant benefits to executing within Typst over what they could perform externally.

With regards to the updated "main" RFC post, here are my thoughts:

Queries are a good midterm solution that fills in a real gap in typst
Plugins are the best long term solution

More about intergration

It should be able to utilize the context of Typst documents, such as state and locate.
It should have access to typst query apis.
It can be able to write to a sub-path in the output directory.
It would be hookable or observable by the external world, similar to localhost webhooks or ipc/rpc events.
Users are able to use their preferred programming languages to handle the serialized output and events produced by plugins.

I agree mostly with @Myriad-Dreamin, except for the following:

If plugins are run in a sandbox, there is no reason to forbid dependencies between plugins, especially where this may reduce code duplication. Having plugins be able to access the symbols of their dependencies should be a feature, since the worst-case scenario here should be that your document gets trashed (due to the strict sandboxing)
I don't particularly like HTTP requests; plugins themselves should export memoization information, and there should be a nice Rust crate to write them

I actually have a use case in mind: you probably haven't heard of Ott, but it's a tool used in the PL community to preprocess LaTeX containing typing rules, while checking the validity of all the rules written down. It can also automatically generate source code for various theorem provers. This is very useful, but the whole "preprocessor" bit is very irritating and may interact unexpectedly with various LaTeX features. More importantly, how many preprocessors can one realistically stack: say I want to use the features of Ott and some other preprocessor at the same time; this will lead to suffering.
Typst, being dynamically typed and intentionally simplified, is simply not the right language to implement such complex type-checking, etc; not to mention the fact that I cannot re-use existing code. But WebAssembly is fine, and can be made to integrate excellently with Rust, so that Typst extensions are usually written in Rust using a set of official API crates. The best part is I can then share code between the Typst plugin and the code generator; the latter of which can use the query API instead of being called by Typst.
While it's such complex features which require (imo) a plugin architecture (especially if, e.g., one wants to interact with layout, dynamically recompute things based on counters, and such), various other simple features would be much more easy to implement using WASM plugins, though all of these can be implemented with queries, including (some of which were previously mentioned):

QR codes, simply using an existing Rust library to generate SVG files. While these could be generated by a tool and cached in a directory, this requires dealing with multiple source files or using an external tool, and while this could be implemented using Typst queries, I just don't like it to be honest.
Loading of unsupported file formats, especially binary formats like MessagePack
Plot generation and graphics, which can be done "efficiently enough" in WASM while avoiding needing to manage multiple source files and tools, but may not be doable in Typst, or may even simply just require porting existing code in another language. Same arguments apply to this as to QR codes.
Advanced syntax highlighting (see e.g. Lean syntax highlighting, which is not regular expression based)

One idea I would propose is to try to erase the distinction between Typst plugins and packages, and instead have a Typst plugin simply be a package with some (not even necessarily all) functions implemented in WebAssembly. Then, the security model is exactly the same as Typst's (with the same API available to Typst code and Typst plugins), but you have the ability to incorporate existing code and do the above things. Then anything which can't be implemented in this way should, I agree, be implemented via a query system.

Thanks for your input @imbrem. Upon further discussion over Discord with some other community members, I believe the best way here is to (at least) support fully sandboxed WASM plugins. This will already make a huge difference, as one will be able to use (many of) the existing Rust libraries to create elements and perform other complex calculations, and access to the network or something shouldn't be necessary here at all.
Another step would be to grant limited access to the filesystem. To be more specific, I think such plugins should be able to read and (perhaps) write (e.g. to export some calculation result) only in the project directory (in the same directory as the main typst file being compiled). Since the plugin won't have access to the network, that should be safe enough to produce output files, which can then be read by other programs if needed. (Note that all typst files can already read the current directory - actually, even more than that!!)
Though, of course, one should be able to block (or otherwise control) filesystem access entirely via some sort of CLI toggle if needed (but this doesn't really pose some sort of huge danger or something anyways).
What do you guys think? This should work just fine with WASM sandboxing, and would also be able to be used in the typst web app (due to sandboxing). I don't believe this would represent a security hole thanks to the power of WASM. (And no network/HTTP is needed!)

I would not grant any file system access at all, I think:

You can read the file in native Typst code and pass the contents to the plugin. This way we don't have to deal with any system interface in WASM.
Direct writing is not a good idea in my opinion because of its interactions with memoization. When would something be written?

You can read the file in native Typst code and pass the contents to the plugin. This way we don't have to deal with any system interface in WASM.

I mean, the plugin will always be able to do at least what normal Typst code can do. I don't see how blocking read access entirely would make sense, especially if e.g. we're planning on having plugins implement custom file parsing abilities. This could just become an annoyance imo (e.g. if the plugin has complicated logic to read or not some file).

Direct writing is not a good idea in my opinion because of its interactions with memoization. When would something be written?

The same argument could probably be held for anything too different, including HTTP requests if that ever becomes a thing, so I think we should find a way around this. I think it's important for plugins to be able to produce output in some way. Maybe the file should be written whenever the plugin actually has to process something? Which would only happen if the user changes the call to the plugin's function.

Sorry to interrupt the back and forth here! I had some thoughts I wanted to share.
I'm strongly in favor of only implementing "queries" - IE a way to extract specific data from a Typst document. I've got a few reasons for this, below.
Plugin systems are complicated & limited
Any plugin system introduces some significant level of complexity to a program - it's unavoidable. You have to embed the language / runtime, add an API, etc. No matter how you do it, there is a large cost. Embedding Lua, JavaScript or even a WASM runtime is fairly huge. But despite this, each way of doing things is significantly limited - without enabling shelling out you can only use libraries compatible with the runtime you chose. And both things are especially aggravated when you take into account security - you have to make sure everything is sandboxed correctly, none of your APIs give methods of escape, etc. and you can no longer do a significant amount of things to maintain security.
I could go on, but that's already a bit of a text wall - the TLDR is that plugins add complex issues and don't do everything you want anyway.
Security is key
People have already mentioned this above quite a lot, but security is super important. Not everyone's a programmer or cybersecurity expert, and if compiling a Typst document can lead to arbitrary code execution, that'll lead to tears. Or IT banning Typst. Just for that, you can completely rule out Shell escape, and I would argue HTTP escape too - that's an easy way to exfiltrate files. Read /etc/passwd or equivalent and stick the result in a URL to evil.com.
Plus, HTTP is a pretty user unfriendly way of going about things in the first place. What are users supposed to do? Start up a local webserver when they want to compile a document? Learn how to use some SaaS API platform?
In the best case, Typst simply reads files and writes to the output you specify, an donthing else. No network requests, no arbitrary code.
People want to use Typst for more than documents
This is the reason why I'm strongly in favor of a query system. A while ago, I can't recall where, I was linked this document, where someone has demonstrated generating tables and code from data using Typst. As you change the data, both the tables and the example code update. If Typst had a query system, then the user could run a script to generate an entire, working, reference implementation from a specification. In fact, I was considering writing a feature request for allowing Typst to output text files before I saw this issue.
It works the other way around, too - the example typst_pyimage could be replaced with a script that queries data from the typst document and then generates images into the filesystem, which can be used when compiling the document. Or you could write a script to generate a CSV that's read into a table or graph in the document.
And this is better for security too - these generated files can be saved with the project so that anyone compiling the project, like the reviewers given as an example above, doesn't have to execute untrusted code, only Typst.

So there are my thoughts. I hope they're helpful :slightly_smiling_face:

I think its important to ask, what task do we want the plugin architecture to fulfill? I see these two use cases:

allow preprocessing content with specialized tools: As stated multiple times in this thread already, users currently utilize the specialized tools directly and import only the ready-to-display results. It is also important to keep the goal of typst focused. Typst is not a build system and it should not add a plugin system to pretend it is one. Making typst easier to integrate with build systems should be the priority. Given that, the main area of improvement is to make the CLI interoperable with other tools (in the order of importance):

provide examples of how to integrate typst with common build systems.
the CLI query system would allow other tools to process parts of a typst document.
allow typst compile to cache intermediate results in a target directory. A build system will likely use typst compile instead of typst watch, so documents should not be built from scratch all the time.
machine-readable compiler output to allow other tools to act on it.

change some internal behavior of typst (e.g. Layouting): I expect the barrier of entry for plugins that allow such things to be similar to developing typst itself. Given that, I believe it's reasonable to expect the users that want a niche feature to develop this in a fork of typst. Then users have no limitations imposed by an arbitrary plugin API and could also contribute the feature. Additionally, no plugin API has to be maintained and instead of spending energy on learning material for the plugin API the onboarding for new typst contributors could be improved.

TLDR: Use Queries+Preprocessing to allow specialized tools to interoperate without spending energy on maintaining plugin APIs.
To quote the Unix Philosophy: "Write programs that do one thing and do it well. Write programs to work together."

@DrRuhe : the problem is now if I want to edit those preprocessed things I need to have all the relevant software installed, vet it for security issues, etc. Whereas with a secure, WASM-based plugin architecture, a plugin can be Just Another Typst Package, even downloaded and run automatically. And if anything is too complex for the plugin interface, then you can still do preprocessing and so on.
The problem with build systems is there's too many of them. Look at Rust itself; you don't use "some build system + rustc", you use Cargo, because it works well with Rust's unique features. Similarly, a Typst version of Cargo is necessary for Typst packages; might as well let those packages be implemented in languages other than Typst. That's all I'd ask for of a plugin system, really. Typst can then be kept a small, focused, and simple scripting language, while allowing us to bring in the software engineering "big guns" (like a proper type system, traits, etc.) when necessary for more complex projects (like my typing rule preprocessing example!)
And the problem with forks of Typst is exactly the UNIX philosophy issue: they don't compose! What if I want two fancy layout-modifying features? Preprocessing actually also has problems, since it suffers from a phase-order problem: in what order do you run the preprocessors. Whereas a plugin system can automatically deal with such issues dynamically at runtime; for an example, see Minecraft mods: multiple mods which drastically change world generation and gameplay can nevertheless co-exist assuming they all use the shared Forge API (note: this knowledge is like 5 years outdated, no idea what they use now, but the example still stands). Imagine instead having a separate fork for every complex mod, with simple mods limited immensely by the need to be hacked together in a simple in-game scripting language. Sounds horrific! And people will hack horrible things together in the Typst language if they are not given the tools to do it properly in an actual programming language. I guarantee it.
Look at LaTeX, after all.

@imbrem I agree that plugins have their use-case. But I strongly believe that the priorization of (wasm-based) plugins should be lower than the query+preprocessing system. A query system with a CLI that properly interoperates with other tools will allow most possible use cases. Plugins can then be added later, once it is apparent, that they are really necessary. But until then, I think spending considerable amounts of the limited development capabilities on a plugin system, that is limited to a sandbox, is a bad idea.
Regarding  a build system specialized for typst, we should make the CLI allow for easy interoperability and can then add a thin wrapper around that when necessary. But creating a full build system seems very out-of-scope for typst itself. On the topic of ordering problems for preprocessing, I think the query system should be able to solve this (idea: typst cli provides the results of queries as an "iterator". Once all things were preprocessed, the actual document can be built)
I also see a separation between (wasm-based) plugins, that essentially act as integrated preprocessors, and plugins that change core parts of typsts functionality. I agree that the multiple-forks approach is not great, but I'd argue that it is better to make the typst language flexible enough to allow these customizations. Similar to how rust has no rustc-plugins because there are mechanisms like proc-macros and build scripts in place to allow for customization.

You've all said it well, and I intend to expand typst in my spare time in the future

Proc macros are essentially just hacky typst plugins though. The real difference after all is just that they call the Typst API directly rather than emitting source.

As a very new typst (and super impressed) user, my input probably isn't worth all that much, but here are my ramblings on the topic.
I am a very new user of typst and love it so far.
The feature I am missing most from daily-driving RMarkdown is the ability to write diagrams and other code in my documents and have them be rendered/executed as I compile/render them.
I currently do this through a shell escape, which is undesirable but works.
This is where I think typst could be really cool.
If typst supported sandboxed WASM plugins, I could realistically write code that looks like the following:
#mermaid! {
sequenceDiagram
    Alice->>+John: Hello John, how are you?
    Alice->>+John: John, can you hear me?
    John-->>-Alice: Hi Alice, I can hear you!
    John-->>-Alice: I feel great!
}

and have the figure be rendered and output straight into my document, of course integrating with the memoization and other nice features of typst, maybe even syntax highlighting provided by defining metadata for how the inner content should be treated.
I honestly do not believe it is the responsibility of typst to implement diagramming or plotting.
It is nice that the primitives are available, and plugins could use those to output diagrams or plots (or they could output it in some other supported format: SVG, PDF, PNG...).
It sounds much nicer to let plugins handle this task.
Plugins have the huge benefit that they can be written in other languages and can be maintained in other repositories.
Uses I can think of right now:

Diagramming (plugins could be created for d2, mermaid, tikz, ...)
Providing renderers for other things (LaTeX equations, chemfigs, or even just LaTeX)
Executing code in other languages and showing the results (quite niche)
Highlighting code blocks can be outsourced to plugins
Image-compression plugin?  Say your source images are very large, and you don't want to use some external tool to compress them before including them in your document
Plotting

None of these uses require accessing the internet or the file system and they should only do processing.
I think it would be interesting if code execution could interact with typst in some way, and perhaps preserve state across executions.
This is perhaps tricky to do safely, but I think there is some reasonable way to let functions "capture" variables for use inside the inner code.
How it could work:
I imagine some package format with a .typ header and a .wasm binary.
For the diagramming, LaTeX, code execution, and highlighting use-cases, typst need only send the raw text straight to the WASM code and somehow take the output and use it in a productive way.
I imagine this to be done by simply returning typst code from the plugin and inserting that in place of the function.
If this is the approach that is taken, one would need to be very careful that it cannot hook into privileged features that are exposed through normal typst code.
For the image-compression plugin, I could imagine a read_binary function that could be assigned to a variable, then passed to a plugin function.
The function could also output binary data which could be included in a figure with something that looks like #image(bfile(data, ".png")) that would treat the binary data as a file with a certain extension.
The plotting may be a little more difficult as it needs access to data, but I don't think an interface needs to be very complicated.
A string/binary serialized version of the appropriate structures could be passed to the wasm.
Plotting functions can output typst primitives (i.e. a string to be interpreted as typst) or a binary (image, pdf...).
Benefits:
This interface is very simple; WASM functions only expect strings or binary data (maybe only binary data?).
There is also no need to access the file system, the internet, or any other privileged functionality, and there should be no way for plugins to output code that would use these features.
It also means there is an easy way to manage dependencies; a package manager could feasibly be created; plugin versions could be listed with their version numbers.
It is easily supported in the web editor with no need to run preprocessing or wrap the compilation with other tools.
Closing remarks
I am not sure I see the need for a plugin system to do more than what I have proposed here (let custom functions be run in WASM).
I don't think WASM plugins should be able to affect document layout, and I don't think they should be able to interact with anything outside of their world (what has been directly passed to them).
I will probably be tinkering with adding this on my own as soon as my exams are over and I have spare time again.

@mumblingdrunkard the only thing I disagree with is I feel it wouldn't be too hard to allow plugins to call arbitrary Typst functions from WASM, allowing us to affect layout that way; similarly to how in the browser WASM modules can call Javascript.

@imbrem

Also, please no string serialization. Strings are slow and ugly.

I agree with this, though WASM is built in such a way that you need to copy data into it to let the WASM work on it.  I totally agree that it should be in a binary format though.

I just read through the whole thing and I'm still in favor of allowing some powerful external system built in Typst.

I think the issue here is that the "users" mentioned here are not always going to be the authors of the document. People will want to share Typst projects with each other. Using an external tool to process data and convert it into a format that Typst can read allows a user A who trusts the tool to share the Typst code and formatted data with user B, even if B does not trust the external tool. Whereas with plugins/shell escape/external tools that run while the document is being compiled, both A and B would need to trust the plugin in order for B to compile the document at all.

This is the only valid reason that I see. I think another reasonable compromise is:

allow typst --shell-escape document.typ to compile with "shell escape" or whatever dangerous powerful feature needed;
typst --shell-escape --dump document.typ to do as above but store the result of all external interactions to some file document.dump,
typst --use-dump=document.dump document.typ to compile the document safely (and deterministically + system-independent) while still allow it to compile.

I think it can solve the problem above, and still allow B to do minor modifications so long as it does not touch the "powerful" part of the document. (of course in order to allow B to modify the powerful parts then B need to trust the plugin)

As for the "query" approach, it works okay but has the disadvantage of back-and-forth overhead (you need to invoke Typst → external tool → Typst), which is probably not ideal for performance and does not compose very well with incremental compiler.

Chiming in here as the creator of typst_pyimage, which has been discussed several times in this thread.
The main thing I'd like to contribute to this discussion: I think it would be a good idea if plugins communicate information back to the .typ file via the file system.
Why? Because I do want the ability to run arbitrary external code -- my current use-case involves training a small machine learning model inside my Typst file! I have a #pyinit(...) directive at the start that runs most of the training code, and then lower down I have a number of #pycontent(...) and #pyimage(...) directives to render the output.
And obviously, no journal under the sun is going to be okay with running that. But that's fine!

Instead of shipping them paper.typ and saying "please run typst compile paper.typ --plugins pyimage"...
...I can simply ship them paper.typ and some_image.png and some_value.txt, and say "please run typst compile paper.typ".

This also meshes well with incremental compilation and caching.

I think your need here is not a plugins system but something outside of
the typst ecosystem like Jupyter. As such, I think that for this
particular use case, it is worthwhile to build an external tool like
Jupyter (which allows the typesetting tool to do what the typesetting tool
is good at, and the python interpreter to do what the python interpreter is
good at).
On Sat, May 13, 2023 at 1:06 PM Patrick Kidger @.***>
wrote:

Chiming in here as the creator of typst_pyimage
https://github.com/patrick-kidger/typst_pyimage, which has been
discussed several times in this thread.
The main thing I'd like to contribute to this discussion: I think it
would be a good idea if plugins communicate information back to the .typ
file via the file system.
Why? Because I do want the ability to run arbitrary external code -- my
current use-case involves training a small machine learning model inside my
Typst file! I have a #pyinit(...) directive at the start that runs most
of the training code, and then lower down I have a number of
#pycontent(...) and #pyimage(...) directives to render the output.
And obviously, no journal under the sun is going to be okay with running
that. But that's fine!

Instead of shipping them paper.typ and saying "please run typst
compile paper.typ --plugins pyimage"...
...I can simply ship them paper.typ and some_image.png and
some_value.txt, and say "please run typst compile paper.typ".

This also meshes well with incremental compilation and caching.
—
Reply to this email directly, view it on GitHub
https://github.com/typst/typst/issues/1056#issuecomment-1546711934, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/A2QMYGK4C6XGMMLINYVRYVDXF65RZANCNFSM6AAAAAAXRT2JWY
.
You are receiving this because you were mentioned.Message ID:
@.***>

Quite possibly. My particular use-case isn't really the point, though. (Perhaps that has been missed.) I'm speaking to the specfics of submitting papers to journals, and this I really don't see any option other than the file system.
For Typst to achieve traction with journals, I expect all involved would find it much simpler if the journal can simply run Typst version x.y.z, and have everything compile -- without any plugins at all.
Otherwise:

as an author, I will have to check the list of plugins that the journal uses, and what versions they are at.
as a journal editor, I would have to maintain a list of such plugins, keep the listed version update-to-date, field user requests to add some new FooPlugin.
worst of all would be to discover that for whatever reason the journal compiles my file with a different version of a plugin than I was expecting -- and that the final version of paper now looks completely different from what I expected!

The file system is a convenient way to avoid all such issues.

Other than this, I'm definitely -1 on a WASM-only based approach. I've not tried, it but a little searching suggestings that NumPy isn't supported on WASM, and that a number of workarounds are instead going to be required. I suspect this kind of containerisation/emulation is going to lead to an endless source of compatibility headaches down the line, for whoever is in charge of maintaining the Typst plugin infrastructure.
In contrast VMs are already a mature technology, so it'd be easy to sandbox things for the web app by just running each instance in a VM. (I understand the security concerns here.) Charge users extra for plugin usage if hosting costs are a concern.

I completely concur here regarding the FS, but for the webapp I think it's
important to recognize that it is fast because compilation is done
in-browser with a WASM-compiled typst.
On Sat, May 13, 2023 at 5:56 PM Patrick Kidger @.***>
wrote:

Quite possibly. My particular use-case isn't really the point, though.
(Perhaps that has been missed.) I'm speaking to the specfics of submitting
papers to journals, and this I really don't see any option other than the
file system.
For Typst to achieve traction with journals, I expect all involved would
find it much simpler if the journal can simply run Typst version x.y.z, and
have everything compile -- without any plugins at all.
Otherwise:

as an author, I will have to check the list of plugins that the
journal uses, and what versions they are at.
as a journal editor, I would have to maintain a list of such
plugins, keep the listed version update-to-date, field user requests to add
some new FooPlugin.
worst of all would be to discover that for whatever reason the
journal compiles my file with a different version of a plugin than I was
expecting -- and that the final version of paper now looks completely
different from what I expected!

The file system is a convenient way to avoid all such issues.
Other than this, I'm definitely -1 on a WASM-only based approach. I've not
tried, it but a little searching suggestings that NumPy isn't supported
on WASM https://github.com/wapm-packages/python/issues/2, and that a
number of workarounds are instead going to be required. I suspect this kind
of containerisation/emulation is going to lead to an endless source of
compatibility headaches down the line, for whoever is in charge of
maintaining the Typst plugin infrastructure.
In contrast VMs are already a mature technology, so it'd be easy to
sandbox things for the web app by just running each instance in a VM. (I
understand the security concerns here.) Charge users extra for plugin usage
if hosting costs are a concern.
—
Reply to this email directly, view it on GitHub
https://github.com/typst/typst/issues/1056#issuecomment-1546757250, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/A2QMYGMQFKZIE3QKU6R2ISLXF77QJANCNFSM6AAAAAAXRT2JWY
.
You are receiving this because you were mentioned.Message ID:
@.***>

Glad we agree on using the file system.
Webapp speed: I think this should still be okay, by simply caching the result of each plugin call and only making a network request for the updated file if the input changes. Each plugin call being assumed to be functionally pure.

for the webapp I think it's important to recognize that it is fast because compilation is done in-browser with a WASM-compiled typst.

That's the problem, inevitably there's some user that wants to use something that is not compilable (*) to JavaScript. I don't think it's feasible for the webapp to support every plugin (even overleaf with shell-escape access does not support PythonTeX at the moment (**)), but just let the user do whatever they want on their machine.
In any case, the --dump option would be a faster communication method than using FS, as it is "built-in" by Typst (the plugin does not need to manage the file path etc.) and everything is packed to one file (the dump itself may be a zip if it needs to be "readable"), plus depends on the implementation it may be faster as well (avoid needing to touch the file system as much / only need to spawn one external process etc.)
(*) because of technical limitation, like numpy at the moment.
(**) because of another technical limitation https://tex.stackexchange.com/questions/534049/pythontex-not-working-on-current-overleaf-compiler although it's possible to modify PythonTeX to not depend on the specific features that overleaf prohibits.

There's a fundamental question here that still hasn't been resolved, and I
think this is one for the founders themselves to . Do we want something
that is a typesetting tool, like TeX, that is brought into the 21st century
with a better compiler/macro system/syntax? Or do we want a better code
notebook tool, like Juptyer.
I think typst is currently marketing itself a little bit closer to the
latter than the former (leading to people wanting plugins/etc) because most
examples of typst demonstrate its vastly superior macro capabilities.
But really, the latter is really difficult for something meant to produce
documents to do sustainably. If you want to run python code alongside
your paper, a Jupyter-but-with-typst-instead-of-tex engine better satisfies
this need way better than typst alone ever could. Instead of the
typst-pyimage model, where typst calls the python interpreter, a
third-party program that uses both typst and python together is very
reasonable.  I don't think that typst is nearly ready enough as a library
to support this, but that is a minor development hurdle (at least relative
to having to perpetually maintain a plugin API).
I've argued my point above for having the user- by whatever existing means
they want- give their images/tables/etc files to typst in reasonable,
#include-able and parsable formats. Compatibility is really not a concern
as far as I know, cross-platform file path weirdness is generally
well-behaved in the rust world.
On Sat, May 13, 2023 at 10:37 PM user202729 @.***>
wrote:

for the webapp I think it's important to recognize that it is fast because
compilation is done in-browser with a WASM-compiled typst.
That's the problem, inevitably there's some user that wants to use
something that is not compilable (*) to JavaScript. I don't think it's
feasible for the webapp to support every plugin (even overleaf with
shell-escape access does not support PythonTeX at the moment (**)), but
just let the user do whatever they want on their machine.
In any case, the --dump option would be a faster communication method
than using FS, as it is "built-in" by Typst (the plugin does not need to
manage the file path etc.) and everything is packed to one file (the dump
itself may be a zip if it needs to be "readable"), plus depends on the
implementation it may be faster as well (avoid needing to touch the file
system as much / only need to spawn one external process etc.)
(*) because of technical limitation, like numpy at the moment.
(**) because of another technical limitation
https://tex.stackexchange.com/questions/534049/pythontex-not-working-on-current-overleaf-compiler
although it's possible to modify PythonTeX to not depend on the specific
features that overleaf prohibits.
—
Reply to this email directly, view it on GitHub
https://github.com/typst/typst/issues/1056#issuecomment-1546792647, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/A2QMYGLS7N5X3WZRLO2JUPDXGBANPANCNFSM6AAAAAAXRT2JWY
.
You are receiving this because you were mentioned.Message ID:
@.***>

@patrick-kidger

I suspect this kind of containerisation/emulation is going to lead to an endless source of compatibility headaches down the line, for whoever is in charge of maintaining the Typst plugin infrastructure.

This is why I proposed that WASM plugins only be simple transformers, receiving text/binary and outputting text/binary, at least in the beginning.

For Typst to best achieve traction with journals, I expect all involved would find it much simpler if the journal can simply run Typst version x.y.z, and have everything compile -- without any plugins at all.

This, I think, is easily solved by having something like typst freeze which would freeze the plugin versions (maybe even typst version somehow?) and bundle them locally, then use those when calling typst compile.  If we have a reliable repository of plugins, they don't even need to be bundled and can instead just be locked to the appropriate version.
Better still, typst prepare could be a command that simply executes all the plugins and collects all the string/binary artifacts produced by plugins and outputs new source .typ files which use these artifacts instead of using plugins.
Overall, I agree with @rpitasky that it sounds like you want something outside of Typst, like Jupyter.  This could also be very useful, but it requires external tooling to get set up and compile documents.  With a reliable plugin system, all you need is typst to compile your document.
@rpitasky

I've argued my point above for having the user- by whatever existing means
they want- give their images/tables/etc files to typst in reasonable,
#include-able and parsable formats. Compatibility is really not a concern
as far as I know, cross-platform file path weirdness is generally
well-behaved in the rust world.

I agree with the premise of this, but I don't think my version of plugins clashes with this.  Simple text/binary -> text/binary transformer plugins would just embed with typst in a better way.  It makes it so that you can edit your diagrams/plots directly in the source files, and I think it's a much better workflow.  I also think this would be preferable to rewriting all the existing tooling in Typst if we can avoid it by simply compiling the existing tooling to WASM and wrapping it with some Typst code to make it a package.

I did a little work today to find out how to make minimal changes to test and run our function call：
https://github.com/Euraxluo/typst

My 2 cents.
There are few distinct features which people call plugins.

A sandbox environment to transform input into desired output possibly by leveraging external libraries
A system which query the docs and allows to produce files in other formats (json/markdown/yaml/anki)
A system to preprocess data
A system to fetch data from internet
A system which adds reactivity to documents by injecting javascript into resulting pdf
API server

It kind of means at least two different plugin systems:

sandboxed wasm execution environment
API server with SDK for few languages

I strongly believe that plugins should be hermetic and shouldn't

access network
1.1. The configuration file should allow only access to whitelisted urls (see a note about bypassing)
1.2. There should be a mechanism to specify digest of the expected file
access filesystem
2.2 There might be a way to allow (via configuration) writing only into specified directories and only files with allowed extensions. There must be no way to change ownership or permissions (set executable bit).
2.3. Except predefined place to allow persistence (LocalStorage???) of intermediate results (plugin shouldn't be able to control filename)
access time and date
randomness (but typst might provide an API for them to call)
cannot call OS commands (I know this is a huge restriction but I'll elaborate latter)
listen on tcp/udp ports (unix sockets are fine (supported in windows 10 and 11))

Note: The restrictions should be bypassable. Options are

command line flag
signed config in the directory of a project
confirmation dialog typst allow (inspired by direnv allow) to allow use of the config file (every change to config would require typst allow).

Auto-download plugins
The auto-download should have a safety mechanism, for example:

Download only signed plugins
Download only plugins mentioned in the signed manifest (configuration file).
Download only from specified registry
Verification of the integrity of downloaded plugin
Use typst allow to confirm each plugin individually. If we would have permission system for plugins the required permissions would need to be printed in the confirmation dialog.

Ability to run OS commands
This is a huge security risk. Also it means that plugins would start requiring arbitrary dependencies installed on the OS.
I think if people want this functionality they should use any of the following:

special build (with different name)
API server with SDK for multiple languages which works over unix sockets. In this case the users would be calling typst from their program and they would have full control over it.

I think WASM plugins would be great addition. There are few crates which can be used to implement them

https://crates.io/crates/wasmer/
https://crates.io/crates/wasmtime
https://crates.io/crates/wasm_plugin_host
https://crates.io/crates/fp-bindgen
https://github.com/ItsEthra/scotch

Even if we would start from input transform use case it would be a good testing ground for the plugin system itself.

@iilyak idk I feel this is really over complicating things for the current state of Typst. We already have a Turing-complete language in Typst; all I propose is to allow it to be low level, via WASM. It’s kinda like JavaScript in the browser vs WASM; Typst being the JavaScript here. It would be incredibly useful even if it had exactly the same capabilities as regular Typst code (and hence the same security model).

@iilyak idk I feel this is really over complicating things for the current state of Typst. We already have a Turing-complete language in Typst; all I propose is to allow it to be low level, via WASM. It’s kinda like JavaScript in the browser vs WASM; Typst being the JavaScript here. It would be incredibly useful even if it had exactly the same capabilities as regular Typst code (and hence the same security model).

Can you elaborate on "I propose is to allow it to be low level, via WASM" what do you mean

use WASM runtime and re-implement typst to work on top of that runtime?
provide bindings for some of the typst APIs so it is available from WASM runtime.

I assume you mean #2 in such case I don't think it is easy to expose all typst API for following reasons

The API surface is big and we have a risk of outdated plugins
Writing so many bindings will be time consuming
The plugins would be using low level typst functions and rely on implementation details. Which would make the codebase hard to change without a risk of breaking all plugins.

If we go this route I suggest to limit the APIs we expose and provide some mechanism to detect when plugin is not compatible with the version of typst. I would start from 0 API and carefully consider what to expose and what not to expose. Initially it could be done based on the plugin needs. Plugin author opens an issue when the required API is missing.
It is hard to tell what is the best way of exposing the layout engine details without leaking implementation details.

I have a use case which wasn't mentioned, yet I think, this issue is getting pretty long, I want to use https://github.com/pdfpc/pdfpc to present my typst presentation, and I want to add notes inside typst.

Great usecase! Using this as some sort of "case study", lets look how well the different possible approaches might realize this:

HTTP Escape: Generating local files via HTTP requests is (if at all possible) rather hacky.
WASM Plugins: As this usecase would require access to the filesystem, either typst provides a filesystem API (lots of effort) in which case this usecase could also be fulfilled in native typst code. Or the plugin accesses the filesystem directly, in which case sandboxed plugins would not allow this. This would also require the use of a plugin (which has to be developed first) for a rather simple usecase.
Shell Escape: Probably something like ``sh("echo "+ notes + " > ./pdfpc")` in your typst code. But now your typst document depends on a specific environment (in this case it only requires echo and bash, but this might quickly get out of hand once more complex dependency setups are present). Also, when does the file get regenerated completely? Suddenly the user has to remember to do that as well or typst will just append the same notes to the file on each compilation.
Preprocessing + Queries:  you could query for all the notes in your document and save that output to a file. I think this is the most elegant solution.

A few notes on the queries approach that came to my mind while doing this:

how is the "build process"? Will typst execute a provided build command?
Queries should allow for retrieval of structured data, lets say JSON, which would be very helpful to provide an easy to use interface for external tools to use these queries. In the case of note generation (The following only explores how this might look like ):

#let note(body) = locate(loc => { typst-preprocessor-query(
  key: "presenter-notes"
  (
    page: #loc.position().page,
    note: body
  )
)});

#note("Hello World!")
#pagebreak()
#note("Note on the second page")

And then running the query would produce the following result:
$typst query presenter-notes
[
  {
    "page":1,
    "note":"Hello World!"
  },
  {
    "page": 2,
    "note": "Note on the second page"
  }
]

Note that I have not delved into a build pipeline to also transform the query output into the desired state.

Wasm Plugin support
Inspired by the previous messages and discussion here and on discord, I think a first approach to wasm plugin could ressemble something like this:
As a host

You can only send strings and receive a singular string
you're aware if the plugin has had an error
As a plugin:
you have to define a few functions

description of the protocol and discussion here #1555
No access to any filesystem but you already get the power of extern tools as long as they compile to wasm.
Emphasis on a simple protocol.
Most things already work but need polishing and better integration.

You can only send strings and receive a singular string
I would agree that being able to read and write more generic Values would be nice, at least, not having to parse numbers, dimensions, etc. yourself

I mean, you don't have to parse if you can eval you're returned string ;)

I have a use case which wasn't mentioned, yet I think, this issue is getting pretty long, I want to use https://github.com/pdfpc/pdfpc to present my typst presentation, and I want to add notes inside typst. For LaTeX there is a pdfpc package that can embed said notes inside the pdf, but there is also a json format for generating those notes inside a .pdfpc file next too the typ file.
I'd like a way to, just using typst code - no need for any additional language support, output a plain text file next to the pdf.
Something like defining that file at the top, and then subsequently writing to it.:
#let file = new_file("main.pdfpc")

#write_to_file("{")
//...
#write_to_file(the notes of a slide)
//...
#write_to_file("}")

Just a word. I have also thought of outputting structured data into an external file, see #1629 for more details and a prototype implementation - just prototype.

I have a use case which wasn't mentioned, yet I think, this issue is getting pretty long, I want to use https://github.com/pdfpc/pdfpc to present my typst presentation, and I want to add notes inside typst.

Great usecase! Using this as some sort of "case study", lets look how well the different possible approaches might realize this:

HTTP Escape: Generating local files via HTTP requests is (if at all possible) rather hacky.
WASM Plugins: As this usecase would require access to the filesystem, either typst provides a filesystem API (lots of effort) in which case this usecase could also be fulfilled in native typst code. Or the plugin accesses the filesystem directly, in which case sandboxed plugins would not allow this. This would also require the use of a plugin (which has to be developed first) for a rather simple usecase.
Shell Escape: Probably something like ``sh("echo "+ notes + " > ./pdfpc")` in your typst code. But now your typst document depends on a specific environment (in this case it only requires echo and bash, but this might quickly get out of hand once more complex dependency setups are present). Also, when does the file get regenerated completely? Suddenly the user has to remember to do that as well or typst will just append the same notes to the file on each compilation.
Preprocessing + Queries:  you could query for all the notes in your document and save that output to a file. I think this is the most elegant solution.

A few notes on the queries approach that came to my mind while doing this:

how is the "build process"? Will typst execute a provided build command?
Queries should allow for retrieval of structured data, lets say JSON, which would be very helpful to provide an easy to use interface for external tools to use these queries. In the case of note generation (The following only explores how this might look like ):

#let note(body) = locate(loc => { typst-preprocessor-query(
  key: "presenter-notes"
  (
    page: #loc.position().page,
    note: body
  )
)});

#note("Hello World!")
#pagebreak()
#note("Note on the second page")

And then running the query would produce the following result:
$typst query presenter-notes
[
  {
    "page":1,
    "note":"Hello World!"
  },
  {
    "page": 2,
    "note": "Note on the second page"
  }
]

Note that I have not delved into a build pipeline to also transform the query output into the desired state.

I really like this solution. It's not clogging the working dir with lots of files, integrates well with different tools.
A few hints:
1.) give the query command a --format parameter to output json, yaml, toml, ... as needed. Seems not a big deal for implementation, but make the tool more versatile
2.) Allow for --list and --one as result ... just in case someone only needs a single object to save postprocessing steps.
3.)

I have a use case which wasn't mentioned, yet I think, this issue is getting pretty long, I want to use https://github.com/pdfpc/pdfpc to present my typst presentation, and I want to add notes inside typst.

Great usecase! Using this as some sort of "case study", lets look how well the different possible approaches might realize this:
* **HTTP Escape:** Generating local files via HTTP requests is (if at all possible) rather hacky.

* **WASM Plugins:** As this usecase would require access to the filesystem, either typst provides a filesystem API (lots of effort) in which case this usecase could also be fulfilled in native typst code. Or the plugin accesses the filesystem directly, in which case sandboxed plugins would not allow this. This would also require the use of a plugin (which has to be developed first) for a rather simple usecase.

* **Shell Escape:** Probably something like ``sh("echo "+ notes + " > ./pdfpc")` in your typst code. But now your typst document depends on a specific environment (in this case it only requires echo and bash, but this might quickly get out of hand once more complex dependency setups are present). Also, when does the file get regenerated completely? Suddenly the user has to remember to do that as well or typst will just append the same notes to the file on each compilation.

* **Preprocessing + Queries:**  you could query for all the notes in your document and save that output to a file. I think this is the most elegant solution.

A few notes on the queries approach that came to my mind while doing this:
* how is the "build process"? Will typst execute a provided build command?

* Queries should allow for retrieval of structured data, lets say JSON, which would be very helpful to provide an easy to use interface for external tools to use these queries. In the case of note generation (**The following only explores how this might look like** ):

#let note(body) = locate(loc => { typst-preprocessor-query(
  key: "presenter-notes"
  (
    page: #loc.position().page,
    note: body
  )
)});

#note("Hello World!")
#pagebreak()
#note("Note on the second page")

And then running the query would produce the following result:
$typst query presenter-notes
[
  {
    "page":1,
    "note":"Hello World!"
  },
  {
    "page": 2,
    "note": "Note on the second page"
  }
]

Note that I have not delved into a build pipeline to also transform the query output into the desired state.

The issue with preprocessing and queries is that while it's a great answer to this particular use case, it falls over when you need the result of the preprocessing to then have an effect on the document, and perhaps you need to do this multiple times mutually recursively. This isn't theory, it's LaTeX bibliographies, and those are a disaster. And don't even get me started on things like Ott, which is my use case (this takes in a bunch of typing rules for a programming language or similar, checks they're well-formed, and typesets them, as well as proofs using them, which are also checked). Typst is just not the right language to design something so complicated, if it wasn't for the fact that it might already be designed, but a native experience for such tools would be a real advantage.

The issue with preprocessing and queries is that while it's a great answer to this particular use case, it falls over when you need the result of the preprocessing to then have an effect on the document

Preprocessing + Queries might not be the solution to all use cases, but it seems to be an elegant solution to some important use cases - and therefore a very promising first step towards interoperability. So, in my opinion, we should take it ;-) It might not be the last step...

The issue with preprocessing and queries is that while it's a great answer to this particular use case, it falls over when you need the result of the preprocessing to then have an effect on the document, and perhaps you need to do this multiple times mutually recursively.

I totally agree that the system I have described in my previous post is only for extracting (the query part) but not actually for inserting the actually preprocessed data. In this post I want to elaborate on how the preprocessing side of this might look. So basically answer the question: "how is the build process? Will typst execute a provided build command?". This is just a collection of my thoughts on how this might work, so bear with me:
First of all, I think it's fair to treat the queries and preprocessing like typst providing the inputs for pure function calls and reimporting the resulting data. This means, the results of queries typst returns will act as the input to the "function" (in this case, some external program that will do the preprocessing) which will produce the same results as long as the inputs don't change. Additionally, the preprocessed data has to be accessible to typst somehow. For this example, let's use the example from my previous post where we had the following query results:
$typst query presenter-notes
[
  {
    "page":1,
    "note":"Hello World!"
  },
  {
    "page": 2,
    "note": "Note on the second page"
  }
]

These are the "inputs" to our function. To give typst a place to reimport the preprocessed data from, let's build a cache on disk containing all the preprocessed data for each query result.   The data is "input adressed", meaning that a query result will be hashed and the preprocessed data is imported from the hash folder in the cache. To illustrate, lets take the query result, "hash" it by using the page property as the hash and use a preprocessor that turns every note to uppercase. We then get something like the following folder structure:
$ls ./typst-preprocessor-queries
1 2
$cat ./typst-preprocessor-queries/1
HELLO WORLD!
$cat ./typst-preprocessor-queries/2
NOTE ON THE SECOND PAGE

Note that reality, the hash should hash everything that is input into the preprocessing logic.
This would have the following properties:

any build system can build such a folder structure. People can get started with this and typst might even provide the build mechanism in the future.
query results are automatically "memoized". This has the added benefit that others do not need to run the preprocessing logic when they have the cache folder.

It still remains a bit unclear how exactly recursive stuff would be implemented, and while I have no straightforward way to solve this, these were a couple ideas:

a typst preprocess subcommand which outputs all the query results that are not yet present. Then just run your preprocessing logic on the query results until the typst preprocess command returns no query results left to preprocess.
compiling/watch with typst compile --preprocess './preprocessor.sh', which would run the ./preprocessor.sh script with each remaining query result.

All in all, I think queries and preprocessing can get us like 90% of the way to third-party-tools integrating with typst with a relatively minimal set of effort involved. That is to say, I also want WASM plugins that integrate nicely into typst, but for now I think the limited time we have is better spent on queries and preprocessing.

Just wanted to note that @astrale-sharp has made great progress on WASM plugins, see #1555.

The issue with preprocessing and queries is that while it's a great answer to this particular use case, it falls over when you need the result of the preprocessing to then have an effect on the document, and perhaps you need to do this multiple times mutually recursively.

I totally agree that the system I have described in my previous post only deals with extracting (the query part) but not actually for inserting the actually preprocessed data. In this post, I want to elaborate on how the preprocessing side of this might look. So basically answer the question: "How is the build process? Will typst execute a provided build command?". This is just a collection of my thoughts on how this might work, so bear with me:
First of all, I think it's fair to treat the queries and preprocessing like typst providing the inputs for pure function calls and reimporting the resulting data. This means the results of queries typst returns will act as the input to the "function" (in this case, some external program that will do the preprocessing) which will produce the same results as long as the inputs don't change. Additionally, the preprocessed data has to be accessible to typst somehow. For this example, let's use the example from my previous post where we had the following query results:
$typst query presenter-notes
[
  {
    "page":1,
    "note":"Hello World!"
  },
  {
    "page": 2,
    "note": "Note on the second page"
  }
]

These are the "inputs" to our function. To give typst a place to reimport the preprocessed data from, let's build a cache on disk containing all the preprocessed data for each query result. The data is "input adressed", meaning that a query result will be hashed and the preprocessed data is imported from the hash folder in the cache. To illustrate, lets take the query result, "hash" it by using the page property as the hash and use a preprocessor that turns every note to uppercase. We then get something like the following folder structure:
$ls ./typst-preprocessor-queries
1 2
$cat ./typst-preprocessor-queries/1
HELLO WORLD!
$cat ./typst-preprocessor-queries/2
NOTE ON THE SECOND PAGE

Note that in reality, the hash should hash everything that is input into the preprocessing logic.
This would have the following properties:
* any build system can build such a folder structure. People can get started with this and typst might even provide the build mechanism in the future.

* query results are automatically "memoized". This has the added benefit that others do not need to run the preprocessing logic when they have the cache folder.

It still remains a bit unclear how exactly recursive stuff would be implemented, and while I have no straightforward way to solve this, these were a couple of ideas:
* a `typst preprocess` subcommand which outputs all the query results that are not yet present. Then just run your preprocessing logic on the query results until the `typst preprocess` command returns no query results left to preprocess.

* compiling/watch with `typst compile --preprocess './preprocessor.sh'`, which would run the `./preprocessor.sh` script with each remaining query result.

All in all, I think queries and preprocessing can get us 90% of the way to third-party tools integrating with typst with a relatively minimal set of effort involved. That is to say, I also want WASM plugins that integrate nicely into typst, but for now, I think the limited time we have is better spent on queries and preprocessing.

Why not expose this API to WASM? Then everything can run cleanly inside Typst, with external preprocessors only required when you need to do IO

A don't want to support APIs directly in Wasm because it's complicated but you can just write Typst code that does the query and pass the results to Wasm (as a serialized string).