Node-based pyiron-base

Summary

Detailed Description

Further Information, Files, and Links

Fundamental concepts
Thanks @liamhuber for all the enhancements. At the IPAM workshop at UCLA, I had the chance to discuss with Jan concepts for bringing together various ideas from pyiron, ironflow and other workflow modules. These ideas are aimed to develop a more modular and easier-to-maintain pyiron. The basic building block will be nodes. The framework must therefore provide all tools to create, manage, and upscale them easily. In the following a brief summary of the concept is given:
    @pyiron.node
    def multiply(x=1, y=2):
           return x * y

The main idea is that a node can be represented by a function with well-defined input, output and execution body. Rather than having to define explicitly the input and output when using a class this structure appears more pythonic and is used e.g. by dask. A main advantage would be the low entry barrier - all a new user would have to do is to write a common python function and decorate it with the pyiron.node. This approach can be also easily extended to provide type/ontology information via typing:
    from pyiron import onto

    @pyiron.node(register(onto.atomistic), log_input=True)
    def multiply(x: onto.types.atomistic.BulkModule=1 , y: int=2): -> float:
        return x * y

The idea would be to provide ontological types like you introduced in the latest versions of ironflow. The nodes can be individually used and connected similar like in dask without having to explicitly define a workflow. A decorated node would be delayed and executed only once the highest level node is executed (similar like the delayed mode in dask):
      c = multiply(2, 3)    # no execution, only delayed object created
      d = multiply(c, 4)    # no execution, only delayed object created
      d.run() # only now c and d will be evaluated

Pyiron-related concepts
While the node-based pyiron could be used without explicitly defining a workflow, a common simulation should probably have one:
from pyiron import Workflow
    wf = Workflow(name='test', onto=onto.atomistic)

    Al = wf.create.structure.bulk('Al')
    job = wf.create.job.Lammps(name='Lammps_job', structure='Al')

The new syntax would be very similar to the existing one, so users should very easily adapt to it. A couple of things are to note:

Workflow provides arguments/functionality that is not in Project (i.e. definition of its core ontology/application)
The entire input to Lammps can be provided by function arguments (e.g. structure='Al'). For convenience and consistency, we should still leave our existing input notation (i.e. job.input['para'] = para) but the preferred style should be the new functional one.
The Workflow object provides on one hand autocompletion to easily access and browse through all available nodes, on the other hand this syntax appends the nodes to the workflow.

Serial execution
The following example is an extreme case, and may be a good test case for checking the concept and code:
    structure = wf.create.structure.bulk('Al')
    job_old = None
    for i_step in range(10):   # cannot be captured by workflow (write as macrocode)
          job = wf.create.job.Lammps(name=‘Lammps_job',
                                   structure=job.get_structure(-1))

          if job_old is not None:  # write decorated function to capture it in workflows
               if np.abs(job.output.energy_tot - job_old.output.energy_tot) < eps:
                      break

Such a construct would fail in dask delay.
VASP example
Register nodes
In contrast to having a complex module like VASP in a single node we should have smaller and more flexible ones. This is sketched below together with ideas regarding node registration:
    @pyiron.node(register(onto.atomistic.code.vasp.exe))
    def VASP_exe(incar:FilePath, poscar:FilePath, potcar:FilePath, kpoints:FilePath):
        work = WorkingDirectory(path='.',
                                files=[incar, poscar, potcar, kpoints]
                               )
        work.run('vasp.exe -f my_mode')
        return work

    @pyiron.node(register(onto.atomistic.code.vasp.parser_outcar))
    def VASP_parser_outcar(outcar:FilePath, select=[], exclude=[]):
        out_dict = my_parser(outcar, select)
        return out_dict  # or iodata object

    @pyiron.node(register(onto.atomistic.code.vasp.parser_incar))
    def VASP_parser_incar(incar:DataIO):
        incar_str = my_parser(DataIO)
        return incar_str

    @pyiron.node(register(onto.atomistic.code.vasp.parser_poscar))
    def VASP_parser_poscar(structure: onto.atomistic.structure):
        poscar_str = my_parser(structure)
        return poscar_str

Create VASP (macro node)
    @pyiron.node(register(onto.atomistic.code.vasp))
    def VASP(incar: DataIO,
             structure: onto.atomistic.structure,
             calculator: onto.node.calculator
            ):

        onto_vasp = onto.atomistic.code.vasp

        vasp = onto_vasp.VASP_exe(incar=onto_vasp.VASP_parser_incar(incar=incar),
                                  structure=onto_vasp.VASP_parser_structure(structure=structure)
                                 )
        out_dict = onto_vasp.VASP_parser_outcar(vasp.outcar, select=['energy_tot'], exclude=[])

        return out_dict

This is only a sketch of first ideas. Comments and suggestions are very welcome.

A few more ideas and pseudo code regarding node based pyiron. The examples demonstrate typical workflows we use.
Example workflows
Single
    wf = Workflow(name='test',
                  onto=onto.atomistic,
                  sql=None, # True - use default, db_connector, ...
                  log_level = 'low', # 'debug', ...
                  store_working_directory = True,
                 )

Note that we could add many options in the workflow creation. Examples are whether to store the nodes in a database, whether and which input and output data to put into hdf5, etc.
    job = wf.create.code.VASP(name='Lammps_job',
                              structure=wf.create.structure.bulk(‘Al’),
                              calc=wf.create.calculator.MD(T=300)
                            )

Note that the new nodes allow to provide all input via function parameters.
Parallel
Run over large number of structure stored in a structure container:
    jobs = wf.create.code.VASP(name='Lammps_job',
                               structure=wf.create.structure_container(‘Al_data’),
                               calc=wf.create.calculator.MD(T=300)
                               )

or run over a list of temperatures:
    structures = wf.create.structure_container(‘Al_data’)
    calculators = wf.create.calculator.MD(T=np.arange(100, 1000, 100))
    STRUC, CALC = wf.np.meshgrid(strctures, calculators)

    jobs = wf.create.code.VASP(name='Lammps_job',
                               structure=STRUC,
                               calc=CALC
                               )

Serial
    @pyiron.node
    def iter_energy(energy_convergence):
        job = wf.create.code.VASP(name='Lammps_job',
                                  structure=wf.create.structure.bulk(‘Al’),
                                  calc=wf.create.calculator.MD(T=300)
                                 )

        for i in range(10):
            job_new = wf.create.code.VASP(name='Lammps_job',
                                      structure=job.get_structure(-1))
            if np.abs(job.output.energy_tot - job_new.output.energy_tot) < energy_convergence:
                break

            job = job_new

        return job.get_structure(-1)

and apply nodes:
    wf.iter_energy(energy_convergence=1e-4)
    wf.run()

Hi Jörg,
Super, yep, I'm on board with this. I'll be on vacation through Wednesday so I won't be able to look at this in depth, but a bunch of the earlier stuff you suggest is already implemented over in contrib, minus the syntactic sugar of doing it with a decorator (which should be easy).
One thing that is notably missing in your examples is labels for outputs. IMO these are absolutely critical for constructing complex graphs where inputs and outputs can be connected with any complexity -- the only way to get around this is to restrict these functions to only return a single output value, which I think is too harsh.
Concretely, look at your example setting a kwarg "structure=wf.create.structure.bulk('Al')" -- in this case "bulk" is a node, and in principal may return multiple values (although in this case it only returns one). So I really feel extremely strongly that we need something like "structure = wf.create.structure.bulk('Al').structure", and to require specifying labels for the output, eg in the decorator like "@pyiron.node('structure')" or "@pyiron.node(('energy_pot', 'forces')".
If you and Jan haven't yet, please run the contrib workflow notebook(s?) to see what parts of your plans the existing infrastructure already already covers. All the starting stuff I think is just a matter of adding decorator syntactic sugar to existing functionality. The later stuff with specifying parallelization, database connection, etc all still needs to be done.
On a first read through, the absence of output naming is really only thing that worries me here, the rest of it looks brilliant so far.
Very practically, I want to first prioritize some performance enhancements for ironflow (decoupling port status model logic from the draw call should be sufficient), but once that's done I am excited to go over to contrib and start integrating the ideas here into the graph infrastructure 👍👍👍

Actually I have a second concern: the use of a global variable ('wf') in the final example. I think this is a closely connected concern to my worries about Marvin's contrib work where he allows entire nodes to be passed as input to other nodes. I completely agree that we need this type of macro functionality, I just think we'll need to be a little out careful about the implementation.

Hi Joerg,
A live chat sounds great!
Re annotations, I think that works nicely for adding onto typing in addition to data typing, although annotations currently do not support kwargs (like "o type=onto.foo"), so we would need to force a fixed ordering in annotations. For this reason I lean towards using a dict, eg as a kwarg in the decorator with keys matching the variable names, to provide this data. The big problem I see using annotations to label output data is that I want output labels to be absolutely mandatory, and forcing people to learn about annotations and use them seems harder to understand than adding a positional arg to the node decorator. Also we could probably enforce it as a requirement, but we'd need to add extra rail guards, where adding it as an arg to the decorator just right away makes it clear.
Re the demo notebook, this is actually not at all integrated with ironflow yet, it's over in contrib in notebooks/workflow_example.ipynb

Missed the global car discussion because of indentation.
It may indeed be possible to provide it by the scope of the class, but then you'd still need something like 'self.wf', and self will (at a minimum) look very strange in the context of a function definition (even if the decorator means we return a class instance). I bet we can find a solution, it will just take some thinking to get it both functional and intuitive.

Let's try to schedule a meeting on Thursday or Friday. I am presently at the DPG meeting in Dresden and will be back in Düsseldorf on Friday.
It would be definitely good to explore possible option to provide the labels for the output variables. Decorators may be a good option. We should only make sure that our solution is as close to standard python, so that the barrier for users is as low as possible.
Thanks also for the latest development on ironflow. The ontologic features are really great and it is exciting to play with it. The only issue is the sluggish behavior, which makes it often hard to know whether a click did not work or something is still happening. I am therefore looking very much forward to the next development on speeding things up. Then one can fully enjoy the really cool features and the great concepts that you have already implemented. Really great work!

Hi Liam, I had now a look at your workflow notebook in pyiron_contrib. Really very nice! I see also the strong links to my thoughts. An important task of the decorator would be to make the following statement more intuitive and python-like:
      def plus_minus_one(x: int | float = 1) -> tuple[int | float, int | float]:
          return x+1, x-1

      node = Node(plus_minus_one, ("p1", "m1"))

With the new formulation this could read like
      @pyiron.node(repository)   # repository where the node would be registered (could be local or global)
      def plus_minus_one(x: int | float = 1) -> tuple[int | float, int | float]:
          return x+1, x-1

      plus_minus.inputs.x.type_hint

The last line is only an example to show that all constructions in your notebook should work, i.e., the decorator converted the function into a node object.

While I am currently at the IPAM workshop, I would like to join the meeting to synchronize the discussions, so just keep me in the loop.

Hi Jan, great that you will join. We have not yet set up a meeting but a good choice may be Friday afternoon when I will be back at home.

I have a couple of recent developments from the IPAM workshop, which might also be helpful for this discussion:

structuretoolkit - move the structure analysis to a separate module so they can be used with ASE Atoms directly. The idea is that if people like this stand-alone module then they might also be more likely to give pyiron a try. The pull request to merge these changes back into pyiron_atomistics is available https://github.com/pyiron/pyiron_atomistics/pull/994
pyiron_lammps - the pure python interface to LAMMPS now supports multiple ASE structures as well as multiple pandas DataFrames for interatomic potentials. Just like numpy if only a single ASE structure and a single potential are provided only a single set of elastic constants is calculated. If either multiple ASE structures or multiple potential dataframes are provided then multiple sets of elastic constants are calculated. Finally, if a list of ASE structures and a list of interatomic potential dataframes are provided with the same length - e.g. as they are generated from np.meshgrid then again a list of elastic constants is calculated. https://github.com/pyiron/pyiron_lammps/pull/12
pympipool - now subtasks can use multiple MPI ranks. For running LAMMPS simulation this gives the user the full flexibility, either a single LAMMPS calculation is using all MPI ranks, or each LAMMPS simulation uses one MPI rank each or any other split of MPI ranks to LAMMPS simulations is possible, as long as all LAMMPS simulation use the same number of MPI ranks and all MPI ranks are used. https://github.com/pyiron/pympipool/pull/28

While these developments focus more on the scalability of pyiron, I could see them being beneficial in simplifying the development of complex workflows and hopefully provide the ideal test bed for the developments discussed above.

I'm just on mobile so my responses are pretty limited in depth, sorry.
Re meeting: Friday sounds good. At present I can be free any time. We should keep @pmrv in the loop here too in case he wants to attend; there is both synergy and some conflict between the graph stuff and the tinybase stuff.
Re standard python/decorators/link: indeed, I think getting the existing graph stuff working with decorators should be super fast, then adding the fancier bits on top can be more iterative. I'm super excited about this direction.
Re lazy evaluation: there is some support for this! My existing node stuff has init flags for turning on/off the initial run and running automatically on update. Definitely not as smooth as your example, but the groundwork exists at least.

On Friday I'll still be in the train by the time @liamhuber and @jan-janssen would be able to join, so I'd prefer Thursday.

From my side any time Thursday is also currently fine.

From my side any time Thursday is also currently fine.

Although if it's going to be first thing Thursday morning (pst) then I'll need to know inside the next five hours, which seems unlikely...

Friday >=1500 CET is good for me.

Friday >=1500 CET is good for me.

For me as well.

I played around and implemented the decorator so that this now works:
from pyiron_contrib.workflow.node import node

@node("sum")
def adder(x: int|float, y: int|float) -> int|float:
    return x + y

I was thinking a bit about how to handle macros and had two concrete thoughts:

Currently node functions are always staticmethod, but nodes should be adapted s.t. the node function can reference self. I think it would be efficient to parse the function to see if the 0th arg is called self, and if so simply don't make an input channel for this, and pass self on execution.
It should be possible to make an @macro decorator somewhat like the @node decorator, but it would decorate a function that builds a little graph. The returned node class would then have this construction done after __init__, and the node function would just be updating the (sub)graph inputs. The constructor would probably need to rely on self.workflow, which isn't necessarily populated for all nodes, but I think it's a fair requirement of a macro that they exist inside workflows.

And one fuzzy thought:

We could consider merging Node and Workflow into the same class. What really pushes me in this direction is that then macro nodes effectively create their own namespace, like my_workflow.my_macro.some_node, because we can nest what are currently workflows. In contrast, if nodes have no true structural hierarchy, then macros are creating new nodes on the parent workflow directly and will need to do some mangling of the "sub"-node labels to avoid conflicts. Merging these two classes is trickier business than the above two points though, and I'm not yet 100% convinced this is wise.

Friday >=1500 CET is good for me.

Sorry, I didn't write again yesterday.  The earliest I can do tomorrow is 7pm CET, @JNmpi can do earlier he told me.  I guess we can use the normal pyiron link.

So what is the actual time then? I will set an alarm for 1445 CET (0545 PST) and check for a concrete reply here, in case the time is 1500 CET... but at that point I would certainly be happy to roll over and go back to sleep until my kids wake me up.

Let's say 1915 CET then, in case there's a train delay or so.

1915 CET works for me.

I will be late.

I tried to summarize and sketch some of the ideas we had over the last few days, particularly with Jan at the IPAM workshop, in schematic graphs. They should serve to sharpen and focus the discussion rather than representing a fixed construction schema.
The first figure below shows the main components of the future node-based implementation of pyiron. An important aspect is the difference in the concepts/terms node and task. The node is the object that has all the information to  translate input into a series of tasks. A simple example would be our Murnaghan object that creates for each fixed volume a separate Lammps or VASP job (task). Another example could be the Lammps library which creates for a structure container a series of jobs.

Below is a specification of the node repository (or node store), which locally or globally stores and provides all information to run a node on any computer. For the other parts, we should construct similar sketches and augment them by pseudo code.

Some notes from our discussion today:
@JNmpi and I chatted today about graph-based pyiron computations, including taking a look at @pmrv's tiny_base PR.
First, @jan-janssen has some concerns about actually providing a snakemake interface.
That's fine, we can always come back to a @snakemake meta-node decorator in the future; for now we can shelve it.
The current thrust is to make this sort of graph computation super easy to use and sufficiently powerful and performant for code users -- forget an actual graphical representation right now.
To that end there were a few topics
Rapid access for simple nodes
After talking a bit about pyiron objects knowing their own history, we came around to the idea of storing recipes for objects in the form of simple graphs.
This lead to thinking about how nodes might be more easily used in a text context, e.g. without always having to have this trailing .outputs.foo.value to get at things.
We came up with the following ideas for the case of very simple nodes that (a) initialize with valid input for all ports, (b) evaluate quickly, and (c) have a single output:

Update these nodes at instantiation and on each input change so that the output is always populated
Modify __getattr__ so that as a last resort it tries self.outputs[0].value.__getattr__
Add syntactic sugar to connections to avoid asking for the full output path
Add syntactic sugar to indexing, and maybe assignment to exploit the output too
Here's the fun/hard part: with some usages of the output, actually go under the hood and transform the node from a single node to a macro

In fact, it might have been a macro of one node all along, so we really just add to the macro

Then we might get an example like this:
from pyiron_contrib import Workflow

wf = Workflow("my_chain_example")

structure = wf.add.node.atomistics.bulk(element="Al", repeat=5)

structure.plot3d()  # == structure.outputs.structure.value.plot3d()

structure.visualize()  # Shows the graph vis for a single node

structure[:5]  # Creates a _new node slice under the hood_
structure.visualize()
# Now we see a two-node graph, with an internal connection

structure.plot3d()  # == structure.outputs.slice_sliced.value.plot3d()
# Shows a structure with just the first five atoms
# Note that we still have only one output, so getattr works fine
# The full path to that output, however, is changed to the dynamically-created
# macro path of {node_label}_{output_channel_label}

structure.undo()
# Pops off the last node in our macro chain
structure.plot3d()  # Shows the original, full-size structure

structure[:5] = "Cu" # ***Hard***
# By some magic, this adds a different new node,
# that changes species and returns a structure, and its input is set
# to match the slice info
structure.plot3d()  # == structure.outputs.change_species_structure
# Shows the Al structure with 5 Cu atoms

structure.inputs.bulk_element = "Mg"
structure.inputs.change_species_element = "Ca"
structure.inputs.change_species_i_end = 6
structure.plot3d()
# Now we have an Mg structure with 5 Ca atoms!

Honestly, I'm not sure how we will get the magic line labeled ***Hard*** working, but I think it's a great goal.
What to do about control loops
We can currently make sophisticated graphs on-the-fly in the notebook, but since the python process and notebook are the ones aware of for/while loops, there is no way to serialize them as part of the graph a-priori.
Ultimately, I would like a fully-nodal graph to be the ground truth for workflows.
However, all the node-based for/while solutions we've seen are super ugly.
So, as previously mentioned it would be nice to offer pythonic for/while loops as a syntactic sugar for building these beasts.
Today Joerg shared some snippets from Lego Mindstorms, where these sort of flow control objects are offered graphically with a drag-and-drop interface.
This may actually work nicely with the above paradigm s.t. after we have for/while as true nodes, and then build pythonic sugar on top of those, we can build graphical sugar on top of that so that GUI users can place a sort of meta node.
E.g. for a for loop I imagine a sort of mega-node with IO like iterator, index, etc. and outputs like accumulated.
Then inside this you build a sub-graph, and connect, e.g. the lattice input of your bulk_structure node to the internal iterator connection point, and the structure output to the internal connection point of accumulator.
Then you pass in a linspace or whatever to the external connection for iterator, and can get a list of structures at the external connection to accumulator.
| o-iterator -o~~~~~~~|o- lattice : bulk_structure : structure -o|~~~~~~o- accumulator -o|
Or something like that.
At any rate, for now let's just jam loops inside the node functionality itself and keep going.
Key missing pieces
We want a stable and useful solution ASAP, that means prioritizing a few things while letting others fall by the wayside.
Priorities:

Node registries -- getting nodes from a python module, live objects from the current interpreter instance, and deserializing a cloudpickle dump (or similar) should be sufficient.

In the long run it would be nice to have some sort of global node store, including unique node identifiers and versioning, and even unwrapping URLs to download node collections, but that can all wait
Other than cloudpickle, this stuff is already in place in Ironflow so that can be used as a launching point

Asychronous execution -- Right now everything just happens on the interpreter head.

I think what we need to do is integrate with Marvin's work in tiny_base such that, when their executor is not None, nodes generate task lists from their functionality and work with an executor to get results from those tasks.
This is in pretty good agreement with Joerg's sketch of the node-task relationship from earlier.

Macros -- this is just a really critical node subclass.

Non-priorities:

Solving all the flow-control problems -- All the solutions we've seen that have node definitions of flow control are super ugly, and this is hard. Instead of getting totally hung-up on it, for now let's dump as much for/while/etc stuff into the node functionality itself

Maybe there is some nice synergy here with Marvin's more complex tasks, s.t. nodes-as-task-generators can be aware of whether they are generating individual tasks or some list of tasks.

Workflow serialization -- Ultimately this is necessary to dethrone the Jupyter notebook as the source of truth for the workflow and replace it with something better defined, but this may change substantially depending on how far we've gotten in representing flows with nodes at the most fundamental level, so just delay it and rely on re-running workflows for now.

@liamhuber, thanks for the excellent summary of our discussion. I fully agree with it. Only a few minor points/thoughts:

Regarding your code example:
- To keep the code as close as possible to regular python I would alter the behaviour of a slice as follows:
        structure_5_atoms = structure[:5]
        structure.visualize()   # visualizes the original structure
        structure_5_atoms.visualize()  # applies the node slice (contains the first 5 atoms)

        structure[:5] = "Cu" # ***Hard*** Like in 'normal' python this changes the first 5 elements, i.e., no changes

        structure.inputs.bulk_element = "Mg"
        structure.inputs.change_species_element = "Ca"
        structure.inputs.change_species_i_end = 6
        structure.plot3d()

        # should be equivalent to the following code block
        structure[:] = "Mg"
        structure[:6] = "Ca"

Workflow serialization: Having the code in the nodes, which can be fully serialized, rather than in the Jupyter notebook is already a big step forward. Since the nodes, including their underlying code are part of the workflow and stored, we would have full serialization. Having also the code in the node expressed as a workflow would be a cool feature but is not mandatory to achieve serialization. It is therefore perfectly fine to have this feature as no-priority.

Notes from 2023.05.17 meeting with Joerg
Raw notes augmented and polished on the 18th.
@JNmpi, you had a nicely updated version of the sketch in this comment, could you upload it in this thread?
Core pyiron 1.0 features:

ease of use
logging (i.e. database interaction)
submission (especially HPC cluster support)

Executors and restarting workflows
Recently @jan-janssen has been getting into flux; I experimented with the most-primitive "executor" in workflow; @pmrv earlier made Executor classes for tinybase.
Joerg and I talked a bit about how to handle restarting workflows when (a) the python process controlling the workflow gets restarted and/or (b) the process handling the task execution gets restarted.
For (a) we need to be able to reconnect nodes to the task manager they were using beforehand and recover the status of their task.
For (b) we need to restart and recover the status of the task manager itself (if we have permissions to do so) and then recover our tasks.
E.g. you can imagine the case where something like a SLURM manager goes down; it must have data serialized for recovering itself, and in principle we should then be able to have our node reconnect with the restarted and recovered SLURM instance and find out what happened to its task -- although in this case the workflow runner probably doesn't have permission to restart SLURM -- some sysadmin needs to do this -- so we need to be able to let users know that the task manager is inaccessible and how to handle this (e.g. by cleaning the node state and restarting tasks with a fresh executor/task manager).
Joerg was also excited about the hierarchical approach of Flux, giving us the option to have something like per-worflow or even per-node task management.
Sitting down to write out these notes the day after the meeting, this is my thinking on the topic -- and it may all be "duh" stuff to Marvin and Jan who have been thinking about pyiron's interaction with task scheduling for longer.
I would define a "task manager" as some python-independent and permanent/recoverable process for executing computations, and an "executor" as a python object that executes tasks generated by nodes in our workflow.
The "executor" may wrap/communicate with one of these external "task managers" -- e.g. SLURM, Flux.
When the python process controlling a workflow is killed and restarted, and we de-serialize the workflow state, we'll need to re-instantiate the "executors" too.
In the case that these "executors" are thin wrappers for "task managers", the re-instantiated executor needs to recover all the statuses of its tasks from the "task manager", e.g. by having stored the PID for the "task manager" and recovering a connection from there.
Here it becomes very useful that the "executors" are also just python objects, because now we can cleanly register callbacks from "executor" to node.
In case the background "task manager" process has also died, we may

Restart this as well, e.g. by having the "executor" run some bash code to restart the other process and de-serialize the "task manager" state and reconnect as usual, in the case that we have permission to do so (maybe for workflow-specific Flux managers? I don't know)
Fail hard, clear, and clean in case we don't have these sort of privileges over the "task manager" process (e.g. a centralized SLURM deployment on some HPC)

When the "executor" very simply runs tasks modally on the main workflow python process, this is all trivial.
When the "executor" packages and submits tasks to a "task manager" the behaviour is fairly clear, as above.
I see one intermediate case, where the "executor" starts background tasks using something like concurrent.multiprocessing, or the node task is running some MPI shell script on local resources; here we probably need to handle resource management (i.e. scheduling multi-core tasks over our finite local number of cores) and "task manager"-restarting (i.e. (de)serialization of the manager state) entirely ourselves.
There is obviously some strong overlap with dask's resiliancy policies, although in my dream-behaviour above, we would find a way to handle (some things like) scheduler failure more robustly.
There is probably also some stuff to be learned from dask_jobqueue.
Ultimately I'm not too worried about overlap with dask capabilities here, as we are looking to be more tailored to HPC environments from the get-go, and to support possibly cyclic graphs (at the cost of graph-execution optimization).
In all cases, the end user should see an extremely similar interface for their Executor class, regardless of how the executor is handling tasks in the background.
GUI stuff
In terms of GUIs, ironflow's dependence on ipycanvas means that more complex features -- like connection lines that automatically bend to flow around objects, or meta-nodes with snappable slots for actual nodes, etc. -- are going to be a huge pain to implement from scratch.
I want a higher level GUI platform, but didn't know of anything that keeps our support for the jupyter environment (which is a nice feature, especially given pyiron's history and current deployment).
Joerg reminded me that in NFDI MatWerk there is some Java library for GUIs that is now accessible inside Jupyter notebooks.
We think we discussed it before, but it's been a while and the details are fuzzy for me already.
This maybe provides a jupyter-compliant route for implementing these more advanced graphical features though!
We did a nice meta-node sketch for loops in the graph, and agreed that under the hood these would probably still construct pure graphs (sort of like Unreal Blueprints flow management, which I like formally, but which are hell to read and write!).
Below is a sketch for what a slottable macro-node might look like.
Note:

Mapping is possible from an iterables onto what to iterate over in the slotted node instance
We should be able to change the number of iterables in and out
We still allow regular connections to the slotted node (these are held constant over the iterations)
Under the hood this creates an uglier pure-node macro like seen in Unreal blueprints for-loop
A similarly convenient macro-node interface should be available to code-based users when writing their workflows, although obviiously there we don't need to worry about the slotted node "snapping" nicely into place.

Maintainability and classes
We want a few interfaces like Executor and Serializer that may be replaced to have different behaviour -- e.g. SLURM vs multiprocess discussed above for the Executor or HDF5 vs S3 for serialization, whatever.
These should look as similar as possible to the end user.
These interfaces should have testing to make sure that what happens under the hood is staying compliant with the thing we're interfacing to!
We want to update the user-facing side of the interface as seldom as possible, but what we do under the hood we can update as much as we want.
In particular, a Workflow instance might hold a collection of Executor instances;
If a Node instance tries to create a new Executor instance, we should check if this executor interface has already been instantiated and is being held by the node and just use that (singleton-esque), otherwise make a new one and add it to the owning workflow's list.
Similarly, we should give convenience methods for updating what type of Serializer is used across an entire workflow at once (although in principle you should still be able to define per-node how serialization is handled).
Language power and usability
Joerg has recently looked into Julia a bit and was particularly keen on how it handles multiple dispatch.
Not that we should switch pyiron1.0 over to Julia, but just that there are some concepts here that we could pull in better to this graph-and-node framework.
E.g., we have talked about having hierarchically defined IO classes, like AtomisticOutput-MDOutuput(AtomisticOutput)-LammpsOutput(MDOutput)-VASPOutput(MDOutput,DFTOutput), etc.
We can imagine then a situation where we have a CalcMD node who simply has MDOutput.
Now suppose we had some FiniteTBulkModulus node that took a bunch of temperatures and pressures;
We could empower the node so that it accepts either different arrays to each of these inputs, or just (the same) MDOutput to each and it knows how to extract temperature and pressure from an MDOutput object.
There is also room to synergize with or replace our ontological typing here, as in my made-up example we clearly want NVT MD.
E.g. 2, the Plot3D node could support a Atoms with NGLView, or automatically provide a different viewer with electron iso-surfaces if you pass data with electron densities, or automatically support animation if you pass a list of structures or MDOutput.
Node packages
We also talked briefly about version control and node packages.
In principle, each serialized node will need to know which version of its node package it is from, but there is no problem mixing-and-matching nodes in a given workflow from different versions of the same node package -- as long as the IO connections are valid, the workflow shouldn't care if it uses nodes from multiple packages, and different versions of the same package is just equivalent to different packages.
We'll ultimately need to support versioning in our package loading/registration, and how to get the namespacing ok, but this is just an implementation detail and not a fundamental barrier.
The one catch is that node package versions will need to be consistent with the version of pyiron being used, e.g. in case we change something like the Executor interface and the node's internal instructions are no longer parseable.
But breaking backwards compatibility between pyiron1.0 and specific nodes should happen much less frequently than updates to a node's behaviour, so that's not a deal breaker.

Today I spent some time playing with the idea of macros. Nothing is running yet, but I have some very rough spec ideas and pseudocode.
As we've discussed before, I am thinking of a macro as a sort of crystalized workflow. As such,

It should have a nodes attribute just like a workflow, as well as the ability to add/create nodes onto itself

At least until the initialization is finished, then we may want to disable this

IO should default to the same thing as it does in workflows: unconnected channels from all the child nodes
But you should be able to provide a map to give additional convenient channels

Maybe to rename something, e.g. suppose some node "foo" has output "y", instead of "foo_y", we can make a map so it's just "y"
Maybe to provide sychronized setting for input, e.g. "node1_x" and "node2_x" can be updated simultaneously by updating the macro's "x" channel
Maybe to break in and provide special access to connected ports -- this one seems a bit dangerous (at least to allow for input, output is fine) given the assumption that macros are "crystalized"

They should be creatable from workflows or a decorator (or from a class instantiation, but that should be a boring and more verbose duplicate of the decorator pathway, so don't worry about it).

Here's the syntax I'm playing around with:
from pyiron_contrib.workflow import Workflow

@Workflow.wrap_as.single_value_node("sum")
def add(x: int = 0, y: int = 0) -> int:
    return x + y

macro = Workflow("plus_minus_one")
macro.p1 = add(y=1)
macro.m1 = add(y=-1)

# Choice 1) Use the default interface
plus_minus_one_default = wf.to_macro()

wf_default = Workflow("double_it_default")
wf_default.start = add()
wf_default.macro = plus_minus_one(
    p1_x=wf_default.start,
    m1_x=wf_default.start
)
wf_default.end = add(
    x=wf_default.macro.outputs.p1_sum,
    y=wf_default.macro.outputs.m1_sum
)

# Choice 2) Define a new interace
plus_minus_one_custom = wf.to_macro(
    # inputs={
    #     "x": (macro.p1.inputs.x, macro.m1.inputs.x)
    # },  # This way?
    inputs={
        macro.p1.inputs.x: "x",
        macro.m1.inputs.x: "x"
    }  # Or this way? For linking two inputs to a single channel
    outputs={
        "p1": macro.p1,
        "m1": macro.m1
    }
)

wf_custom = Workflow("double_it_custom")
wf_custom.start = add()
wf_custom.macro = plus_minus_one_custom(x=wf_custom.start)
wf_custom.end = add(
    x=wf_custom.macro.outputs.p1,
    y=wf_custom.macro.outputs.m1
)

# Choice 3) With a decorator
@Workflow.wrap_as.macro()
def plus_minus_one_deco(macro):
    """
    Macro wrapped functions take the macro as the first and only argument
    (which is a lot like a workflow), create nodes and make connections,
    and return inputs and outputs maps for giving special access
    """
    macro.p1 = add(y=1)
    macro.m1 = add(y=-1)
    return {macro.p1.inputs.x: "x", macro.m1.inputs.x: "x"}, {}

wf_deco = Workflow("double_it_deco")
wf_deco.start = add()
wf_deco.macro = plus_minus_one_deco(x=wf_deco.start)
wf_deco.end = add(
    x=wf_deco.macro.outputs.p1_sum, # We didn't map these
    y=wf_deco.macro.outputs.m1_sum # So use the default
)

for wf in [wf_default, wf_custom, wf_deco]:
    for i in range(5):
        # All cases should do the same boring thing, and should do it
        # automatically since the children are SVNodes
        assert(2 * wf.inputs.start_x.value == wf.outputs.end_y)

The fact that the macro's child node is defined in the notebook really pushes at the question of how to best store macros. Of course I'd love if under the hood they just stored class names and connection lists and re-instantiated nodes from known libraries... but perhaps sometimes we will really need to cloudpickle node instances and reinstantiate by unpicking the whole thing.


⚠ API - Lens has degraded performance

In 29e3b00, API - Lens ($LENS_API_URL) experienced degraded performance:

HTTP code: 502
Response time: 367 ms

Resolved: API - Lens performance has improved in 8d0709d after 6 minutes.


change the method of initializing slice

[x] Do only one thing
[x] Non breaking API changes
[x] Tested

What did this pull request do?
Hi, I'm sorry for the trouble I caused you earlier. I'll be careful next time.
This is a small change, but it will send a PR.
There are places where the slice capacity is not specified.
I thought this would be better for Go's slice specification, since the capacity of the slice to be appended can be allocated in advance by the length.

Oops, the run all tests in the local tests directory went through.
testing sqlite...
ok      gorm.io/gorm    0.845s
?       gorm.io/gorm/callbacks  [no test files]
ok      gorm.io/gorm/clause     2.513s
ok      gorm.io/gorm/logger     0.851s
?       gorm.io/gorm/migrator   [no test files]
ok      gorm.io/gorm/schema     1.325s
ok      gorm.io/gorm/utils      0.909s
?       gorm.io/gorm/utils/tests        [no test files]
go: finding module for package gorm.io/driver/postgres
go: finding module for package gorm.io/driver/mysql
go: finding module for package gorm.io/driver/sqlite
go: finding module for package gorm.io/driver/sqlserver
go: found gorm.io/driver/mysql in gorm.io/driver/mysql v1.0.4
go: found gorm.io/driver/postgres in gorm.io/driver/postgres v1.0.8
go: found gorm.io/driver/sqlite in gorm.io/driver/sqlite v1.1.4
go: found gorm.io/driver/sqlserver in gorm.io/driver/sqlserver v1.0.6
ok      gorm.io/gorm/tests      8.099s
testing mysql...
ok      gorm.io/gorm    0.339s
?       gorm.io/gorm/callbacks  [no test files]
ok      gorm.io/gorm/clause     1.449s
ok      gorm.io/gorm/logger     0.135s
?       gorm.io/gorm/migrator   [no test files]
ok      gorm.io/gorm/schema     0.559s
ok      gorm.io/gorm/utils      0.214s
?       gorm.io/gorm/utils/tests        [no test files]
ok      gorm.io/gorm/tests      44.006s
testing postgres...
ok      gorm.io/gorm    0.172s
?       gorm.io/gorm/callbacks  [no test files]
ok      gorm.io/gorm/clause     1.218s
ok      gorm.io/gorm/logger     0.181s
?       gorm.io/gorm/migrator   [no test files]
ok      gorm.io/gorm/schema     0.554s
ok      gorm.io/gorm/utils      0.134s
?       gorm.io/gorm/utils/tests        [no test files]
ok      gorm.io/gorm/tests      22.010s
testing sqlserver...
ok      gorm.io/gorm    0.183s
?       gorm.io/gorm/callbacks  [no test files]
ok      gorm.io/gorm/clause     1.629s
ok      gorm.io/gorm/logger     0.153s
?       gorm.io/gorm/migrator   [no test files]
ok      gorm.io/gorm/schema     0.815s
ok      gorm.io/gorm/utils      0.135s
?       gorm.io/gorm/utils/tests        [no test files]
ok      gorm.io/gorm/tests      28.078s


Unit tests build time has degraded

Building locally, rebuild after 1 small change in a test takes 3min. This eliminates the advantages of unit tests and should be fixed asap.

https://github.com/rdkcentral/rdkservices/pull/3773

https://github.com/rdkcentral/rdkservices/pull/3778


Support external document linkage for inter-related Metanorma documents

https://asciidoctor.org/docs/user-manual/#inter-document-cross-references
We should support cross-document linkage, for example, for a set of inter-related Metanorma documents.

Ah, this is what I feared. I need not fear it if we do it the way Asciidoctor does it (as does RFC 3, for that matter): include the HTML anchor in the crossreference, rather than try to infer it from the citation.
This requires an extension to the grammar.

The out-of-the-box way to do it (and I'd need a compelling reason for us not to) is to treat this as a link to an HTML anchor. Even if Word understood the links, it would not do anything sensible with them. So:
link:external.html#id3[The document link]

which metanorma-iso already converts, without any change, to
<link target="external.html#id3">The document link</link>

and thence
The document link

If you want to do something extra, you'll need to argue for it. This approach is robust; it only works for HTML, but it only needs to work for HTML. Word documents will produce a hyperlink, but clicking on them will go to an HTML page not a Word document. But then again, do you want it to go looking for a Word document, that likely won't be there at all?

Need feedback on this

We need direct linkage to the source rather than a pre-compiled HTML document.

Why?
I need you to provide me with requirements for this. I'm not seeing them: this is only ever going to work for HTML source and target anyway. What behaviour do you expect under PDF or DOC formats?
Asciidoctor resolves the link to a precompiled HTML document under HTML, and to a precompiled DocBook document under DocBook:
<>
c
<link xl:href="b.xml#x">c</link>
If this is being transformed into Metanorma XML, it will need to choose a suffix to render to in Metanorma XML: <link target="document-b.XXX#section-b">.  We could make XXX be "xml" in Metanorma XML, and then transform it into HTML for HTML output; but right now, I can't see it being anything other than HTML for PDF or DOC output, either.
I'm not comfortable changing the file extension from XML to MNX.

I think for link target="document-b.XXX#section-b we should make XXX as according to the target format, i.e. HTML links to HTML, PDF to PDF.

... Anchors will only work in HTML, but fine.

I am leaving it in Metanorma XML as just <link target="document-b#section-b">: I don't see why we should be specifying a spurious filetype suffix at all. I will inject the appropriate file extension in isodoc for each type.
I continue to regard extending this functionality beyond HTML as pointless.


Query parameter serialization issue between TSED versions

Describe the bug
Description
We recently upgraded from TSED version v7.67.5 to v7.84.1 and are experiencing a change in the way query parameters are serialized, specifically for nested objects in the query string.
Actual Behavior
In version v7.84.1, the query parameters for the nested object are being serialized as a single string, like this:
/api/orders?filter[settled_on]={"gte":"2024-11-25T20:16:16.975Z","lte":"2024-11-25T20:16:16.975Z","gt":"2024-11-25T20:16:16.975Z","lt":"2024-11-25T20:16:16.975Z"}&filter[sale]=ABSD_23

This is causing issues with parsing the query parameters correctly in the backend.
To Reproduce

Use the following code in a TSED controller using the v7.84.1

import { Controller } from "@tsed/di";
import { QueryParams } from "@tsed/platform-params";
import { Format, Get, Name, Nullable } from "@tsed/schema";

class DateFilter {
  @Nullable(Date)
  @Format("date-time")
  gte: Date | null;

  @Format("date-time")
  @Nullable(Date)
  lte: Date | null;

  @Format("date-time")
  @Nullable(Date)
  gt: Date | null;

  @Format("date-time")
  @Nullable(Date)
  lt: Date | null;
}

class OrderFilter {
  @Nullable(DateFilter)
  settled_on: DateFilter | null;

  @Name("sale")
  sale: string;
}

@Controller("/orders")
export class OrderController {
  @Get("/")
  get(@QueryParams("filter") filter: OrderFilter) {
    return;
  }
}

Use Swagger UI to make a request to /orders with the following query parameters as filter:

{
  "settled_on": {
    "gte": "2024-11-25T20:16:16.975Z",
    "lte": "2024-11-25T20:16:16.975Z",
    "gt": "2024-11-25T20:16:16.975Z",
    "lt": "2024-11-25T20:16:16.975Z"
  },
  "sale": "ABSD_23"
}

The API sends the query parameters in the following format

/api/orders?filter[settled_on]={"gte":"2024-11-25T20:16:16.975Z","lte":"2024-11-25T20:16:16.975Z","gt":"2024-11-25T20:16:16.975Z","lt":"2024-11-25T20:16:16.975Z"}&filter[sale]=ABSD_23

Use the same code snippet now using the version v.7.67.5
Use Swagger UI to make the same request to /orders with the same query parameters object.
The API sends the query parameters in the following format

/api/orders?filter[settled_on][gte]=2024-11-25T20:16:16.975Z&filter[settled_on][lte]=2024-11-25T20:16:16.975Z&filter[settled_on][gt]=2024-11-25T20:16:16.975Z&filter[settled_on][lt]=2024-11-25T20:16:16.975Z&filter[sale]=ABSD_23

Notice the difference in how query parameters are serialized between v7.67.5 and v7.84.1.

Expected behavior
Expected Behavior
In version v7.67.5, the query parameters for a nested object (e.g., settled_on in OrderFilter) are serialized as multiple query parameters, such as:
/api/orders?filter[settled_on][gte]=2024-11-25T20:16:16.975Z&filter[settled_on][lte]=2024-11-25T20:16:16.975Z&filter[settled_on][gt]=2024-11-25T20:16:16.975Z&filter[settled_on][lt]=2024-11-25T20:16:16.975Z&filter[sale]=ABSD_23

This behavior is working as expected in v7.67.5 so I expect the same behavior when using the latest version of TSED v.7.84.1
Code snippets
No response
Repository URL example
No response
OS
macOS
Node version
Node 18.18.1
Library version
v7.84.1
Additional context
I tried to downgrade the package @tsed/swagger from v7.84.1 to v7.67.5 to fix this behavior, however, according to TSED framework Docs it is recommendable to have all @tsed/* packages in the same version. https://tsed.io/getting-started/#update-dependencies

Hi @orlandorode97
I don't see pertinent change in the commit history in Ts.ED accepted the swagger-ui bump version dependency.
That I known, deep query parameters style is supported by Ts.ED since a long time (v6.65.0) with Open API 3:
/api/orders?filter[settled_on][gte]=2024-11-25T20:16:16.975Z&filter[settled_on][lte]=2024-11-25T20:16:16.975Z&filter[settled_on][gt]=2024-11-25T20:16:16.975Z&filter[settled_on][lt]=2024-11-25T20:16:16.975Z&filter[sale]=ABSD_23

See documentation here:
https://rc.tsed.dev/docs/model.html#deep-object-on-query or https://tsed.io/docs/model.html#deep-object-on-query
While your version isn't expected and described on Ts.ED documentation. deepObject: style by default since v6.65.0, I don't know how you didn't have the problem before.
Express.js/Koa.js support deepObject: style query parameters parsing and it shouldn't be a problem. I recently used this feature, (and there are some integration test) which proves that it works as expected.
So, I encourage you to create a small project using Ts.ED CLI to have the latest project configuration, adding your models and controller, then start the project and log the received model in the controller method.
You'll see that the instance is correctly parsed. You've have probably something wrong in your side ;)
I change the label of this issue (not a bug), isn't a bug, because deepObject style is enabled since v6.65.0, parsing works, and your version isn't covered by the documentation.
See you

Hello @Romakita
Thanks a lot for the detailed response!
I followed your suggestion and created a new small Ts.ED project using the CLI. After testing with Postman, I can confirm that the query parameters are parsed correctly, and everything works as expected.
The issue, however, seems to be specific to Swagger UI. When I try to test the query parameters directly through Swagger UI, I still face issues with the parsing. I have a few clients that use my API through Swagger UI
The swagger.json generated is different between the versions v7.84.1 and v.7.67.5
v7.84.1:
{
  "openapi": "3.0.1",
  "info": { "title": "Api documentation", "version": "1.0.0" },
  "paths": {
    "/rest/orders": {
      "get": {
        "responses": { "200": { "description": "Success" } },
        "parameters": [
          {
            "in": "query",
            "name": "filter",
            "required": false,
            "style": "deepObject",
            "schema": { "$ref": "#/components/schemas/OrderFilter" }
          }
        ],
        "tags": ["OrderController"],
        "operationId": "orderControllerGet"
      }
    }
  },
  "tags": [{ "name": "OrderController" }],
  "components": {
    "schemas": {
      "OrderFilter": {
        "type": "object",
        "properties": {
          "settled_on": {
            "anyOf": [{ "$ref": "#/components/schemas/DateFilter" }], <--- Here uses anyOf
            "nullable": true
          },
          "sale": { "type": "string" }
        }
      },
      "DateFilter": {
        "type": "object",
        "properties": {
          "gte": { "type": "string", "format": "date-time", "nullable": true },
          "lte": { "type": "string", "format": "date-time", "nullable": true },
          "gt": { "type": "string", "format": "date-time", "nullable": true },
          "lt": { "type": "string", "format": "date-time", "nullable": true }
        }
      }
    }
  }
}

v7.67.5:
{
  "openapi": "3.0.1",
  "info": { "title": "Api documentation", "version": "1.0.0" },
  "paths": {
    "/rest/orders": {
      "get": {
        "responses": { "200": { "description": "Success" } },
        "parameters": [
          {
            "in": "query",
            "name": "filter",
            "required": false,
            "style": "deepObject",
            "schema": { "$ref": "#/components/schemas/OrderFilter" }
          }
        ],
        "tags": ["OrderController"],
        "operationId": "orderControllerGet"
      }
    }
  },
  "tags": [{ "name": "OrderController" }],
  "components": {
    "schemas": {
      "OrderFilter": {
        "type": "object",
        "properties": {
          "settled_on": {
            "nullable": true,
            "oneOf": [{ "$ref": "#/components/schemas/DateFilter" }] <- Here uses oneOf
          },
          "sale": { "type": "string" }
        }
      },
      "DateFilter": {
        "type": "object",
        "properties": {
          "gte": { "type": "string", "format": "date-time", "nullable": true },
          "lte": { "type": "string", "format": "date-time", "nullable": true },
          "gt": { "type": "string", "format": "date-time", "nullable": true },
          "lt": { "type": "string", "format": "date-time", "nullable": true }
        }
      }
    }
  }
}

Would you happen to have any suggestions on how to get Swagger UI to properly handle these parameters?
Thanks again for your help!

Hi @orlandorode97
I'm not sure that anyOf impact the swagger behavior in this case. But I can't change that, because, anyOf is related to an issue with nullable + a schema. So revert it isn't possible.
After adding your second swagger.json I add that:
v7.84.1:

and v7.67.5:

You can see that the form is the same. anyOf and oneOf doesn't impact the query parameters format. the only option that impact the query parameters style is: deepObject

Note: The curl is the same between the first and second swagger on swagger-ui editor. But I don't understand where the problem final, because, the serveur parse correctly the followings curl generated by swagger-ui:
curl -X 'GET' \
  'https://editor.swagger.io/rest/orders?settled_on=%7B%22gte%22%3A%222024-12-02T15%3A16%3A10.177Z%22%2C%22lte%22%3A%222024-12-02T15%3A16%3A10.177Z%22%2C%22gt%22%3A%222024-12-02T15%3A16%3A10.177Z%22%2C%22lt%22%3A%222024-12-02T15%3A16%3A10.177Z%22%7D&sale=string' \
  -H 'accept: */*'

And swagger-ui let you fill a json as value. So this is not a problem.


grafana image

New Image Pull Request Template

Image Size

[X] The Image is smaller in size than its common public counterpart.
[ ] The Image is larger in size than its common public counterpart (please explain in the notes).

Notes:
Original image: 392MB
Chainguard image: 387MB
Image Vulnerabilities

[ ] The Grype vulnerability scan returned 0 CVE(s).
[ ] The Grype vulnerability scan returned > 0 CVE(s) (please explain in the notes).

Notes:
Original image: 22 vulnerabiliites
Chainguard image: 2 Vulnerabilities (Still WIP - This PR is a draft)
Image Tagging

[ ] The image is not tagged with version tags.
[X] The image is tagged with :latest
[ ] The image is not tagged with :latest (please explain in the notes).

Notes:
Basic Testing - K8s cluster

[X] The container image was successfully loaded into a kind cluster.
[ ] The container image could not be loaded into a kind cluster (please explain in the notes).

Notes:
Basic Testing - Package/Application

[X] The application is accessible to the user/cluster/etc. after start-up.
[ ] The application is not accessible to the user/cluster/etc. after start-up. (please explain in the notes).

Notes:
Helm

[X] A Helm chart has been provided and the container image can be used with the chart.  If needed, please add a -compat package to close any gaps with the public helm chart.
[ ] A Helm chart has been provided and the container image is not working with the chart (please explain in the notes).
[ ] A Helm chart was not provided.

Notes:
well maintained helm chart - documented in README clearly with example.
Processor Architectures

[X] The image was built and tested for x86_64.
[ ] The image could not be built for x86_64 (please explain in the notes).
[X] The image was built and tested for aarch64.
[ ] The image could not be built for aarch64. (please explain in the notes).

Notes:
Functional Testing + Documentation

[X] Functional tests have been included and the tests are passing.  All tests have been documented in the notes section.

Notes:
Helm chart test - and also some checks to look for error logs and good known log lines whenever the pod should be running cleanly.
Environment Testing + Documentation

[X] There has not been a request and/or there is no indication that this image needs tested on a public cloud provider.
[ ] The container image has been tested successfully on a public cloud provider (AWS, GCP, Azure).
[ ] The container image has not been tested successfully on a public cloud provider (AWS, GCP, Azure) (please explain in the notes).

Notes:
Version

[X] The package version is the latest version of the package.  The latest tag points to this version.
[ ] The package version is the not the latest version of the package (please explain in the notes).

Notes:
Dev Tag Availability

[X] There is a dev tag available that includes a shell and apk tools (by depending on 'wolfi-base')
[ ] There is not a dev tag available that includes a shell and apk tools (by depending on 'wolfi-base') (please explain in the notes).

Notes:
Access Control + Authentication

[X] The image runs as nonroot and GID/UID are set to 65532 or upstream default
[ ] Alternatively the username and GID/UID may be a commonly used one from the ecosystem e.g: postgres
[ ] The image requires a non-standard username or non-standard GID/UID (please explain in the notes).

ENTRYPOINT

[X] applications/servers/utilities set to call main program with no arguments e.g. [redis-server]
[ ] applications/servers/utilities not set to call main program with no arguments e.g. [redis-server] (please explain in the notes)
[ ] base images leave empty.
[ ] base image and not empty (please explain in the notes).
[ ] dev variants is set to entrypoint script that falls back to system.
[ ] dev variants is not set to entrypoint script that falls back to system (please explain in the notes).

CMD

[ ] For server applications give arguments to start in daemon mode (may be empty)
[ ] For utilities/tooling bring up help e.g. –help
[ ] For base images with a shell, call it e.g. [/bin/sh]

Environment Variables

[ ] Environment variables added.
[X] Environment variables not added and not required.

 Logs DO NOT write to file.

Documentation - README

[X] A README file has been provided and it follows the README template.

This is awesome - let me know when this is ready for review!

I think the README check has been enabled now!


Importers Not Working

I have an importer connected to a Creative Controller (it shows up in the GUI, so I know it's working correctly); however, it will not pull items from a chest placed next to it's input face.

Did you connect a storage device like a storage block or drive with a disk in?

Yes, a drive with disks. I had a functioning network and could add or remove items through the crafting grid.

Do they work again when you break and replace them? If not, can you share a screenshot of your setup?

http://imgur.com/a/r5y5Z

Thanks, will try to reproduce this.

Does it work with a normal chest (so not an iron one)?

I've tried with regular, trapped, and iron. None of them are working for me.

I'm unable to reproduce this. Are you sure that:

there is still space in your system (check the disk drive GUI)
you are on the latest version

Yes to both, I added 8 16k storage and I've only filled a handful of slots and I'm running 0.5.5.
My best guess at this point is a mod conflict so I'll try to do some more testing.

This is my setup used in testing:

Ah ha! It works if I place the importer face up but if I place it facing horizontally and then use the wrench to change the input to face up it doesn't!

There we go :) I'll investigate this.

From what mod is the wrench you were using?


Extending writable sort order to be public

Now writeable_sort_order accepts either BaseEntity objects
or typ strings.  Its tested too.  To be used in citrine-python.

This is getting pulled out of https://github.com/CitrineInformatics/citrine-python/pull/223


Not able to config JDBC input

Hello everyone, below is my configuration -
input {
jdbc {
jdbc_driver_class => "org.apache.derby.jdbc.EmbeddedDriver"
jdbc_connection_string => "jdbc:derby://localhost:3306/test"
jdbc_user => "root"
jdbc_password => "password"
statement => "select * from user"
}
}
When I run the config file, I get the following error -
The error reported is:
 org.apache.derby.jdbc.EmbeddedDriver not loaded
Can anyone please help me resolve it? My current Java version is -
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)
I'm also confused with the line  jdbc_connection_string => "jdbc:derby:memory:testdb;create=true". I have database named test. How do I mention in the above connection_string?

No JDBC driver comes prepackaged within the plugin. To properly load a driver for use with the plugin, you must either add it to your java's classpath or supply the driver jar with the following parameter:
input {
  jdbc {
    jdbc_driver_library => "/path/to/derbyclient.jar"
    jdbc_driver_class => "org.apache.derby.jdbc.EmbeddedDriver"
    ...
  }
}

you can find the client library in the derby release downloads page here: http://db.apache.org/derby/releases/release-<IP_ADDRESS>.cgi.
let me know if that helps!

Thank you Talevy. It did solve my problem partially. I added the line  jdbc_driver_library => "/home/pavan/Downloads/db-derby-<IP_ADDRESS>-bin/lib/derby.jar" to my configuration file but I'm getting a new error now.
The error reported is:
 Sequel::DatabaseError: driver.new.connect returned nil: probably bad JDBC connection string
Java::JavaSql::SQLException: No suitable driver found for jdbc:derby://localhost:3306/memory:test;create=true
I have a database by name test in phpmyAdmin and I want to pull data from it and index in elasticsearch. Can you please provide me the right connection string for it. Here's what I've tried ` jdbc_connection_string => "jdbc:derby://localhost:3306/memory:test;create=true" '

I am not familiar with phpmyAdmin. are you using a different database possibly, maybe MySQL?

I'm using MySQL but not able to overcome that error :(

@PavanRGowda
First you need to download the correct JDBC driver. If you are using a MySQL Engine you the to use the MySQL JDBC Driver instead of Derby JDBC Driver. You can download the correct driver from here: https://dev.mysql.com/downloads/connector/j/5.1.html
After you get the driver you need to use the correct jdbc_connection_string. In you case it should be something like:
jdbc:mysql://localhost/test
With this Connection String you will connect to Database test on your localhost.
Also you need to use the correct driver_class, for MySQL it's: com.mysql.jdbc.Driver
If that don't work, try Java::com.mysql.jdbc.Driver
The driver name should be something like "mysql-connector-java-5.1.36-bin.jar"

Hi, if you have more debugging questions, please move it to discuss. question/discuss here: https://discuss.elastic.co/c/logstash


[FR] Use TypeId for all models

Please verify that this feature request has NOT been suggested before.

[X] I checked and didn't find a similar feature request

Problem statement
Ids/PK between instances are not unique.
Suggested solution
Add TypeIds to all models to make Ids typesafe and unique between instances.
Describe alternatives you've considered
UUID4, GUIDs, there are multiple ways to go around this.
Examples of other systems
No response
Do you want to develop this?

[ ] I want to develop this.

This would also be usable for:
https://github.com/inventree/InvenTree/discussions/4920 / @roman-dvorak
https://github.com/inventree/InvenTree/issues/4562 / @p-fruck

Typeid sounds pretty cool, learned something new, but how do you plan the migration for instances that already use numeric ids?

This would be additional to the pk's we already have as all the database stuff is optimized for numeric pks.

Great!
It looks enough like what I wanted in my FR-Issue. I think that one of the essential criteria of choosing such an identifier is how long the code is and how it can be encoded into different barcodes.
TypeID looks like Interesting extension of UUID. I didn't know it. Thank you.

This is currently just a proposal, implementation is not planned any time soon - at least by me.

Still an interesting idea.

not stale

not stale


RE2 Crashing on startup

re2_framework_log.txt

Is the above what is needed to diagnose the issue? Thanks again for any help hope to get this working

What is your system info? Graphics card, cpu, etc.

Thanks so much for replying! Ive tried for hours. Disabled antivirus re
installed everything. Checked all drivers. Still nothing.
Details below
AMD Ryzen 9 3900 12-Core Processor                3.10 GHz
32.0 GB
64-bit operating system, x64-based processor
NVIDIA GeFore RTX 3070
Need any other details?
Thanks again, It would be amazing if you can help me to get this to work
Ben
Ben Whitmore Motion Graphics
Contact<PHONE_NUMBER>8
Email @.***
http://benwhitmore24.wixsite.com/mgfx
On Fri, Jan 14, 2022 at 8:22 PM dmitsuki @.***> wrote:

What is your system info? Graphics card, cpu, etc.
—
Reply to this email directly, view it on GitHub
https://github.com/praydog/REFramework/issues/187#issuecomment-1013448850,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AXJLI5OPNDS4KDICVW3POY3UWCAZ7ANCNFSM5L4P4M2A
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you authored the thread.Message ID:
@.***>

First things first, have you tried changing to DX11? You can either open the game in non VR and do that or can do it through the reconfig ini file in the root directory of re2.

Thanks again for helping.
Yes set to DX11 in the graphics settings.
Also optimised settings in game by turning off or down
Sent from my iPhone

On 14 Jan 2022, at 20:46, dmitsuki @.***> wrote:

First things first, have you tried changing to DX11? You can either open the game in non VR and do that or can do it through the reconfig ini file in the root directory of re2.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you authored the thread.

[image: IMG_0155.jpg]
Please see the files that downloaded from the link. Maybe I don't have
all the correct files? There is no "ini file"
Thanks again. I will be donating if you can help to get this to work for
sure.
Ben
On Fri, Jan 14, 2022 at 9:02 PM @.***> wrote:

Thanks again for helping.
Yes set to DX11 in the graphics settings.
Also optimised settings in game by turning off or down
Sent from my iPhone
On 14 Jan 2022, at 20:46, dmitsuki @.***> wrote:

First things first, have you tried changing to DX11? You can either open
the game in non VR and do that or can do it through the reconfig ini file
in the root directory of re2.
—
Reply to this email directly, view it on GitHub
https://github.com/praydog/REFramework/issues/187#issuecomment-1013462679,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AXJLI5J3C7TYWNWMEB6XCFTUWCDUDANCNFSM5L4P4M2A
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you authored the thread.Message ID:
@.***>

Are you launching the game directly from Steam?

So I launch occulus client,steam,steam vr. Connect via airlink then launch the game on desktop in steam. Blackscreen then warning message resident evil not supported in vr etc etc
Sent from my iPhone

On 15 Jan 2022, at 12:30, lucasfisher0 @.***> wrote:

Are you launching the game directly from Steam?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you authored the thread.

Last line of the log: VRCompositor failed to initialize.
That is a SteamVR issue. I don't know how the airlink works, but I would try this:

Start Oculus and AirLink
Do not launch SteamVR
Launch RE directly from Steam using the desktop
???

Ok cool. I will try that.
Doesnt steam have to use steam vr to play games in vr?i think it automatically boots steam vr when playing a vr game via steam?
What would be the best setup if not using airlink? Remote desktop?
Do i have all files needed by the way?
Thanks again
Sent from my iPhone

On 15 Jan 2022, at 14:41, lucasfisher0 @.***> wrote:

Last line of the log: VRCompositor failed to initialize.
That is a SteamVR issue. I don't know how the airlink works, but I would try this:
Start Oculus and AirLink
Do not launch SteamVR
Launch RE directly from Steam using the desktop
???
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you authored the thread.

When the mod needs SteamVR, it should automatically start. There are non-Steam games that use SteamVR, but since the game is on Steam but isn't marked as a VR game, it sometimes gives me issues launching.

Ok great will try this when im home and let you know how i get on. I think it will crash again as i have tried it i think without steam vr being active but will try again.any other trouble dhooting i could try before i get back in touch?
Thanks very much
Sent from my iPhone

On 15 Jan 2022, at 15:59, lucasfisher0 @.***> wrote:

When the mod needs SteamVR, it should automatically start. There are non-Steam games that use SteamVR, but since the game is on Steam but isn't marked as a VR game, it sometimes gives me issues launching.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you authored the thread.

It worked!!! Loads in with some stuttering but works!Thanks so much for
your help!
2 more final things. How do you turn of vignette and how do you switch to
first person as a menu did not come up at the beginning even when I point
to my wrist as recommended etc
Thanks again
How do I donate?
Ben
Ben Whitmore Motion Graphics
Contact<PHONE_NUMBER>8
Email @.***
http://benwhitmore24.wixsite.com/mgfx
On Sat, Jan 15, 2022 at 4:08 PM @.***> wrote:

Ok great will try this when im home and let you know how i get on. I think
it will crash again as i have tried it i think without steam vr being
active but will try again.any other trouble dhooting i could try before i
get back in touch?
Thanks very much
Sent from my iPhone
On 15 Jan 2022, at 15:59, lucasfisher0 @.***> wrote:

When the mod needs SteamVR, it should automatically start. There are
non-Steam games that use SteamVR, but since the game is on Steam but isn't
marked as a VR game, it sometimes gives me issues launching.
—
Reply to this email directly, view it on GitHub
https://github.com/praydog/REFramework/issues/187#issuecomment-1013706220,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AXJLI5L3BJ5CHJOKLWB6OMTUWGKU3ANCNFSM5L4P4M2A
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you authored the thread.Message ID:
@.***>

All sorted. Found the wrist menu. Thanks again for your amazing support!
On Sat, Jan 15, 2022 at 7:08 PM ben whitmore @.***>
wrote:

It worked!!! Loads in with some stuttering but works!Thanks so much for
your help!
2 more final things. How do you turn of vignette and how do you switch to
first person as a menu did not come up at the beginning even when I point
to my wrist as recommended etc
Thanks again
How do I donate?
Ben
Ben Whitmore Motion Graphics
Contact<PHONE_NUMBER>8
Email @.***
http://benwhitmore24.wixsite.com/mgfx
On Sat, Jan 15, 2022 at 4:08 PM @.***> wrote:

Ok great will try this when im home and let you know how i get on. I
think it will crash again as i have tried it i think without steam vr being
active but will try again.any other trouble dhooting i could try before i
get back in touch?
Thanks very much
Sent from my iPhone
On 15 Jan 2022, at 15:59, lucasfisher0 @.***> wrote:

When the mod needs SteamVR, it should automatically start. There are
non-Steam games that use SteamVR, but since the game is on Steam but isn't
marked as a VR game, it sometimes gives me issues launching.
—
Reply to this email directly, view it on GitHub
https://github.com/praydog/REFramework/issues/187#issuecomment-1013706220,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AXJLI5L3BJ5CHJOKLWB6OMTUWGKU3ANCNFSM5L4P4M2A
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you authored the thread.Message ID:
@.***>

Cheers, I am not on the dev team so I don't know about donations. This really is an amazing mod though, as much as I could help out it's nothing compared to the work that's been done so far.

Ok mate!Thanks so much again
Sent from my iPhone

On 16 Jan 2022, at 12:23, lucasfisher0 @.***> wrote:

Cheers, I am not on the dev team so I don't know about donations. This really is an amazing mod though, as much as I could help out it's nothing compared to the work that's been done so far.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you authored the thread.

One last thing. Do you know how to smooth out all the aliasing/sparkling.
Seems to be in reflections maybe?
Thanks again
On Sun, Jan 16, 2022 at 12:23 PM lucasfisher0 @.***>
wrote:

Cheers, I am not on the dev team so I don't know about donations. This
really is an amazing mod though, as much as I could help out it's nothing
compared to the work that's been done so far.
—
Reply to this email directly, view it on GitHub
https://github.com/praydog/REFramework/issues/187#issuecomment-1013866201,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AXJLI5PS4YWBBJ45PEJP4JTUWK2FLANCNFSM5L4P4M2A
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you authored the thread.Message ID:
@.***>

This game still looks better than Alyx at the lowest settings. The mod is supposed to overwrite settings but it sounds like they are getting reverted.
All settings to lowest/off except:
Textures to around 3-4GB, whatever you have VRAM for
Bloom ON
Anisotropic filtering to X16 (max)
Ambient Occlusion to SSAO
Then open the MOD MENU by pointing at your left hand, find the sharpening filter and turn it on.
With this, I am getting around ~60fps on a 3080Ti in RE2, and I haven't checked in RE3 but it is smooth. Maybe RE7/8 will perform better since they are made for first person.

Great cheers man
Sent from my iPhone

On 16 Jan 2022, at 16:56, lucasfisher0 @.***> wrote:

This game still looks better than Alyx at the lowest settings. The mod is supposed to overwrite settings but it sounds like they are getting reverted.
All settings to lowest/off except:
Textures to around 3-4GB, whatever you have VRAM for
Bloom ON
Anisotropic filtering to X16 (max)
Ambient Occlusion to SSAO
Then open the MOD MENU by pointing at your left hand, find the sharpening filter and turn it on.
With this, I am getting around ~60fps on a 3080Ti in RE2, and I haven't checked in RE3 but it is smooth. Maybe RE7/8 will perform better since they are made for first person.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you authored the thread.

To get ridd of the lagging stuff, open MOD in game by holding hands togheter or click out of the Window after using MOD in flatscreen with Mouse. You can also Minimize the Mod with the Mouse and Press WIN Key or CTRL+ESC. Than put on the Headset and dont klick with ur Mouse again in the Gamewindow. That fixed laggs or Blackscreen for me.

Everythings working.thanks for your kind help!
Sent from my iPhone

On 23 Jan 2022, at 03:25, takesword @.***> wrote:

To get ridd of the lagging stuff, open MOD in game by holding hands togheter or click out of the Window after using MOD in flatscreen with Mouse. You can also Minimize the Mod with the Mouse and Press WIN Key or CTRL+ESC. Than put on the Headset and dont klick with ur Mouse again in the Gamewindow. That fixed laggs or Blackscreen for me.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you authored the thread.

glad to hear! i knew u fixed the Problem, but wanted to tell you this if you would still lagging. :) <3

Amazing thanks so much!!
Sent from my iPhone

On 23 Jan 2022, at 10:05, takesword @.***> wrote:

glad to hear! i knew u fixed the Problem, but wanted to tell you this if you would still lagging. :) <3
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you authored the thread.

Hello,
Hope you are well.
I have a problem with the Resident Evil village Vr mod. I updated my
machine and it stopped working completely.
I have seen people have mentioned this online but no fixes.
Could you possibly tell me how to fix this?
Thanks so much,
Ben
Ben Whitmore Motion Graphics
Contact<PHONE_NUMBER>8
Email @.***
Showreel and website http://benwhitmore24.wixsite.com/mgfx
Design portfolio https://benwhitmore24.wixsite.com/mgfx/design-portfolio
On Sun, Jan 16, 2022 at 4:56 PM lucasfisher0 @.***>
wrote:

This game still looks better than Alyx at the lowest settings. The mod is
supposed to overwrite settings but it sounds like they are getting reverted.
All settings to lowest/off except:
Textures to around 3-4GB, whatever you have VRAM for
Bloom ON
Anisotropic filtering to X16 (max)
Ambient Occlusion to SSAO
Then open the MOD MENU by pointing at your left hand, find the sharpening
filter and turn it on.
With this, I am getting around ~60fps on a 3080Ti in RE2, and I haven't
checked in RE3 but it is smooth. Maybe RE7/8 will perform better since they
are made for first person.
—
Reply to this email directly, view it on GitHub
https://github.com/praydog/REFramework/issues/187#issuecomment-1013911832,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AXJLI5NI26VEZYRX3JLBC5TUWL2D3ANCNFSM5L4P4M2A
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you authored the thread.Message ID:
@.***>


Change color of the output

, orange is too light color

Hey @mayuri-pansuriya this isn't enough information for me to understand what your issue is.

Please follow the issue template


Create a diagram of the workflow

Create a diagram to help describe all steps and inputs of the workflow.
Related Issues:
#414

File for editing: https://drive.google.com/file/d/15uc39GtDSKdVugF33CJhIQ-pvGU1cAiG/view?usp=sharing
:information_source: when updating this, please comment on the issue what you are changing to keep track and notify the people involved.
Here is the current desired version (last update 22/01/2024):

This is the current workflow, for comparison (last update 22/01/2024):

Is it normal that I have a "Unable to render rich display" error?

Updated with app.diagrams.net / draw.io version.

Works now! 👍

@abelsiqueira @suvayu Maybe this is a good candidate for a wiki? 🙃

Have we decided what use we'll be moving to the wiki? We have 6 places for prose right now:

README.md
README.dev.md
CONTRIBUTING
Docs = Website
Issues
Discussions

So it might be useful to redefine some things to give use to the wiki. Also, it is a good time to bring up https://diataxis.fr, since it relates to documentation and purpose.

Diataxis was a great read!
I agree we're spreading out a bit, but this is something that doesn't seem to really belong in an issue... Maybe it can go into the docs? Right now it's pretty dev-related, but eventually might be useful to the user.

@abelsiqueira  Is there a way we can have this generate with code? That way we can put it in the docs without screenshotting it and keep it updated in the future (maybe add it to our PR checklist?)

Is there a way we can have this generate with code?

Thinking like an engineer @clizbe ;-)
We could explore using Mermaid.js. It's supported by GitHub. You can experiment @ https://mermaid.live/

Great! Let's use that. It'll probably take some work for the first version but updating after that shouldn't be bad.


Refactor: improve type definitions

PR Checklist
Please check if your PR fulfills the following requirements:

[ ] Tests for the changes have been added (for bug fixes / features)

PR Type
What kind of change does this PR introduce?

[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[x] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] CI related changes
[ ] Documentation content changes
[ ] infrastructure changes
[ ] Other... Please describe:

What is the current behavior?
Some of the variables/parameters were implicitly typed as any
What is the new behavior?
Added proper type definitions
Other information

Thanks for doing this. I was already doing it at the same time. Too tricky to merge probably. If you want to clean up what I've done further go for it (Although the Dummy Component is there for a reason).


Always displays the error: type check failed for prop "value" if I put v-model

Vue version 2.5.13
Sample code:
<toggle-button @change="changeRole(user_type)" :sync="true" v-model="user_type.add_clients"/>

@euvl I've manage to remove the type chek error by translating my 1/0 value to true or false. the value of user_type.add_clients is 1. I was just wondering it displays the right result But still throws an error. Just a suggestion, it would be better if vue-toggle can translate 1/0 to true/false. since in tables boolean columns are translated to 1/0

Strict typing helps to eliminate errors, i don't think anything should be changed here.
Also how should know the plugin  better which data type to use if even you didnt know what's your data in that variable?

I'm not forcing to change change something in the plugin. Since its a toggle it only expects true/false right? Well is some cases  1/0 can be interpret as true/false, as when stored in database.

I'm using Laravel + Vue and I just changed my model with:
this.currentClient.status = client.status === 1;
No errors now!


Issues with new theme

@domdomegg @neslihanturan @maskaravivek I've been doing more in-depth testing before releasing the new version (on a Nexus S emulator API 23 and 25, prior to real device testing), and a few things that I noticed with the new themes/icons:

When sent to the overflow menu due to the action bar being full, "From gallery" shows up with its icon on the left. Besides being the only item in the overflow menu with an icon, when the light theme is selected, the icon isn't visible due to being white.

The subtext in the tutorial is black now, while the text is white. IMO keeping the text and subtext roughly the same color is a lot more aesthetic.

I think it might be a good idea to make the dark theme the default instead of the light theme, so that people start out with the theme that they are used to, and have the option to change it if they prefer light. What do you guys think?

Very minor issue, and could just be a personal preference, but in the media details pane, landscape images have a large white band above them. IMO this looks okay in the dark theme, but rather jarring in the light theme.

I agree with all the points except for 4, I think it's fine. However it may be an idea to change that layout of the media_detail_fragment to something more material-ly at some point where we could change the vertical alignment of the image.
Might be good to get some feedback from beta testers on the new theme as well, perhaps vote on some issues.

Yeah, I don't think 4 is a big issue either. Would be good if 1-3 could be fixed before we start publicizing though.
Having a polling option sounds good too. Will make a separate issue for that.

It seems like 2 is already solved by @veyndan . I can solve 1 and 3 but I am having problem about reproducing the case 1. Can you share a scenario @misaochan ?

Thanks @veyndan !
@neslihanturan Case 1 happens when you have a small device (like the Nexus S that I emulated). When the action bar is full, extra items go to the overflow menu, which is when the issue occurs. Test it on a Nexus S emulator and see?

Issue 5: When in light theme, uploading an image with a dark background makes it impossible to read black category text.

I am on issue 5

I recognize another one about spinner theme. Fixing now.

With point no. 1, is it even necessary to have an icon in the overflow menu? As noted before, it feels a bit out of place considering it is the only item with an icon.

Well-spotted @neslihanturan , thanks!
@veyndan I'm fine either way (with removing the icon when it's in overflow or changing its color).

@veyndan I am okay with either way too. However, gallery menu has higher importance than other menus when it is in overflow (to upload). So it can be a good idea to show its icon to make user recognize it easier. If you decide to change it, I am online for a new PR.

I'll make a new PR. I think a better way to show importance is hierarchically, by showing the gallery menu at the top of the overflow.


fix nix flake build for aarch64-darwin

This uses he system gmp to fix the config.guess error

You actually did it, nice!

I see!


Regex ignoring prohibited characters

The bug
I have defined a regex that prohibits certain characters from generating using "[^]" expression. Guidance still generates these characters, and even seems to be biased towards them.
To Reproduce
Give a full working code snippet that can be pasted into a notebook cell or python file. Make sure to include the LLM load step so we know which model you are using.
import guidance
from guidance import models, user, system, assistant, capture, substring, zero_or_more, optional
import json

lm = models.transformers.LlamaChat(model=model, tokenizer=tokenizer, temperature=0.0, echo=False) # model is a qlora fine-tuned huggingface model.

@guidance(stateless=True)
def gen_number(lm):
    return lm + gen(regex=r'-?\d+\.?\d*', stop=',')

@guidance(stateless=True)
def gen_string(lm):
    return lm + '"' + gen(stop='"')  + '"'

@guidance(stateless=True)
def nullable(lm, generator):
    return lm + select([generator, "null"])

@guidance(stateless=True)
def gen_date(lm, prefix="20"):
    return lm + '"' + prefix + gen(regex=r"\d\d-\d\d-\d\d")  + '"'

@guidance(stateless=True)
def gen_list(lm):
    return lm + "[" + gen(regex=r'(("[^>"]*", )*("[^>"]*"))?') + "]"

# @guidance(stateless=True)
# def gen_list(lm):
#     return lm + "[" + gen(stop="]") + "]"

@guidance(stateless=True)
def gen_row(lm):
    out = lm + "{" \
        '"description": ' + nullable(gen_string()) + ", " \
        '"value": ' + nullable(gen_number()) + ", " \
        '"face_amount": ' + nullable(gen_number()) + ", " \
        '"cost": ' + nullable(gen_number()) + ", " \
        '"currency_code": ' + nullable(gen_string()) + ", " \
        '"quantity": ' + nullable(gen_number()) + ", " \
        '"maturity_date_min": ' + nullable(gen_date()) + ", " \
        '"maturity_date_max": ' + nullable(gen_date()) + ", " \
        '"call_date": ' + nullable(gen_date()) + ", " \
        '"coupon_min": ' + nullable(gen_number()) + ", " \
        '"coupon_max": ' + nullable(gen_number()) + ", " \
        '"yield_min": ' + nullable(gen_number()) + ", " \
        '"yield_max": ' + nullable(gen_number()) + ", " \
        '"spread": ' + nullable(gen_number()) + ", " \
        '"percent_net_assets": ' + nullable(gen_number()) + ", " \
        '"footnotes": ' +  gen_list() + "}"
    return out

@guidance(stateless=False)
def format_row(lm, row_json):
    with user():
        lm += row_json

    with assistant():
        lm += capture(gen_row(), "formatted")
    return lm

prediction = (lm + format_row(my_json))["formatted"] # my_json is the input

I am not able to share my fine-tuned model or an example input unfortunately.
Despite telling the regex not to generate ">" characters as list elements in gen_list, I will still get a list like [">", "a"] after "footnotes". Another odd thing is that when I switch to using the commented out gen_list above (which doesn't use a regex), the model never generates ">" characters. The only issues I run into with this approach is that sometimes the model will generate numbers without quotes around them, which is why I'm trying to further constrain the generation.
In my training data, there is never a ">" character in the list, and this character isn't in the prompt. I have also noticed a significant bias towards generating other "symbol" characters like ":", "<", ".", "]", "[" and a few others. Sometimes it will also incorrectly combine a symbol with the correct value like ":a", instead of "a". Again, the training data never shows such examples, and when I use the alternate gen_list function, it doesn't do this. It seems like using the regex option specifically biases the model to generate these characters, even though it should be forbidden.
System info (please complete the following information):

OS (e.g. Ubuntu, Windows 11, Mac OS, etc.): Ubuntu
Guidance Version (guidance.__version__): 0.1.10

hmm...I tried to reproduce this with mistral 7b and could not get violations. Can you try to reproduce with a model we can both use? thanks!

See also: https://github.com/guidance-ai/guidance/issues/710


use SafeTestsets.jl

Purpose
Use SafeTestsets to prevent leaking data between tests. This PR also moves some tests for soil BC calculations from test/standalone/Soil/soiltest.jl to test/standalone/Soil/soil_bc.jl, which I think is a more logical place for them to live.
Closes #197
Content

[x] wrap all tests in test/runtests.jl in @safetestset
[x] move soil BC calculation tests

[x] I have read and checked the items on the review checklist.

bors r+


Add clarifications, fix typos for Dev Container Features proposal

There was some missing info and typos I corrected.  I also linked out to the new feature-template and features repositories. //cc @bamurtaugh

Actually @joshspicer @jkeech @edgonmsft - This still refers to the concept of a collection. Is that still valid?  If so its missing from the template.  https://github.com/devcontainers/feature-template


[Docs]: Missing -n flag explanation for /mv create

Describe the feature
https://github.com/Multiverse/Multiverse-Core/wiki/Command-Reference#create-command
> mv help create
====[ Multiverse Help ]====
Add a '?' after a command to see more about it.
/mv create {NAME} {ENV} -s [SEED] -g [GENERATOR[:ID]] -t [WORLDTYPE] [-n] -a [true|false]
/mv anchor {name} [-d]

-n is not explained in the wiki link
I am willing to contribute to the wiki
How is the feature useful to all Multiverse users?
Would be nice to have a documentation of this option, as its probably a feature
Agreements

[X] I have searched for and ensured there isn't already an open issue regarding this.

Unlikely to be fixed as MV5 is in heavy development and the docs will need to be rewritten for that update


Update pkg, ignoring severity in tests

Adapting tests after knative/pkg@4b704fa.  This is following the
analogous bump made in knative/eventing@7dc7624

cc @mattmoor (from https://github.com/knative/eventing-sources/pull/129#issuecomment-441815135)

/lgtm


Can it be launched on usual VPS? For example, 6 CPU 16 RAM (usual chips)

Sorry for maybe a stupid question, fortunately, I find your product and want to integrate it into social media accounts; I don't release how to use it from the box (via Docker). As it said in the instruction I need a strong PC with GPU chips (which are pretty expensive for me) and I wonder if there is a way to utilize " input text prompt/variables - get a response in console / API"?
Can you please, @artnitolog, comment on this?
For reference, a way how https://porfirevich.ru/ works.

As you can see, you have to use very advanced GPU-hardware in order to run the inference.
On the other hand, it should be possible to run it using also much limited GPU memory via huggingface-accelerate. Generally, you have to use the hook for the pytorch forward method. This hook should load the weights into GPU memory iteratively and save the intermediate representation to disk until a working inference is obtained. There is a discussion here pointing out it is in principle possible.

As you can see, you have to use very advanced GPU-hardware in order to run the inference.
On the other hand, it should be possible to run it using also much limited GPU memory via huggingface-accelerate. Generally, you have to use the hook for the pytorch forward method. This hook should load the weights into GPU memory iteratively and save the intermediate representation to disk until a working inference is obtained. There is a discussion here pointing out it is in principle possible.

Thanks, Evgeny for the help and links, I will listen to discussions and check product updates.
AI is cool 🤘🏻


[Bug]: ModuleNotFoundError: No module named 'llama_index.readers.web.browserbase'

Bug Description
In line 8 of llama_index/readers/web/init.py, it must be from llama_index.readers.web.browserbase_web.base import BrowserbaseWebReader. Please fix it as soon as possible
Version
In main branch
Steps to Reproduce
import something from llama_index.readers.web
Relevant Logs/Tracbacks
No response

I have the same problem even when I am only importing BeautifulSoupWebReader and TrafilaturaWebReader.
Line 8: from llama_index.readers.web.browserbase.base import BrowserbaseWebReader at llama_index/readers/web/__init__.py generates this problem.
Running pip install browserbase does not help either even though there should be no reason to install browserbase when I am not using it.
The import line in my code from llama_index.readers.web import BeautifulSoupWebReader, TrafilaturaWebReader used to work before I upgraded llama_index and its readers.

Should be fixed, just need to run pip install -U llama-index-readers-web (will take one or two minutes to publish)

Thanks @logan-markewich.
This worked for me.
pip install -U llama-index-readers-web