Package Group UI

This is a quick update in order to provide better package experience in the editor. Though ultimately I believe the NodesPackage system needs to be remade and possibly added to ryvencore (Changing the identifier prefix outside ryvencore - along with other things - leads to Ryven specific importing. Furthermore, classes with the same name from different packages break the importing. Couldn't importing eventually be remade to work by using python packages?)
Packages are now added in a list -per name. There's also a trick to include "subpackages". Packages can be named as {root}.{subpackage}. There are no subpackages of subpackages, it only works for one level of grouping. By clicking on the last node of a package, the corresponding nodes are shown for Ryven.
Example: Suppose we want to divide our nodes into simple and advanced.
Option 1: Simply have the folders linalg and std and they'll appear in the editor.
Option 2: Bundle them into a "virtual" package, i.e test. Folders have to be renamed test.lnalg and test.std

Current display problem: Packages imported with the same prefix (i.e test.linalg and test.std) in the menu before creating the session are displaying only the prefix in the UI.

Changing the identifier prefix outside ryvencore - along with other things - leads to Ryven specific importing. Furthermore, classes with the same name from different packages break the importing.

I don't see a problem here. Strictly speaking, ryvencore does not have an import mechanism, so there's nothing to break. Registering nodes in a ryvencore session means providing their class, and I'd like to keep it this way. ryvencore should contain only the absolute core functionality for the idea behind the project. It is not specific to Ryven - in fact ryvencore is used by projects that don't care about Ryven at all. Developer focused features like a packaging system etc. should be defined on a higher level, coupled with a particular implementation, because there's about a million different design choices.

Couldn't importing eventually be remade to work by using python packages?

Theoretically yes, but Python packaging is a pain, and conceptually Ryven node packages are not (and don't have to be) this complicated. I don't see an immediate argument against supporting it, but I think requiring it would significantly raise the entry barrier.

Packages are now added in a list -per name. There's also a trick to include "subpackages". Packages can be named as {root}.{subpackage}. There are no subpackages of subpackages, it only works for one level of grouping. By clicking on the last node of a package, the corresponding nodes are shown for Ryven.

I like this idea! Some immediate thoughts:

I would be wary about arbitrary nesting

unnecessary complexity
basically a Python package at this point
possible UI issues (how do we nicely display a sub-sub-sub-sub-sub-sub-package in the editor)

I think forcing at max 1 level of package nesting (packages and sub-packages) gives a good balance between modularity and simplicity
Two sub-packages contained in the same package (e.g. test.std and test.linalg) will share common types (otherwise they should be separate packages), and in order to not run into import issues, the sub-packages should then rather be contained within the directory of the package (for example test/std, test/linalg).

One idea from the top of my head: define a function export_sub_package() in node_env; now test/std/whatever.py calls export_sub_package(), and if test/nodes.py imports test/std/whatever.py, we can easily attach test.std to test during loading of test.
Do you have other ideas?

I believe these are reasonable thoughts. Upon looking at it more carefully, there indeed seems to be no reason for more than 1 level of depth. It seems to be a really good balance for a graph-like system. I do find some utility functions in ryvencore could be useful though. It would be providing a good system for package handling from the get go.
As per the export_sub_package() idea,  that's what I initially thought about implementing, but I opted for a quicker implementation, without having to intervene with the code as much (didn't want to change a lot of things).  I suppose though that it should be relatively quick to implement something like this. One-two questions:

The root package can still have itws own nodes, correct?
Wouldnt subnodes.py be a good enough name?

I do find some utility functions in ryvencore could be useful though.

That's fair. But why not keeping Ryven-specific ryvencore utils in the Ryven repo? We could possibly detach it from Ryven a little, though, such that others can easily use ryvencore together with only Ryven's packaging. I don't think that's necessary right now but if someone ever needs this, I don't see strong reasons against it.

The root package can still have its own nodes, correct?

I would think so.

Wouldn't subnodes.py be a good enough name?

Hmm, maybe.

You can't have multiple subnodes.py in the same directory, so this forces the user to define sub-directories for the sub-packages. This might be a good or a bad thing, I'm not sure
In your example, I think I would personally prefer something liketest
├── nodes.py   // defines export_nodes()
├── std.py     // defines sub-package "std"
├── linalg.py  // defines sub-package "linalg"
└── utils.py   // not a sub-package, just utils

In the (unexpected, but not impossible) case that we eventually do support deeper package nesting in the future, this might be a pain, you don't want to write subsubsubpackage.py

I can see that working, might give it a try tomorrow.

I noticed there is no apparent difference in the code between  having subfolders or not. The real difference is that a single folder enforces different names. I believe it should be up to the user to arrange them however he likes. If both are done, we simply need to decide on naming conventions.
Firstly, I added sys.path.append(os.path.dirname(file)) in load_from_file. That way, the root nodes.py can be something like this:
import linalg.nodes
import std.nodes

or
import linalg
import std`

depending on the way they are imported. Exporting sub-packages was as simple as:
def export_sub_package(node_types:[Type[Node]], data_types: [Type[Data]] = None, sub_pkg_name:str = None):
    """
    Exports / exposes nodes to Ryven as a subpackage for use in flows. Called from inside a subpackage
    which is imported in nodes.py
    """

    #Fetch the module name
    if (sub_pkg_name == None):
        filename = inspect.stack()[1].filename
        print(filename)

    #Apply a the subpackage name
    NodesEnvRegistry.current_sub_package = sub_pkg_name
    export_nodes(node_types, data_types)
    NodesEnvRegistry.current_sub_package = None

All that's left is the handling of conflict when we have a subfolder with a nodes.py. But I suppose it's as easy as using the folder name when we encounter such  a situation. Any more thoughts or dislikes before pushing this?

Thanks for looking into this! Some notes:

I added sys.path.append(os.path.dirname(file)) in load_from_file. That way, the root nodes.py can be something like this, without having to add the path in every package.

I think we should also remove the path again after the file is loaded. Assume the following structure:
node-packages
├── pkg1
    ├── nodes.py    // "import utils"
    └── utils.py
└── pkg2
    ├── nodes.py    // "import utils"
    ├── special.py
    └── utils.py

if pkg1 is imported first, and then pkg2, the import of utils in pkg2 might import pkg1/utils because it comes first in sys.path. The downside of removing the path is that dynamic imports at runtime (e.g. a node in pkg2/nodes.py executes import special in its update_event()) will break, but these imports are kind of avoidable so I think I would favor avoiding the first issue.

    filename = inspect.stack()[1].filename

Please don't access stack frames. This breaks invariants assumed by lots of surrounding software, such as debuggers. Been there, done that ;) This is the reason why some commands (such as import_guis()) take an argument __file__, because it's generally not possible in Python to determine who called a function without.

I just noticed, importlib.util.spec_from_file_location() (which it uses to import a nodes package) has a submodule_search_locations parameter whose description sounds like exactly what we need, might give it a try, instead of mutating sys.path.

Well, I did quite a bit of digging...
Firstly, the submodule_search_locations is meant to work with exec_module and not load_module, which is deprecated. exec_module works, but you have to add the module using sys.modules[name] = mod before calling it. A full implementation looks something like this:
def get_mod_name(file:str) -> str:
        head, tail = os.path.split(file)
        _, pkg_name = os.path.split(head)
        mod_name = tail.split('.')[0]
        return f"{pkg_name}.{mod_name}" #i.e built_in.nodes

def add_mod(file:str, prefix:str = None, submodule_search_locations:list[str] = None):
    name = get_mod_name(file)
    if prefix != None:
        name = f"{prefix}.{name}"
    spec = importlib.util.spec_from_file_location(name, file, submodule_search_locations= submodule_search_locations)
    mod = importlib.util.module_from_spec(spec)
    sys.modules[name] = mod
    #print(f"{name} {sys.modules[name]}")
    return (spec, mod)

By doing that and also providing a package prefix, i.e. suppose we have pkg_test.std,.nodes you can add a module named pkg_test.std.nodes dynamically to the modules. The downside is that I have not found a way to also include pkg_test as a module using this. So, when you call import pkg_test.std.nodes in the root nodes.py, while the module does exist, the import throws an exception, because it can't find modules pkg_test and pkg_test.std, which must be set in sys.modules for importing to work. I can't find a way to find a module through only the path (essentially a namespace package). Everything I tried through importlib hasn't worked in creating a package module and then executing (importing it). If this worked, each ryven package would not collide with another.
At this point, I'm wondering if it would be just better to include the package location to sys.path and importing the whole package through there (by initialing things in init rather than nodes.py). I tried doing that but it hasn't worked, despite appending the path to sys.path and using importlib.import() or importlib.__import()__ (maybe I'm doing something wrong because I'm tired)
If you don't know how to solve any of the above, the only way I can think of it working is to rename the nodes.py files to the corresponding folder names (i.e a pkg.stk is compromised of pkg/pkg.py, pkg/stk/stk.py).
The other way is it leave it as is, appending to the sys.path and then removing  it to avoid future conflicts.

exec_module works, but you have to add the module using sys.modules[name] = mod before calling it.

Thanks for this, you're right, the previous inspect issue disappears.
I noticed I think I misread your original comment on python packages, I was thinking about packaging to PyPI. Normal Python packages (folders with __init__.py) I avoided for similar conflict reasons, but I already forgot that hadn't solved the package-relative import problem at all (e.g. std/node.py imports its sibling files literally by appending itself to sys.path first, which induces the issue as I described above). After another hour of trying trying to make it work based on module_from_spec(), I arrive at the same point.
I just wrote a quick prototype for a python packages based import mechanism with the following characteristics

package-relative imports must go through the package name

e.g. std/nodes.py calls import std.special_nodes instead of import special_nodes
this should avoid the issue I outlined above (given that package names are unique, which they should be)

a nodes package must have a __init__.py which should be empty by default
ryven package initialization still happens in nodes.py

It's super simple
def load_from_file(file: str = None, components_list: [str] = None) -> tuple:
    """
    Imports specified components from a python module with given file path.
    """
    if components_list is None:
        components_list = []

    dirpath, filename = os.path.split(file)
    parent_dirpath, pkg_name = os.path.split(dirpath)
    mod_name = filename.split('.')[0]
    name = f"{pkg_name}.{mod_name}" # e.g. built_in.nodes

    if parent_dirpath not in sys.path:
        sys.path.append(parent_dirpath)
    mod = importlib.import_module(name)

    comps = tuple([getattr(mod, c) for c in components_list])
    return comps

I think I didn't consider originally that I could just add the package's parent directory to sys.path and hence have package-internal imports still be unique because the package name is then part of it. What do you think?

If you agree this would be easiest and I don't suddenly remember other reasons why I didn't do it this way, I guess we could do that to actually support multi-module node packages, which we need for subpackages.

def export_sub_package(node_types:[Type[Node]], data_types: [Type[Data]] = None, sub_pkg_name:str = None):
    ...

To stay close to the name export_nodes(), I would rather call it export_sub_nodes() I think.
def export_sub_nodes(name: str, node_types:[Type[Node]], data_types: [Type[Data]] = None):
    """
    Exports / exposes nodes to Ryven as a subpackage of the currently loaded package for use in flows.
    Do not call this function in a node package's nodes.py file, call export_nodes instead.
    """
    NodesEnvRegistry.current_sub_package = sub_pkg_name
    export_nodes(node_types, data_types)
    NodesEnvRegistry.current_sub_package = None

Is there any reason for the initialization to go through the nodes.py file? Consider a pkg_test package with sub-packages std and linalg:
pkg_test
init.py
nodes.py
├── pkg1
├── nodes.py    // "import utils"
└── utils.py
└── pkg2
├── nodes.py    // "import utils", "import special"...
├── special.py
└── utils.py
Option 1) Package loading in root nodes.py:
#import root nodes here
...
#import subpackage nodes
from .std import nodes
from .linalg import nodes

Option 2) Package loading in init:
#import root nodes here
from . import nodes
#import subpackage nodes
from .std import nodes
from .linalg import nodes

This way, we simply import a python package, i.e importlib.import('pkg_test') instead of importlib.import('pkg_test.nodes). All you need is the folder and not any files. What do you think?

To stay close to the name export_nodes(), I would rather call it export_sub_nodes() I think.
Yes, that would be a better name.

On another note, I have 3 other separate ideas which I'm working on, in case you want to include them in this request:

Flow editor is the central widget, all the other widgets (Flows, Nodes, Settings, Variables, Consoles etc.) are dockable to it. Much more flexible than it currently is and should be quite fast and easy to implement.

Separation of GUI and logic. Any logic imports shouldn't depend on GUI code.

Inspector GUI. Widgets have GUIs, but I believe a dedicated window for altering a node's properties is a must for any editor.

should be easy to include in this pull request. 2) and 3) might take a few more days.

Regarding your example, I think from . import nodes does not work, but the point still follows:

If we want the package to be able to be used outside of ryven as well though, it would be better to keep the loading in a separate .py file like now to avoid issues while importing it.

I think this would be more problematic than a nodes.py file requirement.

To me it makes a lot of sense intuitively that a nodes package can be identified by a nodes.py file.
The user might want to reserve __init__.py for their custom (Ryven independent) initialization stuff. This would be especially useful if there ever is some deferred package import mechanism at some point in the future (where __init__.py is loaded long before nodes.py). Not currently on my radar but might turn out very useful.
I think we can certainly live with calling importlib.import_module(f"{pkg_name}.{mod_name}") instead of importlib.import_module(pkg_name) internally.

[...] in case you want to include them in this request

I don't have a strong preference on that, I think it's nice to have PRs isolated on specific features, but if you're doing all your work on one branch already and you'd have to pull that apart, I'm also fine with extending this PR.

Flow editor is the central widget, all the other widgets (Flows, Nodes, Consoles etc.) are dockable to it. Much more flexible than it currently is and should be quite fast and easy to implement.

Sure, if that's easy.

Separation of GUI and logic. Any logic imports shouldn't depend on GUI code.

That is the case to a large extent, and the main reason why ryvencore exists. ryvencore-qt defines main GUI classes for ryvencore components, ryven-editor/main GUI independent (or GUI-dependent files are only imported in a GUI environment), ryven-editor/gui assembles all Qt UI for the editor. I'm not sure further separating the UI would help maintainability. But I'm curious, what else are you thinking of separating?

Inspector GUI. Widgets have GUIs, but I believe a dedicated window for altering a node's properties is a must for any editor.

Do you mean something like a node creator app, or a browser-style inspector of instantiated components inside the Ryven editor? Regarding the former I'm wary about abstracting away nodes programming. I intentionally don't want to take away the coding from developing node packages, because there are lots of subtle things to be aware of when writing robust nodes packages. I'm strongly in favor of providing a higher barrier for nodes development, yielding higher quality node packages that are actually scalable. In fact Ryven did have a nodes crator application a while ago, and I decided to remove it.

I agree with the loading. Pushed it to work like we discussed, along with an example in the pkg_test folder. I also pushed docks for the Nodes, Flows and Console widgets. Will see what I can do about the Variables and Settings. Small preview on how it works.

But I'm curious, what else are you thinking of separating?

I did notice that it's the case to a large extent. I also noticed though that, for the linalg test, nodes.py imports the guis and makes the connection inside the nodes themselves. Doesn't that make the nodes package ryven dependent, when we want it to be ryvencore dependent only? Perhaps I've missed something. But wouldn't defining the connection in a separate module solve the problem? (I just noticed the same problem exists by calling export_nodes inside the same module that defines the nodes, but that is trivial to solve by separating the export and the nodes to different files).

Do you mean something like a node creator app, or a browser-style inspector of instantiated components inside the Ryven editor?

I misspoke. Meant to say nodes have guis. I was in fact talking about a browser-style inspector for the instantiated nodes. Something like Unity (https://docs.unity3d.com/Manual/UsingTheInspector.html) or any other Game Engine has. I believe it is essential if a node has many configurable properties. I'm relatively new to python type-hinting, inspecting and serializing (though it mostly works the same in every oop-feature language).

Small preview on how it works

yup, just tried it myself, very cool!

I also noticed though that, for the linalg test, nodes.py imports the guis and makes the connection inside the nodes themselves. Doesn't that make the nodes package ryven dependent, when we want it to be ryvencore dependent only?

Oh no it doesn't! That's why I separated GUI definitions and node definitions, because I want nodes to work without GUI by default. If you look closely, nodes.py files don't import gui.py, they call import_guis(__file__). In the editor, this function will just import the GUI components, but when running in headless mode, it won't.

I was in fact talking about a browser-style inspector for the instantiated nodes. Something like Unity (https://docs.unity3d.com/Manual/UsingTheInspector.html) or any other Game Engine has. I believe it is essential if a node has many configurable properties. I'm relatively new to python type-hinting, inspecting and serializing (though it mostly works the same in every oop-feature language).

Oh I think we're on the same page then. I've long wanted something like a DETAILS section (roughly where the SETTINGS is, which presents node properties and allows configuration. I would love an extension to the nodes API that allows for customized behavior by exposing properties, a bit like in Blender I think.

About the console placement, should we make it dockable everywhere?

sure, or is there any reason against?

I can make it closeable, but then we have to introduce a new item in the View tab or something to re-open it.

why would the way it was before (dragging the splitter handle) not work?

It's just that, despite it working exactly as you say, one still needs to import the node_env from ryven inside nodes.py.

node_env is unrelated to GUI, and will always need to be imported.

Most solutions I've worked with typically make the connection in a separate file or the gui file, in some form of metadata connection.

That's fair, but for Ryven somewhat over-complicated IMO.

At the very least, I believe that the GUI class should have which node it corresponds to, and not the other way around.

Hm I'm not sure about that. One node can have one GUI, and one GUI can be applied to multiple different nodes. But one node should never be applied to multiple different GUIs. But that is indeed only my own intuition, and I don't think there are any strong conventions about that in Python, so I'd be open to look at alternatives but I would rather postpone that for now.

Do you find this overly complex, should we opt for something simpler?

I do have some ideas. Could you open an issue?

But that is indeed only my own intuition, and I don't think there are any strong conventions about that in Python, so I'd be open to look at alternatives but I would rather postpone that for now.

Sure, no problem as of now. It doesn't really matter currently. It would probably matter if one were to use the nodes outside of Ryven.

I do have some ideas. Could you open an issue?

Sure! I'll probably be working on this a lot though (at least a prototype), since I need a quick implementation for a project I'm currently working on.
I'm leaving two examples of the various functionalities from the last push. I believe the updates are substantial and stable enough. If I'm missing something, do tell.
UI Quality of Life

Loading a Project with saved window states

Also, I'd like to mention that what I find missing from Ryven the most, apart from the Inspector GUI, is the ability to run the flow only once with the click of a button and preventing any gui changes from updating the flow or executing it again. I'll also be working on something like this, but I don't know how generic enough it could / should be to be included in vanilla Ryven.
The idea is configuring a graph and then running it only once or providing a custom script that determines how the graph is updated (i.e. in a for loop). This would be extremely helpful in situtations where you want to define Input Nodes that acquire streamed data, go through these specific nodes in a for loop and update them every N Frames / second. Or simply for running your flow only once and outputting the results.

Apparently, it's a problem regarding qt compilation from .ui to .py. I'm using 5.15.10. Instead of compiling to self.logger_doc.setWidget(self.logs_scrollArea), it compiles self.logger_doc.setWidget(self.loggerWidgetContents). I have no idea if that's my fault for not using qt creator correctly or the compiler's fault.

I actually have no clue, if the problem is reducible to a simple example someone in the qt forum can maybe resolve it in a matter of seconds.

I'll probably be working on this a lot though (at least a prototype), since I need a quick implementation for a project I'm currently working on.

Just make sure you do that on a different branch and not clutter this PR too much, otherwise it will take long until I can merge.

I'm leaving two examples of the various functionalities from the last push.

What you show looks cool! If I notice sth once I try, I will tell.

should I also make pyside6 the default?

pyside6 is not the default because currently it breaks a few things in buggy ways, but I think I didn't take the time yet to track everything down. If you're working on pyside6 and come across particular bugs, fixes would be highly appreciated, but preferable in another PR.

Also, I'd like to mention that what I find missing from Ryven the most, apart from the Inspector GUI, is the ability to run the flow only once with the click of a button and preventing any gui changes from updating the flow or executing it again.

As I noted in my email response, I don't think this kind of flow execution control should be part of Ryven right now, since it's highly use-case specific and can be implemented within node packages. For example, instead of using connections to distribute actual streams of data, it can make sense to instead use connections to distribute handles to the data streams which are controlled externally (like a socket, or a file descriptor).
If strong patterns and repeating use cases emerge in the future, I'm totally open to look at possible integrations, but for now I'd stick with node packages (and rather extend node package capabilities - as we're doing right now - instead of the editor).

Is there any urgency on your side to have changes committed here be merged into master soon?

Just make sure you do that on a different branch and not clutter this PR too much, otherwise it will take long until I can merge.

Already doing that!

If strong patterns and repeating use cases emerge in the future, I'm totally open to look at possible integrations, but for now I'd stick with node packages (and rather extend node package capabilities - as we're doing right now - instead of the editor).

I believe the case for a one-time play button mode is a strong one. A flow can include big data processing, so it wouldn't be meaningful to have changes induce a new evaluation; it would slow down the editor too much. You could pretty much emulate this I guess by packages and custom guis, but I think it's too generic a case. Anyways, I'll be doing that on another branch, so no worries, just thought I'd mention it.

Is there any urgency on your side to have changes committed here be merged into master soon?

Not really, but I'll be basing other branches on this one, so I don't know if any conflicts will arise later on (just trying to be cautious). Time I'll be attempting to utilize Ryven is roughly 2 months; After that it's indefinite on how much I'll be able to contribute.
I've pretty much finished what I wanted to do in this PR with the last few pushes. When you have some time go through them and ask me anything if you need clarification (I may have some typos in there). Some questions for a last push:

Should set_performance_mode() in Design.py emit the signal even if the same performance mode is being set? Currently, it's that way. I could change it to guard against the same value if you want to.
Which is the correct setting to connect dock animations (currently they are always animated). Is it performance mode or animations?

Time I'll be attempting to utilize Ryven is roughly 2 months; After that it's indefinite on how much I'll be able to contribute.

cool

Should set_performance_mode() in Design.py emit the signal even if the same performance mode is being set?

Sure, you can change that if I didn't comment that it's required in the code.

2. Which is the correct setting to connect dock animations (currently they are always animated). Is it performance mode or animations?

performance mode

As I was testing, I noticed that, despite having successfully implemented one layer sub-packaging, the import_guis function leads to some problems. Consider this structure:
pkg_test
├──__init__
├──nodes.py
├── linalg
    ├── nodes.py
    └── gui.py
linalg
    ├── nodes.py
    └── gui.py

Due to the nature of import_guis, which relies on the new load_from_file, the file which will be imported is the one that resides on the root linalg folder and not the subpackage one. Personally, import_guis was quite unintuitive to me and this problem has made me consider another solution. As it currently stands, my solution is this:

import_guis and export_guis are no longer needed. We define a node_widget or node_gui decorator:

def node_widget(node_cls:type[Node]):
    """
    Registers a node widget as the GUI for a specific node. Works with inheritance
    """
    if not issubclass(node_cls, Node):
        raise Exception(f"{node_cls} is not of type {Node}")

    def register_gui(gui_cls:type[NodeGUI]):
        print(f"Registering node gui: {gui_cls} for {node_cls}")
        node_cls.GUI = gui_cls
        GuiClassesRegistry.exported_guis.append(gui_cls)
        GuiClassesRegistry.exported_guis_sources.append(inspect.getsource(gui_cls))
        return gui_cls
    return register_gui

The connection happens where we define the GUI classes. As an example, I'll use the linalg packages classes. This is the gui.py

from .nodes import *

@node_widget(MatrixNodeBase)
class MatrixNodeBaseGui(NodeGUI):
   ...

@node_widget(EditMatrixNode)
class EditMatrixNodeGui(MatrixNodeBaseGui):
   ...

Finally, importing happens in the root nodes package. This is nodes.py directly under the folder pkg_test.

import os

from .std import nodes
from .linalg import nodes

if os.environ.get('RYVEN_MODE') == 'gui':
    from .std import gui
    from .linalg import gui

I believe this is much better than what we currently have. It removes having to call a method to import the guis and allows for plain old python importing to make the connections. For a final touch, I would probably have the definition of the nodes and the exporting happen in different files as well. That way, the definition file is completely ryven independent and can be used outside of it without having to worry about import errors.
I have this already working. What do you think?

Thanks for the example, that's true. I like the decorator approach visually, it seems more pythonic.
Side-note: node_widget should be called node_gui IMO, it's not a widget (as opposed to e.g. main widget).

import os

from .std import nodes
from .linalg import nodes

if os.environ.get('RYVEN_MODE') == 'gui':
    from .std import gui
    from .linalg import gui

It removes having to call a method to import the guis [...]

...and replaces this requirement by boilerplate code that depends on ryven environment variables?
Concerns I have:

runtime dependency on environment variables

...defined by another (part of the) application is generally a bad idea down the line
runtime behavior should not depend on environment variables, but on the API (node_env and gui_env)
an easy solution might be to define another decorator gui_init() and decorate a function in nodes.py which imports the gui files

complexity

doing the import centrally somewhere else (nodes.py) potentially decreases modularity

I still have the concern from above, which I don't see addressed here

I've already called it node_gui in my implementation. I can't quite see these concerns as valid. Consider the following structure:
built_in
├── nodes.py
├── nodes_def.py
└── gui.py

nodes_def.py defines the nodes, as well as a list of the potentially useful nodes for exporting, called nodes. It only depends on ryvencore and not anything else.
gui.py depends on the nodes_def and ryven to assign the gui to different nodes. Essentially, the node_gui decorator imports the node to ryven. At this point, gui_env isn't really needed. From what I could see, it just stored the guis in a list so that last one could be used and also defined a function to return a container for the gui importing inside the file where the nodes were defined.
The nodes.py file handles all the importing. Essentially, it is the file where users define their package. As for the runtime dependency on env variables, should be easily solvable. E.g:

from ryven import in_gui_mode #e.g for environment checking
from ryven.node_env import export_nodes
from .nodes_def import nodes

export_nodes(nodes)
if in_gui_mode():
    from .gui import *

#import sub packages here which will be defined like above, but with export_sub_nodes instead
from .linalg import nodes
....

I believe this is more clear and concise. Logic (i.e nodes) aren't dependent on ryven and don't import guis, Guis register themselves to nodes and depend on the nodes and ryven; And a single file defines the package.

also, how do you override gui classes for node subclasses? doesn't that depend on the order in which you declare (i.e. decorate) your gui classes?

Re-check the node_gui / node_widget decorator. It assigns a GUI class attribute to the node class which refers to the gui class. It's exactly the same thing we previously had via import_guis but transferred to the gui file and "type-checked". Reposting my example with the linalg for clarity:
from .nodes import *

@node_gui(MatrixNodeBase)
class MatrixNodeBaseGui(NodeGUI):
   ...

@node_gui(EditMatrixNode)
class EditMatrixNodeGui(MatrixNodeBaseGui):
   ...

EditMatrixNode is a subclass of MatrixNodeBase. The @node_gui(EditMatrixNode) overrides the GUI for the EditMatrixNode. If I remove that line, then the EditMatrixNode will use the MatrixNodeBaseGui its MatrixNodeBase base class uses.

I've already called it node_gui in my implementation.

good

At this point, gui_env isn't really needed.

I'd like to keep it, to potentially extend GUI API in the future.

from .nodes_def import nodes_to_export

For simple examples, I would stick with node definitions in nodes.py (questionable name otherwise), you are of course free to do that differently in your packages.

if in_gui_mode():
    from .gui import *

Let's make it a function so that we could defer GUI imports.

The other way I can think of this happening is by doing this automatically when importing a package.

I'm not opposed to that, but if explicit imports are possible without boilerplate, explicit import sounds like less problems.

Re-check the node_gui / node_widget decorator.

I explained that poorly, I was thinking towards multi-inheritance, but I this the same issues arise in the current system.

I still have the concern from above, which I don't see addressed here

...still have that, but after consulting other popular uses of decorators it seems common practice to expect unique usage, so it's fine for me.

Let's make it a function so that we could defer GUI imports.
Could you give an example?

...still have that, but after consulting other popular uses of decorators it seems common practice to expect unique usage, so it's fine for me.

I'm missing something. The node_gui decorator can be stacked multiple times. We could even make it be node_gui(*args) if we want to connect more nodes with the same gui. I can see the problem with multiple inheritance, but then the developer just has to explicitly define which gui is to be used for that node.
Regarding this feature, would you like me to push it in this PR?

Could you give an example?

def load_ui():
    from .std import gui
    from .linalg import gui

or
@on_load_ui()
def whatever():
    from .std import gui
    from .linalg import gui

I can see the problem with multiple inheritance, but then the developer just has to explicitly define which gui is to be used for that node.

@node_gui(Node1)
class MyGui...

# (lots of code)

@node_gui(Node1)
class SomeOtherGui...

or much worse even
@node_gui(MutableMatrixNode)
class SomeNodeGui...

@node_gui(DataFrameExtensionsBase)
class SomeOtherGui...

class MutableDFMatrix(MutableMatrixNode, DataFrameExtensionsBase)...

These mistakes are not fully obvious to me. As I said, the same is possible currently, it just seems slightly less likely.

Regarding this feature, would you like me to push it in this PR?

I would prefer another PR, if that doesn't complicate things significantly for you.

I don't quite understand the examples. What benefit would the developer of a package have by wrapping the imports of the UI in a function? The function must still be called. I can see the decorator doing the work of the if in my example, but someone still has to call that function in nodes.py. Is there a more important reason I'm missing?
As for the decoration of the GUI, I think at that point it's a developer's mistake. All we can do is keep a set of all the nodes types whose guis have been explicitly set and throw an exception for safety purposes. As you said, though, I don't think it's that big a problem.
I'll be waiting for this PR to close before I start a new one.

I don't quite understand the examples. What benefit would the developer of a package have by wrapping the imports of the UI in a function? The function must still be called. I can see the decorator doing the work of the if in my example, but someone still has to call that function in nodes.py. Is there a more important reason I'm missing?

Ryven would call it. Currently this would happen at package import time but it doesn't have to, deferring GUI imports has already been on my radar. It would open some use cases and push the developer to avoid GUI-dependent semantics.
I think I prefer the decorator option, it doesn't really matter for the package dev but gives Ryven more control.

As for the decoration of the GUI, I think at that point it's a developer's mistake. All we can do is keep a set of all the nodes types whose guis have been explicitly set and throw an exception for safety purposes. As you said, though, I don't think it's that big a problem.

yeah, we might just implement some sanity checks here, but it's not top priority

I'll be waiting for this PR to close before I start a new one.

I'll try to look over all the code asap as soon as the packaging changes are done.

Ryven would call it

Got it. Should look nicer.

I'll try to look over all the code asap as soon as the packaging changes are done.

The last push in this PR had all the functionality I wanted to add with the packages and the UI quality of life. If I've forgotten anything, do tell. Otherwise, should be ready for reviewing.

notes on semantics:

fix auto discover (I can do it after merge) [^1]
be careful with Python features of recent versions, Ryven should run on 3.6+

T1 | T2 union type notation was introduced in Python 3.10
please re-check the rest of your changes, in case I missed sth

whenever you add a new feature, you should add basic backwards compatibility

your window geometry loading crashes on old projects which don't have the entry

notes on styling:

please try to stay closer to the rest of the code base
a PEP 8 editor plugin might help you
includes a space between a colon and type hints, a space in a comment after "#", docstrings (instead of comments) to annotate functions, starting multi-line dicts with a newline (and then a single indentation), and consistent number of newlines between methods

I can fix the styling on your branch if you prefer.
[^1]: package auto discover is broken with sub-packages, should replace
python         for top, dirs, files in os.walk(packages_dir):             path = pathlib.Path(top)
in StartupDialog.py with
python         for entry in filter(lambda e: e.is_dir(), os.scandir(packages_dir)):             path = pathlib.Path(entry.path)

Do you find using the black formatter disruptive? It's PEP 8 compliant. The only thing that one might dislike is that it forces the use of double quotes for strings. (I have no particular preference)
If not, I'm using VS Code to develop. If you have a specific formatting plugin you're using, tell me and I'll use that.

A formatter is just as good as its configuration and I don't have a formal configuration of Ryven code base styling. There are many choices even beyond PEP 8 that require additional circumspection. Of course you are free (and encouraged) to use it, but you should still make sure the styling matches the rest of the code base.

fix auto discover (I can do it after merge)

Fixed

Ryven should run on 3.6+

Removed any new union type notation. Furthermore, I had a lot of generic standard type hinting, i.e dict[str, str] which, as I understand, is recommended from 3.9+, as the typing module has been deprecated for standard collections. Fixed them.

your window geometry loading crashes on old projects which don't have the entry

fixed, tested by loading the example projects

please try to stay closer to the rest of the code base

Formatted using the aforementioned formatter. Styling obeys the rules you mentioned. The way some imports are typed have been affected as well [from . import N1, N2, .... Nn] are broken in lines if there are too many imports happening
As per the 3.9+ type hinting, I find it quite intuitive and a must for modern python. I commented with a should be... in all the files what the type hint would be if it were 3.9+ (I guess the typing module could be used instead). Is there any particular reason you're constricting to 3.6 or any plans to update the min requirement to 3.9? (3.8 will be reach end-of-life in 2024)

I've checked my last commits for Ryven. Tried it for 3.6, 3.7, 3.8 (3.9+ is working as expected).

For python 3.6, 3.7 it crashes with a syntax error: SynatxError: annotated name 'instance' can't be global in line 69 of config.py. Haven't attempted to resolve it.
For python 3.8, it crashes with a type error in std_input_widgets.py because type[Data] is used, which is allowed in 3.9+

These aren't changes I made. I've tested this PR in 3.8 after fixing the issue. Haven't tested in 3.6, 3.7. As it currently stands, the main branch for Ryven works with 3.9+ and above.
I have finished with my changes. Please look them over. I'm leaving the 3.6, 3.7 and 3.8 issues to you.

I might do minor cosmetic changes to the restyling but not in this PR

For python 3.6, 3.7 it crashes with a syntax error: SynatxError: annotated name 'instance' can't be global in line 69 of config.py. Haven't attempted to resolve it.

thanks that's good to know

do you have any plans to update the min requirement to 3.9?

I don't have plans to move further into the present than what's currently alive (e.g. 3.8). Now 3.6 has reached EOL long ago (3.7 also) but I still see people using it sometimes, so if it's not a major difference for us I guess we can keep it flexible.

Good to know!

I might do minor cosmetic changes to the restyling but not in this PR

Yeah, I'm also not a fan of some of the things it did.
Regarding af91367, there's another type hint I forgot right under the one you fixed. I can fix it in the PR for the gui decorators if you want to.

I still see people using it sometimes, so if it's not a major difference for us I guess we can keep it flexible

I was mainly talking about the library I mention in #173 which has a 3.7 min req, but other than that, I don't think it's a problem. Regarding that issue, I believe we should avoid ryven specific stuff and simply require a function or a class be implemented that attaches the Inspector it creates to a ryven window. I'll comment on that issue again and I'll wait for your reply there, I'm currently testing this.
As for the GUI decorators and deferring of gui loading, it's ready. I'll be making a new PR unless there is something you want to clarify first.

there's another type hint I forgot right under the one you fixed. I can fix it in the PR for the gui decorators if you want to.

oh how did I not see that; I'll fix it when I look into the other version compatibility issues

I was mainly talking about the library I mention in #173 which has a 3.7 min req, but other than that, I don't think it's a problem. Regarding that issue, I believe we should avoid ryven specific stuff and simply require a function or a class be implemented that attaches the Inspector it creates to a ryven window. I'll comment on that issue again and I'll wait for your reply there, I'm currently testing this.

The concept of a mutable tree of properties could be very useful beyond the UI. To make future extension easy, this system should be really simple, and optimally prevents misuse by design. Looking forward to see what you're thinking of.

As for the GUI decorators and deferring of gui loading, it's ready. I'll be making a new PR unless there is something you want to clarify first.

Cool! Nothing from my side.

Sorry for posting on a closed PR, but I just realized... Lets say we have this structure.
pkg # root pkg
├── nodes.py # root nodes for pkg
├── sub_nodes.py # a sub nodes grouping of pkg
├── sub_nodes_2.py # another sub nodes grouping
└── sub_pkg  # a sub pkg
    ├── nodes.py  # root nodes for sub_pkg
    └── sub_nodes.py  #  sub nodes grouping of sub_pkg

Don't we already have a completely defined tree - as nested as we like - for any package? If cls  is the class name, then by using
{cls.__module__}.{cls.__name__}, we automatically create the following:
pkg.nodes.<class>
pkg.sub_nodes.<class>
pkg.sub_nodes_2.<class>
pkg.sub_pkg.nodes.<class>
pkg.sub_pkg.sub_nodes.<class>

With this, we can easily define a tree:
pkg
├── Node1 # from pkg.nodes
 ...
├── NodeN
└──sub nodes
   ├──Node1 # from pkg.sub_nodes
    ...
   └──  NodeN
└── sub pkg
   ├── Node1 # from pkg.sub_pkg.nodes
    ...
   ├── NodeN
   └──  sub nodes
        ├── Node1 # from pkg.sub_pkg.sub_nodes
         ...
        └──  NodeN

In this context, the string nodes. (and hence the nodes.py file) is special. Do you find any fault in this?

What's the issue here? Of course you can have arbitrary sub-python-package nesting, but not sub-ryven-package nesting.[^1]

In this context, the string nodes. (and hence the nodes.py files) is special.

not sure what you mean by this
[^1]: we currently don't enforce this, though. it's on my TODO to enforce that a Ryven package calls export_nodes() exactly once, while export_sub_nodes() can be called arbitrarily often

also, I'm wondering now, in order to increase flexibility and decrease confusion potential between python and ryven packaging, maybe should relax export_sub_nodes() to receive the name directly and allow it to be called anywhere.
e.g.: nodes.py
from ryven.node_env import ...

# some basic node defs
export_nodes(...)

# some node defs
export_sub_nodes('special_nodes', ...)

# some more node defs
export_sub_nodes('more_special_nodes', ...)

what do you think?

What's the issue here? Of course you can have arbitrary sub-python-package nesting, but not sub-ryven-package nesting.

The proposed method above does the following:

Removes the need of the export_sub_nodes function.
Provides multi-nesting out of the box by aligning ryven's package system with python's package system. The two are kind of similar, so merging them could make sense.
Defines a clear and concise way to define packages. This can be good and bad, depending on how you see it.

not sure what you mean by this

Using the method above, a python package class that is pkg.sub_pkg.nodes.<class> will result in ryven package that is pkg.sub_pkg.<class>. Hence, any .nodes generated by the python module should be excluded in ryven.
Essentially it creates a ryven package tree using python package rules and the special nodes.py file.

also, I'm wondering now, in order to increase flexibility and decrease confusion potential between python and ryven packaging, maybe should relax export_sub_nodes() to receive the name directly and allow it to be called anywhere.

The export_sub_nodes() function already provides an optional parameter for the sub-package name. I proposed this method to conveniently merge python and ryven packages and provide arbitrary nesting, since python already provides a tree-like structure for its packages.
I believe it solves a lot of problems. If you think the tight coupling with the python package system or the special rule of the nodes.py file do more bad than good, then it's ok. Otherwise, we pretty much have complete nested packaging without the need of an export_sub_nodes() or any other system, as showcased in the example above.

Sorry, I'm still not completely sure what you're proposing. Certainly one can also just export a sub-package with name pkg.sub.sub.subpkg which makes no sense so we should actively prevent it, just as we should prevent multiple export_nodes() in the same package. Certainly we can technically implement arbitrary ryven package nesting, but the point is we we didn't want to, right? Are you re-considering that or am I misunderstanding?

actually, if I understand your proposal, I can also see benefits because it's conceptually just much more simple... I think I'll try to get a third opinion on this

Are you re-considering that or am I misunderstanding?

I indeed am reconsidering this. I also believe it won't have a problem with UI issues. Let the developers define their packages as well as they like.

Is that correct? In that case export_nodes() will require file, right?

export_nodes can be called from anywhere the way I see it. But no, the __file__ is no longer required. By exporting a node you have access to the full python generated module path. In simple python code, if you have a node_type, then
s = f'{node_type.__module__}.{node_type.__name__}' # same thing can be applied to data types.

That way, a class FooA residing in a nodes.py file of package pkg will have  a path of pkg.nodes.FooA. While another class FooB residing in sub_group.py file of the same package will have  a path of pkg.sub_group.FooB. Repeat that for any exported nodes defined in any sub-folders and you get what I have here.
Or almost the same thing. Every class' path residing in a package's or sub-(sub-...)-package's root is "polluted" with the .nodes substring, which is the result of the nodes.py file which is used in the importing process. That's why .nodes is a special substring. Removing it gets you to:

pkg.FooA // a class in the root pkg
pkg.sub_group.FooB // a class in the sub_group of pkg, sub_group comes from a .py file
pkg.sub_pkg.FooC // a class in the sub_pkg of pkg, sub_pkg comes from a folder (pkg.sub_pkg.nodes.FooC)
pkg.sub_pkg.sub_group.Food // class in sub_group of sub_pkg of pkg
etc...

We're essentially letting python create the package tree by deciding on a simple rule. Only export_nodes is needed and can be called from anywhere. The fun part is that this method is less code than what we currently have.
(Also, I can even argue that export_nodes isn't really needed, but I think doing anything like that needs inspecting and will slow things down)


Fix code example in readme

The stdSerializers are defined in pinoHttp and this was mixed up with pino in one code example.

Closing again. I had an instantiated pino logger where stdSerializers are not defined. That's what confused me. I saw that the stdSerializers are defined both on pino and pinoHttp.


Kucoin fix for multiple buys

Description
When the API would return empty or incorrect data, bot could improperly evaluate last action as a sell.  Added some additional checks to the Kucoin code that would prevent that and a WAIT state check in Strategy.py that could be reference by other exchange APIs or other code for a similar reason.
Fixes # (issue)
Type of change
Please make your selection.

[ x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ x] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update
[ ] This change requires a new dependency
[ x] Code Maintainability / comments
[ ] Code Refactoring / future-proofing

How Has This Been Tested?
Running multiple bots myself for more than 2 days without issue occurring again.  Previously would happen frequently when bot process restarts or gets a "Too many requests" error.
Checklist:

[ x] My code follows the style guidelines of this project
[ x] I have performed a self-review of my own code
[ x] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ x] My changes generate no new warnings
[ ] I have added pytest unit tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[ ] Any dependent changes have been merged and published in downstream modules
[ x] I have checked my code and corrected any misspellings

Thanks @chriskoepf, is this a Kucoin specific fix? Should the same fix not be implemented for Binance and Coinbase Pro as well?
There are some conflicts in models/AppState.py. Can you please review and resolve the conflicts? Thanks

I believe it’s Kucoin specific, but could possibly work on Coinbase and Binance as well.  I have not seen any comments about multiple buy issues on the other 2 exchanges.  Pretty much though, on shared account with a large quote pot, the evaluation could happen any time.
I will look at the conflicts later or in the next day or so.  I did know that there were a ton of changes coming through after seeing the pending PRs and talking with Mark.
I will make sure I get fully updated and look at the changes again.
I have another big fix for Kucoin API errors.  Jegb gave me an idea that seems to have worked very well.    He’s going to reach out to you to see if there was a problem or if he should recreate the PR.
CC (aka Chris K)
On Nov 5, 2021, 7:38 AM -0400, Michael Whittle @.***>, wrote:

Thanks @chriskoepf, is this a Kucoin specific fix? Should the same fix not be implemented for Binance and Coinbase Pro as well?
There are some conflicts in models/AppState.py. Can you please review and resolve the conflicts? Thanks
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

Conflicts resolved, ready to merge.


Shutdown handler not working

Bug Description
Code in shutdown_handler is never executed at shutdown. If a database connection is left open at time of shutdown/restart and --dev is given, a process is left hanging, preventing further use of the port.
Steps to Reproduce
For example, no message is printed upon receiving SIGINT:
from robyn import Robyn

app = Robyn(__file__)

@app.get('/')
async def index(request):
    return 'hello world'

@app.startup_handler
async def startup_handler():
    print('Starting up...')

@app.shutdown_handler
async def shutdown_handler():
    print('Shutting down...')

app.start(port=8081)

Your operating system
Linux
Your Python version (python --version)
3.12
Your Robyn version
latest
Additional Info
No response

@VishnuSanal , can you have a look at this?

aight, I can reproduce this on my machine too.
@app.shutdown_handler
def shutdown_handler():
    print('Shutting down...')

but, works normally if I remove the async keyword. @matt24smith can you please try it & report back?

Thanks for looking into this!
My use case requires the async keyword, as I'm trying to terminate an async database connection. I can confirm that the non-async case works as expected

Many thanks for your work on this, the fix is working great! @VishnuSanal @sansyrox

I am glad you like this @matt24smith 😄
Thanks for the great work @VishnuSanal ✨


capture screen to htmlimageelement htmlcanvaselement or imagedata

http://stackoverflow.com/questions/24265361/phantomjs-capture-screen-to-htmlimageelement-htmlcanvaselement-or-imagedata

Please upgrade to PhantomJS 2. 1.9.x is no longer supported, and known to have many crasher bugs that are fixed in 2 version.


The masked password text is removed/hidden after restarting Joplin.

The masked password text for protected backups is removed/hidden after restarting Joplin.
There's no UI indication whether backups are going to be encrypted with my previously entered password or not.  Looks like the file will be encrypted with an empty password, because the field is empty.
I would expect the password field to appear just like it did prior to restarting Joplin, with the password field masked.

Enter password.

Apply/Ok settings.

Exit and restart Joplin.

Joplin 2.1.9 (prod, win32)
Simple Backup v1.0.2

This sounds like the bug in Joplin < 2.1.3

Could you please set the log to verbose and post the logfile
Could you also post the Joplin debug console messages and logfile from a start and backup run
https://joplinapp.org/debugging/

I'm having a similar issue.
Password set:

When Joplin is restarted the password is reset:

backup.log (verbose, backup successful before restart):
[2021-12-08 19:57:29.408] [info] Backup started
[2021-12-08 19:57:29.408] [verbose] loadSettings
[2021-12-08 19:57:29.408] [verbose] loadBackupPath
[2021-12-08 19:57:29.459] [verbose] Plugin settings:
[2021-12-08 19:57:29.470] [verbose] path: /home/shbach/Filen Sync/Backups/Desktop/Joplin
[2021-12-08 19:57:29.472] [verbose] singleJex: true
[2021-12-08 19:57:29.473] [verbose] backupRetention: 5
[2021-12-08 19:57:29.474] [verbose] backupInterval: 24
[2021-12-08 19:57:29.476] [verbose] onlyOnChange: true
[2021-12-08 19:57:29.489] [verbose] usePassword: true
[2021-12-08 19:57:29.490] [verbose] lastBackup:<PHONE_NUMBER>453
[2021-12-08 19:57:29.491] [verbose] fileLogLevel: verbose
[2021-12-08 19:57:29.492] [verbose] zipArchive: no
[2021-12-08 19:57:29.494] [verbose] compressionLevel: 0
[2021-12-08 19:57:29.506] [verbose] exportPath:
[2021-12-08 19:57:29.508] [verbose] backupSetName: {YYYYMMDDHHmm}
[2021-12-08 19:57:29.509] [verbose] backupInfo: [{"name":"202112071826","date":1638930352999},{"name":"202112081912","date":1639019533842},{"name":"202112081918","date":1639019928873},{"name":"202112081937","date":1639021011180},{"name":"202112081945","date":1639021517453}]
[2021-12-08 19:57:29.509] [verbose] activeBackupPath: /home/shbach/.config/joplin-desktop/tmp/joplin_active_backup_job
[2021-12-08 19:57:29.510] [verbose] backupBasePath: /home/shbach/Filen Sync/Backups/Desktop/Joplin
[2021-12-08 19:57:29.510] [verbose] logFile: /home/shbach/.config/joplin-desktop/cache/io.github.jackgruber.backup/activeBackup.log
[2021-12-08 19:57:29.510] [verbose] showDoneMsg: true
[2021-12-08 19:57:29.539] [verbose] installationDir: /home/shbach/.config/joplin-desktop/cache/io.github.jackgruber.backup
[2021-12-08 19:57:29.580] [info] Enable password protection: true
[2021-12-08 19:57:29.580] [verbose] Backup path: /home/shbach/Filen Sync/Backups/Desktop/Joplin
[2021-12-08 19:57:29.580] [verbose] Active backup path (export path): /home/shbach/.config/joplin-desktop/tmp/joplin_active_backup_job
[2021-12-08 19:57:29.582] [verbose] Create folder /home/shbach/.config/joplin-desktop/tmp/joplin_active_backup_job
[2021-12-08 19:57:29.584] [info] Backup Profile Data
[2021-12-08 19:57:29.584] [verbose] Create folder /home/shbach/.config/joplin-desktop/tmp/joplin_active_backup_job/profile
[2021-12-08 19:57:29.595] [verbose] Copy /home/shbach/.config/joplin-desktop/settings.json
[2021-12-08 19:57:29.596] [verbose] Copy /home/shbach/.config/joplin-desktop/keymap-desktop.json
[2021-12-08 19:57:29.596] [verbose] Copy /home/shbach/.config/joplin-desktop/userchrome.css
[2021-12-08 19:57:29.596] [verbose] Copy /home/shbach/.config/joplin-desktop/userstyle.css
[2021-12-08 19:57:29.610] [info] No templateDir, Joplin >= v2.2.5
[2021-12-08 19:57:29.611] [info] Select notebooks for export
[2021-12-08 19:57:29.645] [verbose] Add 'Recipes' (2c268ab43f7946a5a4211bde3270a96a)
[2021-12-08 19:57:29.646] [verbose] Add 'Ideas' (3f5d2a79a15b45a0b87a2284d63485e5)
[2021-12-08 19:57:29.671] [info] Create single file JEX backup
[2021-12-08 19:57:36.938] [info] Create zip archive
[2021-12-08 19:57:36.939] [verbose] Add /home/shbach/.config/joplin-desktop/tmp/joplin_active_backup_job/all_notebooks.jex to zip /home/shbach/.config/joplin-desktop/tmp/joplin_active_backup_job/all_notebooks.jex.7z
[2021-12-08 19:57:39.024] [verbose] Add /home/shbach/.config/joplin-desktop/tmp/joplin_active_backup_job/profile/* to zip /home/shbach/.config/joplin-desktop/tmp/joplin_active_backup_job/profile.7z
[2021-12-08 19:57:39.776] [info] Move finished backup
[2021-12-08 19:57:39.795] [verbose] deleteOldBackupSets
[2021-12-08 19:57:39.799] [verbose] Remove backup set /home/shbach/Filen Sync/Backups/Desktop/Joplin/202112071826
[2021-12-08 19:57:40.097] [info] Backup finished to: /home/shbach/Filen Sync/Backups/Desktop/Joplin/202112081957
[2021-12-08 19:57:40.097] [info] Backup completed

When I restart Joplin and attempt a backup, backup.log does not have any new entries. However, activeBackup.log is updated and contains the following:
[2021-12-08 20:08:02.068] [info] Backup started
[2021-12-08 20:08:02.068] [verbose] loadSettings
[2021-12-08 20:08:02.069] [verbose] loadBackupPath
[2021-12-08 20:08:02.116] [verbose] Plugin settings:
[2021-12-08 20:08:02.117] [verbose] path: /home/shbach/Filen Sync/Backups/Desktop/Joplin
[2021-12-08 20:08:02.119] [verbose] singleJex: true
[2021-12-08 20:08:02.120] [verbose] backupRetention: 5
[2021-12-08 20:08:02.134] [verbose] backupInterval: 24
[2021-12-08 20:08:02.137] [verbose] onlyOnChange: true
[2021-12-08 20:08:02.151] [verbose] usePassword: true
[2021-12-08 20:08:02.152] [verbose] lastBackup:<PHONE_NUMBER>406
[2021-12-08 20:08:02.154] [verbose] fileLogLevel: verbose
[2021-12-08 20:08:02.168] [verbose] zipArchive: no
[2021-12-08 20:08:02.170] [verbose] compressionLevel: 0
[2021-12-08 20:08:02.183] [verbose] exportPath:
[2021-12-08 20:08:02.184] [verbose] backupSetName: {YYYYMMDDHHmm}
[2021-12-08 20:08:02.186] [verbose] backupInfo: [{"name":"202112081957","date":1639022249406},{"name":"202112081945","date":1639021517453},{"name":"202112081937","date":1639021011180},{"name":"202112081918","date":1639019928873},{"name":"202112081912","date":1639019533842}]
[2021-12-08 20:08:02.186] [verbose] activeBackupPath: /home/shbach/Filen Sync/Backups/Desktop/Joplin/joplin_active_backup_job
[2021-12-08 20:08:02.186] [verbose] backupBasePath: /home/shbach/Filen Sync/Backups/Desktop/Joplin
[2021-12-08 20:08:02.187] [verbose] logFile: /home/shbach/.config/joplin-desktop/cache/io.github.jackgruber.backup/activeBackup.log
[2021-12-08 20:08:02.187] [verbose] showDoneMsg: true
[2021-12-08 20:08:02.204] [verbose] installationDir: /home/shbach/.config/joplin-desktop/cache/io.github.jackgruber.backup
[2021-12-08 20:08:02.232] [error] Passwords do not match!

Joplin version: <IP_ADDRESS> (appimage)
Simple backup version: 1.0.5
OS: Solux Linux
Let me know if I need to provide any more info.

Update your Joplin to > 2.6.2
https://github.com/laurent22/joplin/releases/tag/v2.6.2
https://github.com/laurent22/joplin/issues/5720

Update your Joplin to > 2.6.2

Thanks!
Interesting that Joplin isn't telling me to upgrade.
Joplin 2.5.12 (prod, win32)


[JSDoc] Add missing documentation for adaptive-expressions/builtinFuntions files 10/10

Addresses # 2602
Description
This PR adds the missing documentation in adaptive-expressions/builtinFunctions/ files.
Some documentation is ported and adapted from botbuilder-dotnet
Specific Changes

Added JSDoc comments in all public methods.
Added @private attribute to private methods.
Fixes linter spacing warnings.

Testing
The following image shows the ESLint Report with no JSDoc errors.

Promoted


Complete implementation of Book Reviews webapp

Description
In this pull request, we have successfully implemented the essential features of our Book Reviews webapp. The following functionalities have been added:

Fetching and Display: The app dynamically fetches books from an external API and displays them on the homepage.
Likes Feature: Users can like their favorite books. The count of likes is updated in real-time.
Comments Popup: A popup that shows comments for a selected book and allows users to add new comments.
Reservations Popup: (For groups of 3 students) A popup that shows reservations for a selected book and lets users add new reservations.
Counters: The homepage displays the count of items, the comments popup shows the number of comments, and the reservations popup displays the number of reservations.
Styling: The app is styled according to the given wireframes, with additional design improvements for a better user experience.
Testing: Added unit tests for items, comments, and reservations counters.

All team members have collaborated effectively to complete the group tasks, and each student focused on individual tasks to ensure the functionalities were implemented correctly. The project adheres to best practices for HTML, CSS, and JavaScript, and the repository is set up with Webpack and Jest for bundling and testing, respectively.
We kindly request a review of our project and appreciate any feedback provided.

Hello Reviewer!

We'd like to bring to your attention that due to an oversight, our project was already merged into the main branch before we submitted the pull request. We apologize for this confusion. As a result, the deployed branch for our project is the main branch.
We assure you that everything should be functioning as expected, and you can verify the features and functionalities using the live demo link. We have thoroughly tested the application to ensure all requirements have been met.
We appreciate your understanding and kindly request a review based on the current state of the main branch.
Thank you for your time and feedback!


fancyzones not good one

Microsoft PowerToys version
V0.46.0
Running as admin

[X] Yes

Area(s) with issue?
FancyZones
Steps to reproduce
When I am editing fancy zone and I save it when I restart the cumputer the save existing but this it not this one was selected and when I open a new window.
✔ Expected Behavior
nothing
❌ Actual Behavior
nothing
Other Software
nothing

Duplicate of #13069

Thanks for reporting this. This issue is a duplicate. We'll track this in #13069 to centralize the discussion thread.


Conversion fails with unspported op FusedDepthwiseConv2dNative

Quite a few new models from TFHub fail on coversion:
vlado@Spectre:~/dev/models $ tfjs_graph_converter imagenet-efficientnet-b5 imagenet-efficientnet-b5.pb
TensorFlow.js Graph Model Converter

Graph model:    imagenet-efficientnet-b5
Output:         imagenet-efficientnet-b5.pb
Target format:  tf_frozen_model

Converting.... Error: Node module_apply_default/efficientnet-b5/model/blocks_0/depthwise_conv2d/depthwise: unsupported op FusedDepthwiseConv2dNative

Model in question is published at https://tfhub.dev/google/efficientnet/b5/classification/1
Same error with few other models published recently by Google, such as all AIY models and all EfficientNet models on TFHub.

Nice! Thanks :)


fix: add overrides for some of the content schemas

This will allow to have some better discrimination between schemas

This will only work for referenced schemas.
I'm not sure how you would proceed to fix this for the de-referenced schema

Fixed upstream


BadRequest400Exception in collection with id 17017150

JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 10)): has to be escaped using backslash to be included in string value\n

fixed in 25c2dbdd063eb4cc7b069c23270c5e5bd5b725e5 with replacing all "\r", "\n", "\t", "\v"


Add package publish Github action

Create a Github workflow that builds, tests and publishes the package on NPM.
Configure to make it run on test branch.

Github Actions for automated test and build added.


cpu-o3: fix false positive in AddressSanitizer

AddressSanitizer found a new-delete-type-mismatch because of the custom new operator for DynInst.
Adding a custom delete operator for DynInstPtr fixes this issue. It has been fixed the same way in Mozilla:
https://bugzilla.mozilla.org/show_bug.cgi?id=1391500
Change-Id: I0ab4cb6d79cac88069cc2374a1deb499cdb15f02
Reviewed-on: https://gem5-review.googlesource.com/c/public/gem5/+/68357
Maintainer: Jason Lowe-Power<EMAIL_ADDRESS>Reviewed-by: Jason Lowe-Power<EMAIL_ADDRESS>Tested-by: kokoro<EMAIL_ADDRESS>

In convention, we use merge for multi-commit pr, while use squash for single-commit pr


Installation troubleshooting / FAQ page

📚 Documentation
Would be great to have an FAQ section in our installation docs with known issues and workarounds for things like colab or kaggle install. See the FiftyOne docs for inspiration: https://voxel51.com/docs/fiftyone/getting_started/troubleshooting.html
Also see #804 where this discussion began

Of the top of my head we have:

torchtext troubles (need uninstall of torchtext first):

pip uninstall -y torchtext
pip install lightning-flash[...]

fiftyone + PL issue (some research needed here to gather possible error messages this can give): no workaround, just can't do it for now, see the issue here: https://github.com/voxel51/fiftyone/issues/1376


Feature/matching

This is a draft, code for matching should be finished first.
This has been done with @Visssarion , we might refactor the code a bit before we merge this

@Visssarion @HaiKonofanDesu Feel free to review

Ok, I'm really done now, going to sleep :')

I'm gonna need to look into timer stuff, I feel like current implementation isnt right.


Package the application as an .app bundle for macOS

This might require the use of cargo-bundle, although it appears that not everything works as expected on that. It's probably not worth the effort to also package it for Intel Macs unless I can get access to one.

Commit 032c4038e27b2640aab8d829f3224f45eb7ff216 enables compiling a bundle on macOS using cargo bundle.


Unnecessary use of pipes disables terminal features

onchange always pipes the stdout of the child process, which causes the child process to think it's not running in a terminal (as that's how most utilities detect whether they are running in on).
This disables useful features such as color highlighting (in log messages etc.).
Ideally, stdout should not be piped unless it's necessary (due to the other command-line options),.

@jaens Wouldn't you always want the output of your executed command to be printed? Changing this would be a backward incompatible change, if I understand what you're asking for. I also believe, but am not 100% familiar, that modules don't detect color based on whether they're being piped, but through isTTY. Can you show an example that would be fixed by avoiding piping?

Did a test, this comes down to incorrect options for stdio in spawn. I can open a PR, it seems to work as expected in the new case.

Released in 6.1.1, let me know if this resolves your issue!

Yeah, what I meant is exactly what you did - not calling the .pipe() method, instead just passing through stdout directly.
Colors (isTTY) works now, thanks!


Fix 93dub on Thursdays

This should fix a crash for 93dub that only shows up on Thursdays. When this happens it even seems to disable navigation back to the Settings app from the button on a Banglejs 2 when set as the default clock.
I've tested this on the emulator, and it appears to do the correct thing now.

Great - thanks! I think if it errors while executing, Bangle.setUI never gets called so the launcher button doesn't work.


Feature request:  enable ignore 404 - File not found

I'm working with an openUI5 project in the SAP ecosystem, and it happens that UI5 is notorious for both the number of files used at runtime, and the number of 404 resource load failures which are expected to fail gracefully.  It would be great if there was an option --ignore-errors which allowed the image generation to complete, regardless of the state of the context.
Here's an example of the error I'm seeing (only visible in the CLI -- seems to hang in pageres if delay is > 14-15, else completes without 'done'):
$ pageres localhost:3003/SAPUI5Tutorials/webapp/index.html 375x677 --crop --scale=2 --delay=2 --verbose
failed to load 'sap/ui/demo/wt/Component-preload.js' from ./Component-preload.js: 404 - File not found
Caused by: undefined → http://localhost:3003/SAPUI5Tutorials/resources/sap-ui-core.js on line 88 in function f1

I see this 404 error in Chrome also, but there it is just logged and ignored.

Some additional details:  I dug into pageres/dist/index.js, and compared the runtime sequence of a successful execution (apple.com) vs. the error case.  I found that in run(), both cases correctly progress to case 7, after setting context$2$0.next = 7 in case 2, and both correctly set context$2$0.next = 9 in case 7.  But in the failure case, the switch evaluates (context$2$0.prev = context$2$0.next) as 'end', while in the success case it is evaluated to 9.
Oddly, in both cases, context$2$0.next = 9, and context$2$0.prev = 7.
Edit:  Looks like there is probably a barf in the babel-runtime AsyncIterator...?

In babel-runtime/regenerator/runtime.js Line 152 block:
if (value instanceof AwaitArgument) {
          return _Promise.resolve(value.arg).then(function (value) {
            invoke("next", value, resolve, reject);
          }, function (err) {
          console.log('AwaitArgument Err:  ' + JSON.stringify(err, null, 2));
            invoke("throw", err, resolve, reject);
          });
        }

prints:
AwaitArgument Err:  {
  "noStack": true
}

It would be great if there was an option --ignore-errors which allowed the image generation to complete, regardless of the state of the context.

No need for an option. That's the default. If not, it's a bug.

I can test this locally but I need to understand the issue correctly :).
So the problem is that your page is including a script that results in a 404 because the script can't be found?

I have a similar issue @sindresorhus @SamVerschueren.
Using the Shots package, I am seeing the following pageres error due to an on-page uncaught JS error.
(env)➜  shots2 shots philly.com -c 1 -o ~/Desktop/shotsoutput/ --verbose
  shots dest directory set to: /Users/davis/Desktop/shotsoutput/ +0ms
  shots pmkdirp(pages) +3ms
  shots pmkdirp(screenshots) +1ms
  shots pmkdirp(diffs) +0ms
  shots pmkdirp(output) +0ms
  shots rglob('/Users/davis/Desktop/shotsoutput/pages/*.html') +5ms
  shots resolving download stage +0ms
  shots waybackpack philly.com -d /Users/davis/Desktop/shotsoutput/pages --start<PHONE_NUMBER>2940 +8s
  shots rglob('/Users/davis/Desktop/shotsoutput/pages/*.html') +8s
  shots rglob('/Users/davis/Desktop/shotsoutput/screenshots/*.html.png') +1ms
  shots resolving screenshots stage +0ms
  shots shot chunk(sources) and filter, len=11831 +4s
  shots shot chunk reducer, len=11830 +0ms
  shots reducing chunk 1/11830, len=1 +11ms
  shots creating pageres instance +0ms
  shots adding pageres src /Users/davis/Desktop/shotsoutput/pages/philly-com-20160507070636.html, sizes=1024x768 +0ms
  shots ERR! Error: /web/20160507070754/http://requirejs.org/docs/errors.html#scripterror → https://web.archive.org/web/20160507070754js_/http://launch.newsinc.com/132/js/embed.js on line 69 in function defaultOnError +6s
{ [Error: /web/20160507070754/http://requirejs.org/docs/errors.html#scripterror → https://web.archive.org/web/20160507070754js_/http://launch.newsinc.com/132/js/embed.js on line 69 in function defaultOnError] noStack: true }

is there a recommended media to ignore or skip pages with JS error gracefully?

I'll have a look if it's possible to ignore those errors. Apparently lots of sites are having crappy scripts which breaks our library.

@samverschueren That is exactly the case :). But maybe even crappy sites deserve to be remembered/screenshotted?
If I can help please let me know. I had thought that you were using PhantomJS as it's in the package.json but I didn't see it used in pageres, perhaps you could point me to that as I think we should be able to give PhantomJS a nulled out onError as outlined here http://phantomjs.org/api/phantom/handler/on-error.html

We are using screenshot-stream which is a nice interface to PhantomJS.

This bug should be fixed in screenshot-stream. You have to reinstall pageres so it installs the latest screenshot-stream version and then try it again. I'll close this issue for now. If it's still not resolved, let us know.


[Feature Request]: Support for Objects and Array data types

What would you like to happen?
Migration: https://issues.apache.org/jira/browse/HOP-2476
As a developer I come across objects and arrays stored as part of a record fairly regularly. I'd like Hop to natively support them with dedicated data types so that extracting values becomes easier (no need to use any scripting or coding).
In example, similarly to Trifacta, Hop could offer easy to use functions like these ones then:

"Flatten array values into new rows":
[{"type":"primary","colour":"green","product_number":3899},{"type":"secondary","colour":"red","product_number":3899}]

returns two rows:
{"type":"primary","colour":"green","product_number":3899} {"type":"secondary","colour":"red","product_number":3899}

"Unnest values into new columns". The unnest transformation takes a nested key-value pair in the object and extracts it to a new column. e.g. unnest product_number:{"type":"secondary","colour":"red","product_number":3899} => 3899

Issue Priority
Priority: 3
Issue Component
Component: Metadata

+1


RackHD API may not reply with all running workflows

from Megan:
We’re using the workflows API to poll for monorail workflows. Basically, we take the list of workflows, look for the one we’re running and check it’s status to determine if it’s still pending and which tasks are pending.
We’re seeing an issue occasionally where the query will return without our workflow in the list. On our end, it causes an error and we exit from our provisioning here as this is unexpected. Manual queries later will have the workflow back in the list.
We were wondering if this was a known issue or if this is something we can provide some logs/debug around? Or, are there any known issues with number of requests? (Maybe some tuning we can do on our side).

When the workflow “disappears” from the list, does it appear to come back on a further request – i.e. the request to list a workflows isn’t returning data consistently? If so, that’s definitely not a known issue and we should open up a bug on that and track that down.

From a sidebar with @king-jam we have seen some occasional failures recently that may be similar.  Tagged him here for alignment.

We're doing a GET on nodes//workflows and then looking for the workflow within the workflows list. We poll on this request, so the workflow is there originally. Sometimes, the workflow is not found within the list, which causes our logic to exit.
After the failure, we can typically manually query the workflows and see the workflow in the response.

Adding to Megan's comment: what I've seen happen is that the GET for the workflow will occasionally come back empty, and waiting 5-10 minutes and doing the same query manually gets the right result.

@tysonmcnulty thanks! That's awesome detail. Any chance you noted how long that http request took to complete?
@benbp ^^ this sounds like the messenger maybe timed out or something while retrieving the list of active workflows (timeout in the communications between on-http and on-taskgraph)

Yeah, sounds like a messenger timeout. Depending on the number of active workflows, this can be a LOT of data coming out of the database and over the wire, so I wouldn't be surprised if we're timing out.
I think you'll have better results if you poll /nodes/<node id>/workflows/active: that will give you only the workflow that's actively running for the node, and the request is less likely to time out.

@meganmurawski @tysonmcnulty any luck with the route alternative ^^, or still struggling?

We can change the way we poll for the workflows, but I would feel better about getting an error from this query if there is a timeout instead of incomplete/inconsistent results.
Can we be certain that querying the active workflows won't also result in a timeout?

@meganmurawski there is an issue with using the GET /nodes/<node id>/workflows/active route since at workflow completion you will get a 204 not found which may be a problem.
I think the right solution is to take the response from POST /nodes/<node id>/workflows and grab the workflow id, then pass that into a poll of GET /workflows/<workflow id>.
If you do that then you aren't querying as much data. You can simplify your WorkflowFetcher code, and have a better control loop.

Agreed @meganmurawski, didn't mean to suggest this isn't a real issue. Active workflows will be much less likely to result in a timeout since the data size is limited to a single workflow, but as @king-jam said you will have issues with a 204 on completion.
@heckj @RackHD/corecommitters I think the proper fix here might be to go directly to the database from the on-http server instead of passing this large amount of data over a single AMQP message from on-taskgraph (going directly to the db will also make it easier to paginate the data, or add logic to the route for better filtering/limiting).

@benbp @king-jam it sounds like the right solution for us will be grabbing /workflows/<workflow_id>
is there any possibility that we'll see timeouts on that resource if there are a bunch of workflows in general? or should it be ok since it's a small amount of data?
Pagination sounds like a good idea for the workflows. We can probably avoid using it for our polling, but may need to query all the workflows at some point.

@meganmurawski I don't think you'll see timeout issues for the /workflows/<workflow_id> route, partly because it's a small amount of data, and also because the logic for that route goes straight to the database rather than receiving data over AMQP (which would be the fix for the other routes).
One caveat:
The nodes/<id>/workflows route also goes straight to the database, however it will return the entire workflow history, and I don't think the output is sorted in any way, so you aren't necessarily guaranteed to know which workflow is the latest one without doing a createdAt/updatedAt sort yourself (which is kind of annoying).

@benbp could you verify - will this be effectively resolved when the workflow refactor work merges in?
RackHD/on-taskgraph#56, https://github.com/RackHD/on-tasks/pull/116, and RackHD/on-http#112

@heckj we didn't make any updates to that route with the refactor, but it would be trivial to add some code in there that sorts the output as its own item of work.

@heckj wasn't this resolved to be a unit test bug? Or is there still an issue on our side?

I never heard a full resolution, but I believe it was an mismatch in the client code using the APIs.


set python stdio encoding to utf-8

should fix #503

Should we put this in the image instead?

Done.

Pull Request Test Coverage Report for Build 1751

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 66.809%

Totals

Change from base Build 1748:
0.0%

Covered Lines:
6820

Relevant Lines:
9606

💛  - Coveralls


feature request: addition of pyspark.SparkConf

Currently config can be set only at SparkSession Building time .
using
builder = SparkSession.builder.appName("MyApp").config("spark.driver.memory", "15g")
But it look for specific keys
https://github.com/eakmanrq/sqlframe/blob/13f1c11c283583303535a67db583618ab3d0cde3/sqlframe/base/session.py#L615-L645
maybe we can keep the config in simple dict ?.
and have them spark.conf.get | spark.conf.set working
related to this when we have conf we can pass creds to duckdb
spark.conf.set("fs.azure.account.oauth.provider.type.<storage-account>.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
spark.conf.set("fs.azure.account.oauth2.client.id.<storage-account>.dfs.core.windows.net", "<application-id>")
spark.conf.set("fs.azure.account.oauth2.client.secret.<storage-account>.dfs.core.windows.net", service_credential)
spark.conf.set("fs.azure.account.oauth2.client.endpoint.<storage-account>.dfs.core.windows.net", "https://login.microsoftonline.com/<directory-id>/oauth2/token")

[x] Yes. I would be willing to contribute this feature with guidance from the maintainers.

Hey @sherlockbeard thanks for the suggestion.
Some context as to why SQLFrame is written this way currently is that is needs to know which of the config is relevant to it and which can be discarded. For example we wouldn't want to pass spark.driver.memory kwarg to the construction of a duckdb connection right? That is why SQLFrame checks what config is passed in, stores the values relevant to it, and then discards the rest. Do you have thoughts on how to address this if we start passing all config into the connection constructor?
One idea would be to just discard spark.* config but I don't think it is safe to assume that anything spark.* can be passed into the connection.
Another way to solve this problem is to construct the DuckDB connection outside of SQLFrame and then pass that in. For example:
import duckdb
from adlfs.spec import AzureBlobFileSystem
from sqlframe.duckdb import DuckDBSession

connection = duckdb.connect()
connection.register_filesystem(AzureBlobFileSystem(account_name='<storage_account>', connection_string='<storage_account_key>'))
session = DuckDBSession(conn=connection)

...

Let me know what you think.

Another way to solve this problem is to construct the DuckDB connection outside of SQLFrame and then pass that in. For example:

this option is good .
I was thinking if we can dynamically pass specific config
spark.conf.set("fs.azure.account.oauth.provider.type.<storage-account>.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
spark.conf.set("fs.azure.account.oauth2.client.id.<storage-account>.dfs.core.windows.net", "<application-id>")
spark.conf.set("fs.azure.account.oauth2.client.secret.<storage-account>.dfs.core.windows.net", service_credential)
spark.conf.set("fs.azure.account.oauth2.client.endpoint.<storage-account>.dfs.core.windows.net", "https://login.microsoftonline.com/<directory-id>/oauth2/token")

to duckdb SECRET
query = connection.sql(f'''
CREATE SECRET tf12test (TYPE AZURE, CONNECTION_STRING '${Azure_key}', ACCOUNT_NAME 'tf12test');
''')
But now I realize it will be too much work without much meaning 😴.


SAP Mobile Cards - Tutorial Update

Hello,
Scenario link: https://developers.sap.com/tutorials/cp-mobile-cards-action.html#b0de6140-460f-47d5-a7c7-811d29bfe4b4
Step 8: Cross verify status update in the back end ->In the Mobile Services cockpit, click Connectivity under the Features tab. ->
statement should be "click MOBILE Connectivity" not just Connectivity.
Also, at step : If you   cannot find particular data set, click Entity Properties to select some   properties, then click OK. -> click on "SUBMIT" is there instead of "OK"
Thanks,
Shubham
Landscape label: trail1 & trail2 & trial3
Scenario label : SCPTest-mcard
Priority : Low-prio
Sprint Label: SCPTest-2009A

@i353590 Thank you for raising the issue. I have updated our document.
Changes will be publically accessible by early next week.

Thank you for your feedback. The issue seems to be resolved, so I am closing the issue.
If you still have questions, feel free to reopen the issue.


young adult group edit

Young adult leaders are now Kish Larson & Camden Kerby.
Not sure if Smiths will continue to host this group, Ben will confirm that for us soon.
Here is Kish's pic, I'll get Camden's soon. Since they're not a couple, I'd like 2 separate photos...are we able to do that?

@keelyjay Yes, that's doable.

I used this photo for now, I'll add the second when available
https://www.lifestonechurch.net/connect/lifegroups/


Types Parser

Hi,
I have written a plugin to integrate a C parser with Binary Ninja, for parsing C header files:
https://github.com/chame1eon/binaryninja-types-parser
Let me know if you want me to make any changes to it.
Thanks.

Closing for real this time. :-P
I pulled the plugin, though I'm curious (I haven't looked closely) how is this different from https://api.binary.ninja/binaryninja.platform-module.html?highlight=parse#binaryninja.platform.Platform.parse_types_from_source_file ?

Oh :(. It is no different at all. I wish I had seen that part of the API, then I would not have wasted my time making that plugin! Oh well. Sorry about this.  Seems little point in having a duplicate feature.

Yeah, I'll go ahead and pull it. No worries, consider it a useful learning experience. :-)