HTML refactoring

This PR adds all missing HTML5 nodes, adds support for node namespaces, to add support for SVG rendering, and adds support for non-standard CSS properties

Codecov Report
Patch coverage: 94.82% and project coverage change: +0.78 :tada:

Comparison is base (6f691bb) 68.96% compared to head (ceadd04) 69.74%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #380      +/-   ##
==========================================
+ Coverage   68.96%   69.74%   +0.78%
==========================================
  Files          68       81      +13
  Lines        5168     5355     +187
  Branches     1164      876     -288
==========================================
+ Hits         3564     3735     +171
- Misses       1326     1347      +21
+ Partials      278      273       -5

Impacted Files
Coverage Δ

lona/html/nodes/forms/select.py
79.20% <ø> (ø)

lona/html/nodes/forms/select2.py
68.07% <ø> (ø)

lona/html/node.py
75.38% <73.33%> (-0.13%)
:arrow_down:

lona/html/nodes/text_content.py
84.21% <84.21%> (ø)

lona/html/nodes/inline_text_semantics.py
93.33% <93.33%> (ø)

lona/html/nodes/forms/__init__.py
94.44% <94.44%> (ø)

lona/html/parsing.py
90.90% <95.45%> (+5.19%)
:arrow_up:

lona/html/__init__.py
100.00% <100.00%> (ø)

lona/html/nodes/content_sectioning.py
100.00% <100.00%> (ø)

lona/html/nodes/demarcating_edits.py
100.00% <100.00%> (ø)

... and 10 more

... and 10 files with indirect coverage changes
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

This PR is finally done! I fixed the HTML parsing code, so it works with SVGs, and added lona.html.RawHTML which can render  raw HTML on the client without the server parsing it first.
For interactive SVGs that contain Javascript, we will need another solution, but with these patches we will have initial SVG support in Lona \o/


⚠ Bofrost.de has degraded performance

In b90691c, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1358 ms

Resolved: Bofrost.de performance has improved in 27b5006 after 6 minutes.


pipenv lock fails when executed under python 3.12

Be sure to check the existing issues (both open and closed!), and make sure you are running the latest version of Pipenv.
Check the diagnose documentation for common issues before posting! We may close your issue if it is very similar to one of them. Please be considerate, or be on your way.
Make sure to mention your debugging experience if the documented solution failed.
Issue description
Runing pipenv lock on python 3.12 causes pipenv to fail with:
  File "/tmp/pip-build-env-cxa535v1/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 10, in <module>
    import distutils.core
ModuleNotFoundError: No module named 'distutils'

Expected result
pipenv lock would create or update a Pipenv.lock file
Actual result
When possible, provide the verbose output (--verbose), especially for locking and dependencies resolving issues.
[rvodden:~/Source/maaspower]└2 [maaspower] fix_linting+* 4s 1 ± pipenv lock --verbose
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
INFO:pipenv.patched.pip._internal.operations.prepare:Obtaining file:///home/rvodden/Source/maaspower (from -r
/tmp/pipenv-wnki6z45-requirements/pipenv-r79h4iyg-constraints.txt (line 2))
INFO:pipenv.patched.pip._internal.cli.spinners:Installing build dependencies: started
INFO:pipenv.patched.pip._internal.cli.spinners:Installing build dependencies: finished with status 'done'
INFO:pipenv.patched.pip._internal.cli.spinners:Checking if build backend supports build_editable: started
INFO:pipenv.patched.pip._internal.cli.spinners:Checking if build backend supports build_editable: finished with status 'done'
Traceback (most recent call last):
  File "/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/resolver.py", line 675, in <module>
    main()
  File "/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/resolver.py", line 661, in main
    _main(
  File "/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/resolver.py", line 645, in _main
    resolve_packages(
  File "/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/resolver.py", line 612, in resolve_packages
    results, resolver = resolve(
                        ^^^^^^^^
  File "/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/resolver.py", line 592, in resolve
    return resolve_deps(
           ^^^^^^^^^^^^^
  File "/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/utils/resolver.py", line 908, in resolve_deps
    results, hashes, internal_resolver = actually_resolve_deps(
                                         ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/utils/resolver.py", line 681, in
actually_resolve_deps
    resolver.resolve()
  File "/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/utils/resolver.py", line 440, in resolve
    results = resolver.resolve(constraints, check_supported_wheels=False)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File
"/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/resolver.py"
, line 76, in resolve
    collected = self.factory.collect_root_requirements(root_reqs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File
"/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/factory.py",
line 513, in collect_root_requirements
    reqs = list(
           ^^^^^
  File
"/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/factory.py",
line 474, in _make_requirements_from_install_req
    cand = self._make_candidate_from_link(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File
"/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/factory.py",
line 190, in _make_candidate_from_link
    self._editable_candidate_cache = EditableCandidate(
                                           ^^^^^^^^^^^^^^^^^^
  File
"/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/candidates.p
y", line 322, in __init__
    super().__init__(
  File
"/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/candidates.p
y", line 157, in __init__
    self.dist = self._prepare()
                ^^^^^^^^^^^^^^^
  File
"/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/candidates.p
y", line 226, in _prepare
    dist = self._prepare_distribution()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File
"/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/patched/pip/_internal/resolution/resolvelib/candidates.p
y", line 332, in _prepare_distribution
    return self._factory.preparer.prepare_editable_requirement(self._ireq)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/patched/pip/_internal/operations/prepare.py",
line 696, in prepare_editable_requirement
    dist = _get_prepared_distribution(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/patched/pip/_internal/operations/prepare.py",
line 71, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(
  File "/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/patched/pip/_internal/distributions/sdist.py",
line 52, in prepare_distribution_metadata
    self.req.isolated_editable_sanity_check()
  File "/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/patched/pip/_internal/req/req_install.py", line
544, in isolated_editable_sanity_check
    and not self.supports_pyproject_editable()
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/patched/pip/_internal/req/req_install.py", line
247, in supports_pyproject_editable
    return "build_editable" in self.pep517_backend._supported_features()
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/patched/pip/_vendor/pyproject_hooks/_impl.py",
line 153, in _supported_features
    return self._call_hook('_supported_features', {})
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/patched/pip/_vendor/pyproject_hooks/_impl.py",
line 321, in _call_hook
    raise BackendUnavailable(data.get('traceback', ''))
pipenv.patched.pip._vendor.pyproject_hooks._impl.BackendUnavailable: Traceback (most recent call last):
  File
"/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/patched/pip/_vendor/pyproject_hooks/_in_process/_in_proc
ess.py", line 77, in _build_backend
    obj = import_module(mod_path)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rvodden/.pyenv/versions/3.12.0/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name, package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1381, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1304, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1381, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1325, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 929, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 994, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/tmp/pip-build-env-fgmj84tv/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 10, in <module>
    import distutils.core
ModuleNotFoundError: No module named 'distutils'
✘ Locking Failed!
⠙ Locking...
Traceback (most recent call last):
  File "/home/rvodden/.pyenv/versions/maaspower/bin/pipenv", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/cli/options.py", line 58, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/vendor/click/decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/vendor/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/cli/command.py", line 340, in lock
    do_lock(
  File "/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/routines/lock.py", line 65, in do_lock
    venv_resolve_deps(
  File "/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/utils/resolver.py", line 849, in venv_resolve_deps
    c = resolve(cmd, st, project=project)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rvodden/.pyenv/versions/3.12.0/envs/maaspower/lib/python3.12/site-packages/pipenv/utils/resolver.py", line 718, in resolve
    raise RuntimeError("Failed to lock Pipfile.lock!")
RuntimeError: Failed to lock Pipfile.lock!

Steps to replicate
Provide the steps to replicate (which usually at least includes the commands and the Pipfile).

install python 3.12
set python 3.12 to be your current python
install pyenv
create python project
run pipenv lock

Please run $ pipenv --support, and paste the results here. Don't put backticks (`) around it! The output already contains Markdown formatting.
If you're on macOS, run the following:
$ pipenv --support | pbcopy

If you're on Windows, run the following:
> pipenv --support | clip

If you're on Linux, run the following:
$ pipenv --support | xclip

pipenv support is here:
https://gist.github.com/rvodden/7c39c1984c36d1f45f788bbf5ec91ee3

The test runner runs on python 3.12 and there are plenty of operations that lock.

Ok. I can still replicate this reliably by using the steps I gave in the report.

hey @rvodden
I had the same issue. This error has nothing to do with the Pipenv itself. The problem lies in a particular library specified in your Pipfile. The newly released Python 3.12 removed distutils after it has been deprecated since Python 3.10. You need to upgrade some libraries to support Python 3.12.


Typo fix

@johnnychen94 hey! Just going through some of my older PRs, would you consider merging this or should I close it?


Bump rust version to 1.68

https://github.com/rust-lang/cargo/pull/11224

https://blog.rust-lang.org/2022/06/22/sparse-registry-testing.html#details

in ~/.cargo/config.toml
[registries.crates-io]
protocol = "sparse"

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

1 of 1 (100.0%)  changed or added relevant line in 1 file are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage increased (+0.07%) to 98.839%

Files with Coverage Reduction
New Missed Lines
%

core/src/executor/validate.rs
1
99.24%

Totals

Change from base Build<PHONE_NUMBER>:
0.07%

Covered Lines:
39740

Relevant Lines:
40207

💛  - Coveralls


[Update your ComfyUI to work with current AnimateDiff-Evolved version] help ！ cannot import name 'model_sampling' from 'comfy.model_base'，when update comfyui

Traceback (most recent call last):
File "D:\ProgramData\ComfyUI\ComfyUI\nodes.py", line 1735, in load_custom_node
module_spec.loader.exec_module(module)
File "", line 883, in exec_module
File "", line 241, in call_with_frames_removed
File "D:\ProgramData\ComfyUI\ComfyUI\custom_nodes\ComfyUI-AnimateDiff-Evolved_init.py", line 2, in
from .animatediff.nodes import NODE_CLASS_MAPPINGS, NODE_DISPLAY_NAME_MAPPINGS
File "D:\ProgramData\ComfyUI\ComfyUI\custom_nodes\ComfyUI-AnimateDiff-Evolved\animatediff\nodes.py", line 17, in
from .motion_module import eject_params_from_model, inject_params_into_model, load_motion_lora, load_motion_module
File "D:\ProgramData\ComfyUI\ComfyUI\custom_nodes\ComfyUI-AnimateDiff-Evolved\animatediff\motion_module.py", line 18, in
from .model_utils import ModelTypesSD, calculate_file_hash, get_motion_lora_path, get_motion_model_path, get_sd_model_type, is_checkpoint_sd1_5
File "D:\ProgramData\ComfyUI\ComfyUI\custom_nodes\ComfyUI-AnimateDiff-Evolved\animatediff\model_utils.py", line 13, in
from comfy.model_base import SDXL, BaseModel, model_sampling
ImportError: cannot import name 'model_sampling' from 'comfy.model_base' (D:\ProgramData\ComfyUI\ComfyUI\comfy\model_base.py)
Cannot import D:\ProgramData\ComfyUI\ComfyUI\custom_nodes\ComfyUI-AnimateDiff-Evolved module for custom nodes: cannot import name 'model_sampling' from 'comfy.model_base' (D:\ProgramData\ComfyUI\ComfyUI\comfy\model_base.py)

ComfyUI had an update that I had to push a code change for to work with. The change makes it not backwards compatible with previous ComfyUI versions. Update your ComfyUI, and the error will go away.

I will in advance admit I am just getting back into this and novice and I might be doing something dumb. I have everything up to date and even went as far as re-installing my whole computer.
Traceback (most recent call last):
File "C:\Users\datin\ComfyUI_windows_portable\ComfyUI\nodes.py", line 1735, in load_custom_node
module_spec.loader.exec_module(module)
File "", line 883, in exec_module
File "", line 241, in call_with_frames_removed
File "C:\Users\datin\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-AnimateDiff-Evolved_init.py", line 3, in
from .animatediff.model_utils import get_available_motion_models, Folders
File "C:\Users\datin\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-AnimateDiff-Evolved\animatediff\model_utils.py", line 9, in
from comfy.model_base import SDXL, BaseModel, model_sampling
ImportError: cannot import name 'model_sampling' from 'comfy.model_base' (C:\Users\datin\ComfyUI_windows_portable\ComfyUI\comfy\model_base.py)
Cannot import C:\Users\datin\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-AnimateDiff-Evolved module for custom nodes: cannot import name 'model_sampling' from 'comfy.model_base' (C:\Users\datin\ComfyUI_windows_portable\ComfyUI\comfy\model_base.py)
Loading: ComfyUI-Manager (V0.36)
ComfyUI Revision: 1627 [aac8fc99] | Released on '2023-10-28'
I haven't been able to fix this but it might be something dumb I did. I have my other computer working with everything the same the only difference is that its AMD

Nvmd, it was the Comfy Revision, I had to update comfy through the Manager and it worked! Thanks for a great program it's excellent on my less powerful computer and im ready to play now :)

Thank！！！，This solved my problem

thank!!!


[feat] Allow wheel to accept choices via query string URL

I'd like to be able to save a wheel with my customizations and then set the default choices by passing in a URL parameter:
https://wheelofnames.com/jmj-a7b?choices=Leo,Fred,Monica,Jane
This would set up the wheel with the following choices:
Leo
Fred
Monica
Jane

Great idea

Keeping the security aspects aside, a simple approach to get started would be:
At ./static/WheelConfig.js line 20
  // this.names = [
  //   'Hamburger', 'Lasagna', 'Fish and chips', 'فلافل - Falafel',
  //   '炒麵 - Chow mein', 'Tortilla española', 'Crêpes', 'Pierogi', 'Feijoada',
  //   'ผัดไทย - Pad thai', 'Döner kebab', 'Phở'
  // ];
  // Run something like: http://localhost:port/?names=Leo,Fred,Monica,Jane
  this.names = window.location.search.match(/names=(.[^&]+)/i)[1].split(',');

I just started looking at the code, and seems like, at code loads the default config, and then relies upon LocalStorage "LastWheelConfig" so the code shared above will only work at the initial load.
There are more better alternatives like use of hash based routing and change the LocalStorage based on hash change. Also a better more secured query string or hash route parser can be used to load the whole config using query string params.
It could also be a base64 encoded string that can be deserialized, and one way to share the wheels would be to share the urls with whole config as base64 encoded string.
However, be careful with user provided input and regex it could be very insecure. The sample code is just an example.


fix!: migrate argo_archived_workflows.workflow to jsonb

Motivation

As explained in https://github.com/argoproj/argo-workflows/issues/13601#issuecomment-2420499551, I believe https://github.com/argoproj/argo-workflows/pull/12912 introduced a performance regression when listing workflows for PostgreSQL users due to TOAST issues. Reverting that PR would solve that particular issue, but it could re-introduce the memory issues mentioned in the PR description. Instead, this mitigates the impact by converting the workflow column to be of type jsonb.
Modifications
Initially workflow was of type text, and was migrated to json in https://github.com/argoproj/argo-workflows/pull/2152. I'm not sure why jsonb wasn't chosen, but based on this comment in the linked issue, I think it was simply an oversight. It's possible compatibility with older PostgreSQL versions was a concern. Support for jsonb was introduced in PostgreSQL 9.4, and PostgreSQL 9.3 became end-of-life on November 8, 2018. I don't think we should be concerned about breaking compatibility with EOL PostgreSQL versions, especially versions that have been EOL for nearly 6 years.
Here's what the relevant docs say about choosing between json and jsonb:

The json and jsonb data types accept almost identical sets of values as input. The major practical difference is one of efficiency. The json data type stores an exact copy of the input text, which processing functions must reparse on each execution; while jsonb data is stored in a decomposed binary format that makes it slightly slower to input due to added conversion overhead, but significantly faster to process, since no reparsing is needed. jsonb also supports indexing, which can be a significant advantage.
Because the json type stores an exact copy of the input text, it will preserve semantically-insignificant white space between tokens, as well as the order of keys within JSON objects. Also, if a JSON object within the value contains the same key more than once, all the key/value pairs are kept. (The processing functions consider the last value as the operative one.) By contrast, jsonb does not preserve white space, does not preserve the order of object keys, and does not keep duplicate object keys. If duplicate keys are specified in the input, only the last value is kept.
In general, most applications should prefer to store JSON data as jsonb, unless there are quite specialized needs, such as legacy assumptions about ordering of object keys.

I'm pretty sure we don't care about key order or whitespace. We do care somewhat about insertion speed, but archived workflows are read much more frequently than written, so a slight reduction in write speed that gives a large improvement in read speed is a good tradeoff. As the benchmarks below show, this gives a ~75% performance improvement for calls to ListWorkflows().
The biggest problem with this change is the time needed for the migration. With my local DB populated with 100,000 records, it takes ~70s (see below). The good news is this is fully backwards-compatible, and users can opt to manually run alter table argo_archived_workflows alter column workflow set data type jsonb using workflow::jsonb in their DB before upgrading. We'll probably want to add a prominent warning about this.
Verification
Here's steps to test this:

Use https://github.com/argoproj/argo-workflows/pull/13715 to generate 100,000 randomized workflows, with https://gist.github.com/MasonM/52932ff6644c3c0ccea9e847780bfd90 as a template:
$ time go run ./hack/db fake-archived-workflows  --template<EMAIL_ADDRESS>--rows 100000
Using seed<PHONE_NUMBER>624432788
Clusters: [default]
Namespaces: [argo]
Inserted 100000 rows

real    18m35.316s
user    3m2.447s
sys     0m44.972s

Run the benchmarks using https://github.com/argoproj/argo-workflows/pull/13767:
make BenchmarkWorkflowArchive > postgres_before_10000_workflows.txt

Run the migration the DB CLI:
$ time go run ./hack/db migrate
INFO[0000] Migrating database schema                     clusterName=default dbType=postgres
INFO[0000] applying database change                      change="alter table argo_archived_workflows alter column workflow set data type jsonb using workflow::jsonb" changeSchemaVersion=60
2024/10/17 18:07:42     Session ID:     00001
        Query:          alter table argo_archived_workflows alter column workflow set data type jsonb using workflow::jsonb
        Stack:
                fmt.(*pp).handleMethods@/usr/local/go/src/fmt/print.go:673
                fmt.(*pp).printArg@/usr/local/go/src/fmt/print.go:756
                fmt.(*pp).doPrint@/usr/local/go/src/fmt/print.go:1208
                fmt.Append@/usr/local/go/src/fmt/print.go:289
                log.(*Logger).Print.func1@/usr/local/go/src/log/log.go:261
                log.(*Logger).output@/usr/local/go/src/log/log.go:238
                log.(*Logger).Print@/usr/local/go/src/log/log.go:260
                github.com/argoproj/argo-workflows/v3/persist/sqldb.ansiSQLChange.apply@/home/vscode/go/src/github.com/argoproj/argo-workflows/persist/sqldb/ansi_sql_change.go:11
                github.com/argoproj/argo-workflows/v3/persist/sqldb.migrate.applyChange.func1@/home/vscode/go/src/github.com/argoproj/argo-workflows/persist/sqldb/migrate.go:295
                github.com/argoproj/argo-workflows/v3/persist/sqldb.migrate.applyChange@/home/vscode/go/src/github.com/argoproj/argo-workflows/persist/sqldb/migrate.go:284
                github.com/argoproj/argo-workflows/v3/persist/sqldb.migrate.Exec@/home/vscode/go/src/github.com/argoproj/argo-workflows/persist/sqldb/migrate.go:273
                main.NewMigrateCommand.func1@/home/vscode/go/src/github.com/argoproj/argo-workflows/hack/db/main.go:50
                github.com/spf13/cobra.(*Command).execute@/home/vscode/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:985
                github.com/spf13/cobra.(*Command).ExecuteC@/home/vscode/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117
                github.com/spf13/cobra.(*Command).Execute@/home/vscode/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041
                main.main@/home/vscode/go/src/github.com/argoproj/argo-workflows/hack/db/main.go:39
                runtime.main@/usr/local/go/src/runtime/proc.go:272
                runtime.goexit@/usr/local/go/src/runtime/asm_amd64.s:1700
        Rows affected:  0
        Error:          upper: slow query
        Time taken:     69.12755s
        Context:        context.Background

real    1m10.087s
user    0m1.541s
sys     0m0.410s

Note: the upper: slow query error is harmless and doesn't impact the migration.

Re-run the benchmarks:
make BenchmarkWorkflowArchive > postgres_after_10000_workflows.txt

Compare results using benchstat:
$ benchstat postgres_before_10000_workflows3.txt postgres_after_10000_workflows2.txt
goos: linux
goarch: amd64
pkg: github.com/argoproj/argo-workflows/v3/test/e2e
cpu: 12th Gen Intel(R) Core(TM) i5-12400
                                                     │ postgres_before_10000_workflows3.txt │  postgres_after_10000_workflows2.txt  │
                                                     │                sec/op                │    sec/op     vs base                 │
WorkflowArchive/ListWorkflows-12                                              183.83m ± ∞ ¹   24.69m ± ∞ ¹        ~ (p=1.000 n=1) ²
WorkflowArchive/ListWorkflows_with_label_selector-12                          192.71m ± ∞ ¹   25.87m ± ∞ ¹        ~ (p=1.000 n=1) ²
WorkflowArchive/CountWorkflows-12                                              13.04m ± ∞ ¹   11.75m ± ∞ ¹        ~ (p=1.000 n=1) ²
geomean                                                                        77.31m         19.58m        -74.68%
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05

                                                     │ postgres_before_10000_workflows3.txt │  postgres_after_10000_workflows2.txt  │
                                                     │                 B/op                 │     B/op       vs base                │
WorkflowArchive/ListWorkflows-12                                              497.2Ki ± ∞ ¹   497.5Ki ± ∞ ¹       ~ (p=1.000 n=1) ²
WorkflowArchive/ListWorkflows_with_label_selector-12                          503.1Ki ± ∞ ¹   503.9Ki ± ∞ ¹       ~ (p=1.000 n=1) ²
WorkflowArchive/CountWorkflows-12                                             8.972Ki ± ∞ ¹   8.899Ki ± ∞ ¹       ~ (p=1.000 n=1) ²
geomean                                                                       130.9Ki         130.7Ki        -0.20%
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05

                                                     │ postgres_before_10000_workflows3.txt │ postgres_after_10000_workflows2.txt  │
                                                     │              allocs/op               │  allocs/op    vs base                │
WorkflowArchive/ListWorkflows-12                                               8.373k ± ∞ ¹   8.370k ± ∞ ¹       ~ (p=1.000 n=1) ²
WorkflowArchive/ListWorkflows_with_label_selector-12                           8.410k ± ∞ ¹   8.406k ± ∞ ¹       ~ (p=1.000 n=1) ²
WorkflowArchive/CountWorkflows-12                                               212.0 ± ∞ ¹    212.0 ± ∞ ¹       ~ (p=1.000 n=1) ³
geomean                                                                        2.462k         2.462k        -0.03%
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05
³ all samples are equal

I've marked the title as breaking here (fix!) as it is a schema change and requires a long migration, as you mentioned

Fixes #13601

I also don't think this fully fixes #13601 since it does not affect MySQL, is an order of magnitude gain over JSON in Postgres, but can still take multiple seconds, and doesn't resolve the description of the issue either, but another related performance issue brought up in the thread

@agilgur5 Okay, I removed references to https://github.com/argoproj/argo-workflows/issues/13601 from the title and description. The reason I referenced it is because nearly all the comments in that issue (and the user reports) are about general performance issues with PostgreSQL, which this PR addresses, but you're right it doesn't address MySQL or the index optimization mentioned (CREATE INDEX argo_archived_workflows_i5 ON argo_archived_workflows (clustername, startedat).
I did try benchmarking the index optimization, but the performance improvement was very small, which is why I didn't include it here:
$ benchstat before.txt after.txt
goos: linux
goarch: amd64
pkg: github.com/argoproj/argo-workflows/v3/test/e2e
cpu: 12th Gen Intel(R) Core(TM) i5-12400
                                                     │  before.txt  │              after.txt               │
                                                     │    sec/op    │    sec/op     vs base                │
WorkflowArchive/ListWorkflows-12                       185.9m ± ∞ ¹   188.9m ± ∞ ¹       ~ (p=1.000 n=1) ²
WorkflowArchive/ListWorkflows_with_label_selector-12   199.3m ± ∞ ¹   185.0m ± ∞ ¹       ~ (p=1.000 n=1) ²
WorkflowArchive/CountWorkflows-12                      13.67m ± ∞ ¹   13.24m ± ∞ ¹       ~ (p=1.000 n=1) ²
geomean                                                79.71m         77.35m        -2.96%
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05

Also

but can still take multiple seconds

I'm pretty sure that's solely due to the JSON cast issue fixed in https://github.com/argoproj/argo-workflows/pull/13777. I spent over a week trying to reproduce that and I can't.

I did try benchmarking the index optimization, but the performance improvement was very small, which is why I didn't include it here:

It would still be better and less complex than a whole subquery IMO

I'm pretty sure that's solely due to the JSON cast issue fixed in #13777 (and this PR)

You believe the cast itself adding multiple seconds to a query? Certainly possible, but that would be even more surprising than the index misses or existing JSON inefficiencies with this issue in the past 😅

You believe the cast itself adding multiple seconds to a query? Certainly possible, but that would be even more surprising than the index misses or existing JSON inefficiencies with this issue in the past 😅

It slows it down by over 2x on my machine. But you don't have to take my word for it, because you can reproduce this yourself by following in the instructions in the PR. Just revert the changes made to persist/sqldb/workflow_archive.go.
Here's the benchstat output I get:
 $ benchstat before.txt after.txt
goos: linux
goarch: amd64
pkg: github.com/argoproj/argo-workflows/v3/test/e2e
cpu: 12th Gen Intel(R) Core(TM) i5-12400
                                                     │  before.txt  │               after.txt               │
                                                     │    sec/op    │    sec/op     vs base                 │
WorkflowArchive/ListWorkflows-12                       181.4m ± ∞ ¹   417.0m ± ∞ ¹        ~ (p=1.000 n=1) ²
WorkflowArchive/ListWorkflows_with_label_selector-12   181.6m ± ∞ ¹   417.0m ± ∞ ¹        ~ (p=1.000 n=1) ²
WorkflowArchive/CountWorkflows-12                      44.88m ± ∞ ¹   11.59m ± ∞ ¹        ~ (p=1.000 n=1) ²
geomean                                                113.9m         126.3m        +10.90%
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05

Sure, I just don't think we should necessarily close out the issue until we've ironed out all the remaining loose items, especially as I made the issue intentionally to cover all the remaining loose items after the previous ones

Okay, that's reasonable.

It slows it down by over 2x on my machine.

The cast alone? 👀 👀 As far as I could tell, here and in https://github.com/argoproj/argo-workflows/pull/13777 you mentioned both removing the cast and moving to JSONB

But you don't have to take my word for it, because you can reproduce this yourself by following in the instructions in the PR

I'm pretty sure

You had also said "pretty sure" before, so I thought you hadn't narrowed that down yet

I am not sure how I feel about this change. Whilst I completely agree with the goal of speed, it changes the behaviour of archiving in subtle but surprising ways.
We would end up have two different behaviours depending upon whether you're using mysql or postgres. If you want a 'perfect archive' (which for some users may be something they care about, perhaps compliance reasons), you must use mysql.
I can't see a mechanism whereby we would end up with bugs with duplicate keys, but archived workflows in postgres would hide them after this change.
We should document this change because it's reasonable to expect the contents of an archived workflow to exactly match that of that workflow before archiving, and that won't be the case. Have you considered an option (which most people wouldn't use) to store and present the textual version, but query via jsonb?

According to this documentation mysql performs some kind of json normalization during json insertion: https://dev.mysql.com/doc/refman/8.4/en/json.html
I checked this also on https://www.db-fiddle.com/. Maybe someone with mysql access could also verify it. However, it seems that currently mysql does not perform perfect archive and this is different behaviour from what current implementation in postgres does.

The cast alone? 👀 👀 As far as I could tell, here and in #13777 you mentioned both removing the cast and moving to JSONB

The comment you linked (https://github.com/argoproj/argo-workflows/issues/13601#issuecomment-2376579589) said "I tested converting json to jsonb and observed times approximately 9s." I'm assuming this was measured with the cast present (i.e. the current behavior), since that's the only scenario where I could reproduce such a slowdown. There may be other scenarios, but there's too many variables involved for me to narrow it down further.

I am not sure how I feel about this change. Whilst I completely agree with the goal of speed, it changes the behaviour of archiving in subtle but surprising ways.
We would end up have two different behaviours depending upon whether you're using mysql or postgres. If you want a 'perfect archive' (which for some users may be something they care about, perhaps compliance reasons), you must use mysql.

As @kodieg correctly pointed out, MySQL is already stripping whitespace and eliminating duplicate keys, so this PR actually makes the behavior more consistent, not less. You can verify this yourself by following these steps:

Save the following file as test.sql:insert into argo_archived_workflows (uid, instanceid, name, namespace, phase, clustername, workflow) values ('0b3ec561-bc78-400c-9772-ce11c69b3d65', '', 'dg92nw8qr4k', 'argo', 'Succeeded', 'default', '{ "key2": 1, "key2":  2, "key1": 1}');
select workflow from argo_archived_workflows where uid='0b3ec561-bc78-400c-9772-ce11c69b3d65';

Run make start PROFILE=mysql
Run make mysql-cli < test.sql . Output:$ make mysql-cli < test.sql
GIT_COMMIT=c9f0376594aed5ae0ebc53d3bf2970300406860f GIT_BRANCH=feat-cli-db-generator2 GIT_TAG=untagged GIT_TREE_STATE=dirty RELEASE_TAG=false DEV_BRANCH=true VERSION=latest
KUBECTX=k3d-k3s-default DOCKER_DESKTOP=false K3D=true DOCKER_PUSH=false TARGET_PLATFORM=linux/arm64
RUN_MODE=local PROFILE=minimal AUTH_MODE=hybrid SECURE=false  STATIC_FILES=false ALWAYS_OFFLOAD_NODE_STATUS=false UPPERIO_DB_DEBUG=0 LOG_LEVEL=debug NAMESPACED=true
kubectl exec -ti `kubectl get pod -l app=mysql -o name|cut -c 5-` -- mysql -u mysql -ppassword argo
Unable to use a TTY - input is not a terminal or the right kind of file
mysql: [Warning] Using a password on the command line interface can be insecure.
workflow
{"key1": 1, "key2": 2}

Run make start PROFILE=postgres
Run make postgres-cli < test.sql once on the main branch and once with this migration applied.
Output on main:$ make postgres-cli < test.sql
GIT_COMMIT=c9f0376594aed5ae0ebc53d3bf2970300406860f GIT_BRANCH=feat-cli-db-generator2 GIT_TAG=untagged GIT_TREE_STATE=dirty RELEASE_TAG=false DEV_BRANCH=true VERSION=latest
KUBECTX=k3d-k3s-default DOCKER_DESKTOP=false K3D=true DOCKER_PUSH=false TARGET_PLATFORM=linux/arm64
RUN_MODE=local PROFILE=minimal AUTH_MODE=hybrid SECURE=false  STATIC_FILES=false ALWAYS_OFFLOAD_NODE_STATUS=false UPPERIO_DB_DEBUG=0 LOG_LEVEL=debug NAMESPACED=true
kubectl exec -ti `kubectl get pod -l app=postgres -o name|cut -c 5-` -- psql -U postgres
Unable to use a TTY - input is not a terminal or the right kind of file
INSERT 0 1
              workflow
-------------------------------------
 { "key2": 1, "key2":  2, "key1": 1}
(1 row)

Output with migration applied:$ make postgres-cli < test.sql
GIT_COMMIT=c9f0376594aed5ae0ebc53d3bf2970300406860f GIT_BRANCH=feat-cli-db-generator2 GIT_TAG=untagged GIT_TREE_STATE=dirty RELEASE_TAG=false DEV_BRANCH=true VERSION=latest
KUBECTX=k3d-k3s-default DOCKER_DESKTOP=false K3D=true DOCKER_PUSH=false TARGET_PLATFORM=linux/arm64
RUN_MODE=local PROFILE=minimal AUTH_MODE=hybrid SECURE=false  STATIC_FILES=false ALWAYS_OFFLOAD_NODE_STATUS=false UPPERIO_DB_DEBUG=0 LOG_LEVEL=debug NAMESPACED=true
kubectl exec -ti `kubectl get pod -l app=postgres -o name|cut -c 5-` -- psql -U postgres
Unable to use a TTY - input is not a terminal or the right kind of file
INSERT 0 1
        workflow
------------------------
 {"key1": 1, "key2": 2}
(1 row)

Ok, so if we're going to get this in for 3.6 it should have some notes as to its effect in docs/upgrading.md, and how to do it prior to upgrade.

and how to do it prior to upgrade.

Migrations happen automatically during upgrades, so it only needs a note that it will happen, not how to do it.

and how to do it prior to upgrade.

Migrations happen automatically during upgrades, so it only needs a note that it will happen, not how to do it.

Ah, sorry, that was unclear. I was meaning referring to this in the original issue description:

The good news is this is fully backwards-compatible, and users can opt to manually run alter table argo_archived_workflows alter column workflow set data type jsonb using workflow::jsonb in their DB before upgrading. We'll probably want to add a prominent warning about this.

@Joibel Sorry for the delay. I was researching ways that users with a very large number of workflows could do this migration with minimal downtime. I entered https://github.com/argoproj/argo-workflows/pull/13816 with my findings.


2022-07-11 - KDB Project Call Meeting Minutes

Date
Monday, July 11, 2022 - 9:30am ET / 2:30pm UK
// Second Monday of every month
Attendees

Fullname
Affiliation
GitHub Username

Meeting notices

FINOS Project leads are responsible for observing the FINOS guidelines for running project meetings. Project maintainers can find additional resources in the FINOS Maintainers Cheatsheet.

All participants in FINOS project meetings are subject to the LF Antitrust Policy, the FINOS Community Code of Conduct and all other FINOS policies.

FINOS meetings involve participation by industry competitors, and it is the intention of FINOS and the Linux Foundation to conduct all of its activities in accordance with applicable antitrust and competition laws. It is therefore extremely important that attendees adhere to meeting agendas, and be aware of, and not participate in, any activities that are prohibited under applicable US state, federal or foreign antitrust and competition laws. Please contact<EMAIL_ADDRESS>with any questions.

FINOS project meetings may be recorded for use solely by the FINOS team for administration purposes. In very limited instances, and with explicit approval, recordings may be made more widely available.

Agenda

(5 min) Convene, roll call, welcome new people.
[ ] Display FINOS Antitrust Policy summary slide
[ ] Review Meeting Notices (see above)
(5 min) Approve previous meeting minutes.
(5 min) Updates on kdb Studio / qPython / other projects?
(5 min) Update on searching code on forked repos.

https://github.com/finos/kdb/issues/76
GitHub Code Search improvements will include support.

https://github.blog/2021-12-08-improving-github-code-search/

(5 min) Update on WhiteSsource integration?

https://github.com/finos/kdb-studio/pull/126
https://github.com/finos/qPython/issues/8

(15 min) Review of Enterprise kdb+ vs. Small Team kdb+. and HDB Well-Formedness to see if that helps generate ideas for:

Additional best practices docs
Code collaborations

(10 min) Discuss https://github.com/finos/kdb/tree/main/q offerings.

Suggested improvements to those offerings?
Any code you have that you'd like to contribute?

(5 min) AOB, Q&A, Adjourn.

Decisions Made

Decision 1
Decision 2
...

Action Items

Action 1
Action 2
...

WebEx info
Webex

https://finos.webex.com/finos/j.php?MTID=me61719ca3c3295a7cc76762d78392bc3

Dial-in and Project Materials

US: +1-415-655-0003 US Toll
UK: +44-20319-88141 UK Toll
Access code: 2704 417 0390
GitHub Repos:

https://github.com/finos/kdb
https://github.com/finos/kdb-studio

Wiki Page:

https://finosfoundation.atlassian.net/wiki/spaces/DT/pages/329383945/kdb+Project

KDB Metrics on FINOS

https://metrics.finos.org/goto/033b15c80f0d08cab70f40f4358932f6

Hello 👋🏻

Hello

hi

hi

hi :fox_face:

Howdy!

WhiteSource config docs for @alvin-c-shih - https://community.finos.org/docs/development-infrastructure/code-validation/whitesource


feat(spanner): add support for LCI

This PR adds support for low cost instances. By using the new processing units option instead of nodes, users are able to create and manage instances that uses less resources than a single node.
Note: 1000 processing units = 1 node

I think the underlying protos have been published for this. I assume this PR hasn't been rebased since the generated code is not yet merged into the repo?

@dwsupplee We get the following error in php5.5:
PHP Fatal error:  Call to undefined method Google\Cloud\Spanner\Tests\Unit\InstanceTest::expectException() in /tmpfs/src/github/google-cloud-php/Spanner/tests/Unit/InstanceTest.php on line 298

It seems that $this->expectException() is not supported in php5.5. Do you know how to deal with this issue?

I think I found a solution which may solve this issue:
https://github.com/googleapis/google-cloud-php/blob/8611b2337883fb3693d5295d5364059f9f223c77/Storage/tests/Unit/SigningHelperTest.php#L499-L505

@dwsupplee Could you please take a review for this PR? Thanks.

@dwsupplee Please take another look!


Extra padding on top of table view

I have a table view in a standard UIViewController.  The ViewController is in a UINavigationController with a constraint pinning to the top layout guide.  When I do the pull to refresh it looks like it's adding top space for the navigation bar.  So everything is offset.  After a refresh it keeps that extra inset at the top.

This is still there.  I can hide the tableview behind the navigation and adjust the content inset to fix it.

Change Line 69 and 70 in UIScroll+UzysCircularProgressPullToRefresh.m file to as follows:
view.portraitTopInset = 0.0;
view.originalTopInset = 0.0;
It will work just fine.
Happy Coding... :)

Thanks.  Can you add a setter for these properties so I don't have to edit source code that will break on update?

please use method "- (void)addTopInsetInPortrait:(CGFloat)pInset TopInsetInLandscape:(CGFloat)lInset; "


Available balance is not checked against outgoing pending STX transfers and fees

Steps to reproduce:

Click send in the wallet
Select Stacks
Click send max
transmit
Click send in the wallet
Select Stacks (even though there is 0 balance)
Click send max
transmit
See error, the transaction is transmitted without any error getting thrown.
A negative balance is shown here, but not in the lower right corner on the "send" flow.

https://github.com/leather-wallet/extension/assets/33360391/b6de301f-0065-4e50-a2c8-8a682b31389e

An additional issue, related, when increasing the fee available balance after adjusting for pending tx (outgoing STX and fees) is also not checked. I can click transmit

even though the address doesn't have the balance to cover this. After clicking transmit, it turns out it isn't actually transmitted: I see an error thrown "Not enough funds" and:

But I expect the user to be shown a warning rather than letting them transmit the impossible transaction.
Expected
I expect the user to be prevented from creating a new transaction and shown guidance on why it is prevented. Not enough funds to cover transfer + fee.

@alter-eggo any idea if this is a regression caused by other recent balance-related work?

@markmhendrickson might be caused by recent refactoring of stacks related code, need to investigate