zarr.errors.ContainsGroupError suddenly showing up in different workflows

I'm running a few tests in parallel at FMI and suddenly, I keep getting zarr.errors.ContainsGroupError at different places of workflows. Normally, we'd get this error if we didn't delete an old output file. I made sure that it's in fresh files though.
Very weirdly, this error occurs for different parts. Currently at FMI, I can reproducibly get the 01 example to fail at the parsing step with an error like this:
anding/fractal/fractal-envs/.fractal/fractal-tasks-core0.6.5/venv/lib/python3.8/site-packages/anndata/_io/utils.py", line 214, in func_wrapper
    return func(elem, key, val, *args, **kwargs)
  File "/ENV/FOLDER/fractal-tasks-core0.6.5/venv/lib/python3.8/site-packages/anndata/_io/specs/registry.py", line 175, in write_elem
    _REGISTRY.get_writer(dest_type, t, modifiers)(f, k, elem, *args, **kwargs)
  File "/ENV/FOLDER/fractal-tasks-core0.6.5/venv/lib/python3.8/site-packages/anndata/_io/specs/registry.py", line 24, in wrapper
    result = func(g, k, *args, **kwargs)
  File "/ENV/FOLDER/fractal-tasks-core0.6.5/venv/lib/python3.8/site-packages/anndata/_io/specs/methods.py", line 279, in write_mapping
    g = f.create_group(k)
  File "/ENV/FOLDER/fractal-tasks-core0.6.5/venv/lib/python3.8/site-packages/zarr/hierarchy.py", line 853, in create_group
    return self._write_op(self._create_group_nosync, name, overwrite=overwrite)
  File "/ENV/FOLDER/fractal-tasks-core0.6.5/venv/lib/python3.8/site-packages/zarr/hierarchy.py", line 827, in _write_op
    return f(*args, **kwargs)
  File "/ENV/FOLDER/fractal-tasks-core0.6.5/venv/lib/python3.8/site-packages/zarr/hierarchy.py", line 859, in _create_group_nosync
    init_group(self._store, path=path, chunk_store=self._chunk_store,
  File "/ENV/FOLDER/fractal-tasks-core0.6.5/venv/lib/python3.8/site-packages/zarr/storage.py", line 633, in init_group
    _init_group_metadata(store=store, overwrite=overwrite, path=path,
  File "/ENV/FOLDER/fractal-tasks-core0.6.5/venv/lib/python3.8/site-packages/zarr/storage.py", line 680, in _init_group_metadata
    raise ContainsGroupError(path)
zarr.errors.ContainsGroupError: path 'B/03/0/tables/well_ROI_table/varm' contains a group

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/ENV/FOLDER/fractal-tasks-core0.6.5/venv/lib/python3.8/site-packages/fractal_tasks_core/create_ome_zarr.py", line 396, in <module>
    run_fractal_task(
  File "/ENV/FOLDER/fractal-tasks-core0.6.5/venv/lib/python3.8/site-packages/fractal_tasks_core/_utils.py", line 104, in run_fractal_task
    metadata_update = task_function(**task_args.dict())
  File "/ENV/FOLDER/fractal-tasks-core0.6.5/venv/lib/python3.8/site-packages/fractal_tasks_core/create_ome_zarr.py", line 361, in create_ome_zarr
    write_elem(group_tables, "well_ROI_table", well_ROIs_table)
  File "/ENV/FOLDER/fractal-tasks-core0.6.5/venv/lib/python3.8/site-packages/anndata/_io/utils.py", line 214, in func_wrapper
    return func(elem, key, val, *args, **kwargs)
  File "/ENV/FOLDER/fractal-tasks-core0.6.5/venv/lib/python3.8/site-packages/anndata/_io/specs/registry.py", line 175, in write_elem
    _REGISTRY.get_writer(dest_type, t, modifiers)(f, k, elem, *args, **kwargs)
  File "/ENV/FOLDER/fractal-tasks-core0.6.5/venv/lib/python3.8/site-packages/anndata/_io/specs/registry.py", line 24, in wrapper
    result = func(g, k, *args, **kwargs)
  File "/ENV/FOLDER/fractal-tasks-core0.6.5/venv/lib/python3.8/site-packages/anndata/_io/specs/methods.py", line 221, in write_anndata
    write_elem(g, "varm", dict(adata.varm), dataset_kwargs=dataset_kwargs)
  File "/ENV/FOLDER/fractal-tasks-core0.6.5/venv/lib/python3.8/site-packages/anndata/_io/utils.py", line 220, in func_wrapper
    raise type(e)(
zarr.errors.ContainsGroupError: path "path 'B/03/0/tables/well_ROI_table/varm' contains a group\n\nAbove error raised while writing key 'varm' of <class 'zarr.hierarchy.Group'> to <zarr.storage.DirectoryStore object at 0x7fc539503df0>" contains a group

Sometimes this is varm part of the table, sometime this is var, sometimes obs.
And other workflows failed when building the cellpose pyramid with the same error. I don't currently understand what's happening here. Will try to reproduce this on the UZH side with example 01 next.

I can also reproduce this error on the UZH side with example 01, here it also fails during cellpose pyramid building:
2022-12-15 15:29:40,611; INFO; [B_03_0] End cellpose_segmentation task for /net/nfs4/pelkmanslab-fileserver-jluethi/data/homes/jluethi/v1_fractal/fractal-demos/examples-v1/01_cardio_tiny_dataset/tmp_workflow_test-4/output/20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr/B/03/0/, now building pyramids.
Traceback (most recent call last):
  File "/net/nfs4/pelkmanslab-fileserver-common/data/homes/fractal/joel/fractal_v1/fractal-demos/examples-v1/server/FRACTAL_TASKS_DIR/.fractal/fractal-tasks-core0.6.5/venv/lib/python3.8/site-packages/fractal_tasks_core/cellpose_segmentation.py", line 533, in <module>
    run_fractal_task(
  File "/data/homes/fractal/joel/fractal_v1/fractal-demos/examples-v1/server/FRACTAL_TASKS_DIR/.fractal/fractal-tasks-core0.6.5/venv/lib/python3.8/site-packages/fractal_tasks_core/_utils.py", line 104, in run_fractal_task
    metadata_update = task_function(**task_args.dict())
  File "/net/nfs4/pelkmanslab-fileserver-common/data/homes/fractal/joel/fractal_v1/fractal-demos/examples-v1/server/FRACTAL_TASKS_DIR/.fractal/fractal-tasks-core0.6.5/venv/lib/python3.8/site-packages/fractal_tasks_core/cellpose_segmentation.py", line 476, in cellpose_segmentation
    build_pyramid(
  File "/data/homes/fractal/joel/fractal_v1/fractal-demos/examples-v1/server/FRACTAL_TASKS_DIR/.fractal/fractal-tasks-core0.6.5/venv/lib/python3.8/site-packages/fractal_tasks_core/lib_pyramid_creation.py", line 95, in build_pyramid
    previous_level = newlevel_rechunked.to_zarr(
  File "/data/homes/fractal/joel/fractal_v1/fractal-demos/examples-v1/server/FRACTAL_TASKS_DIR/.fractal/fractal-tasks-core0.6.5/venv/lib/python3.8/site-packages/dask/array/core.py", line 2905, in to_zarr
    return to_zarr(self, *args, **kwargs)
  File "/data/homes/fractal/joel/fractal_v1/fractal-demos/examples-v1/server/FRACTAL_TASKS_DIR/.fractal/fractal-tasks-core0.6.5/venv/lib/python3.8/site-packages/dask/array/core.py", line 3663, in to_zarr
    z = zarr.create(
  File "/data/homes/fractal/joel/fractal_v1/fractal-demos/examples-v1/server/FRACTAL_TASKS_DIR/.fractal/fractal-tasks-core0.6.5/venv/lib/python3.8/site-packages/zarr/creation.py", line 169, in create
    init_array(store, shape=shape, chunks=chunks, dtype=dtype, compressor=compressor,
  File "/data/homes/fractal/joel/fractal_v1/fractal-demos/examples-v1/server/FRACTAL_TASKS_DIR/.fractal/fractal-tasks-core0.6.5/venv/lib/python3.8/site-packages/zarr/storage.py", line 426, in init_array
    _init_array_metadata(store, shape=shape, chunks=chunks, dtype=dtype,
  File "/data/homes/fractal/joel/fractal_v1/fractal-demos/examples-v1/server/FRACTAL_TASKS_DIR/.fractal/fractal-tasks-core0.6.5/venv/lib/python3.8/site-packages/zarr/storage.py", line 485, in _init_array_metadata
    raise ContainsArrayError(path)
zarr.errors.ContainsArrayError: path '1' contains an array

My first suspicion was an update in e.g. the Zarr library or the AnnData library. But neither have been updated recently. There were lots of changes to the poetry.lock today, but those aren't in the released version yet. Any ideas @tcompa ?
This is currently blocking all our tests and makes workflows fail at weird places. Haven't observed this before today though. I suspect it came with the 0.6.5 release somehow? Did something change in the dependencies that could be causing this?

For reference, this was with server 1.0.0b5 and client 1.00a6. I'm testing with server 1.0.0b6 & b4 now to see if it makes a difference

To make things even more confusing: While at FMI, I had my workflows result in a failed status with these errors in the logs, at UZH, the workflows actually appear to complete successfully. But the logs still contain those messages.
Thus, while those error tracebacks are worrying, maybe they are not the actual thing that made the workflow fail on the FMI setup. But we should certainly still figure out what causes them and fix that

I take it back again. It also fails at UZH sometimes. I don't know why it runs through sometimes, but fails other times. I ran it another 3 times and all 3 runs of the 01 test failed. Two at the cellpose stage, one in the initial parsing

For the record, example 01 in fractal-demos runs through locally (on my machine), when using the process backend. This is with versions server 1.0.0b6, client 1.0.0a6, tasks 0.6.5 (all on python 3.8).
Other (unrelated) info:

In tasks 0.6.5 we switched from napari-workflows 0.2.3 to 0.2.8. This causes no obvious harm, but it can be checked explicitly.
The tasks 0.6.4 release seems an innocent one (mostly creating the tools submodule).
Some changes for the server from 1.0.0b2 to 1.0.0b4, for example, include:

Removing parsl
A bunch of related changes in poetry.lock, which don't look like they should really matter (to be verified)
The new job endpoint (irrelevant, I'd guess)
The capture of task errors
Refactor of env-variable names (this would be a typical gotcha, but I don't see an obvious connection with the current issue)
Customization of names for tasks input/output/error/slurm files

Nothing obvious comes to mind as a possible reason for the current issue - will dig further.

For the record, example 01 in fractal-demos runs through locally (on my machine), when using the process backend. This is with versions server 1.0.0b6, client 1.0.0a6, tasks 0.6.5 (all on python 3.8).

Does it run through consistently and without showing those errors in the log? For me, it runs through sometimes at the moment, but I always see some errors in the logs. And often, it fails either during the initial parsing or cellpose.

I've been testing it with different server & tasks versions. Server 1.0.0b4, 1.0.0b5 & 1.0.0b6 with the 0.6.5 tasks all seem to have the same issues.
But the 0.6.4 fractal-tasks-core version does not appear to have this issue. I ran example 01 4x and it always completed successfully. Plus, it does not show any of those errors above in the logs

Does it run through consistently and without showing those errors in the log?
Yes. Ran it once, reinstalled everything, restarted the server, ran it twice. No errors in the logs, only warnings.

For me, it runs through sometimes at the moment, but I always see some errors in the logs. And often, it fails either during the initial parsing or cellpose.

This difference on the process backend is unexpected.
Same exact package versions?
Which python version?
Fresh install in new envs?

Same exact package versions?
I've always run with client 1.0.0a6. Server side tested b4 to b6. Tasks always used 0.6.5, except in the last tests.

Which python version?
Always Python 3.8.13

Fresh install in new envs?
Yes, always fresh installs using the install scripts

In tasks 0.6.5 we switched from napari-workflows 0.2.3 to 0.2.8. This causes no obvious harm, but it can be checked explicitly.

I guess that would be the main culprit to test now, since it appears like a 0.6.5 issue (but not always and potentially only on slurm? Would be super weird)

Another (currently not understood) observation.
The line
            group_well = group_plate.create_group(f"{row}/{column}/")

in the create-ome-zarr task is often related to this issue.
If we add a raise ValueError("GOT HERE") right after that line, we could observe logs like the one below. What is unclear is that the ValueError is correctly handled, and then there is an additional (corrupted) buffer of logs that is printed.
/data/homes/fractal/tommaso/fractal-demos/examples-v1/server/FRACTAL_TASKS_DIR/.fractal/fractal_tasks_core0.6.5/venv/lib/python3.8/site-packages/anndata/experimental/pytorch/_annloader.py:18: UserWarning: Сould not load pytorch.
  warnings.warn("Сould not load pytorch.")

2022-12-16 09:33:21,596; INFO; START create_ome_zarr task
2022-12-16 09:33:21,597; INFO; Creating 20200812-CardiomyocyteDifferentiation14-Cycle1.zarr
2022-12-16 09:33:21,649; INFO; removing FOV overlaps for well='B03'
/data/homes/fractal/tommaso/fractal-demos/examples-v1/server/FRACTAL_TASKS_DIR/.fractal/fractal_tasks_core0.6.5/venv/lib/python3.8/site-packages/fractal_tasks_core/lib_remove_FOV_overlaps.py:309: FutureWarning: In a future version, `df.iloc[:, i] = newvals` will attempt to set the values inplace instead of always setting a new array. To retain the old behavior, use either `df[df.columns[i]] = newvals` or, if columns are non-unique, `df.isetitem(i, newvals)`
  df.loc[well, :] = df_well.values

2022-12-16 09:33:21,664; ERROR; GOT HERE
Traceback (most recent call last):
  File "/net/nfs4/pelkmanslab-fileserver-common/data/homes/fractal/tommaso/fractal-demos/examples-v1/server/FRACTAL_TASKS_DIR/.fractal/fractal_tasks_core0.6.5/venv/lib/python3.8/site-packages/fractal_tasks_core/create_ome_zarr.py", line 409, in <module>
    run_fractal_task(
  File "/data/homes/fractal/tommaso/fractal-demos/examples-v1/server/FRACTAL_TASKS_DIR/.fractal/fractal_tasks_core0.6.5/venv/lib/python3.8/site-packages/fractal_tasks_core/_utils.py", line 110, in run_fractal_task
    raise e
  File "/data/homes/fractal/tommaso/fractal-demos/examples-v1/server/FRACTAL_TASKS_DIR/.fractal/fractal_tasks_core0.6.5/venv/lib/python3.8/site-packages/fractal_tasks_core/_utils.py", line 107, in run_fractal_task
    metadata_update = task_function(**task_args.dict())
  File "/net/nfs4/pelkmanslab-fileserver-common/data/homes/fractal/tommaso/fractal-demos/examples-v1/server/FRACTAL_TASKS_DIR/.fractal/fractal_tasks_core0.6.5/venv/lib/python3.8/site-packages/fractal_tasks_core/create_ome_zarr.py", line 296, in create_ome_zarr
    raise ValueError("GOT HERE")
ValueError: GOT HERE

e_group(f"{row}/{column}/")
  File "/data/homes/fractal/tommaso/fractal-demos/examples-v1/server/FRACTAL_TASKS_DIR/.fractal/fractal_tasks_core0.6.5/venv/lib/python3.8/site-packages/zarr/hierarchy.py", line 853, in create_group
    return self._write_op(self._create_group_nosync, name, overwrite=overwrite)
  File "/data/homes/fractal/tommaso/fractal-demos/examples-v1/server/FRACTAL_TASKS_DIR/.fractal/fractal_tasks_core0.6.5/venv/lib/python3.8/site-packages/zarr/hierarchy.py", line 827, in _write_op
    return f(*args, **kwargs)
  File "/data/homes/fractal/tommaso/fractal-demos/examples-v1/server/FRACTAL_TASKS_DIR/.fractal/fractal_tasks_core0.6.5/venv/lib/python3.8/site-packages/zarr/hierarchy.py", line 859, in _create_group_nosync
    init_group(self._store, path=path, chunk_store=self._chunk_store,
  File "/data/homes/fractal/tommaso/fractal-demos/examples-v1/server/FRACTAL_TASKS_DIR/.fractal/fractal_tasks_core0.6.5/venv/lib/python3.8/site-packages/zarr/storage.py", line 633, in init_group
    _init_group_metadata(store=store, overwrite=overwrite, path=path,
  File "/data/homes/fractal/tommaso/fractal-demos/examples-v1/server/FRACTAL_TASKS_DIR/.fractal/fractal_tasks_core0.6.5/venv/lib/python3.8/site-packages/zarr/storage.py", line 680, in _init_group_metadata
    raise ContainsGroupError(path)
zarr.errors.ContainsGroupError: path 'B/03' contains a group

For the record, I thought https://github.com/fractal-analytics-platform/fractal-server/issues/361 was the reason for the corrupted log files, but my local fix was not sufficient to systematically get clean log files (that is, "without broken words").
Will keep digging.

Yet another partial observation (many more will keep flowing here, I guess)
I confirm that putting a return statement (something like return {"some": "metadata"}) in the create-ome-zarr task, right before the block below leads to "successful" execution (i.e. the 0.err logfile has nothing unexpected)
        for row, column in well_rows_columns:
            group_well = group_plate.create_group(f"{row}/{column}/")

Also:
I removed the two anndata-related lines (write_elem(group_tables, "FOV_ROI_table", FOV_ROIs_table) and write_elem(group_tables, "well_ROI_table", well_ROIs_table)) and the corresponding import, to make sure that the current issue is not related to anndata.
I confirm the issue is still present, thus pointing to a twofold issue:

"Something" goes wrong in the group_well = group_plate.create_group(f"{row}/{column}/") statement.
"Something" goes wrong because the exception is not raised correctly, and the err file is corrupted.

Another hypothesis we are testing: cloudpickle and ExceptionProxy may be failing with non-standard python exceptions (e.g. the zarr one).

Another hypothesis we are testing: cloudpickle and ExceptionProxy may be failing with non-standard python exceptions (e.g. the zarr one).

We can confirm the issue is somewhere here.
Using server 1.0.0b3 (from github release) does not have the current issue.


Developed Create sub admin and Sub admin profile pages

🛠 Fixes Issue
Closes #172

👨💻 Changes proposed

✔ Check List (Check all the applicable boxes)

[x] My code follows the code style of this project.
[x] This PR does not contain plagiarized content.
[x] The title of my pull request is a short description of the requested changes.

📄 Note to reviewers

📷 Screenshots

Good to merge


Segmentation fault running master 3.01 with default config

Describe the bug
Running weed master with the configuration from weed scaffold causes a segmentation fault.
Running weed master without a configuration file still works.
/tmp$ mkdir defaultmaster
/tmp$ cd defaultmaster
/tmp/defaultmaster$ weed scaffold -config=master -output=.
/tmp/defaultmaster$ weed master
I0502 11:06:24 00127 file_util.go:23] Folder /tmp Permission: -rwxrwxrwx
I0502 11:06:24 00127 master.go:255] current: <IP_ADDRESS>:9333 peers:
I0502 11:06:24 00127 master_server.go:131] Volume Size Limit is 30000 MB
I0502 11:06:24 00127 master_server.go:257] adminScripts:   lock
  ec.encode -fullPercent=95 -quietFor=1h
  ec.rebuild -force
  ec.balance -force
  volume.deleteEmpty -quietFor=24h -force
  volume.balance -force
  volume.fix.replication
  s3.clean.uploads -timeAgo=24h
  unlock
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x18ea923]

goroutine 1 [running]:
github.com/chrislusf/seaweedfs/weed/shell.NewCommandEnv(0xc000668600)
        /github/workspace/weed/shell/commands.go:50 +0xa3
github.com/chrislusf/seaweedfs/weed/server.(*MasterServer).startAdminScripts(0xc000632360)
        /github/workspace/weed/server/master_server.go:276 +0x3f1
github.com/chrislusf/seaweedfs/weed/server.NewMasterServer(0xc00093fdb0?, 0xc00081c100, 0xc000799cb0)
        /github/workspace/weed/server/master_server.go:165 +0xfa5
github.com/chrislusf/seaweedfs/weed/command.startMaster({0xc00030e058, 0xc00030e070, 0xc00009d230, 0xc00009d250, 0xc00009d260, 0xc00009d280, 0xc00030e158, 0xc00030e16d, 0xc00009d290, 0xc00030e178, ...}, ...)
        /github/workspace/weed/command/master.go:145 +0x237
github.com/chrislusf/seaweedfs/weed/command.runMaster(0x387ed38?, {0xc00013a010?, 0x0?, 0x0?})
        /github/workspace/weed/command/master.go:121 +0x290
main.main()
        /github/workspace/weed/weed.go:81 +0x383

System Setup

Ubuntu 20.04 amd64, where seaweedfs 3.00 worked fine
version 30GB 3.01 73961e24d8b311b938dfb692f93e172866813487 linux amd64

Expected behavior
Default configuration should work (and invalid config probably shouldn't segfault.
Additional context
The config file, `master.toml`
# Put this file to one of the location, with descending priority
#    ./master.toml
#    $HOME/.seaweedfs/master.toml
#    /etc/seaweedfs/master.toml
# this file is read by master

[master.maintenance]
# periodically run these scripts are the same as running them from 'weed shell'
scripts = """
  lock
  ec.encode -fullPercent=95 -quietFor=1h
  ec.rebuild -force
  ec.balance -force
  volume.deleteEmpty -quietFor=24h -force
  volume.balance -force
  volume.fix.replication
  s3.clean.uploads -timeAgo=24h
  unlock
"""
sleep_minutes = 17          # sleep minutes between each script execution

[master.sequencer]
type = "raft"     # Choose [raft|snowflake] type for storing the file id sequence
# when sequencer.type = snowflake, the snowflake id must be different from other masters
sequencer_snowflake_id = 0     # any number between 1~1023

# configurations for tiered cloud storage
# old volumes are transparently moved to cloud for cost efficiency
[storage.backend]
[storage.backend.s3.default]
enabled = false
aws_access_key_id = ""         # if empty, loads from the shared credentials file (~/.aws/credentials).
aws_secret_access_key = ""     # if empty, loads from the shared credentials file (~/.aws/credentials).
region = "us-east-2"
bucket = "your_bucket_name"    # an existing bucket
endpoint = ""
storage_class = "STANDARD_IA"

# create this number of logical volumes if no more writable volumes
# count_x means how many copies of data.
# e.g.:
#   000 has only one copy, copy_1
#   010 and 001 has two copies, copy_2
#   011 has only 3 copies, copy_3
[master.volume_growth]
copy_1 = 7                # create 1 x 7 = 7 actual volumes
copy_2 = 6                # create 2 x 6 = 12 actual volumes
copy_3 = 3                # create 3 x 3 = 9 actual volumes
copy_other = 1            # create n x 1 = n actual volumes

# configuration flags for replication
[master.replication]
# any replication counts should be considered minimums. If you specify 010 and
# have 3 different racks, that's still considered writable. Writes will still
# try to replicate to all available volumes. You should only use this option
# if you are doing your own replication or periodic sync of volumes.
treat_replication_as_minimums = false

Removing the [master.maintenance] scripts= section allows the master to start.

Confirmed, I see this same behaviour and removing [master.maintenance] scripts= allows it to start.


Publish placeholders for static pages

Currently the static pages are not present in the new publishing platform and
do not have content ids. This ensures they are present as placeholder_[format], and mainstream_browses_pages are included.
There will be some follow up work to add other static pages that are not currently in rummager.
https://trello.com/c/PbHiHeBa/576-add-how-government-works-and-get-involved-to-publishing-api
cc @jackscotti

I think we shouldn't send links every time this task is run (which is on deploy if I'm not mistaking). It would wipe out any links set by content-tagger or others.
Perhaps we could do the initial patch_links in a data migration or do it manually.

Why have you chosen publication and policy as formats?

Good point about the links, I'll extract that to a rake task.
Format came from the content api:
https://www.gov.uk/api/government/how-government-works.json
https://www.gov.uk/api/government/get-involved.json
Everything else we pass I entered manually based on the schema so please flag if anything looks wrong.

@tijmenb We've separated the links call out into a separate task as discussed (actually a data migration because we don't expect to run it again - ultimately we'll be able to use content tagger instead).
One thing that came up when discussing with @benhyland is whether the existing rake task for this is called automatically on deploy or not. We couldn't find any evidence of this. Also if it is, do we still need the rummager call at this point (we've been assuming we do)?

I was assuming that too, but from https://trello.com/c/7qw4TRiC it looks like we ran the rake task manually once. Perhaps it was assumed that whenever this data would change, the developer would re-run that task? At the very least it should have been documented in that rake task.

Discussed offline: better to put the definition of the links into the data migration (because it is temporary), and add a comment that says we should run the publish_static_pages rake task after changing the data.

I'm not sure what these pages are, but would it be appropriate to use the special_route format like we do with other non-publishing-api pages?

@danielroseman these are just https://www.gov.uk/government/how-government-works and https://www.gov.uk/government/get-involved
There is also https://www.gov.uk/government/history which we are going to do separately.
I did notice the special_routes task but I'm not sure if it applies to these. What's the intended use case of these? We're using placeholder here just for the linking aspect, as documented here: https://github.com/alphagov/content-store/blob/master/doc/placeholder_item.md

:+1: good to merge when green


Update sendResponse() to process parameters such as status code

https://github.com/snowplow-incubator/snowplow-gtm-server-side-client/blob/0d83bf24eec697eb2f8cf15b777a08b26d173e0d/template.tpl#L171
It might be useful to turn this into an error handler as well, so for example if the container fails to download the sp.js from the SP CDN, then sendResponse() could be called with a 404 status code, which is then set as the response status before the response is sent back.

Closed by https://github.com/snowplow-incubator/snowplow-gtm-server-side-client/pull/3


Slavikz/download drawings list json

@obergrd @erymski  I tried to do it the same way we download BOM.json, but I'm missing a piece somewhere.
There's also a difference from BOM.json, as the drawingsList.json is not dependent on a hash and is stored in attributes (e.g. attributes/Wheel/drawingsList.json).
Any help appreciated.

Wrong target branch, closing.


Duplicate PubSubClient library found

Report malware or abuse
Please fill out the form to report malware or abuse in a package to the PlatformIO team.
The PlatformIO team will review the report and act accordingly. In cases where malware or abuse is confirmed, the package will be removed and the namespace will be blocked.
Sensitive information should be sent to<EMAIL_ADDRESS>included this issue link.
Please note that this report goes to the PlatformIO team for triage, not the package maintainer. To report bugs or vulnerabilities in a package please contact the maintainer directly.
Package information

Package name: drk/PubSubClient
Package version: 2.8
Package link: https://registry.platformio.org/libraries/drk/PubSubClient

Description of vulnerability
Please include any references, screenshots, commits, or code examples that would be useful in reproducing the security issue.
It has a different owner than knolleary/PubSubClient, but the content is the same.

Thanks, removed!


Hyper 3.0.0 deleted settings/configuration (.hyperjs)

[x] I am on the latest Hyper.app version

[x] I have searched the issues of this repo and believe that this is not a duplicate

OS version and name: Windows 10

Hyper.app version: 3.0.0

Link of a Gist with the contents of your .hyper.js:

Relevant information from devtools (CMD+ALT+I on macOS, CTRL+SHIFT+I elsewhere):

The issue is reproducible in vanilla Hyper.app:

Issue
I was on 2.x.x when it prompted me to restart hyper as 3.0.0 was ready.
Once i did, it wiped my .hyperjs configuration.
Is there any way to get it back?

Same here 😞 . Unfortunately, it seems to be gone.
It could be restored if you have backup.

Fortunately I'm on WSL, so I could go into Preferences and change the shell to my bash shell, and copy the contents of my WSL config and paste it over.

Basically the same issue, but it didn't wipe the configuration for me, it's just not using it. My .hyper.js is safe in my home directory.
Weirdly enough, if I execute hyper in hyper, it opens up a window with my configuration loaded.

Weirdly enough, if I execute hyper in hyper, it opens up a window with my configuration loaded.

I copy pasted the config from this window into the config of the original window, closed, re-opened and was good to go.

Basically the same issue, but it didn't wipe the configuration for me, it's just not using it. My .hyper.js is safe in my home directory.
Weirdly enough, if I execute hyper in hyper, it opens up a window with my configuration loaded.

Is it loading the previous version? I noticed that the CLI tool uses the old config rather than the new one.

Basically the same issue, but it didn't wipe the configuration for me, it's just not using it. My .hyper.js is safe in my home directory.
Weirdly enough, if I execute hyper in hyper, it opens up a window with my configuration loaded.

Is it loading the previous version? I noticed that the CLI tool uses the old config rather than the new one.

Yes, the CLI tool is opening up the previous version from what I can tell.

[x]  I am on the latest Hyper.app version
[x]  I have searched the issues of this repo and believe that this is not a duplicate
OS version and name: Windows 10
Hyper.app version: 3.0.0
Link of a Gist with the contents of your .hyper.js:
Relevant information from devtools (CMD+ALT+I on macOS, CTRL+SHIFT+I elsewhere):
The issue is reproducible in vanilla Hyper.app:

Issue
I was on 2.x.x when it prompted me to restart hyper as 3.0.0 was ready.
Once i did, it wiped my .hyperjs configuration.
Is there any way to get it back?

Looks like the new version is pointing to the settings file in %APPDATA%\Hyper.hyper.js instead of the old one %USERPROFILE%.hyper.js.
Just grab your old file from the old location and paste it into the new location to get all your settings back.


Yq breaking changes with v4

We could run into issues with our usage of yq, because yq v4 has breaking changes to v3.
https://github.com/mikefarah/yq#v4-released
Currently the setup action is installing yq v3.32 by default, so we will not run into problems until they change that.
https://github.com/chrisdickinson/setup-yq

We could adjust our documentation for setup-yq to set install version hardcoded to an v3 so we will not get autoupdate of v4.
We could also adjust our helm actions to validate that the major version of yq is matching the version is the version we want to have.
We could also update our helm action to use v4 of yq. But the two points above would still apply ;)


[GLUTEN-3922][CH] Fix incorrect shuffle hash id value when executing modulo

What changes were proposed in this pull request?
Fix incorrect shuffle hash id value when executing modulo. In CH Backend, the data type of the shuffle split num is a UInt32 and the returned type of the hash function is a UInt64, when the returned value of the hash function is more than 2^31 - 1, the modulo value of the hash value and the shuffle split num is different from the one of the vanilla spark.
Close #3922.
(Fixes: #3922)
How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

perf tests:
main + cityHash64:       127， 121   (mean, min)
main + murmur3:          123， 118
pr-3923 + cityHash64:  129，121
pr-3923 + murmur3:     124，118

Run Gluten Clickhouse CI

perf tests:
main + cityHash64: 134， 120 (mean, min)
main + murmur3: 127， 118
pr-3923 + cityHash64: 121，118
pr-3923 + murmur3: 122，122

perf tests:
main + cityHash64: 133， 123 (mean, min)
pr-3923 + murmur3: 121，117

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query
log/native_master_12_07_2023_time.csv
log/native_master_12_06_2023_d6b1e9138_time.csv
difference
percentage

q1
33.97
33.13
-0.839
97.53%

q2
24.95
25.88
0.936
103.75%

q3
37.91
36.12
-1.793
95.27%

q4
37.39
38.73
1.336
103.57%

q5
71.30
72.19
0.881
101.24%

q6
5.38
7.01
1.635
130.39%

q7
85.81
84.85
-0.963
98.88%

q8
85.80
87.08
1.279
101.49%

q9
123.51
127.07
3.562
102.88%

q10
45.87
44.57
-1.293
97.18%

q11
20.29
20.26
-0.032
99.84%

q12
26.73
21.15
-5.579
79.13%

q13
45.73
48.39
2.661
105.82%

q14
16.69
18.86
2.168
112.98%

q15
28.46
27.18
-1.278
95.51%

q16
15.56
15.92
0.370
102.38%

q17
103.37
102.54
-0.832
99.19%

q18
149.01
152.02
3.011
102.02%

q19
12.84
13.90
1.055
108.21%

q20
27.78
27.91
0.128
100.46%

q21
223.73
223.84
0.104
100.05%

q22
13.39
13.24
-0.146
98.91%

total
1235.47
1241.84
6.370
100.52%


Add Test Coverage

I've already got things sorted just need to know how to update the coverage to codecov and then use the badge in the readme

I should also add some golden tests


Items loaded from the SPH

Hi, please is it possible to expand the items loaded from the SPH inverter, for example "EnergyToday" and others? Otherwise it works great, I have SPH 10000 ;-) Thank you.

If you can find out the correct registers for your inverter then you should be able to make a copy of the mapping file already there and add the mappings yourself.

Do you have updated file, petrzma? I have added some items but I need more and I have enough free time for this... :(


Accessing Application.Resources in Application constructor throws COMException

Describe the bug

We are a commercial UI control development company and have been working on a theming system for WinUI.  Part of this theming system is that we need to inject a ResourceDictionary into the Application.Resources.
In WPF, customers almost always access and use our theming system in the application's constructor.  However accessing the Application.Resources getter in WinUI's application constructor throws a COMException.  An exception when trying to do so is confusing and will make our theming system look bad when customers try and use our themes in their own apps, only to find it crashes due to us accessing the Application.Resources within our code.
Sure we can write documentation about NOT executing any theme-related stuff in the application constructor and only in OnLaunched, but you know how much people read documentation.  It seems like this is a definite bug and there should be no problem with accessing Application.Resources in the constructor.
Steps to reproduce the bug

Use this Application constructor in a WinUI app project:
public App() {
	this.InitializeComponent();
	this.Suspending += OnSuspending;

	var resources = this.Resources;  // Triggers COMException
}

That code throws:
System.Runtime.InteropServices.COMException
  HResult=0x8000FFFF
  Message=Catastrophic failure (0x8000FFFF (E_UNEXPECTED))
  Source=WinRT.Runtime
  StackTrace:
   at WinRT.ExceptionHelpers.ThrowExceptionForHR(Int32 hr)
   at ABI.Microsoft.UI.Xaml.IApplication.global::Microsoft.UI.Xaml.IApplication.get_Resources()
   at Microsoft.UI.Xaml.Application.get_Resources()
   at WinUITestApp.App..ctor() in S:\Code\Test\WinUITestApp\WinUITestApp\App.xaml.cs:line 35
   at WinUITestApp.Program.<>c.<Main>b__0_0(ApplicationInitializationCallbackParams p) in S:\Code\Test\WinUITestApp\WinUITestApp\obj\x86\Debug\net5.0-windows10.0.18362.0\App.g.i.cs:line 29
   at ABI.Microsoft.UI.Xaml.ApplicationInitializationCallback.<>c__DisplayClass10_0.<Do_Abi_Invoke>b__0(ApplicationInitializationCallback invoke)
   at WinRT.ComWrappersSupport.MarshalDelegateInvoke[T](IntPtr thisPtr, Action`1 invoke)
   at ABI.Microsoft.UI.Xaml.ApplicationInitializationCallback.Do_Abi_Invoke(IntPtr thisPtr, IntPtr p)

Expected behavior

We should be able to access the Application.Resources in the application's constructor without any issues.
Version Info

NuGet package version:
Microsoft.WinUI 3.0.0-preview4.210210.4

Windows app type:

UWP
Win32

Yes

Windows 10 version
Saw the problem?

Insider Build (xxxxx)

October 2020 Update (19042)

May 2020 Update (19041)
Yes

November 2019 Update (18363)

May 2019 Update (18362)

October 2018 Update (17763)

April 2018 Update (17134)

Fall Creators Update (16299)

Creators Update (15063)

Device form factor
Saw the problem?

Desktop
Yes

Xbox

Surface Hub

IoT

@chrisglein and @MikeHillberg FYI


Feature/user auth

User authentication and management
Changelog
**New**

* login authentication
* roles
* bcrypt passwords
* edit users
* add users
* delete users

**Changed**

* dep upgrades

Resolves: #8
Resolves: #9
DCO 1.1 Signed-off-by: Scott Nath<EMAIL_ADDRESS>

@scottnath Your coverage dropped too much. Please increase your code coverage.

@snugug agree and aware. Can you please go over the pr anyway (as time
allows) so I can tackle en masse mañana?
On May 25, 2016 6:02 PM, "Sam Richard"<EMAIL_ADDRESS>wrote:

@scottnath https://github.com/scottnath Your coverage dropped too much.
Please increase your code coverage.
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/punchcard-cms/punchcard/pull/37#issuecomment-221721516

Lots of comments here. Many apply multiple times, but didn't mark them multiple times. Please address and make sure you don't drop in code coverage.

@Snugug great comments. Only question (for now) is where should I put all that messaging? Make a new object in config, yes? Separate messaging file?

How does this relate to #44


[AS3 Service Discovery]: Need support for AS3 Service Discovery append mode

Do you already have an issue opened with F5 support?
Github Issues are consistently monitored by F5 staff, but should be considered as best effort only and you should not expect to receive the same level of response as provided by F5 Support. Please open an case with F5 if this is a critical issue.
ISSUE TYPE

Feature Idea

AS3 BUILD/ VERSION
3.14
BIGIP VERSION
13.1
SUMMARY
Recently opened github issue #188 where Service discovery doesnt allow 70+ endpoints to be POSTed to service discovery EP in same go.
Service discovery endpoint always requires the entire final list of nodes to be sent to it. So it works in replace mode rather than append mode, where we cant send it a list of node ips to add or remove (append mode) from original list.

Supporting PATCH requests on the service discovery node endpoints is being tracked in our internal product backlog as AUTOTOOL-621.

Found during testing integration with F5 ACI ServiceCenter


Use Arc in SystemCaller

should close #12264

Sorry for the back and forth. I'm not sure if the error is due to a missing import in the Cargo.toml file


Distillation and multi-gpu training with tf.keras

I use https://github.com/keras-team/keras-io/blob/master/examples/vision/knowledge_distillation.py and single gpu everything is right.
When I use https://keras.io/guides/distributed_training/#singlehost-multidevice-synchronous-training and knowledge_distillation, the val_loss will become very big and the acc is very low.
If I only use distributed_training to train, everything is right too.
When I define test_step and train_step in class Distiller(keras.Model), should I use https://tensorflow.google.cn/guide/distributed_training#using_tfdistributestrategy_with_custom_training_loops?

As this is a specific  Keras.io guide I suggest you to open a ticket on https://github.com/keras-team/keras-io to check if they are interested to expand that example.

@bhack Looks like keras was imported from tensorflow (in both the examples). I guess you can use https://tensorflow.google.cn/guide/distributed_training#using_tfdistributestrategy_with_custom_training_loops to define train_step and test_step.
@nikitamaia Can you please take a loot at it. Thanks!

@bhack Looks like keras was imported from tensorflow (in both the examples). I guess you can use https://tensorflow.google.cn/guide/distributed_training#using_tfdistributestrategy_with_custom_training_loops to define train_step and test_step.
@nikitamaia Can you please take a loot at it. Thanks!

Yes the new keras doc hosted there is always related to Keras in TF namespace but It Is a distinct repository

@bhack @jvishnuvardhan Thanks! I will try it soon.


VSP efficiency + retrieving efficiency directly

The modifications to the code solve the issue of Energy calculations for variable speed pumps #71.
Changes to the getenergy function (in hydraulic.c) allow to compute the
correct efficiency of variable speed pumps according to the affinity
laws.
Changes to epanet2.h and epanet.c are for retrieving the efficiency of
pumps through the ENgetlinkvalue function (code 16 for efficiency)

Thanks @AngelaMarchi. Looks fine to me. Will wait for another per of eyes before we merge.

Thank you @eladsal and sorry for the late in doing this. Also, please, let me know if there are problems (I am not sure what the "All checks have failed" means).
Thank you again!

Thanks @AngelaMarchi - I'm looking at this with @jamesuber this morning. In the meantime, would you mind tweaking the code style to match existing code? particularly,
if {
  ...
}
else {
  ...
}

... and using spaces around the assignment of variables, i.e. v = a rather than v=a

also, the failed tests are a reminder for us to fix the tests. sorry for the confusion.

Thanks @AngelaMarchi ! The section of your commit -
		  if (Pump[j].Upat>0) {
			  p = (Htime+Pstart)/Pstep;
			  temp = p % (long) Pattern[Pump[j].Upat].Length;
			  patMult = Pattern[Pump[j].Upat].F[temp];
		  }

is used in lieu of the linkSetting[] when there is a utilization pattern. This seems to duplicate the code in https://github.com/OpenWaterAnalytics/EPANET/blob/master/src/hydraul.c#L627 that actually modifies the linkSetting. So from the perspective of not wanting the same code in two different places, do you know if this code is actually needed - i.e. that your code in getEnergy() can be called prior to the above referenced existing code in demands()? If so, then probably this is another small example where the code base should be designed a bit differently... not sure that's a valid discussion here cause presumably those larger efforts will be focused at epanet-dev (a.k.a. Epanet3). So if it certainly must be in two separate places, then probably just leave it in two separate places.

Thank you, @samhatchett : I have updated  the style (and sorry, I didn't pay attention to it before!)
Thank you also to @jamesuber for letting me know about the LinkSetting: I have updated the code, so only LinkSetting is used now (LinkSetting has the value of the pump pattern at each time step).
Please, let me know if there are problems.

Thanks @AngelaMarchi - this looks good. we'll give this PR about 24 hours more in case others would like to comment.

I think I have been able to change the previous version:

Thank you @eladsal : vb and bas files are updated and the code to retrieve the efficiency is improved
all the previous suggestions from @samhatchett and @jamesuber are incorporated.
Please, let me know if there are other problems and thanks to all for your help!

looks like no further comments. Thanks so much @AngelaMarchi


Forking the lifecycle

Thank you for this project.
I was curious:  the Execute annotation has the effect of forking the lifecycle, which is interfering with my Maven workflow. Hopefully it's just me, but what is the rationale?

Fixed in 1.2.2,


Why doesn't this fail?

Perhaps I'm doing something wrong, but I have code that looks very similar to the examples, and I can't get it to panic or otherwise respond to failpoints in the environment:
The full code is in https://github.com/sourcefrog/fail-repro
main.rs is
use fail::fail_point;

fn main() {
    println!("Has failpoints: {}", fail::has_failpoints());
    println!(
        "FAILPOINTS is {:?}",
        std::env::var("FAILPOINTS").unwrap_or_default()
    );
    fail_point!("main");
    println!("Failpoint passed");
}

When I run this:
$ FAILPOINTS=main=panic cargo +1.61 r --features fail/failpoints
    Updating crates.io index
...
     Running `target/debug/fail-repro`
Has failpoints: true
FAILPOINTS is "main=panic"
Failpoint passed

$ FAILPOINTS=main=print cargo +1.61 r --features fail/failpoints
    Finished dev [unoptimized + debuginfo] target(s) in 0.01s
     Running `target/debug/fail-repro`
Has failpoints: true
FAILPOINTS is "main=print"
Failpoint passed

In case this was broken by a later Cargo change, I tried it on both 1.76 and 1.63 and they both show the same behavior.
This is on x86_64 Linux.

Looking at the code, I think the problem is that the environment variable is only read by the FailScenario constructor, so binaries that just directly have failpoints won't ever fail.
I guess I would have expected that you could just embed fail_point! and then exercise it interactively or from tests that call the binary, by setting $FAILPOINTS: at least, the docs somewhat give this impression.
Maybe the macro could lazily load the environment variable? I got the idea that FailScenario was a convenience for test code.
At least, maybe this could be more clear in the docs. But, perhaps the division of responsibility between them could be reconsidered...


brew cask not supported: should be reflected in search & command-outputs, or implement

brew cask * should return something like "Casks are not (yet) supported for linuxbrew", and searches should only return results relevant to Linuxbrew, and not those relevant to Homebrew.
Alternative: implement casks, create and support parallel binary packages for ...some or many (would this be a hairy part)... linux distributions.
What would be required for that (implementing cask)?
Should it be part of an effort to depend on Homebrew/brew in a more modular, isolated, or configuration-based way?
Binary installs for slow or difficult to compile projects is useful for any user who like the convenience of brew but comes across projects which are distributed in binary format (or, possibly, projects with very tight versioning...like Mac Brew recommendations to use cask for GUI applications). Interest is proven by the existence of the feature in Mac Brew, and the existence of this port.

I think some devs are already working on this upstream: https://github.com/Homebrew/brew/pull/3325

❯❯❯  brew cask install foo
Error: Homebrew Cask is only supported on macOS

Are you seeing something different?
The equivalent project on Linux is Snappy.
https://snapcraft.io
https://www.ubuntu.com/desktop/snappy
Consider installing Linuxbrew in /home/linuxbrew/.linuxbrew/ if possible so that you can use precompiled binary packages (known as bottles) for non-relocatable formula like util-linux.
Another possible workaround for you is brew install --force-bottle util-linux, but no promises.

There are not only snaps, but also flatpak: https://flatpak.org/

And both of those already have package managers, but there is a third format that does not currently have a package manager: AppImage - https://appimage.org/
AppImages are the closest equivalent to .app and .dmg on Linux, so they would be the perfect match for Linuxbrew!
AppImages work on all Linux distributions and do not require root permission to be installed on a per-user basis. In fact, they are fully portable and do not require any kind of installation at all, hence the lack of a package manager. A package manager would still be useful though, simply to automate the process of navigating to the application's upstream website, downloading it, checking the hash, and making it executable, and adding a shim in the user's $PATH - in short all the functionallity that Homebrew Cask provides.
(cc. @probonopd, @TheAssassin)

Sounds like a good fit. You can create a custom to tap to house your favourite AppImage apps.
See https://docs.brew.sh/How-to-Create-and-Maintain-a-Tap
and https://docs.brew.sh/Taps
Let me know if you have any questions!

In case it is helpful to someone, AppImageHub has a feed of projects that have AppImages, but we do not keep track of versions.
https://appimage.github.io/feed.json
Let me know if something else is needed.

I think that if such a project would exist for a year at least, with a stable user base, and at least two maintainers, we would consider making it part of the project. But unless a team steps up and does the thing, there will not be much support from us, as we are busy enough running linuxbrew.


circleci: fix workflow version

Only version 2 is supported at the moment as per
https://circleci.com/docs/2.0/configuration-reference/#version-1
Not to confuse with top-level version which really can be 2.1.

hm. so strange to see failing circle ci jobs. Do you have an idea why is that so?

Now it's this:
Configuration errors: 2 errors occurred:

* Configuration version 2.1 requires the "Enable Pipelines" project setting. Enable pipelines under Project Settings -> Advanced Settings. In order to retrigger pipelines, you must push a new commit.
* Cannot find a job named `test_1_12` to run in the `jobs:` section of your configuration file.
If you expected a workflow to run, check your config contains a top-level key called 'workflows:'

I think enabling pipelines (which I already did) should fix it all. Maybe.

Maybe if I'm to force-push something here (without any real changes) it will try again... Let's check it out.

Yay! https://circleci.com/gh/nspcc-dev/neo-go/2693

The only thing left failing is lint, but that's a separate issue (#213).


[GSoC] Interactive mode should only run in an interactive shell

Tests are currently blocked on the feature/interactive branch because the interactive mode is always on. It should only run when the command is executed in an interactive shell.

This was fixed by #485, yes?


Angular

Hello,
I'm trying to load an excel file on my Angular project and use Lucksheet to render it (like it is working on the demo)
When the file is changed, the ts function is well-called, but nothing happens to the grid...
Can someone help me?

I have added the import like this:
import { LuckyExcel } from '../../../../node_modules/luckyexcel/dist/luckyexcel.umd.js'
But then, in the load excel file handler function, the following line crashes:
LuckyExcel.transformExcelToLucky(files[0], function(exportJson, luckysheetfile){
Because LuckyExcel is undefined.
Can you tell me how to load it properly?

I do not remember who suggested me to use a window service, like:
let nativeWindow = this.windowService.nativeWindow;
nativeWindow.luckysheet.destroy();

Where the service is:
import { Injectable, Inject, PLATFORM_ID } from '@angular/core';
import { isPlatformBrowser } from '@angular/common';

function _window(): any {
  return window;
}

@Injectable({providedIn: 'root'})
export class WindowService {
  constructor(@Inject(PLATFORM_ID) private platformId: object) { }
  get nativeWindow(): any {
    debugger
    if (isPlatformBrowser(this.platformId)) {
      return _window();
    }
  }
}

But I'm still looking for a better solution...
HELP

And if I directly call lucksheet.create(options) I have the error:
````Cannot read properties of undefined (reading `'create')```

I finally find a solution:

I remove the ngOnInit and the ngAfterViewInit
On ngOnInit I only call this.render function
here is the function render:

private render() {
    if (this.hasViewLoaded) {
      return;
    }
    this.hasViewLoaded = true;

    LuckyExcel.transformExcelToLuckyByUrl(
      this.downloadUrl,
      this._document.name + this._document.extension,
      function (exportJson, luckysheetfile) {
        if (exportJson.sheets == null || exportJson.sheets.length == 0) {
          alert("Failed to read the content of the excel file, currently only support xlsx files!");
          return;
        }

        let options = {
          container: "react-excel-id", //react-excel-id
          showinfobar: false,
          showsheetbar: true,
          data: exportJson.sheets,
          enableAddRow: false,
          showtoolbar: false,
          row: 26,
          column: 26,
          allowUpdate: false,
          enableAddBackTop: false,
          lang: "en"
        };
        ReactDOM.render(
          React.createElement(ExcelReactComponent, options),
          document.getElementById('react-excel-id')
        );
      },
      function (err) {
        console.error("Import failed. Is the file a valid xlsx?");
      }
    );
  }

And finally, I replace the Sheet type in the react component by any:

import * as React from 'react';
import { FunctionComponent, useEffect, useRef, useState } from 'react';
import { Workbook } from '@fortune-sheet/react'

export const ExcelReactComponent: FunctionComponent<any> = (props: any) => {
  debugger
  return (
    <Workbook data={props.data} config={props} />
  );
};

But I do not understand how to block the file edition, it looks like the config is not working :(

I described my solution on https://github.com/ruilisi/fortune-sheet/issues/74
This ticket can be closed ;)


Investigate components being initialised more than once

We had a case of a user initialising the components twice by accident. This doesn't warn or error, but creates a broken state (the accordion for example opens then closes on click, and creates two 'Open all' buttons.

My approach I've shared already puts the instance information on the components themselves, but we could also consider an approach similar to web component custom elements.
Where we have a single registry (could be an array), that we update and check:
if (!window.customElements.get('x-component')) {
  window.customElements.define('x-component', ComponentElement)
}

Seen this again from a Prototype Kit user.
Their tabs component was skipping two tabs at once since they accidentally included their application.js bundle twice in the page which meant the component was initialised twice.

As discussed in the JavaScript effort/value session, this is happening because we don't have any guards against initialising a component more than once. This doesn't feel right, but it's also a bit of an unusual case which doesn't come up very often.

Having this functionality may come in useful with our current work relating to localisation and programatic APIs.
For example, a service developer may only want to pass configuration options to a specific instance of a component, but use the default options for all other components. In attempting to do so, they may write something like this:
new window.GOVUKFrontend.Accordion($element, { config }).init()
window.GOVUKFrontend.initAll()

However, unless the developer takes care to remove the data-module attribute from the custom instance (which isn't possible if using our Nunjucks macros), this will initialise the same Accordion twice—resulting in multiple buttons and event handlers being created.
The simplest way to do this is probably by recording which elements have already been initialised and checking against that record when components are initialised again. This could be done by adding a data-* attribute to the element, recording it in the element's dataset (this approach won't work in IE8–10), or maintaining a persistant array of initialised elements.

Discussed this today and decided it's not necessary for MVP internationalisation work, but could be a stretch goal. Otherwise, we might look at this as part of our later JavaScript work anyway.

In a call this week about upcoming changes to the Prototype Kit, it was asked whether it was possible to run Frontend's initAll function multiple times without side-effects, as they were investigating doing so as a way of implementing certain features (I think extensions?)

happened again on support today

Another instance of this happening on support today, this time in an app using Turbolinks.
Turbolinks caches the DOM state of a page when navigating away, restoring that state if the page is navigated back to (in a similar, but not the same, way as bfcache). This was resulting in DOM manipulations made through our JavaScript happening multiple times if the page was navigated away from and back to again.

Another instance of this happening on support today, at: https://github.com/DFE-Digital/npq-prototype/blob/main/app/views/registration-status/accordion.html

was this fixed in recent work?


It doesnt support Latest XCode

Hello Guys as it doesn't support the latest XCode and iOS version, I created another repo that can perform a similar function
https://github.com/hemal08ce094/InstaFeed
PS: It is very basic and needs major improvement as I created it in spare time. Feel free to raise PR if you can improve it for good. :)

Looks great! I'll add a link to it in the README :)

Looks great! I'll add a link to it in the README :)


@oijm17 i will try to reproduce and fix ASAP.

Edit

this command will not output the names to stdout / terminal, but will save the names to a file.

Originally posted by @r0oth3x49 in https://github.com/r0oth3x49/udemy-dl/issues/315#issuecomment-445518630
Thank you very much, now and generate the list, but again I found an inconvenient: What happens is the same as the previous example. of the course. Here is a fragment of the content of the text file that generated the script:

001 Presentation
001 Presentation
002 Objectives and organization of the course.
003 Key points for a better use of the course
004 Definitions and basic terms.
005 Introduction to the module
006 Basic notions
007 Using Mailinator and how to evade controls
008 attack vectors
009 Security devices
010 methodologies
010 methodologies
010 methodologies
010 methodologies
010 methodologies
011 Introduction of the module
011 Introduction of the module
011 Introduction of the module
012 Main Distributions of Ethical Hacking
012 Main Distributions of Ethical Hacking
013 Main vulnerable distributions
013 Main vulnerable distributions
013 Main vulnerable distributions
013 Main vulnerable distributions
013 Main vulnerable distributions
013 Main vulnerable distributions

@oijm17 again i cannot reproduce for me it works fine i might need your account to reproduce or else i will close the issue because you want get the names only and for you now it is serving the purpose.


Add Kayobe Workflows [GitHub]

Add support to SKC to deploy GitHub runners and generate compatible workflows for performing Kayobe Automation tasks.

One last suggestion, do we want this in Yoga or should it be a Zed/Antelope thing?

One last suggestion, do we want this in Yoga or should it be a Zed/Antelope thing?

It will work with zed/antelope should just be able to cherry-pick it. Might also want to define github_kayobe_base_image: quay.io/rockylinux/rockylinux:9 for those two branches.

There are still quite a few unresolved conversations.


Add khiops-env to conda packages

Update of the khiops_env script on Linux/Mac and Windows to be compatible with conda environments (even without CONDA_PREFIX). Conda packages are updated with khiops_env (on all OS). It has an impact on packaging (cmake, NSIS) and github actions.
The important comments are in commit  a98e684f40185467c99cc9b15c3cb1754e52490c
Add a new variable KHIOPS_MPI_HOST_FILE (in Linux only) to set up the MPI command line on multi-machines
Remove process affinity on windows
close #225

Test report for Conda on Ubuntu:

the khiops-core package builds OK [as seen from the CI as well]
the khiops-core installs OK on different Conda (on Ubuntu) environment
which {khiops,khiops_coclustering,khiops_env,MODL_mpich,MODL_coclustering all point to the correct /bin directory inside the Conda environment
khiops_env --env yields (N is the correct number of physical cores; comments have been added below each listed variable; "~" means "almost"):KHIOPS_PATH /path/to/conda/test/environment/bin/MODL_mpich
# OK
KHIOPS_COCLUSTERING_PATH /path/to/conda/test/environment/bin/MODL_Coclustering
# OK
KHIOPS_MPI_COMMAND mpiexec   -n N
# ~OK; should be the full path to mpiexec
KHIOPS_JAVA_PATH /usr/lib/jvm/java-11-openjdk-amd64/lib/server
# KO; should be either empty, or at least the full path to Conda-local JDK for correctness
KHIOPS_CLASSPATH /usr/share/khiops/norm.jar:/usr/share/khiops/khiops.jar
# KO; should be empty on Conda
KHIOPS_LAST_RUN_DIR /tmp/khiops/...
# OK
KHIOPS_PROC_NUMBER N
# OK
KHIOPS_TMP_DIR
# OK
KHIOPS_MEMORY_LIMIT
# OK
KHIOPS_API_MODE
# OK
KHIOPS_RAW_GUI
# OK
KHIOPS_JAVA_ERROR
# OK
KHIOPS_MPI_ERROR
# OK


Skipping Word , because of missing timings

Hello,
thanks for sharing this software - I have a problem with german songs:
Error: Skipping Word 1000, because of missing timings
Error: Skipping Word 300.000, because of missing timings
Error: Skipping Word 4, because of missing timings
Error: Skipping Word 5, because of missing timings
....
My cli:
py UltraSinger.py -i "C:......mp3" --whisper large-v2 --force_whisper_cpu --force_separation_cpu
Can anybody help me with that. Are there better models for german songs?
Thank you
Kind Regards
Roger

Hi,
can you provide a YouTube link for testing?
I will later try to make a better Error message that actually show the word 😅
Maybe whisper found some words in noise which are extrem short.
I found --whisper large-v2 for German quite good so far. But you can try others on Huggingface. Maybe there are better ones.
Please post them here: Discussion

Hi,
thank you for the fast response. So far the translation is fine but numbers won't work.
For example
https://m.youtube.com/watch?v=uC08L4xxjNM
Or Nena 99 Red Balloons
Other Huggingface models seem to be problematic in post processing. The words are not placed accordingly to the tones. Verly long sentences are shown in ultrastar.
Kind Regards and thank you for your work
Roger

hmm nothing i can do much. It's an issue from WhisperX. I opened an issue #409. Maybe you can add some more info there.
I made a workaround. Now the word is not simply removed but added after the previous one with 0.2 seconds length. You have to adjust it manually.

The PR #135 handles the numbers by translating it to english words.
The problem here is that it will probebly only work for english words.

PR created for multiple language support using num2words


Make travis test the build

Doesn't look like this will work without access to the internal yum repos...


Will be updated to adapt to new PL?

Thanks for contributing such a good tool!

I'm using the PL in ver 1.8.2, and I got the following Runtime Error. I think the PL update leads to this issue. I would like to ask that would you have plan to update the callback plugin? Thanks in advance!
=====================
RuntimeError: The Callback.on_epoch_start hook was removed in v1.8. Please use Callback.on_<train/validation/test>_epoch_start instead.


fix: translate boolean values on filter

The filter component of boolean property was missing translation.

The target branch should be master since this is not a breaking change


Stix1.x conversion: missing type="URL" for indicator

For given a stix2.1 bundle, upon converting it to stix1.x the final xml is missing type="URL" in cybox:Properties"> property. This is observed in case of indicator with URL.
Example:
{       "type": "bundle",       "id": "bundle--a9d9a0c5-8e15-42b8-9795-45f32a003161",       "objects": [         {           "type": "marking-definition",           "spec_version": "2.1",           "id": "marking-definition--f88d31f6-486f-44da-b317-01333bde0b82",           "created": "2017-01-20T00:00:00.000Z",           "definition_type": "tlp",           "name": "TLP:AMBER",           "definition": {             "tlp": "amber"           }         },         {           "type": "indicator",           "spec_version": "2.1",           "id": "indicator--f9ca508e-3b75-4d25-955a-e3f150974691",           "created": "2024-01-10T22:08:13.811Z",           "modified": "2024-01-10T22:08:14.811Z",           "name": "aklab3.com/favicon.ico",           "indicator_types": [             "anomalous-activity"           ],           "pattern": "[url:value = 'aklab3.com/favicon.ico']",           "pattern_type": "stix",           "pattern_version": "2.1",           "valid_from": "2024-01-10T22:08:10.07Z",           "valid_until": "2024-01-17T22:08:10.07Z",           "labels": [             "Recently Reported Spam or Unwanted Content",             "Recently Detected Malware Distribution",             "Historically Reported Spam or Unwanted Content",             "Historically Detected Malware Distribution"           ],           "confidence": 65,           "object_marking_refs": [             "marking-definition--f88d31f6-486f-44da-b317-01333bde0b82"           ],           "x_ctix_confidence_score": 25         }       ]     }
After convert it using stix2slider the output is:

`<stix:STIX_Package \n\txmlns:URIObj="http://cybox.mitre.org/objects#URIObject-2"\n\txmlns:tlpMarking="http://data-marking.mitre.org/extensions/MarkingStructure#TLP-1"\n\txmlns:marking="http://data-marking.mitre.org/Marking-1"\n\txmlns:cybox="http://cybox.mitre.org/cybox-2"\n\txmlns:indicator="http://stix.mitre.org/Indicator-2"\n\txmlns:stix="http://stix.mitre.org/stix-1"\n\txmlns:cyboxCommon="http://cybox.mitre.org/common-2"\n\txmlns:stixCommon="http://stix.mitre.org/common-1"\n\txmlns:example="http://example.com"\n\txmlns:xlink="http://www.w3.org/1999/xlink"\n\txmlns:ds="http://www.w3.org/2000/09/xmldsig#"\n\txmlns:xs="http://www.w3.org/2001/XMLSchema"\n\txmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"\n\t id="bundle--a9d9a0c5-8e15-42b8-9795-45f32a003161" version="1.2">\n
<stix:STIX_Header>\n
    <stix:Handling>\n
        <marking:Marking>\n
            <marking:Marking_Structure id="example:markingstructure-f88d31f6-486f-44da-b317-01333bde0b82" xsi:type=\'tlpMarking:TLPMarkingStructureType\' color="AMBER"/>\n
        </marking:Marking>\n
    </stix:Handling>\n
</stix:STIX_Header>\n
<stix:Indicators>\n
    <stix:Indicator id="example:indicator-f9ca508e-3b75-4d25-955a-e3f150974691" timestamp="2024-01-10T22:08:14.811000+00:00" xsi:type=\'indicator:IndicatorType\' version="2.2">\n
        <indicator:Title>aklab3.com/favicon.ico</indicator:Title>\n
        <indicator:Type>anomalous-activity</indicator:Type>\n
        <indicator:Valid_Time_Position>\n
            <indicator:Start_Time precision="second">2024-01-10T22:08:10.070000+00:00</indicator:Start_Time>\n
            <indicator:End_Time precision="second">2024-01-17T22:08:10.070000+00:00</indicator:End_Time>\n
        </indicator:Valid_Time_Position>\n
        <indicator:Observable id="example:Observable-b6adb005-791f-4107-b2bc-caaf5a6fbc59">\n
            <cybox:Keywords>\n
                <cybox:Keyword>Recently Reported Spam or Unwanted Content</cybox:Keyword>\n
                <cybox:Keyword>Recently Detected Malware Distribution</cybox:Keyword>\n
                <cybox:Keyword>Historically Reported Spam or Unwanted Content</cybox:Keyword>\n
                <cybox:Keyword>Historically Detected Malware Distribution</cybox:Keyword>\n
            </cybox:Keywords>\n
            <cybox:Object id="example:URI-22e7239a-31a7-4d29-8b1a-8c54cc7f51ca">\n
                <cybox:Properties xsi:type="URIObj:URIObjectType">\n
                    <cyboxCommon:Custom_Properties>\n
                        <cyboxCommon:Property name="x_ctix_confidence_score">25</cyboxCommon:Property>\n
                    </cyboxCommon:Custom_Properties>\n
                    <URIObj:Value condition="Equals">aklab3.com/favicon.ico</URIObj:Value>\n
                </cybox:Properties>\n
            </cybox:Object>\n
        </indicator:Observable>\n
        <indicator:Handling>\n
            <marking:Marking>\n
                <marking:Controlled_Structure>../../../descendant-or-self::node() | ../../../descendant-or-self::node()/@*</marking:Controlled_Structure>\n
                <marking:Marking_Structure xsi:type=\'tlpMarking:TLPMarkingStructureType\' color="AMBER"/>\n
            </marking:Marking>\n
        </indicator:Handling>\n
        <indicator:Confidence timestamp="2024-01-12T13:23:28.957386+00:00">\n
            <stixCommon:Value>Medium</stixCommon:Value>\n
        </indicator:Confidence>\n
    </stix:Indicator>\n
</stix:Indicators>\n

</stix:STIX_Package>\n`

Hi @aryabharat,
I wrote the slider years ago, and haven't used STIX 1 in ages - so in order to help you, can you send me what you think the "correct" XML would be.  Why isn't this what you want?
<cybox:Object id="example:URI-22e7239a-31a7-4d29-8b1a-8c54cc7f51ca">\n
<cybox:Properties xsi:type="URIObj:URIObjectType">\n
cyboxCommon:Custom_Properties\n
<cyboxCommon:Property name="x_ctix_confidence_score">25</cyboxCommon:Property>\n
</cyboxCommon:Custom_Properties>\n
<URIObj:Value condition="Equals">aklab3.com/favicon.ico</URIObj:Value>\n
</cybox:Properties>\n
</cybox:Object>\n

type="URL"
Hi @rpiazza Thanks for a quick response:
The response should contains a property type in cyboxCommon:Property field.
In the given case:
<cybox:Object id="example:URI-22e7239a-31a7-4d29-8b1a-8c54cc7f51ca">\n <cybox:Properties xsi:type="URIObj:URIObjectType"  type="URL">\n cyboxCommon:Custom_Properties\n <cyboxCommon:Property name="x_ctix_confidence_score">25</cyboxCommon:Property>\n </cyboxCommon:Custom_Properties>\n <URIObj:Value condition="Equals">aklab3.com/favicon.ico</URIObj:Value>\n </cybox:Properties>\n </cybox:Object>\n
Type is optional field according to Stix documentation. So i need a way to add this field.
I am getting type in case of indicator of domain type.

Hi @rpiazza
Created a PR for the same, Please check.
https://github.com/oasis-open/cti-stix-slider/pull/66

@aryabharat - I've looked into this - it does appear to be necessary, but I'm not sure why.  Do you have any insight?  What doesn't work if you don't have type="URL

@rpiazza So typo is a optional fileld in stix1.x cybox:Properties
The issue we faced while polling by a Qradar server, the URL were not getting passed if they don't have a type in the cybox:Properties field in case of 1.x polling.

Can you sign the CLA?

@rpiazza Already signed.  Please let me know if something else ids needed.

Please look at https://github.com/oasis-open/cti-stix-slider/pull/66.  It doesn't appear to be signed...

Please look at #66. It doesn't appear to be signed...

I have signed it, but the recheck is not updating it.

@rpiazza The CLA is signed.


Added support for absolute __filename and __dirname

In NodeJS __filename uses an absolute path while the duktape-node extra had only relative paths. This was mostly done to avoid file system access when this code was integral part of duktape. Now that it is an extra I believe we can relax the no-filesystem rule and make __filename absolute. Also this patch adds __dirname, as this is also used frequently.

I actually use a variant of this extra in miniSphere where this change doesn't make sense.  My engine is sandboxed such that the canonical representation of a filename is relative to the root of the game directory--and in the case of SPK-packaged games, the OS filesystem is not even directly used at all.  Thus I'm otherwise Node-compatible but __filename is not the actual file system path, but instead something like bin/foo.js--which is "absolute" as far as the asset resolver is concerned.
Note that technically duktape-node extra doesn't use "relative paths" specifically--the modResolve function is supposed to return the canonical module ID from whatever was passed in--which in most cases would of course mean returning the absolute path. (since in Node, module.id === __filename).
So just based on sandboxing concerns alone, and the fact that people may want to use this on platforms that might not have a standard file system (or one at all, for that matter), I don't think this change makes sense.  @svaarala, what do you think?

Well, that seems to be a specific requirement for your game engine, but that code is supposed to mirror what NodeJS does and this is what's in the patch. Btw, in NodeJS module.id is also an absolute path while in this patch I left it what was used to load the module. Otherwise the duktape debugger extension in vscode stops working :-(

I wrote the extra.  The intent was to be compatible (as far as possible) with Node modules, not be a full emulation.  If the user wants the NodeJS resolving behavior, that's meant to be done in modResolve, provided by the user.  Even if full Node compatibility was a goal, the point about use on systems without a standard file system stands.  Thus the current more generic, pluggable setup makes most sense, I think.  I'll wait to see what @svaarala thinks though. :)

Ah, and that code works fine for me, thanks! I implemented the NodeJS module resolution on top of your code (using the load and resolve calls) and that works great. I can load already quite a number of existing node modules. I don't need modeResolve and I think that was removed in 2.x anyway.

Yeah, load and resolve, that's right.  I forgot they were renamed, hehe.  I do agree that __dirname should be supported by the way; I just couldn't figure out how to do it in a generic manner.  Now might be a good time to revisit that, maybe it could be worked into the API for resolve somehow.

Generally speaking Duktape tries to be an embeddable engine -- which to me means that the application writer should be in control of how Duktape ties to its environment. Otherwise it will be necessary in some environments (sandboxed, virtualized, etc) to patch the core or extra code to avoid direct file I/O etc which is a hassle and contrary to the goal of being embeddable.
But in case there's a feature that can't be generalized at all, then I'd prefer the feature to be optional so that it's not forced on environments that do need sandboxing.
I'm not sure about this feature however (I'd need to dig into it a bit more to say something useful :). Or maybe you guys already worked it out? :)

I thought, since this is an extra, it is already optional. So the patch would qualify :-D

I object to the patch only on the grounds that it takes freedom away from the resolve callback, which for sandboxing reasons is designed to be the ultimate authority on what the application sees as the "canonical" module filename.
Duktape is commonly used in sandboxing scenarios, so I believe the current approach makes most sense as those same developers might want Node-style module compatibility within the sandbox.
Hm... what if we provided the Node-like FS resolution as a default behavior?  In other words, leave the extra set up the way it is now, but provide a default resolve implementation that resolves against the standard file system.  Then developers that need the sandboxing behavior (like me) can plug in a suitable resolve, while more Node-like environments such as dukluv could use the default one.

Maybe a default implementation of load as well, that reads modules from the file system.

@mike-lischke The components bundled as extras are not as strictly portable or embeddable as core Duktape, but it doesn't really make sense to bundle a non-embeddable component with an embeddable engine. It would make more sense in such a case to move the extra into a separate repo and just link to it, just like any other Duktape-related project.
The main reason to bundle is when some functionality is not part of the core, but commonly needed, and when bundling it's awkward if the component won't be generally usable. Module loaders are part of ES2015 so one reason for bundling them (instead of providing them as separate repos) is that a user will almost always need one, at least when using ES2015. Some of the other extras could be in separate repos and I might move some of them elsewhere (like the logging framework which was previously part of the core, but because there isn't a general logging standard, isn't needed in the same way as module loaders).
In any case, extras are intended to be quite widely usable in scenarios where Duktape is used, and that includes embedding. I've personally had to fight Lua and a host of Lua libraries for embedding and it's no fun when you need to patch everything to get hard I/O dependencies out of internals :-P

But back to this pull/issue, it'd be ideal to find a way to provide the end result @mike-lischke wants but in some embedding friendly manner. Can the functionality be abstracted to a provider callback?

can the functionality be abstracted to a provider callback?

Yeah, that's what I meant with my suggestion for a default implementation of resolve and load.

@fatcerberus Hmm, so can the desired functionality be implemented in the resolve/load callbacks, as module-node stands right now? If so, maybe it'd be easiest to provide an example resolve/load provider that had the functionality in a nicely commented form?
I'd be okay with a default resolve/load too, but I'm just wondering whether a single default provider can ever really work for practical use cases; unlike Node.js, Duktape is quite commonly used in embedding and there are very often some special constraints like translating paths to a virtual root (say, prepending a /foo/bar/quux/ path to "absolute" paths), access control, mapping file I/O to a ZIP/TAR file, etc.

One particular thing I'd like to avoid is linking against fopen() et al even if they're not used. Some embedded systems (several that I work with for example) don't provide fopen() at all, so it's important that the default code is not compiled in if you don't want to use it. In those systems it's common to e.g. bundle modules into a single ZIP file that gets compiled in, and modules are loaded from the ZIP file (I'm using https://github.com/svaarala/lowzip for example), with no actual I/O.
That suggest two alternatives:

Allow some define to be used to disable the default load/resolve so that no unnecessary symbols are referenced.
Or, make the default load/resolve a separate file that can be omitted from compilation entirely.

Hmm, if the load/resolve example with direct filesystem bindings was in a separate file that could easily be omitted for embedding, this would also have the nice feature that it'd be straightforward to add DLL support to that file for Windows, Linux, and OSX. Having an easy-to-use DLL loader example is definitely something that's missing now.

I think I'd agree that a separate file is best.  Maybe I'll try to work up a pull later.

Hmm, I'm not sure what you have in mind. Maybe you can show an example, but I could also imagine a simple #ifdef which wraps the part that accesses the file system and make it so configurable.

What I mean is to provide a pre-made resolve and load implementation that replicates the Node behavior, then you'd just call duk_module_node_init() and provide those instead of your own ones.

Done as you wish.


Fix LXD market importer

Replaces #3138

centos 6 and 7 enable sshd at startup after installing openssh-server
alpine was missing curl, so it couldn't get the context package
packages were being left on container /root/ directory

Since we won't add ssh dependency for alpine on context-package, we could install openssh here on alpine case.

Consider also my comment to #3148 about custom package installation. This commit does not include ssh installation?

Consider also my comment to #3138 about custom package installation. This commit does not include ssh installation?

Done, we had openssh stated but it was confirmed as a missing dep for context package. I guess until we release a new context version we can state here the missing deps such as openssh and dbus for ubuntu.
@vholer do you think stating openssh here on alpine it is correct, taking into account my unmerged packages PR


Support ignore patterns for formatters

Currently there's an option for linters that allows files to be excluded (python.linting.ignorePatterns), but there's not a similar option for formatters as far as I can see.
Would it be possible to implement a similar python.formatting.ignorePatterns, so that we can configure VSCode to not format, e.g., files from the standard library, or 3rd party packages? In my experiments, the Python extension does invoke the formatter for files in site-packages. I think it's more valuable to protect those kinds of files against accidental formatting, rather than accidental linting.
There's a similar issue at microsoft/vscode-python#7047, but that one's about black's inability to hard-exclude files (including files passed on the command line). I'm interested in not invoking the formatter in the first place.

Black supports # fmt: off markers to turn formatting off. It also supports --exclude and --extend-exclude options to control what files to format. And you can turn off the extension per-workspace or use profiles if you don't want to use it in other situations if you have configured Black as your formatter and turned on format-on-save universally instead of per-project.


gdm_format function not found

Hello,
When running the following line, I get an error that the function doesn't exist.
gdm_format(euc_dists2, lipa_coords2, env_pcs2$map, scale_gendist = TRUE)

Error in gdm_format(euc_dists2, lipa_coords2, env) :
could not find function "gdm_format"

Other GDM functions work. I tried to reinstall algatr but that also did not fix the issue.
Do you have any suggestions?
Thank you for your help!

Hello! I can't seem to reproduce your error on my end. Did you restart your R session/reload the algatr library after you reinstalled algatr? Maybe try reinstalling again and confirm that the reinstall was successful?


CsvReport: missing values converted to empty strings

instead of having "null" in your csv an empty string makes more sense
this fixes #246

Thank you.


feat: Update facebookLoginSdk library

Description
LEARNER-9199

Update the SDK version to the latest
Added Client_Token for initialization
Manually initialized FacebookSdk

Inspiration: https://developers.facebook.com/docs/facebook-login/android
PS: Please rebase Config-Repo before testing

These query params used for content sharing on facebook app.

Please add these query params, so we can open the Facebook app for login when it's available. Otherwise, it will always redirect us to the mobile web. Test it with android:exported true and false. Inspiration: Warning.

Loggin with facebook app is working fine if we generate the build using release-keystore.

The new SDK isn't working with our SHA (probably) because we are getting an error while logging through Facebook App. Possible solution.

The solution somehow working fine but it has callback issues with UI changes so we will fix this in another ticket

I have found a solution for the onActivityResult() warning. Shared as a patch on slack. It's working but I'm doubtful of its authenticity.


Fix messaging when approved with too few admins

A host can approve an application to join despite the collective having too few admins with the implication that financial contributions will be blocked (for hosts with this policy), however the email that goes out to the collective says that it is now ready to receive contributions:

There is also nothing on the collective page telling the user that their collective is blocked from receiving contributions until enough admins join:

It is first when clicking Contribute that you get this message:

Perhaps something like this could work:

However, the policies field is not available in the v1 GraphQL schema, which the collective page uses..


Update prefect agent values and end user inputs

Summary

Non-breaking feature that updates the prefect-agent plural configs and values to include two critical items for an end user to get started with a prefect agent

Adding their accountId and workspaceId to allow the agent to successfully connected to their account
Some restructuring on the values file to align with the values file for the prefect agent
Including the user provided values in the values template

Added some documentation on how a user can override the default work queue with one or multiple custom work queues of their own which the agent can generate on startup or the user can create in their prefect cloud account

Test Plan
N/A
Checklist

[x]  No images hosted from dockerhub
[ ]  Are dashboards present to understand the health of the application.  There must be at least 1 of these

[ ]  all databases should have dashboards
[ ]  ideally also have at least cpu/mem utilization dashboards for webserver tier of the app
[ ]  you can use plural from-grafana to convert a grafana dashboard found via google to our CRD

[ ]  Are scaling runbooks present

[ ]  all databases must have scaling runbooks
[ ]  you can use the charts in pluralsh/module-library to accelerate this

[ ]  do you need to add config overlays?

[ ]  inputing secrets
[ ]  configuring autoscaling

[ ]  If there’s a web-facing component to the app, we need to support OIDC authentication and setting up private networks if no authentication option is viable
[x]  All major clouds must be supported

[x]  Azure
[x]  AWS
[x]  GCP

Actually totally forgot, you'll need to bump the chart version in prefect-agent/helm/prefect-agent/Chart.yaml for it to upload (they're immutable in the chart registry)

Plural chart patch version updated 😎


allow a symfony file instance in validate dimensions

As @boobooking mentioned in #29962
Illuminate/Validation/Concerns/ValidatesAttributes::isValidFileInstance allows an instance of Symfony\Component\HttpFoundation\File\UploadedFile as well as Symfony\Component\HttpFoundation\File\File. But the method getClientMimeType – that is called by validating dimensions – only exists on UploadedFile.
This PR fixes this issue by using the getMimeType method that exists on both objects.
resolve #29962

I will check the issues in PHP 7.1 tomorrow. Maybe thats an issue related to https://bugs.php.net/bug.php?id=77973
Travis runs the check on PHP 7.1.27, but Laravel 5.8 requires PHP 7.1.3

Please send this to 6.x. :)

The failing test on PHP 7.1 was caused by a missing  tag in the test svg file. ref #29995 29995


Lesson 13: React - Changing code does not recompile the development server and does not refresh localhost page

After creating the react app within the existing folder (npx create-react-app front_end --template typescript), installing yarn and running yarn start, absolutely nothing happens when I change the code in for example App.tsx, App.css, index.tsx, etc.
I've been trying to look for a solution in different forums but I'm not having any luck. It seems like there is no watchers checking if there is a change in the code, so the development server stays the same. I've tried runnig npm start and yarn start and no luck.
I also tried adding a .env file next to package.json and tsconfig.json, setting the variables FAST_REFRESH=false and CHOKIDAR_USEPOLLING=true as per some recommendations in the forum, but still no luck.
OS: Windows 10, using WSL in Visual Studio Code v1.63.2
npm -v
8.1.4
node -v
v17.2.0
yarn -v
1.22.17

I solved the problem moving the files from Windows to WSL

You can open the wsl filesystem in explorer with \wsl$, and then you can copy-paste all your project files there

@MeLlamoValen, beautiful!!! not only did your suggestion solve my problem, I see the compilation time dropped significantly as well! You rock dude! Thanks :)

Thank you! I'm glad I could help you :)


Suggest to CKM

Given that these mappings are interesting to a wider international community, and CKM just released support for OMOCL mappings I’d like these mappings to be suggested to be adopted by CKM.openEHR.org. https://openehr.atlassian.net/wiki/spaces/healthmod/pages/2356085068/CKM+Release+1.20.0

Yep, feel free to add them already if you want.
Otherwise i will do it or @yampeku as soon as i find the time.


Added Plant Watering Calculator

Fixes Issue🛠
Closes #1193
Description👨💻
The Plant Watering Calculator is a web application that helps you calculate the amount of water needed for your plants based on the plant type, size, soil type, and sunlight exposure.

Select the plant type (Succulents, Ferns, Flowering Plants, or Vegetables).
Enter the size of the plant in cm.
Select the soil type (Sandy, Loamy, or Clay).
Enter the sunlight exposure in hours.
Perform the calculation and display the result in ml.

Type of change📄

[x] New feature (non-breaking change which adds functionality)

How this has been tested✅
I have reviewed a few water calculators available online. I have cross-checked the calculations provided by these tools and manually checked them to ensure its working.
Checklist✅

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have added demonstration in the form of GIF/video file
[x] I am an Open Source Contributor

Screenshots/GIF📷

@Rakesh9100
Please review this PR :)
Thank you


no capability to add ORDER BY table models

a user in dbt Slack asks:

Hi all, is there a way to specify "sort" column(s) or order by for models created in SQL Server? When I put in an order by clause I get the following error:
[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]
The ORDER BY clause is invalid in
views, inline functions, derived tables, subqueries, and common table expressions,
unless TOP, OFFSET or FOR XML is also specified.

Because we're using the SELECT * INTO DML for our model generation, if you include an ORDER BY in the SELECT query of a model, you'll get the above error. But I'm confused because the docs say it should be allowed, but isn't guaranteed...

When used with a SELECT...INTO or INSERT...SELECT statement to insert rows from another source, the ORDER BY clause does not guarantee the rows are inserted in the specified order.

https://github.com/dbt-msft/dbt-sqlserver/blob/679b3515e9f056879c6609c78bb62791716c450a/dbt/include/sqlserver/macros/adapters.sql#L160-L161

I think this is due to that we are creating a temp-view in the middle of this. The tmp_relation above. A view can not have order by.

There is only one scenario where ORDER BY in an INSERT makes sense, and that's for guaranteeing that IDENTITY columns are assigned values in order. In all other cases this does not do what you might want it to do -- in particular, every time you SELECT from a table you must specify an ORDER BY if you want an order, otherwise the engine is free to rearrange things as it sees fit (and it will, if the table is a columnstore, or parallelism is involved, or it can reuse a scan in progress from another query), so whatever order you think you may have gotten from the insert will not be useful anyway (and that's before we consider rows being moved through deletes and updates). People often fall into the trap of thinking there's a "natural physical order" that you're guaranteed to get if you do SELECT * with no ORDER BY, but there isn't.
So this is probably not useful to enable unless there really is a use case of building models with IDENTITY columns (unlikely, since ROW_NUMBER() gives you predictable values as well), and even then, as Mikael notes, you'd need to not use a view. (In olden days SELECT TOP(100) PERCENT ... ORDER BY could be used as a trick to circumvent this restriction, but while that's still allowed syntactically, the optimizer simply pretends you didn't write an ORDER BY in that case.)

I thought it would be possible (and maybe even useful) to add a custom configuration, order_by, that could be used like below. However I tried it just now and the ORDER BY did nothing and the rows are returned in ascending order.
@mikaelene can you remember why we don't use temp tables and instead use views? I can't remember...
-- models/example.SQL
{{
  config(
    order_by = 'id DESC,
    )
}}

SELECT 1 as id
UNION
SELECT 2 as id

calling dbt run -m example would run the following SQL:
create view "ajs"."example__dbt_tmp_temp_view" as

SELECT 1 as id
UNION
SELECT 2 as id;

SELECT * INTO "SalesOps"."ajs"."example__dbt_tmp"
FROM "SalesOps"."ajs"."example__dbt_tmp_temp_view"
-- this would be the line magically "injected"
ORDER BY id DESC

I originally requested a sorting mechanism in the slack channel because I thought it would be convenient to my data mart's end users to have a default row ordering on specific tables/models. However, learning more about the SQL behavior makes me realize that this may not be feasible or as useful as I had thought (@jeroen-mostert's  comment was very helpful in that regard). Alternatively, I can perhaps use a clustered index to achieve what I want?
In any case, if you decide that allowing for the ORDER BY isn't worth it, then I am perfectly happy to drop my original request. I really appreciate you looking into this though!

Adding a clustered index to the table doesn't convey any (intrinsic) order either. It will often be the case that you will get the clustered index order when you do a SELECT * with no ORDER BY (especially if the table is small and is never modified), but this is not guaranteed, and your end users should not count on it -- they should always write their own ORDER BY, otherwise they might get in the habit of only looking at the top rows and assuming that it's all there when it might not be.
If you really want a default order you can do something like a view that does SELECT TOP(9223372036854775807) ... FROM table ORDER BY ... (that number is the largest value that fits a BIGINT and should suffice for any table :P), unlike TOP(100) PERCENT the optimizer does not ignore this, and if the clustered index supports that order the ordering is "free". It's not exactly elegant, though.

Closing this, as it is not really feasible, and not desirable.


Closes #1

Add dotenv and docker support.

I like the idea, however, I think that there is a more simple solution.
Instead of loading a whole separate file, you can add a default copy of "settings.json" which has its JSON values set to the names of the environment variables.
Then you can use run ExpandEnvironmentVariables right after the file is read in, but before it is deserialized.

Oh that's much better, I wasn't aware of that. Thank you! I'll change it.

Looks good! Feel free to merge when ready.

I can't seem to start the merge to close #1 (I accidentally closed the PR for a moment... My bad). I guess you'll have to do it.


Diff editor not working properly

Issue Type: Bug
1.In Version: 1.43.1, the word is no highlight.

2.But in Version:1.39.2, it has.

VSCode Version: 1.43.1 (user setup)
OS Version: Windows_NT x64 10.0.18363

Does this issue occur when all extensions are disabled? Yes
Here is test case:
File1

-rw-r--r-- 1 www root<PHONE_NUMBER> Mar 11 00:00 20200310.access.log
-rw-r--r-- 1 www root<PHONE_NUMBER> Mar 12 00:00 20200311.access.log
-rw-r--r-- 1 www root<PHONE_NUMBER> Mar 13 00:00 20200312.access.log
-rw-r--r-- 1 www root<PHONE_NUMBER> Mar 14 00:00 20200313.access.log
-rw-r--r-- 1 www root<PHONE_NUMBER> Mar 15 00:00 20200314.access.log
-rw-r--r-- 1 www root<PHONE_NUMBER> Mar 16 00:00 20200315.access.log
-rw-r--r-- 1 www root<PHONE_NUMBER> Mar 17 00:00 20200316.access.log
-rw-r--r-- 1 www root<PHONE_NUMBER> Mar 18 00:00 20200317.access.log
-rw-r--r-- 1 www root<PHONE_NUMBER> Mar 19 00:00 20200318.access.log
-rw-r--r-- 1 www root<PHONE_NUMBER> Mar 20 00:00 20200319.access.log
-rw-r--r-- 1 www root<PHONE_NUMBER> Mar 21 00:00 20200320.access.log
-rw-r--r-- 1 www root<PHONE_NUMBER> Mar 11 00:00 20200310.access.log
-rw-r--r-- 1 www root<PHONE_NUMBER> Mar 12 00:00 20200311.access.log
-rw-r--r-- 1 www root<PHONE_NUMBER> Mar 13 00:00 20200312.access.log
-rw-r--r-- 1 www root<PHONE_NUMBER> Mar 14 00:00 20200313.access.log
-rw-r--r-- 1 www root<PHONE_NUMBER> Mar 15 00:00 20200314.access.log
-rw-r--r-- 1 www root<PHONE_NUMBER> Mar 16 00:00 20200315.access.log
-rw-r--r-- 1 www root<PHONE_NUMBER> Mar 17 00:00 20200316.access.log
-rw-r--r-- 1 www root<PHONE_NUMBER> Mar 18 00:00 20200317.access.log
-rw-r--r-- 1 www root<PHONE_NUMBER> Mar 19 00:00 20200318.access.log
-rw-r--r-- 1 www root<PHONE_NUMBER> Mar 20 00:00 20200319.access.log
-rw-r--r-- 1 www root<PHONE_NUMBER> Mar 21 00:00 20200320.access.log

File2

-rw-r--r-- 1 root root<PHONE_NUMBER> Mar 11 02:34 20200310.access.log
-rw-r--r-- 1 root root<PHONE_NUMBER> Mar 12 02:54 20200311.access.log
-rw-r--r-- 1 root root<PHONE_NUMBER> Mar 13 03:06 20200312.access.log
-rw-r--r-- 1 root root<PHONE_NUMBER> Mar 14 03:06 20200313.access.log
-rw-r--r-- 1 root root<PHONE_NUMBER> Mar 15 03:14 20200314.access.log
-rw-r--r-- 1 root root<PHONE_NUMBER> Mar 16 02:58 20200315.access.log
-rw-r--r-- 1 root root<PHONE_NUMBER> Mar 17 03:10 20200316.access.log
-rw-r--r-- 1 root root<PHONE_NUMBER> Mar 18 02:59 20200317.access.log
-rw-r--r-- 1 root root<PHONE_NUMBER> Mar 19 03:04 20200318.access.log
-rw-r--r-- 1 root root<PHONE_NUMBER> Mar 20 03:06 20200319.access.log
-rw-r--r-- 1 root root<PHONE_NUMBER> Mar 21 03:10 20200320.access.log
-rw-r--r-- 1 root root<PHONE_NUMBER> Mar 11 02:34 20200310.access.log
-rw-r--r-- 1 root root<PHONE_NUMBER> Mar 12 02:54 20200311.access.log
-rw-r--r-- 1 root root<PHONE_NUMBER> Mar 13 03:06 20200312.access.log
-rw-r--r-- 1 root root<PHONE_NUMBER> Mar 14 03:06 20200313.access.log
-rw-r--r-- 1 root root<PHONE_NUMBER> Mar 15 03:14 20200314.access.log
-rw-r--r-- 1 root root<PHONE_NUMBER> Mar 16 02:58 20200315.access.log
-rw-r--r-- 1 root root<PHONE_NUMBER> Mar 17 03:10 20200316.access.log
-rw-r--r-- 1 root root<PHONE_NUMBER> Mar 18 02:59 20200317.access.log
-rw-r--r-- 1 root root<PHONE_NUMBER> Mar 19 03:04 20200318.access.log
-rw-r--r-- 1 root root<PHONE_NUMBER> Mar 20 03:06 20200319.access.log
-rw-r--r-- 1 root root<PHONE_NUMBER> Mar 21 03:10 20200320.access.log

/confirm
Diff obtained using File: Compare Active File With...
Also reproduced on today's Insiders
Version: 1.44.0-insider (user setup)
Commit: 0926b534553f3ff2f0cff0a741550a97df199706
Date: 2020-03-23T08:40:59.737Z
Electron: 7.1.11
Chrome: 78.0.3904.130
Node.js: 12.8.1
V8: 7.8.279.23-electron.0
OS: Windows_NT x64 10.0.18363

I found it easier to save both files (file1.log and file2.log) and then compare them. I can reproduce too, I am not seeing diffs for these 2 files even though lines differ.

Duplicate of #87054 . That issue did not reach 20 upvotes, so it got closed by the bot.
This is not a bug. Our diff editor is not optimized to be used with tabular data, so it does not execute char level diff when a diff chunk is over 20 lines. This is done for performance reasons and makes sense for comparing source code.

@alexdima so it sounds like something changed since 1.39.2, right? Also, maybe reopen this issue now that it's a Backlog Candidate.

@gjsjohnmurray Yes, this was an intentional change -- see https://github.com/microsoft/vscode/issues/72823 -- the computation of the character level differences would end up stalling the entire diff computation up to leading to a timeout. And that happened with real source code. This sample does not contain source code, it contains the output of ls.
Please also see #87054 -- This stayed in Backlog candidates and did not receive 20 upvotes. Now this issue is a 100% duplicate of #87054 from the same OP. Issue #87054 did not receive a single community upvote in the past 3 months.


Changeset::new ignores difference in blank lines

First of all, thanks a lot for this library, it makes testing complex structures a breeze! :)
And here is a repro of a bug:
let difference = difference::Changeset::new("foo\nbar", "foo\n\nbar", "\n");
println!("{}", difference);
println!("difference.diffs = {:?}", difference.diffs);

It prints
foo
bar

difference.diffs = [Same("foo\nbar")]

without any colors. It would be nice to indicate that a blank line is missing!

Hm, yes, I agree, that might be better. It's a bit tricky when splitting on \n I imagine.

This is related to #11