Segmentation Fault in OpenROAD During Multi-Corner STA (sky130A PDK)

Describe the bug
OpenROAD crashes with a segmentation fault during the execution of the multi-corner static timing analysis (STA) step in OpenLane. The crash occurs when using the sky130A PDK, and the log reports a segmentation violation in libc. The issue is reproducible across multiple runs with the same configuration.
Steps I do:

Clone the OpenLane repository and set up the environment.
Install the sky130A PDK and set the PDK_ROOT environment variable.
Run the following command:
./flow.tcl -design <your_design> -run_full_flow
The segmentation fault occurs during the STA step in OpenROAD.

Expected Behavior
OpenROAD should complete the multi-corner STA step without crashing and generate timing reports.
OpenROAD Environment
I am using OpenROAD through the OpenLane Docker image: efabless/openlane-tools:openroad_app-cdf98f3571ef892d091b6825c5d473bcfbf6e167-centos-7-amd64 with OpenROAD.

OpenLane Environment
open_pdks cdf98f3571ef892d091b6825c5d473bcfbf6e167
Kernel: 5.15.0-122-generic
Description: Ubuntu 20.04.6 LTS
Docker version 27.3.1, build ce12230
Python 3.11.10
cdf98f35 Thu Jul 18 14:28:46 2024 +0300 Propagate the clock in some STA reports (#2134)
python-venv: INSTALLED

To Reproduce

Clone the OpenLane repository.
Install the sky130A PDK and set PDK_ROOT to point to the installation.
Navigate to the OpenLane directory.
run make mount
Run the following command:./flow.tcl -design adder_depth_2023 -init_design_config -add_to_designs
./flow.tcl -design adder_depth_2023 -init_design_config -add_to_designs -config_file config.tcl
./flow.tcl -design adder_depth_2023 ```

The segmentation fault occurs during the multi-corner STA step in OpenROAD.

I've also attached a zip of the issue reproducible folder.
issue_reproducible.zip
Relevant log output
OpenLane cdf98f3571ef892d091b6825c5d473bcfbf6e167
All rights reserved. (c) 2020-2023 Efabless Corporation and contributors.
Available under the Apache License, version 2.0. See the LICENSE file for more details.

[INFO]: Using configuration in 'designs/adder_depth_2023/config.tcl'...
[INFO]: Process Design Kit: sky130A
[INFO]: PDK Root: /home/mvguser/.volare
[INFO]: Standard Cell Library: sky130_fd_sc_hd
[INFO]: Optimization Standard Cell Library: sky130_fd_sc_hd
[INFO]: Run Directory: /openlane/designs/adder_depth_2023/runs/RUN_2024.11.24_14.53.20
[INFO]: Saving runtime environment...
[INFO]: Preparing LEF files for the nom corner...
[INFO]: Preparing LEF files for the min corner...
[INFO]: Preparing LEF files for the max corner...
[INFO]: Running linter (Verilator) (log: designs/adder_depth_2023/runs/RUN_2024.11.24_14.53.20/logs/synthesis/linter.log)...
[INFO]: 0 errors found by linter
[INFO]: 0 warnings found by linter
[STEP 1]
[INFO]: Running Synthesis (log: designs/adder_depth_2023/runs/RUN_2024.11.24_14.53.20/logs/synthesis/1-synthesis.log)...
[STEP 2]
[INFO]: Running Single-Corner Static Timing Analysis (log: designs/adder_depth_2023/runs/RUN_2024.11.24_14.53.20/logs/synthesis/2-sta.log)...
[STEP 3]
[INFO]: Running Initial Floorplanning (log: designs/adder_depth_2023/runs/RUN_2024.11.24_14.53.20/logs/floorplan/3-initial_fp.log)...
[INFO]: Floorplanned with width 1488.56 and height 1476.96.
[STEP 4]
[INFO]: Running IO Placement (log: designs/adder_depth_2023/runs/RUN_2024.11.24_14.53.20/logs/floorplan/4-io.log)...
[STEP 5]
[INFO]: Running Tap/Decap Insertion (log: designs/adder_depth_2023/runs/RUN_2024.11.24_14.53.20/logs/floorplan/5-tap.log)...
[INFO]: Power planning with power {VPWR} and ground {VGND}...
[STEP 6]
[INFO]: Generating PDN (log: designs/adder_depth_2023/runs/RUN_2024.11.24_14.53.20/logs/floorplan/6-pdn.log)...
[STEP 7]
[INFO]: Running Global Placement (skip_io) (log: designs/adder_depth_2023/runs/RUN_2024.11.24_14.53.20/logs/placement/6-global_skip_io.log)...
[STEP 8]
[INFO]: Running Single-Corner Static Timing Analysis (log: designs/adder_depth_2023/runs/RUN_2024.11.24_14.53.20/logs/placement/8-gpl_sta.log)...
[STEP 9]
[INFO]: Running IO Placement (log: designs/adder_depth_2023/runs/RUN_2024.11.24_14.53.20/logs/placement/9-io.log)...
[STEP 10]
[INFO]: Running Global Placement (log: designs/adder_depth_2023/runs/RUN_2024.11.24_14.53.20/logs/placement/9-global.log)...
[STEP 11]
[INFO]: Running Single-Corner Static Timing Analysis (log: designs/adder_depth_2023/runs/RUN_2024.11.24_14.53.20/logs/placement/11-gpl_sta.log)...
[STEP 12]
[INFO]: Running Placement Resizer Design Optimizations (log: designs/adder_depth_2023/runs/RUN_2024.11.24_14.53.20/logs/placement/12-resizer.log)...
[STEP 13]
[INFO]: Running Detailed Placement (log: designs/adder_depth_2023/runs/RUN_2024.11.24_14.53.20/logs/placement/13-detailed.log)...
[STEP 14]
[INFO]: Running Single-Corner Static Timing Analysis (log: designs/adder_depth_2023/runs/RUN_2024.11.24_14.53.20/logs/placement/14-dpl_sta.log)...
[INFO]: ::env(CLOCK_PORT) is not set
[WARNING]: Skipping CTS...
[STEP 15]
[INFO]: Running Placement Resizer Timing Optimizations (log: designs/adder_depth_2023/runs/RUN_2024.11.24_14.53.20/logs/cts/15-resizer.log)...
[STEP 16]
[INFO]: Running Global Routing Resizer Design Optimizations (log: designs/adder_depth_2023/runs/RUN_2024.11.24_14.53.20/logs/routing/16-resizer_design.log)...
[STEP 17]
[INFO]: Running Single-Corner Static Timing Analysis (log: designs/adder_depth_2023/runs/RUN_2024.11.24_14.53.20/logs/routing/17-rsz_design_sta.log)...
[ERROR]: during executing openroad script /openlane/scripts/openroad/sta/multi_corner.tcl
[ERROR]: Log: designs/adder_depth_2023/runs/RUN_2024.11.24_14.53.20/logs/routing/17-rsz_design_sta.log
[ERROR]: Last 10 lines:
12# Tcl_EvalEx in /nix/store/p16s3zg96afj68mfp17m03svhwmdgcz8-tcl-8.6.13/lib/libtcl.so
13# Tcl_Eval in /nix/store/p16s3zg96afj68mfp17m03svhwmdgcz8-tcl-8.6.13/lib/libtcl.so
14# sta::sourceTclFile(char const*, bool, bool, Tcl_Interp*) in openroad
15# ord::tclAppInit(Tcl_Interp*) in openroad
16# Tcl_MainEx in /nix/store/p16s3zg96afj68mfp17m03svhwmdgcz8-tcl-8.6.13/lib/libtcl.so
17# main in openroad
18# 0x00007FEB8734910E in /nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52/lib/libc.so.6
19# __libc_start_main in /nix/store/k7zgvzp2r31zkg9xqgjim7mbknryv6bs-glibc-2.39-52/lib/libc.so.6
20# _start in openroad
child killed: segmentation violation

[ERROR]: Creating issue reproducible...
[INFO]: Saving runtime environment...
OpenLane TCL Issue Packager

EFABLESS CORPORATION AND ALL AUTHORS OF THE OPENLANE PROJECT SHALL NOT BE HELD
LIABLE FOR ANY LEAKS THAT MAY OCCUR TO ANY PROPRIETARY DATA AS A RESULT OF USING
THIS SCRIPT.

Parsing config file(s)…
Setting up /openlane/designs/adder_depth_2023/runs/RUN_2024.11.24_14.53.20/issue_reproducible…
Done.
[INFO]: Reproducible packaged: Please tarball and upload 'designs/adder_depth_2023/runs/RUN_2024.11.24_14.53.20/issue_reproducible' if you're going to submit an issue.
[ERROR]: Step 17 (routing) failed with error:
-code 1 -level 0 -errorstack {INNER {invokeStk1 throw_error} CALL {run_tcl_script -tool openroad -no_consume /openlane/scripts/openroad/sta/multi_corner.tcl -indexed_log /openlane/designs/adder_depth_2023/runs/RUN_2024.11.24_14.53.20/logs/routing/17-rsz_design_sta.log -save to=/openlane/designs/adder_depth_2023/runs/RUN_2024.11.24_14.53.20/results/signoff,noindex,sdf,lib} CALL {run_openroad_script /openlane/scripts/openroad/sta/multi_corner.tcl -indexed_log /openlane/designs/adder_depth_2023/runs/RUN_2024.11.24_14.53.20/logs/routing/17-rsz_design_sta.log -save to=/openlane/designs/adder_depth_2023/runs/RUN_2024.11.24_14.53.20/results/signoff,noindex,sdf,lib} CALL {run_sta -propagate_all_clocks -estimate_global -no_save -log /openlane/designs/adder_depth_2023/runs/RUN_2024.11.24_14.53.20/logs/routing/rsz_design_sta.log} CALL run_resizer_design_routing CALL run_routing CALL run_routing_step CALL {run_non_interactive_mode -design adder_depth_2023}} -errorcode NONE -errorinfo {
    while executing
"throw_error"
    (procedure "run_tcl_script" line 219)
    invoked from within
"run_tcl_script -tool openroad -no_consume {*}$args"
    (procedure "run_openroad_script" line 2)
    invoked from within
"run_$arg_values(-tool)_script $::env(SCRIPTS_DIR)/openroad/sta/multi_corner.tcl {*}$arg_list"
    (procedure "run_sta" line 109)
    invoked from within
"run_sta -propagate_all_clocks -estimate_global -no_save -log $::env(routing_logs)/rsz_design_sta.log"
    (procedure "run_resizer_design_routing" line 13)
    invoked from within
"run_resizer_design_routing"
    (procedure "run_routing" line 8)
    invoked from within
"run_routing"
    (procedure "run_routing_step" line 7)
    invoked from within
"run_routing_step"} -errorline 1
[INFO]: Saving current set of views in 'designs/adder_depth_2023/runs/RUN_2024.11.24_14.53.20/results/final'...
[INFO]: Generating final set of reports...
[ERROR]: Failed to create manufacturability and metric reports:
[ERROR]: Traceback (most recent call last):
  File "/openlane/scripts/generate_reports.py", line 184, in <module>
    cli()
  File "/nix/store/vr8dx2vs7nhz4cz2vfyypx69vbwwrmk5-python3-3.11.9-env/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/vr8dx2vs7nhz4cz2vfyypx69vbwwrmk5-python3-3.11.9-env/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/nix/store/vr8dx2vs7nhz4cz2vfyypx69vbwwrmk5-python3-3.11.9-env/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/vr8dx2vs7nhz4cz2vfyypx69vbwwrmk5-python3-3.11.9-env/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/openlane/scripts/generate_reports.py", line 51, in cli
    report = Report(design, tag, design_name, params, run_path).get_report()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/openlane/scripts/report/report.py", line 809, in get_report
    *self.extract_all_values(),
     ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/openlane/scripts/report/report.py", line 359, in extract_all_values
    power_report = Artifact(
                   ^^^^^^^^^
  File "/openlane/scripts/report/report.py", line 62, in __init__
    self.pathname = matches[0]
                    ~~~~~~~^^^
IndexError: list index out of range
[INFO]: Saving runtime environment...
[ERROR]: Flow failed.
[INFO]: The failure may have been because of the following warnings:
[WARNING]: Skipping CTS...

Screenshots
No response
Additional Context
No response

I can't find that image
Unable to find image 'efabless/openlane-tools:openroad_app-cdf98f3571ef892d091b6825c5d473bcfbf6e167-centos-7-amd64' locally
docker: Error response from daemon: manifest for efabless/openlane-tools:openroad_app-cdf98f3571ef892d091b6825c5d473bcfbf6e167-centos-7-amd64 not found: manifest unknown: manifest unknown.

what version of OpenROAD is in that image?  If it is not very current then please try with the head of OR first.

You need to make a self contained test case.  The steps to reproduce should be untar & run, not install OL.

Sorry for the confusion, I am not using a separate Docker image of OpenROAD. Instead, I use it as a dependency within OpenLane. The version of OpenROAD in my setup is: OpenROAD b16bda7e82721d10566ff7e2b68f1ff0be9f9e38.

That commit is from Jun 3.  Please try with the head of OR.

I encountered an error while building OpenROAD with the head of OR.
here's the steps:

Edited flake.nix to update the rev

openroad = pkgs.openroad.override {
  rev = "e3ab3fb6e073ec23e499537350008fb2aaaead2e";
  sha256 = ""; # Temporarily empty
  openroad-abc = self.openroad-abc;
  opensta = self.opensta;
};

Ran the build command nix build .#openroad --extra-experimental-features nix-command --extra-experimental-features flakes

error: hash mismatch in fixed-output derivation '/nix/store/qyp5zdcziw8arbp02g5ml8wpdba9gw0r-source.drv':
         specified: sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
            got:    sha256-cYt9Ua9gafWsukB+FBXCuA1leQoXJlDvvfdZtBwGRYI=

Updated flake.nix with the fetched sha256

openroad = pkgs.openroad.override {
  rev = "e3ab3fb6e073ec23e499537350008fb2aaaead2e";
  sha256 = "sha256-cYt9Ua9gafWsukB+FBXCuA1leQoXJlDvvfdZtBwGRYI=";
  openroad-abc = self.openroad-abc;
  opensta = self.opensta;
};

Re-ran the build command nix build .#openroad --extra-experimental-features nix-command --extra-experimental-features flakes

warning: Git tree '/home/mvguser/hdd/OL/OpenLane' is dirty
warning: ignoring untrusted substituter 'https://openlane.cachix.org', you are not a trusted user.
Run `man nix.conf` for more information on the `substituters` configuration option.
warning: Ignoring setting 'auto-allocate-uids' because experimental feature 'auto-allocate-uids' is not enabled
warning: Ignoring setting 'impure-env' because experimental feature 'configurable-impure-env' is not enabled
error: builder for '/nix/store/k2d3pmaqgg8wmzmbkcqrd238cp4wbabh-openroad.drv' failed with exit code 2;
       last 25 log lines:
       > In file included from /build/source/src/gui/src/gui.cpp:46:
       > In file included from /build/source/src/gui/src/mainWindow.h:43:
       > /build/source/src/gui/src/findDialog.h:46:8: warning: 'accept' overrides a member function but is not marked 'override' [-Winconsistent-missing-override]
       >    46 |   void accept();
       >       |        ^
       > /nix/store/p8cr0k5g2jbjy6k80sdhfyr7vp4zzl8w-qtbase-5.15.12-dev/include/QtWidgets/qdialog.h:97:18: note: overridden virtual function is here
       >    97 |     virtual void accept();
       >       |                  ^
       > In file included from /build/source/src/gui/src/gui.cpp:46:
       > In file included from /build/source/src/gui/src/mainWindow.h:43:
       > /build/source/src/gui/src/findDialog.h:47:8: warning: 'reject' overrides a member function but is not marked 'override' [-Winconsistent-missing-override]
       >    47 |   void reject();
       >       |        ^
       > /nix/store/p8cr0k5g2jbjy6k80sdhfyr7vp4zzl8w-qtbase-5.15.12-dev/include/QtWidgets/qdialog.h:98:18: note: overridden virtual function is here
       >    98 |     virtual void reject();
       >       |                  ^
       > 7 warnings generated.
       > [ 47%] Building CXX object src/gui/CMakeFiles/gui.dir/src/globalConnectDialog.cpp.o
       > 1 error generated.
       > make[2]: *** [src/gui/CMakeFiles/gui.dir/build.make:436: src/gui/CMakeFiles/gui.dir/src/staGuiInterface.cpp.o] Error 1
       > make[2]: *** Waiting for unfinished jobs....
       > 2 warnings generated.
       > 2 warnings generated.
       > make[1]: *** [CMakeFiles/Makefile2:4448: src/gui/CMakeFiles/gui.dir/all] Error 2
       > make: *** [Makefile:136: all] Error 2
       For full logs, run 'nix-store -l /nix/store/k2d3pmaqgg8wmzmbkcqrd238cp4wbabh-openroad.drv'.

Since OpenROAD is used as a dependency, I'm not entirely sure how to properly update it without manually modifying flake.nix. Is there a better approach?

The steps @weafo took were correct, slightly undermined by how the latest version of OpenROAD doesn't build with the flake yet and I have not had the chance to submit a patch for it. Something about boost::optional and fmt.


adds support of onDisplayedDateChangeEnd and onDisplayedDateChangeStart callbacks

Solves #202
This pull request adds support of two callbacks on Calendar and DatePicker components.
onDisplayedDateChangeStart will be fired when transition to new month starts, or new month/year is selected from dropdown.
onDisplayedDateChangeEnd will be fired when transition to new month ends, or new month/year is selected from dropdown.
This pull request also adds ref to Calendar (and calendarRef to DatePicker), by which it will be possible to change displayed year and month simply by calling selectYear or selectMonth. In case of selectMonth, it is also possible to pass "NEXT" and "PREVIOUS" strings.

Let me note that github shows a lot of changes in Calendar.js. It is because of linting. To add support of ref to Calendar, I used React.forwardRef and wrapped component in it. That's why indentation of the whole component increased.

@thomasmikava Thanks for the PR, I tried using it, the issue with this is the animation - whenever you want to update the state during these events, the calendar rerenders and swaps back month/year and date cells to the previous selected month, then after the animation completes, it swaps it to the correct one, creating a weird looking jumpy animation.
It's not my call, but I think this is something that should be handled if the functionality was to be merged.
https://imgur.com/CUDejlW
Simple demonstration could be
  const [days, setDays] = useState<EventDay[]>([]);
  const changeCb = () => {
    setDays([]);
    setTimeout(() => {
       setDays([
         {year: 2020, month: 12, day: 15, className: 'anything'}
       ]);
    }, 2500);
  }
  return <Calendar customDaysClassName={events} onDisplayedDateChangeStart={changeCb} slideAnimationDuration={'5s'}) />;

Wish this had been merged


Inlay hints render randomly incorrectly after edits and the cursor goes to an inlay hint location

Does this issue occur when all extensions are disabled?: Yes

VS Code Version: 1.85.0 and
Version: 1.85.0-insider (user setup)
Commit: 182f5125f2f8a8469e9dac4ed4795b804671bf7d
Date: 2023-12-07T21:22:07.460Z
Electron: 27.1.3
ElectronBuildId: 25612240
Chromium: 118.0.5993.159
Node.js: 18.17.1
V8: <IP_ADDRESS>-electron.0
OS: Windows_NT x64 10.0.22621

Steps to Reproduce:

Create a test.ts file with

// aaaaaaaaaaaaaaaaaaaa
// aaaaaaaaaaaaaaaaaaaa
// aaaaaaaaaaaaaaaaaaaa
function foo()
{
// aaaaaaaaaaaaaaaaaaaaa
// aaaaaaaaaaaaaaaaaaaaa
}

and use settings
"typescript.inlayHints.enumMemberValues.enabled": true,
"typescript.inlayHints.functionLikeReturnTypes.enabled": true,
"typescript.inlayHints.parameterTypes.enabled": true,
"typescript.inlayHints.variableTypes.enabled": true

Enter some number of new lines at the top of the file like 3 or so (you may need to add/remove newlines and try again) and then move the cursor to the location of the inlay hint.

Bug: The inlay hint will render randomly duplicated.

FYI, we have a repro with our C++ extension as well, but it seems to me like it's the same root cause.

Thanks for creating this issue! We figured it's covering the same as another one we already have. Thus, we closed this one as a duplicate. You can search for similar existing issues. See also our issue reporting guidelines.
Happy Coding!

fixed in 1.86 insiders


Convenience class for checkbox

Similary as we have WebDriverSelect for <select>, it may be handy to have also something like WebDriverCheckbox for <input type="checkbox"> elements with helper methods:

check
uncheck
isChecked

and maybe some more.

What about radio input? Should checkbox and radio be treated as one?

@deleugpn True. I have some of this WIP already prepared, and it includes both checkboxes and radio buttons.  I hope to send it soon :).


5.15.15

Checklist

[x] Used a personal fork of the feedstock to propose changes
[ ] Bumped the build number (if the version is unchanged)
[x] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[ ] Ensured the license file is being packaged.

this should probably wait for webengine aarch64 before continuing

Hi! This is the friendly automated conda-forge-linting service.
I wanted to let you know that I linted all conda-recipes in your PR (recipe/meta.yaml) and found some lint.
Here's what I've got...
For recipe/meta.yaml:

Selectors are suggested to take a <two spaces>#<one space>[<expression>] form. See lines [10]

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe/meta.yaml) and found it was in an excellent condition.

@hmaarrfk

I think you are hitting: https://github.com/conda-forge/status/issues/188?notification_referrer_id=NT_kwDOAAFfmLExMzA2Nzc3NjA2Mzo5MDAwOA#issuecomment-2440111350

btw doens't azure have native aarch64 runners? why does conda-forge not use those?


Introduce edit job modal

Introduce a job edit modal.

Reviewer: @pierlo-upitup

Integration test are passing locally:

131 passing (4m)
19 pending

Code looks good 💯

Nice! Some minor comments.

I'm merge-friendly!


Add LICENSE

Hello,
thanks for your awesome work :)
Currently this project is missing a license though. Would you mind adding a license file to the repo (GPL, MIT, or whatever you prefer) so it's clear under which conditions it's possible to contribute?

I have added a license. Thank you!


Tool monitoring not working in virtualbox.

Seems to have something to do with shared filesystem - tool monitoring of other directories in that setup works fine. Probably this issue discovered by @kellrott (https://www.virtualbox.org/ticket/10660?cversion=0&cnum_hist=1).
Using from watchdog.observers.polling import PollingObserver Instead of using from watchdog.observers import Observer seems to fix the problem. Just need to get that into Galaxy somehow.

I'm not sure if the tool monitoring is working in the docker image either. Maybe the same issue?

Just pulled latest from hub and it seemed to work for me. You have refresh the browser for the tool to show up - did you try that? Also it appears as though the IDE problem was fixed.
As Galaxy is currently homeless I am doing a build against the following commit:
https://bitbucket.org/jmchilton/galaxy-central-fork-1/commits/19ab1cae127886c06058c09525796140057f0b2b
which adds a new option to watch tools based on polling.
https://github.com/galaxyproject/planemo-machine/compare/watch_test?expand=1
I'll let you know if that build works any better.

Still having a problem. I start the instance with:
docker run -p 8010:80 -v /data/tools:/opt/galaxy/tools planemo/box
Then I log into the IDE and open a terminal and type:
planemo project_init dpc
cd dpc/
planemo tool_init --id dpc
Galaxy logs an event:
watchdog.observers.inotify_buffer DEBUG 2015-02-22 06:24:08,599 in-event <InotifyEvent: src_path=/opt/galaxy/tools/dpc, wd=1, mask=IN_CREATE|IN_ISDIR, cookie=0, name=dpc>
But I don't see any other logging info about tool loading or any other events related to dpc. When I make changes to the file, I don't see any other logs about the watchdog, even when I made changes to the XML file. And the tool never appeared in the browser, even after refreshing the screen.

project_init doesn't really do anything when used that way and not sure the recursive thing works for "new" directories. Just do a tool_init from tools directly and it should work.

Ok that was the different. I've been try to create new project folders in /opt/galaxy/tools or doing git clones of existing tools into the directory.
If its not doing recursive searches on newly created directories, that way my tools were never being picked up.
I ran 'planemo tool_init' in the base directory, and it picked up the tool correctly.
We should make sure to note this in the manual.

Indeed - we should also get recursive addition of directories working - I think it does work for scanning tools right.
Okay - new box - fixes tool monitoring though it has become very expensive to run the machine on my laptop :(.
https://images.galaxyproject.org/planemo/20150222.box
Want to give a shot and let me know if you have any feedback.
Tonight if I have time I will try another pass with following changes -

Just one Galaxy process (instead of separate web and handlers processes)
Fewer tool directories to monitor (trim up the tool panel)
Installing a older version of codebox (0.8.1 instead 0.8.3) - the terminal on the new version seems hosed (what do you think?). It could just be everything is very bogged down in my current setup.

Made those changes and I like them I think - new box available at following locations:
https://images.galaxyproject.org/planemo/20150222.1.box
https://images.galaxyproject.org/planemo/latest.box


[NG] Tree View

Fixes: #239
@youdz and I implemented the TreeView component with the following features and the API mentioned in #239 (with an exception to the last example in the API - selection is of the type TreeSelection[] instead of TreeSelection):
Note that these are not the final examples

Basic Tree Nodes

Tree Node Expanded Input/Output

Building Trees Dynamically

Lazy Loading Tree Nodes

Selectable Tree Nodes

Toggling Individual Tree Nodes

Routing using Tree Nodes

Prepopulating Trees

Added an experimental feature compact trees in addition to the API mentioned in #239. Have to test this on smaller screens before releasing
Please note that this PR is not final. I am creating this PR so that reviewers can get a head start to check out the code and suggest any major changes in the logic. I still need to:

Still need to make styling changes to the component. The classnames are not final and might change after I discuss them with @mathisscott
The tree node examples have to be updated and cleaned up.
Add more tests
Browser test styling
CSS Regression Tests

Tested on: Chrome only. Will browser test this week.
Unit Tests:
Tree View Directives
Tree Node
Tree Node Basics
✔ projects content
✔ projects child tree nodes under .clr-tree-node-children
✔ projects content under .clr-tree-node-content
✔ shows caret under .clr-tree-node-caret only for tree nodes having child tree nodes
✔ has hasChildren set to true for the parent tree node
✔ has hasChildren set to false for the child tree node
✔ has expandable set to false all tree nodes by default
✔ has selectable set to false all tree nodes by default
✔ has expanded set to false on tree nodes by default
✔ toggles 'expanded' on the tree node when its expanded & collapsed
✔ has caretDirection set to 'right' on tree nodes by default
✔ toggles caretDirection on the tree node when its expanded & collapsed
Tree Node Default Expand Option
✔ sets the tree node 'expanded' property to the input provided by the user
✔ emits the 'expanded' event back to the user when the caret button is toggled
Tree Node Expandable Option for Lazy Loading
✔ does not show the caret when expandable is set to false
✔ shows the caret when expandable is set to true
✔ shows the spinner when expandable & loading is set to true
✔ hides the caret when spinner is visible and vice-versa
Building Trees Dynamically
✔ projects content
Basic Tree Node Selection
✔ enables the selection based on the clrTreeSelected input
✔ accepts user input for tree node selection
✔ emits a change event after tree node selection
Parent Child Tree Node Selection
✔ initializes all tree nodes to selected false
✔ initializes using the user input for model
✔ checks all children when the parent is checked
✔ checks all grand children when the parent is checked
✔ sets the parent as indeterminate when the one child is selected while others are not
✔ selects the parent when all the children are selected
✔ emits event when node is selected
Tree View
Pre-populating Trees
✔ pre populates tree based on the tree selection input
Tree Selection
✔ does not emit anything when first initialized
✔ emits an array with all the root nodes when selection is updated
✔ emits an array with right selection

Not being close to this, everything I read makes sense.

What I read looks good. I'll try it out if I have time.


click-command-tree: 1.1.0 -> 1.1.1

Description of changes
https://github.com/whwright/click-command-tree/releases/tag/1.1.1
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[x] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 237209 run on x86_64-linux 1

10 packages built:

pynitrokey
pynitrokey.dist
python310Packages.click-command-tree
python310Packages.click-command-tree.dist
python310Packages.spsdk
python310Packages.spsdk.dist
python311Packages.click-command-tree
python311Packages.click-command-tree.dist
python311Packages.spsdk
python311Packages.spsdk.dist


Bui Cong Nam - Tet Nay Con Se Ve

Song Name
Bui Cong Nam - Tet Nay Con Se Ve
Artist Name
Bui Cong Nam
Source
Youtube
Youtube Link
https://www.youtube.com/watch?v=YGhE2HZ8g8M
Direct File Link
No response
Song ID
827483
Extra
No response

good vn nong song

reject
replaces an unrelated and used song with no popular level for it


For what me need a mysql connection?

Hi@all,
It is unnecessarily created a mysql connection. I have a phpunit test and do the following:
        $data = $this->toArray();

        $model = new AppModel();
        $model->fill($data);

Then I start the test and he is building a mysql connection even though I have never been saved!
PDOException: SQLSTATE[28000] [1045] Access denied for user 'USER'@'localhost' (using password: YES)

In order to run the tests, you need a mysql connection. Just create one named translatable_test and run the tests. See the creds used here: https://github.com/dimsav/laravel-translatable/blob/master/tests/TestsBase.php

Okay
But why i need mysql connection on a fill method?
I think the mysql connection need to be create on ->save method.

Because before each test we reset the migrations to make sure the database is on a clean state: https://github.com/dimsav/laravel-translatable/blob/master/tests/TestsBase.php#L59


Migration to tf2

Before releasing into Noetic, I would like to migrate from tf1 to tf2. I think I managed all required changes, but I'm not sure how to correctly migrate getLatestCommonTime().
The corresponding function in tf2 is private and the public alternative getLatestCommonTime(), which I used for now, is marked as reserved for tf1 compatibility. @tfoote, can you provide a hint? Should the function become public?

The PR LGTM but can you have a look to this line in rviz ? These new changes are breaking the compilation

Of course, I will commit appropriate API adaptions to rviz as well. Actually, I have them prepared for the IM change, but there are other tf2 changes pending for rviz. As soon as all upstream packages required by rviz are released for noetic, I will prepare the rviz release as well (hopefully next weekend).

sounds good, thank you!


build Jansi-Native for aarch64

how to build Jansi-Native for aarch64
I ran following to build Jansi-Native for aarch64
mvn -Dplatform=${platform} package
mvn -Dplatform=aarch64 package
libjansi.so build inside aarch64 is static library for x86-64 platform, how to build it for aarch64

This is question more related to https://github.com/fusesource/hawtjni because jansi only calls maven plugin to generate native code.
Could you try to build with  -Dplatform=arm64 ?


Default image size

When you insert an image it is usually too large in size and you have to reduce it manually.
Is there an option to define the default size in the plugin preferences?

Thank you for the enhancement idea.
The feature added at 1c71eeec41b27af690d7820287bcce272e848d80
For now, I just added a setting for the default image size. In the future, I'd like to make it possible to modify when inserting.

Thanks


relationship between "tab-slug" and "redirect_from" properties, and the filename is unclear / inconsistent.

They should be harmonized / clarified, and possibly merged (i.e. using the filename instead of the tab-slug property, or renaming the files according to the tab-slug, or something else along those lines).
Here's a table of the current values, to allow a better overview:

filename
tab-slug
redirect_from
category

agpl-3.0
agpl-v3
agpl
gpl

apache-2.0

apache

artistic-2.0

artistic

bsd-2-clause
bsd
bsd
bsd

bsd-3-clause
bsd-3

bsd

cc0-1.0
cc0
cc0
pd*

epl-1.0

eclipse

gpl-2.0
gpl-v2
gpl-v2
gpl

gpl-3.0
gpl-v3
gpl-v3
gpl

isc
isc

bsd

lgpl-2.1
lgpl-v2_1
lgpl-v2.1
lgpl

lgpl-3.0
lgpl-v3
lgpl-v3
lgpl

mpl-2.0

mozilla

ofl-1.1

ofl

unlicense
unlicense

pd*

* - shortened from "Public Domain Dedication" to make the table shorter. See #289 for a proposal to change this.

Looking at the "name" and nickname properties, one can also notice a trend:
Additionally, it seems that where the nickname is the same as the tab-slug or redirect_from, it is not included, the exception being the Apache license. So either the nickname should be removed in that case, or added to all licenses regardless of coinciding with the tab-slug/redirect_from version.

nickname
title

GNU Affero GPL v3.0
GNU Affero General Public License v3.0

Apache
Apache License 2.0

                | Artistic License 2.0

Simplified BSD      | BSD 2-clause "Simplified" License
New BSD             | BSD 3-clause "New" or "Revised" License
CC0 1.0 Universal   | Creative Commons Zero v1.0 Universal
| Eclipse Public License 1.0
GNU GPL v2.0        | GNU General Public License v2.0
GNU GPL v3.0        | GNU General Public License v3.0
| ISC License
GNU LGPL v2.1       | GNU Lesser General Public License v2.1
GNU LGPL v3.0       | GNU Lesser General Public License v3.0
| MIT License
| Mozilla Public License 2.0
| No License
| SIL Open Font License 1.1
| Open Software License 3.0
| The Unlicense
| "Do What The F*ck You Want To Public License"

To clarify what the different fields do:

Filename - The SPDX ID
Redirect from - Any previous names the license has had (generally pre-SPDX compliance) to prevent breaking old URLs
Category - means of grouping similar licenses (e.g., GPL, BSD)
Tab slug - For the UI for the grouping, we need a URL-friendly means of describing / linking to the license. Jekyll doesn't easily expose the filename/spdx id, so we need to expose our own.
Nickname - shorter, human-readable version of the license name so that it better fits in the UI / matches expectations

Thanks for the clarifications. I guess I would then suggest merging redirect_from into tab-slug, calling it simply slug, and automatically redirecting from the slug string for all licenses. This would preserve all old links except "lgpl-v2.1".
By the way, I'm not sure whether this would cause problems where the actual permalink equals the current tab-slug value, e.g. "isc", "unlicense". But it doesn't sound like something that would be hard to address.
We could also redirect from the all the slug values with _replaced by . which would solve the lgpl-v2.1 broken link, but I'm not sure it's worth the extra complexity to save a single link, when all others would keep working normally.

Thanks for the clarifications. I guess I would then suggest merging redirect_from into tab-slug, calling it simply slug, and automatically redirecting from the slug string for all licenses.

I think this would confuse things more than it would solve them. The slug, is technically the filename (what's seen in the URL). The redirect_from field is a dummy page that Jekyll creates to redirect users from the old URL to the new URL. Further, not all posts with tab-slugs have redirects and vise versa. What are you trying to solve for here?

The slug, is technically the filename (what's seen in the URL)

True, and the fact that these would automatically redirect actually makes that association work, even if they aren't the canonical url. It's a pity that html IDs can't contain ., otherwise we could harmonize all three :)
In fact, we still can, if we swap things and make the current canonical urls into redirects. To prevent having to define the correspondence manually for each license, we'd need an unambiguous mapping between the slug and the current filename, which requires including the v in the canonical slug, but not the period. We could then make automatic redirects with (1) the v removed, (2) the period added, and (3) doing both. All current URLs would still work, and we'd only have a single "slug" property defined per license, which would be the same as the filename, and serve as its tag-slug if necessary.
I'm not sure if I'm explaining this well, but I can make another table showcasing how this would look like. It would be a simpler and more consistent model, with less special cases.

not all posts with tab-slugs have redirects and vise versa. What are you trying to solve for here?

Exactly, the metadata model is more complex than necessary IMO, and this makes understanding/manipulating the properties in the licenses harder. For instance, when a new license is added, or if someone is using this data elsewhere. The increased number of properties and the very fact that they are only present for some licenses (as you point out) are avoidable.

It's a pity that html IDs can't contain ., otherwise we could harmonize all three :)

You could use the replace liquid filter:

replace - replace each occurrence e.g. {{ 'foofoo' | replace:'foo','bar' }} #=> 'barbar'

I don't see need for nickname going away, and redirects are needed to not break links. tag-slug is gone and the documentation is a bit clearer. Is there anything else that can be done to address the lack of clarity you noted, @waldyrious? If you think so, please re-open and explain. Thanks!

I'll have to recreate a table like the one on top, to see if there's any redundancy or opportunities to automate partial patterns. I will reopen if I manage to do so.


SAK-29694 Filtered Requested Sites

I have added the Jira reference to the title to conform to general practice.

Thanks for the patch. For doing an intersection in Java you can use Collection.retainAll(Collection).

@steveswinsburg I am new to github and struggling. Can you please confirm whether the reverts are fine? Please help me out.

Yeah that line that was added is now gone, but after this PR you might want to read up on git and how it treats the commits.
Last outstanding bit is to fix up the iteration code and then we can get this in. Thanks for your hard work on this!

I've taken your commit and added a test for it on a new pull request #1253 and cleanup up the indentation.
You're still the author on the fix.

Thanks for digging into this and finding the problem.


Show more S3 logs and less verbove WAL logs

During https://zenith-esr3609.slack.com/archives/C033A2WE6BZ/p1648492609014039 log explorations, I've stumbled onto a few verbose places I'd decided to suppress:

After filtering out the previous message:

Also, even without both of the logs, no S3 logging was found despite it was one, so add some basic logging there.


Can I convert nnU-Net pytorch model to MLX?

Hi all!
Is it possible to convert a medical image segmentation model trained using the nnU-Net framework and stored as a .pth file into an MLX compatible format?
Thanks!

My guess is some of the U Net wants  are not well supported yet in MLX (pooling / upsampling / transpose conv).  We will add those ops in due time but some of them require new backend kernels to be reasonably efficient so it's not a very quick fix.

Just a follow-up: when trying to run the trained nnU-Net model directly using pytorch using torch.device("mps"), I get ConvTranspose 3D is not supported on MPS.
  File "/Users/valosek/miniconda3/envs/nnunet/lib/python3.9/site-packages/dynamic_network_architectures/building_blocks/unet_decoder.py", line 109, in forward
    x = self.transpconvs[s](lres_input)
  File "/Users/valosek/miniconda3/envs/nnunet/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/Users/valosek/miniconda3/envs/nnunet/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/valosek/miniconda3/envs/nnunet/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 1104, in forward
    return F.conv_transpose3d(
RuntimeError: ConvTranspose 3D is not supported on MPS

Related to https://github.com/pytorch/pytorch/issues/130256

Ok.. well that's a PyTorch issue. In MLX we have a 3D convolution, so that should work


[zh] Sync /pods/init-containers.md

Sync one of en #43346:
content/zh-cn/docs/concepts/workloads/pods/init-containers.md

See preview please.

Happy New Year! Could we move this forward after months have passed?
@tengqm @my-git9

/lgtm


Consider publishing images to Amazon ECR Public Gallery

Issue/Feature description
The images are available on https://hub.docker.com/u/chainguard, but not on Amazon's public cache:
https://gallery.ecr.aws/chainguard
Consider publishing them there as well? Great free option for users who easily get rate-limited on DH:
https://docs.aws.amazon.com/AmazonECR/latest/public/public-getting-started.html

I believe we are looking into this! I can't say anything about timeline.

Thanks!


Speed up TSDataset.to_flatten

🚀 Feature Request
Make TSDataset.to_flatten faster.
Motivation
Now there is cycle over segments, which seems slow. In some practical cases almost half of the time was spent on select_dtypes inside this cycle.
Proposal
Make TSDataset.to_flatten vectorized.
Prototype:
import pandas as pd
from etna.datasets import generate_const_df, TSDataset

regressors_timestamp = pd.date_range(start="2021-06-01", periods=50)
df_regressors_1 = pd.DataFrame(
    {"timestamp": regressors_timestamp, "regressor_1": 1, "regressor_2": 2, "segment": "segment_0"}
)
df_regressors_2 = pd.DataFrame(
    {"timestamp": regressors_timestamp, "regressor_1": 3, "regressor_2": 4, "segment": "segment_1"}
)
df_exog = pd.concat([df_regressors_1, df_regressors_2], ignore_index=True)
df_exog_ts_format = TSDataset.to_dataset(df_exog)

# current implementation
df_exog_flattened = TSDataset.to_flatten(df_exog_ts_format)

# new implementation prototype
df_unstacked = df_exog_ts_format.unstack().reset_index()
df_exog_flattened_alternative = df_unstacked.pivot(index=["timestamp", "segment"], columns=["feature"], values=[0])[0].reset_index()
df_exog_flattened_alternative.columns.name = None

Test cases

Make sure that existing tests work correct.
Add test with regressors.
Add test with category dtypes.

Alternatives
No response
Additional context
No response
Checklist

[ ] I discussed this issue with ETNA Team

Version in #475 is up to 3 times faster then original and seems to have very low changes. Test for performance
def test_speed_flatten():
    # vectorized (broken): [1.38, 1.33, 1.35]
    # old faster: [11.76, 11.70, 11.98]
    # old: [33.28, 33.88, 36.35]
    import time

    df = generate_ar_df(start_time="2020-01-01", periods=100, n_segments=10_000)
    df_exog = df.copy()
    df_exog.rename(columns={"target": "exog_1"}, inplace=True)
    df_exog["exog_2"] = [1, 2, 3, 4, 5] * (df_exog.shape[0] // 5)
    df_exog["exog_2"] = df_exog["exog_2"].astype("category")

    ts = TSDataset(df=TSDataset.to_dataset(df), df_exog=TSDataset.to_dataset(df_exog), freq="D")

    time_start = time.time()
    df_flat = ts.to_pandas(flatten=True)
    time_finish = time.time()
    elapsed_time = time_finish - time_start
    print(f"{elapsed_time:.2f}")

Implementation at #475 running times are labeled with "old faster".
There is also a prototype vectorized implementation which gives times around 1.5s which is much faster then current solution. However, it has some problems with the second test. It leads to conversion of int category values to float category values.
Vectorized prototype implementation:
# cast categories to object
dtypes = df.dtypes
category_columns = dtypes[dtypes == "category"].index.get_level_values(1).unique()
# flatten dataframe
df_flat = df.stack(level=0, dropna=False).reset_index().sort_values(by=["segment", "timestamp"]).reset_index(drop=True)
# mark category as category
df_flat[category_columns] = df_flat[category_columns].astype("category")
df_flat.columns.name = None
return df_flat

The problem occurs when we try to call stack: it leads to conversion of category values to int, which as a result leads to conversion to float because of NaNs in the beginning of one of the segments. Possible solutions:

Use conversion of category dtypes to object before calling stack. It solves the problem, but running times become about 30s, which gives us no improvement.
Somehow change int64 to Int64 dtype before calling stack and revert it later. The second dtype can handle NaNs without casting to float64.

Pandas PR that can be relevant.

I consider this task closed since we are not going to touch this issue in a near future.

I consider this task closed since we are not going to touch this issue in a near future.


Add assert to castTo

Alternative considered:
Instead of assertion, we could add a unsafe = false named parameter to castTo and throw an JClassCastError instead.
Reason against going with it: This forcefully adds a call to instanceof but with assertion, we don't have any overhead in production while still getting the benefit while development.
Closes #986.

coverage: 92.009% (-1.7%) from 93.677%
when pulling 4a856258e73b453f5b5642000ca1019c2d2d7a1f on safe-cast
into 71ada4b5c8001e6b8207ed40331d158e0912cd94 on main.


Add LogJuicer artifacts for failed build

This change adds a new build artifacts for failed job to help debug build failure.
Fixes: https://github.com/packit/dashboard/issues/375

You can see the new artifacts being proposed in https://softwarefactory-project.io/zuul/t/packit-service/build/63afb3dd29824fd78ce84be8a0f07a3d/artifacts .


Update Promise/Symbol/fetch polyfills for Edge

Probably need to update some other APIs as well, but this is at least a start. Fixes #736.

Since Edge 13 is a browser we already test in CI, support is actually already indicated on our feature matrix.  Looks like it works in 13, but not in IE7:

If you want to make the polyfill work in IE7, great, otherwise you probably want to start targeting at IE8.
To merge a change like this we normally ask for you to follow the simple steps in
https://cdn.polyfill.io/v2/docs/contributing/common-scenarios#adding-browsers. But in this case both browsers concerned are actively tested by our CI, so the testing steps are not required.

Whoops, my bad. I guess I hadn't thought about IE < 8.
Question, though: if the polyfills don't work at all in IE7, then why even specify it? Are people supposed to BYOP (bring your own polyfill :wink:) for IE <8?

Many of the polyfills do work in IE7, but this one doesn't.

Tests for this are running at - https://circleci.com/gh/Financial-Times/polyfill-service/312

It passed! 🎉

Great.  Thanks for this.  It will be released in 3.8.

Awesome, thanks for the help and for the great service! 💃


Loading netCDF file with long_name attribute, iris ignores variable name

I have a netCDF with several variables, each of which has a long_name, attribute.
When I try to load the file so:
cubes = iris.load(nc_file, variable_name)
This works only if variable_name is long_name and not if I use the variable name assigned to that variable.

This is by design, see the user guide section on constrained loading, in particular the section on how name is defined. Iris supports many file formats but the var_name attribute is specific to the netcdf interface, and therefore we use the more general concept of name to refer to cubes.
It isn't actually all that easy to load based on variable name, but if you know the value standard_name or long_name then you won't need to.

Thanks! this is very helpful information.


Support for Internet Explorer 11

The chat widget is not displayed in IE11.

Can someone help me with the Internet explorer issue?

found the resolution. for those looking for the solution. You can do this in 2 ways,

You can add transpileDependencies for "msgdown", "escape-goat", "vue-beautiful-chat".
You can change the code to es5 in these modules.


Add option product attribute to a product.

I cannot find anywhere an example how to add an options ProductAttribute to a Product object.
A code example like this one for entity here would be great:
http://django-oscar.readthedocs.org/en/latest/topics/modelling_your_catalogue.html#product-attributes

you can add product attribute in product type from catalogue menu
example above depicts same

Thanks but I am not looking for any UI based solutions. I want to do this via code. I found out how to do it in the meanwhile, I'll create a PR for the documentation to add that as well.


Cross Domain Issue

This is more like an question for you.  As from the documentation it seems that it is compatible with cross domain iframes. My question is what if i want to use a third party domain. I mean to say a domain of which i do not have any control like let say i want to include the gtihub.com in the iframe. Will it be able to re-size the height of the iframe to be the exact same of height of github.com  so that i do not have a vertical scroll bar. If yes then can you please let me know how to do this.
Thanks
Adi

Nope require JS at both ends.


PathInfo regex should not match on GET parameters

I'm of the opinion that routes should not take GET parameters into consideration when matching. Currently, the PathInfo is checked against RawUrl which includes GET parameters. This makes it hard to write effective routes, because order of parameters matter.
My routes are all starting to take this kind of shape, especially since they will be used in JSONP calls:
[RESTRoute(Method = HttpMethod.GET, PathInfo = @"^/poll/(\?[^/]*)?$")]

If users need to use GET parameters, then they should be extracted from within the route handler and validated there. NotFound can be returned on missing parameters.
Unfortunately, there aren't any properties that I can find on context.Request that expose just the path portion of the URL. If this is accepted, I'd probably split on ? and take the first match.
Note that I'm actually using GrapevinePlus (great work by the way), but I figured that issues relating to both should be raised here.
Thanks.

+1

Note that I'm actually using GrapevinePlus (great work by the way), but I figured that issues relating to both should be raised here

You are absolutely right. Issues that relate to both Grapevine and GrapevinePlus should be raised here.

While it certainly isn't uncommon for GET request to contain query string parameters, let's first make sure we are using proper URI design. To quote from O'Reilly's RESTful Web Services, page 233, under the header URI Design (emphasis mine):

When designing URIs, use path variables to separate elements of a hierarchy, or a path through a directed graph. Example: /weblogs/myweblog/entries/100 goes from the general to the specific. From a list of weblogs, to a particular weblog, to the entries in that weblog, to a particular entry. Each path variable is in some sense "inside" the previous one.
Use punctuation characters to separate multiple pieces of data at the same level of hierarchy. Use commas when the order of the items matters, as it does in  latitude and longitude: /Earth/37.0,-95.2. Use semicolons when the order doesn't matter: /color-blends/red;blue.
Use query variables only to suggest arguments being plugged into an algorithm, or when the other two techniques fail. If two URIs differ only in their query variables, it implies that they're the different sets of inputs into the same underlying algorithm.

The big takeaway here is that our URIs, generally speaking, should only use the query string to pass arguments to an algorithm. If, indeed, we are expecting query string parameters in our methods, those should be validated by the method the request is routed to, and our PathInfo regular expressions should reflect the possibility of receiving such arguments.
Example : When not to use query string
Suppose you wanted to request user data given a specific numeric user id, let's say 632. This is a scenario where it might be tempting to use the query string, but it would be better not to.

Less Correct URI
More Correct URI

/user?id=632
/user/632

The RESTRoute for the more-correct URI would then look like this:
[RESTRoute(Method = HttpMethod.GET, PathInfo = @"^/user/\d+$")]
public void GetUser(HttpListenerContext context)
{
	// GrabFirst is a String extension method that comes bundled with Grapevine
	// See https://github.com/scottoffen/Grapevine/wiki/Getting-Started-Syntactic-Sugar
	var userid = context.RawUrl.GrabFirst(@"^/user/(\d+)$");

	if (!object.ReferenceEquals(userid, null))
	{
		// you got your user id, do something
	}
	else
	{
		// invalid or missing user id
	}
}

Example : Correctly using the query string
If you wanted to create a REST route that multiplied two integers together, then - setting aside that URI are suppose to represent resources, not operations on a resource - using a query string might be more suitable.

Probably Less Correct URI
Probably More Correct URI

/2/product/3
/product?x=2&y=3

The RESTRoute for the probably-more-correct URI would then look like this:
[RESTRoute(Method = HttpMethod.GET, PathInfo = @"^/product")]
public void MultiplyTwoIntegers(HttpListenerContext context)
{
	var x = context.Request.QueryString["x"];
	var y = context.Request.QueryString["y"];

	// I'll let you verify the inputs and do the math.
}

Note that the PathInfo regular expression omits the trailing $, which would normally indicate the end of the string, and we let the route method handle getting the parameters passed. If we really wanted to be a stickler, we could also write it like this:
[RESTRoute(Method = HttpMethod.GET, PathInfo = @"^/product\?.+$")]

Which would ensure that it at least looked like there might be some parameters coming in the query string, but isn't really necessary since we are going to do that checking anyway. If you are accepting an arbitrary number of parameters, we probably want to abstract the parameter retrieval out to another method.
If I've misunderstood your issue or if this example doesn't make sense, let me know. Meanwhile, although I agree with your assertion that

[...] routes should not take GET parameters into consideration when matching.

This should be solved by properly constructing our URIs and regular expression, not via a code change or enhancement.

Let me provide a use-case that probably shouldn't be too uncommon; optionally supporting JSONP.
/weblogs/myweblog/entries/100/?callback=jsonp
/weblogs/myweblog/entries/?callback=jsonp
Now we could use CORS rather than JSONP for most browsers, but let us assume that JSONP is a viable method for cross domain ajax.
[RESTRoute(Method = HttpMethod.GET, PathInfo = @"^/weblogs/myweblog/entries/\d+/")]
[RESTRoute(Method = HttpMethod.GET, PathInfo = @"^/weblogs/myweblog/entries/")]

The problem above is that we explicitly need to order the methods in order to work around the lack of a trailing $ as a terminator. We can't have a terminating $ because there are now GET parameters we need to support. In this contrived case we only have a single GET parameter, but it's quite easy to imagine a situation where there are two or more.
If these URLs are in different types, then you're at the mercy of reflection loading the types in the correct order.
As a small data point, I'm unaware of any other routing library that includes query strings in route matching. I have not done an exhaustive search to confirm this though since it doesn't really matter too much.
I don't see any value added by including the query string in the PathInfo match, yet I see quite a lot of value in excluding the query string. I don't think the (simplified) argument of get parameters aren't that useful to real RESTful webservices is compelling in any way.

That's a fair point. Thank you for taking the time to explain a use case I had not considered; I now have a better understanding of the value add of this type of change.

Thanks! I appreciate you reconsidering.

Might I add that query parameters are required if the supplied values might contain forward slashes (ie, using names instead of numbers as unique IDs) . Even urlencoded, slashes tend to get unencoded en route/by the web server/etc, so am currently using url parameters in places where they do not represent queries or metadata. :(


Problem in changing the spreading factor SF to for instance SF12

I could not change the spreading factor to for instance SF12.
Solution so far:
1,) Set _CAD=0. This is documented but can easily be overlooked.
2.) In the main .INO file called "ESP-sc-gway.ino", the following lines can interfere with the SF factor:
if (gwayConfig.sf != (uint8_t) 0)
{
  sf = (sf_t) gwayConfig.sf;
}

The lines can be found in the function/method WlanReadWpa(). This function not only reads the WiFi settings from the SPIFFS, but also LoRa settings.
I do not know if I had old LoRa settings in my SPIFFS, but even after deleting these, existed the problem. BE AWARE to eliminate this potential error also if you have the same trouble as I. Deletion could be done by deselecting a SPIFFS and flash. Then reselecting having a SPIFFS and flash.
The following code works for me right now. This is though just a short hand fix.
// Fix so that I can change to SF12
gwayConfig.sf = 0;

Serial.print("Before checking gwayConfig.sf != (uint8_t) 0 - SF=");
Serial.println(sf);

if (gwayConfig.sf != (uint8_t) 0)
{
  sf = (sf_t) gwayConfig.sf;
}

Serial.print("After checking gwayConfig.sf != (uint8_t) 0 - SF=SF");
Serial.println(sf);

To me, it seems that gwayConfig.sf is not set at all. I mean not defined at all, and hence does the program set a default value (SF7).
I do not know what the supposed functionality has to be, but reports my findings for others to change them - according to wished use.
Regards

I hope your problem is solved in later versions.


Custom controller for reset password using pow

I am using the reset password extension.
As far as I know I can reset a token using this code PowResetPassword.Plug.create_reset_token(user_params)
But how can I send an email after token reset?

You can find an example here: https://github.com/danschultzer/pow/issues/61#issuecomment-451779111
In short, call PowResetPassword.Phoenix.Mailer.reset_password/3 and Pow.Phoenix.Mailer.deliver/2:
  defp deliver_email(conn, user, token) do
    url = reset_password_url(conn, token)
    email = PowResetPassword.Phoenix.Mailer.reset_password(conn, user, url)

    Pow.Phoenix.Mailer.deliver(conn, email)
  end

  defp reset_password_url(conn, token) do
    Routes.pow_reset_password_reset_password_url(
      conn,
      :edit,
      token
    )
  end

@danschultzer Thank you for your quick response!
I'm trying to integrate with Phoenix LiveView and there is no conn (only socket value) there,
do you happen to know how  to handle it?

The issue I linked with https://github.com/danschultzer/pow/issues/61#issuecomment-451779111 has the example without conn. It’s for absinthe integration.
You may be able to just pass in the socket though, since it closely relates to conn. You just have to set the config with `Pow.Plug.

Thanks @danschultzer ! It works like a charm :)


Documentation/exrtact definitions

Thinking about separating view and data for documentation to give us some more flexibility in the future, and force the style to be consistent. What do you think?
It looks as expected:

I like it, and the types you defined seem to be generic enough so that they can also be used for the other documentations (like dump) that are due in the future.
Two notes:

there is a type in the branch name, so rejected
it is very in-character for you to add a change like this when only a few lines of documentation was expected (which is a good thing of course, you boyscout :))

I'd say, I reject your reason to reject the PR, because you have a typo in it.

I actually even tried a version using reflection today to get the signatures and stuff out of the functions themselves, but getting the  'nice signature' is actually quite cumbersome (ended up reading how it is done in the Ionide repo for the inline code hints, hehe), so I'd say, let's stick to a bit of manual labor :)


[Feature]: Add more allowed images domains for Github

Related Problem
I tried to use images I uploaded on a GitHub wiki page in my readme, but the URLs seem not to be accepted.
The one that is already in the alowed domain list that is nearest to what I need is :
https://user-images.githubusercontent.com/...image
But when I drop the image in the wiki, the link I get at first is:
https://github.com/User/Repo/assets/...image
Or if I get the image address again later:
https://private-user-images.githubusercontent.com/...image
Both versions seem to be valid
The Elevator Pitch
Does it make sense to add the following domains to the allowed domain list?

Add https://github.com/...image
(it sems to be ok for https://gitlab.com)
Add https://private-user-images.githubusercontent.com/...image
(since it sems to be the default way GitHub now stores these images internally??)

Additional Context and Details
A bit off topic:
I would also be happy if someone could point me to how I upload an image on GitHub as non private?
I sadly found nothing about this.

@Grille
Could please list of all of those image domains?
@lyndaidaii
Could you provide us playbook on how to determine which ones are ok or not?

We do have a playbook for this.

Sorry, I’m not entirely sure what you asking? @erdembayar
This is the full URL of one of the images in question, in case you meant that.
https://private-user-images.githubusercontent.com/26384012/333878765-b4541b00-b4d4-4e0e-a9a6-983a1ff05d5e.gif?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTcwMTUzMTYsIm5iZiI6MTcxNzAxNTAxNiwicGF0aCI6Ii8yNjM4NDAxMi8zMzM4Nzg3NjUtYjQ1NDFiMDAtYjRkNC00ZTBlLWE5YTYtOTgzYTFmZjA1ZDVlLmdpZj9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA1MjklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNTI5VDIwMzY1NlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTE5NjQ2ZGRjNWJlOGI2NWQ2YTdmZDgwODRkMjg5ODNhMTBjY2Q2YzdkYTlkZWI3YjBiMmIwNjFmYjgxYWE3ZjMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.hLFEU-lRPPcQs5mGxUSSNCov_JzUlSXWtMRBd22yc7o
Btw, I resolved my personal problem by using https://raw.githubusercontent.com/ instead.
But adding https://private-user-images.githubusercontent.com/ in general may still makes sense?

Sorry, I’m not entirely sure what you asking? @erdembayar
This is the full URL of one of the images in question, in case you meant that.
https://private-user-images.githubusercontent.com/26384012/333878765-b4541b00-b4d4-4e0e-a9a6-983a1ff05d5e.gif?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTcwMTUzMTYsIm5iZiI6MTcxNzAxNTAxNiwicGF0aCI6Ii8yNjM4NDAxMi8zMzM4Nzg3NjUtYjQ1NDFiMDAtYjRkNC00ZTBlLWE5YTYtOTgzYTFmZjA1ZDVlLmdpZj9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA1MjklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNTI5VDIwMzY1NlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTE5NjQ2ZGRjNWJlOGI2NWQ2YTdmZDgwODRkMjg5ODNhMTBjY2Q2YzdkYTlkZWI3YjBiMmIwNjFmYjgxYWE3ZjMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.hLFEU-lRPPcQs5mGxUSSNCov_JzUlSXWtMRBd22yc7o
Btw, I resolved my personal problem by using https://raw.githubusercontent.com/ instead.
But adding https://private-user-images.githubusercontent.com/ in general may still makes sense?

I couldn't open the image link above. If it's not accessible to anyone except you, then how could it be used inside the package?

Hm, did the link change?
Does GitHub create a new link every time the image is accessed on the wiki or something?
Here is the wiki page:
https://github.com/Grille/2D-isometricRenderer/wiki
And the current link:
https://private-user-images.githubusercontent.com/26384012/333878765-b4541b00-b4d4-4e0e-a9a6-983a1ff05d5e.gif?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTcwMTcxNzgsIm5iZiI6MTcxNzAxNjg3OCwicGF0aCI6Ii8yNjM4NDAxMi8zMzM4Nzg3NjUtYjQ1NDFiMDAtYjRkNC00ZTBlLWE5YTYtOTgzYTFmZjA1ZDVlLmdpZj9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA1MjklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNTI5VDIxMDc1OFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTlmYjYxMTZmODI3OGZkM2YxNDk1OWViMTdjMDZjNjQ0MDYwYTcwMmIyMzc1YTAzMjg2NzVlZDQ4YmE2YTA4OTgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.8lGtf9aGE6BiB6KiBMTgSN7D8Q3v7XNCAWj3TrA4Rrw

@erdembayar
Yea tested a bit, and the link stays public only for a short time, after that the image link can only be opened by the owner.
following the image link on the wiki page gets you a new link then.
So that makes anything under https://private-user-images.githubusercontent.com/ obviously useless as an image link in a package.
on the other hand, the link you get on upload sems to stays valid indefinitely and forwards to an temporary public https://private-user-images.githubusercontent.com/ link, so these could probably be used in a package.
here an example of such a link:
https://github.com/Grille/RetroLine3D/assets/26384012/a0192c26-a4de-4b94-acc2-4e028eb95151

on the other hand, the link you get on upload sems to stays valid indefinitely

What is the sems in this context?

forwards to an temporary public https://private-user-images.githubusercontent.com/ link, so these could probably be used in a package. here an example of such a link:
https://github.com/Grille/RetroLine3D/assets/26384012/a0192c26-a4de-4b94-acc2-4e028eb95151

I still don't understand what is the connection between  https://private-user-images.githubusercontent.com/ link and https://github.com/Grille/RetroLine3D/assets/26384012/a0192c26-a4de-4b94-acc2-4e028eb95151
If it's hard to reason or implement, there are several other options already available from GitHub. No sure benefit of adding this one.

What is the sems in this context?

"seems" At this point I’m pretty sure, the example link I gave is still accessible (not only by me)
The connection is that if you upload an image on GitHub wiki/readme you get an https://github.com/User/Repo/assets/ image link.
That link doesn’t directly contain the image but forwards to a temporary https://private-user-images.githubusercontent.com/ link.
The first link could be used from NuGet (if NuGet can handle being forwarded).
The second link cannot be used since it’s just a temporary link.
So my proposal at this point is adding https://github.com/ just as https://gitlab.com/ is already added.
To be able to use any images that are under the GitHub domain.
That I mentioned https://private-user-images.githubusercontent.com/ in the beginning, was based on a misunderstanding on my side on what that link actually does, sorry for that.


[question] How to use html-webpack-plugin with express jade view engine?

How to use html-webpack-plugin with existing express templates?
I've installed the express-generetor npm module and like how the project is structured with the jade files used as views using the express view engine. However I also like to use the HTML-webpack-plugin which generates its own files, what is the best way to go about integrating both approaches?

Please move this question to stackoverflow :)

@jantimon , sorry to re-open this but that question is on stackoverflow with no response:
http://stackoverflow.com/questions/39501555/how-to-use-html-webpack-plugin-with-existing-express-templates
After a lot of searching, I can't find a definitive answer on whether this is even possible. Any chance we could get a hint or a definitive no? Been trying to get this work for a few hours now and I can't get this to work at all with express view engine 😅 It works perfectly if you just need to inject in a public index.html file.
Right now my only workaround is to go back to using gulp just for this specific part :(

I don't have the time right now to dive into the express generator so I can't help you on that. Sorry :(


Old TDs in AURORAL project throw validation error since last update

Due to the change on the discovery link for the WoT specification, all old TDs are throwing validation error.
Is it possible to update them somehow? Rather than delete old TDs?

Solved in release v0.2.10


click.command and build binary file over pyinstaller

How to assemble a script using click.command to make it work?

Check out briefcase https://github.com/pybee/briefcase it could work for your use case. Please post a more specific example of what you are trying to solve, and reopen if you are still having trouble


Enhancement to the import command

The wattpm import command now behaves like this:

When doing wattpm import rootDir local/folder/somewhere, the folder is added in the services of the root watt.json. If the target folder has no watt.json, it also makes sure one exists and that the proper package is installed in package.json.
When doing wattpm import rootDir http://..., the URL is added in watt.json.
When doing wattpm import rootDir, the command scans for all services (autoloaded or explicitly specified in the watt.json) and performs the same checks made in 1.

Why is wattpm import rootDir and not wattpm import everywhere?

watt tests are failing


Table "encrypted_keys" does not exist error during init db command.

I try to set up lemur from scratch (on empty db).
When i run python manage.py init -p password then I see following error in logs when migrations scripts are executed:
lemur-web_1   | sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) table "encrypted_keys" does not exist
lemur-web_1   |  [SQL: '\nDROP TABLE encrypted_keys']

When I remove all files from migrations/versions folder then init set up database without any errors.

Yes, I have an issue open to remove those old migration files as it's causing an issue for quite a few people. #221


secure deallocations from cx_table

adding a new allocation to cx_table is clear and secure. returning the index of the new allocation is fine.
when an entry in the cx_table is to be de-allocated, we have problems. it seems we must mark the entry as invalid, and trap on use to detect errors. however, this means the cx_table will quickly fill if there is any expectation from software to allocate/deallocate.
further, if we ever mark a cx_table entry as valid again (reusing the same entry contents, ie same context id), then do we need to make sure that any past cx_index values cannot be again reused? or does the OS simply clean the context state (security reasons)? can an application allocate cx_values, and then deallocate them, only to have them allocated by another process causing a security leak when the original application attempts to reuse deallocated cx_values?
this part of the security model is flawed.

I disagree this issue identifies some security flaw within the privileged access control design in the spec.
The spec defines bare mechanism, but not policy, nor user programming models, for privileged access control to CX state contexts. Also it briefly covers CX resource managers and what happens at context save/restore with respect to CX state contexts. Absent much more detail on these topics, it is easy to conclude there is some security flaw. But that depends on how software programming models apply the bare mechanism, and how operating systems implement these programming models.
First, what must happen when a privileged resource manager (e.g., an OS) must securely revoke access from a user process or thread or hart to a CX state context that it had previously granted and provided indirectly to the user software as a CX selector index?

The OS writes an invalid selector value into the corresponding entry in the hart's CX selector table (or tables plural if access to the CX state context has been granted across multiple harts).
If a hart's current CX selection (i.e., cx_index) is being revoked, the OS also writes the invalid selector value into the hart's mcx_selector CSR.

If the CX state context access revocation is done synchronous to the user thread (e.g. system call or interrupt) this is easy. But if the revocation originates asynchronously to the hart, e.g. from another thread, the hart must be interrupted to do #2. Indeed, revoking access to a CX state context is similar to page table entry changes, and in the worst case may require the analog of a TLB shootdown -- depending upon whether the CX programming model allows a CX state context to be shared by multiple threads within one process, and/or allows a thread that is using CX state context(s) to be rescheduled to a different hart.
Anyway, once the cx_table(s) entry and the hart(s) mcx_selector CSR are set to the invalid selector, as described, any subsequent attempt to access the revoked CX state context, or to reselect (csrrw cx_index,index) and access the revoked CX state context, all fail, setting an error bit in cx_status as desired. Once access is revoked, no access is possible. No security flaw here.
Now the issue above seeks "trap on use". The current spec does not ever trap on any CX error. The CX error mechanism is currently to set cx_status error bits. This option instead of taking an exception was intentionally adopted to keep the CX hardware simple and frugal. That said, please see issue #19 where this design is being reconsidered.
If we ever do trap on CX error, the revocation process described above will also trap on first use (first custom instruction, first custom CSR access) after revocation, as desired.
The next part of the issue pertains to reclaimation and reuse of CX selector index values. Again, this is a policy rather than mechanism issue.
First note that the spec'd mechanism is per-hart cx_tables. Not per system, per process, or per thread. (You can share them across threads in a process or across processes in a system, but you don't have to. Mechanism vs. policy.) So we have to be careful and precise when we talk about "allocated to another process" because each process will (best practice) have its own distinct cx_table and thus its own disjoint scope of CX selector indices.
Moving on, in one reasonable programming model, OS managed CX selector indices are analogous to OS managed file descriptors. The programming model for these is: open a file, receiving a file descriptor. Perform file operations, specifying the file descriptor. Deal with errors. Close the file, releasing the file descriptor. Be not surprised when a subsequent open file call returns a previously released file descriptor value. And if you never close file descriptors you opened, eventually you will run out of file descriptors.
Similarly, for CX state contexts, the programming model might be: open (request) a CX or a particular CX ID, receiving a CX selector index that identifies that CX and its state context. Perform CX custom instructions (first selecting that CX by CSR-writing cx_index). Deal with errors (CSR-read cx_status). Close (deallocate) the CX, releasing its CX selector index. Be not surprised when a subsequent request to open some CX returns a previously released CX selector index. And if you never close CX selector indices you opened, eventually you will run out of CX selector indices.
I do not believe whether the programming model for resource management is cooperative or adversarial (I have it til I release it vs. the resource manager (OS) can take it away from me at any moment) changes any of this. You can lose access to a CX state context and be left with holding a tombstoned CX selector index (note: with no access to the CX state context you used to be able to access), an index that is nonetheless yours until you return it to the OS. And just as with recycled OS file descriptor values, there is no surprise, no problem, and definitely no security flaw when a subsequent request for a CX returns a recycled CX selector index.
In summary, I believe the right programming models and the right OS implementations can use the spec'd mcx_table / cx_index mechanism to provide the desired privileged access control including secure revocation of access.

The issue also talks of reuse of cx_values (not a term defined in the spec). Per-hart mcx_table entry indices can be reused, as discussed above. But CX state contexts themselves can also be reused and time-multiplexed. Consider a simple system with one CX/CXU with one state context and one hart, running a multiprogrammed OS, scheduling two software threads to the one hart. It is possible for the two threads to each enjoy multiprogrammed access to the one CX state context.
The first thread is scheduled to hart # 0 and requests access to the CX. The OS grants this, initializes (resets) its CX state context # 0 (IStateContext::cf_write_status), sets up the hart's mcx_table to address the first thread's CX selector table with entry # 0 for that cxu_id=0 and state_id=0, and returns CX selector index # 0. The thread selects that CX (csrw cx_index, index) and peforms various CX custom instructions.
The timeslice expires. The OS saves the first thread context, including the thread's CXs' state contexts state (IStateContext::cf_read_{status,state}) and its current cx_index and cx_status.
The second thread is scheduled to hart # 0 and requests access to the same CX. The OS grants this, initializes (resets) its CX state context # 0 (IStateContext::cf_write_status), sets up the hart's mcx_table to address the second thread's CX selector table with entry # 0 for that cxu_id=0 and state_id=0, and returns CX selector index # 0. The thread selects that CX (csrw cx_index, index) and peforms various CX custom instructions.
The timeslice expires. The OS saves the second thread context, including the second thread's CXs' state contexts state (IStateContext::cf_read_{status,state}), and its current cx_index and cx_status.
The OS reschedules the first thread to the hart. It reloads the thread context from the save area, including reloading the thread's CXs' state contexts' state (IStateContext::cf_write_{status,state}). It sets the hart's mcx_table to once again address the first thread's CX selector table. It reloads the hart's cx_index (and thus mcx_selector) and cx_status. Context reload complete, it resumes execution of the second thread.
(The fact that both threads's mcx_tables happen to address the same CX state context is a coincidence and at any rate is benign due to the way the OS manages CX state contexts across context switches.)

The previous comment described dynamic reuse/sharing of CX state context as part of OS thread multiprogramming . But different resource managers may perform such resuse and multiprogramming in other situations.
For example, if a given CX provides fewer state contexts than harts, if threads running on every hart all request that CX, the OS may decide to fail requests when CX state contexts are oversubscribed (a more cooperative resource management policy). Or it may grant a CX state context to each new request by unilaterally revoking some previous grant -- a more adversarial resource management policy. (Either way, when a CX state context is reused by a different thread it must be initialized (reset) or reloaded from a previously saved context record.) Both cooperative and adversarial resource management models have been used previously. The basic mechanism must support both.
At the other dimension, when a system is configured with a CX itself configured with more state contexts than the system has harts, this allows for example, more efficient context switching. In a system with one CX with two state contexts, one hart, and two software threads using the CX, it may not be necessary to laboriously save and restore the CX state context data when context switching from one thread to another on the hart. Instead, each software thread might just use a separate CX state context, only switching mcx_tables and mcx_selector values on a context switch.

Another comment that may be helpful. Depending upon policy, the CX user programming model on Linux-class systems might be that the CX selector indices obtained from the OS are thread-scoped, or process-scoped.
If thread-scoped, that means the indices retrieved by one thread are meaningless/wrong on another thread in the process. In this case, each thread scheduled to a hart would use a unique disjoint CX selector table.
If process-scoped, indices retrieved by one thread would be usable by other threads in the process. In this case, each thread of a process, scheduled to a hart, would share the same per process CX selector table.
(Process scoped CX selector indices behave more analgously to OS file descriptors, which are process scoped: if one thread opens a file and shares the file descriptor with another thread in the process, either or both of the threads can perform file I/O using the file descriptor.)
Process scoped CX selector indices make it easier for software to intentionally (or accidentally!) share a CX state context. This may be good or bad.
Thread scoped CX selector indices require a programming model that mandates finer grained CX state ownership (per-client or per-thread state data structures) vs. process coped CX selector indices which may be kept and shared more simply in "global" data structures.
Thread scoped CX selector indices do not preclude sharing a CX state context if that is the desired use case. (Example: a CX that provides event counter state that we wish to share across threads of a process.) It just requires that each thread request the specific CX state context from the OS, each is granted its own CX selector index to the same CX state context. (Behind the scenes, the same specific CX selector value for the specific shared CX state context will found in both threads' harts' CX selector tables and (at times) in both hart's mcx_selector CSRs.)
The spec'd limited size of the CX selector tables (=1024 entries) reflects past design assumptions that CX selector indices would be thread scoped. If process scoped, 1024 entries per process could be limiting in high-hart-count systems with many CXs times many threads.

There is much to learn from the comments in this this Issue thread but TL/DR there is no flaw in the CX security model, albeit preivously in the spec there was no clean way to force a trap on use of an OS-reclaimed CX selector index.
Now PR #32 adds mcx_selector.cte, custom operation trap enable, to CX selectors, so that an OS may tombstone a closed/released selector index by setting cte=1 on the corresponding selector value in mcx_table[]. If the user thread subsequently reselects this tombstoned CX selector index, then issues a custom operation, this raises an illegal instruction exception. The exception handler can exit the process, or take some other action, as appropriate.


ecto.migrate cannot migrate on RDS database

I am currently deploying a Phoenix app in production and encounter some issue while running Ecto's migrations.
Locally, the migrations work perfectly, however, when changing the parameters in config.ex to point to the production database, I have the following error:
[error] Could not retrieve migrated versions. This error usually happens due to the following:

  * The database does not exist
  * The "schema_migrations" table, which Ecto uses for managing
    migrations, was defined by another library

To fix the first issue, run "mix ecto.create".

To address the second, you can run "mix ecto.drop" followed by
"mix ecto.create". Alternatively you may configure Ecto to use
another table for managing migrations:

    config :sapiens, Sapiens.Repo,
      migration_source: "some_other_table_for_schema_migrations"

The full error report is shown below.

** (Postgrex.Error) ERROR 3F000 (invalid_schema_name): no schema has been selected to create in
    (ecto) lib/ecto/adapters/sql.ex:200: Ecto.Adapters.SQL.query!/5
    (ecto) lib/ecto/adapters/postgres.ex:96: anonymous fn/4 in Ecto.Adapters.Postgres.execute_ddl/3
    (elixir) lib/enum.ex:1925: Enum."-reduce/3-lists^foldl/2-0-"/3
    (ecto) lib/ecto/adapters/postgres.ex:96: Ecto.Adapters.Postgres.execute_ddl/3
    (ecto) lib/ecto/migrator.ex:44: anonymous fn/2 in Ecto.Migrator.migrated_versions/2
    (ecto) lib/ecto/migrator.ex:300: Ecto.Migrator.verbose_schema_migration/3
    (ecto) lib/ecto/migrator.ex:157: Ecto.Migrator.run/4
    (ecto) lib/mix/tasks/ecto.migrate.ex:83: anonymous fn/4 in Mix.Tasks.Ecto.Migrate.run/2
    (elixir) lib/enum.ex:765: Enum."-each/2-lists^foreach/1-0-"/2
    (elixir) lib/enum.ex:765: Enum.each/2
    (mix) lib/mix/task.ex:316: Mix.Task.run_task/3
    (mix) lib/mix/cli.ex:79: Mix.CLI.run_task/2
    (elixir) lib/code.ex:767: Code.require_file/2

The commands mix ecto.drop and mix ecto.create work as expected, but impossible to run the migrations.
The database is a Postgresql, hosted on RDS.
Environment

Elixir version (elixir -v):

Erlang/OTP 21 [erts-10.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe] [dtrace]

Elixir 1.7.4 (compiled with Erlang/OTP 21)

Database and version (PostgreSQL 9.4, MongoDB 3.2, etc.):

PostgreSQL 9.6.6 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.2 20140120 (Red Hat 4.8.2-16), 64-bit

Ecto version (mix deps):

ecto 2.2.11 (Hex package) (mix)
locked at 2.2.11 (ecto) 4bb8f117
ok

Database adapter and version (mix deps):

* postgrex 0.13.5 (Hex package) (mix)
  locked at 0.13.5 (postgrex) 3d931aba
  ok

Operating system:

uname -a
Darwin Thibauts-MacBook-Pro-2.local 18.2.0 Darwin Kernel Version 18.2.0: Mon Nov 12 20:24:46 PST 2018; root:xnu-4903.231.4~2/RELEASE_X86_64 x86_64

I believe it is a permission issue: https://dba.stackexchange.com/questions/106057/error-no-schema-has-been-selected-to-create-in
Is this a fresh RDS instance? IF you can't have those permissions, then you may need to use the after_connect callback to change the default schema for each connection.


Standalone language server (LSP)

Hello!
Is it possible to use dbt-language-server as a standalone language server according to the language server protocol spec? I would very much like to use this with helix (a vim-like text editor), in the same way that I use e.g. pyright.

Hello @thomasaarholt!
We use language server protocol for communication between the client and server parts. In theory it should be possible to use dbt language server with other editors that support LSP, but I haven't tried it yet. Could you explain how do you use pyright or any other language server in helix?

Sure! Today I use pyright to highlight code and type errors as well as navigation like "go to function definition". One installs the language server (something like brew install pyright), and then adds a configuration like what is shown here for helix/pyright: https://github.com/helix-editor/helix/wiki/How-to-install-the-default-language-servers#python---pyright
Whenever I do something in helix, pyright is called according to the config: pyright-langserver --stdio <some lsp command here>.

Hey @thomasaarholt! Thank you for detailed explanation.
I prepared npm package for dbt-language-server. And it can be used for helix. But not all features will work as in VS Code. We recommend using autosave in VS Code. So all changes in extension are happen after file save event.
I tested this and it looks like go to definition, errors highlighting, SQL to ref conversion features work.
First you need to install dbt-language-server npm package:
npm install -g @fivetrandevelopers/dbt-language-server

Then you can check the version of the package (0.30.2 today):
dbt-language-server --version

To use it in helix the following ~/.config/helix/languages.toml config should work:
[[language]]
name = "sql"
scope = "source.sql"
file-types = ["sql"]
roots = ["dbt_project.yml"]
language-server = { command = "dbt-language-server", args = ["--stdio"]}
config = { pythonInfo = { path = 'python' }, lspMode = 'dbtProject', enableSnowflakeSyntaxCheck = true }

https://github.com/fivetran/dbt-language-server/assets/3650304/7f2d60be-509f-4b9f-9ce1-4c7019c3166e

This is incredible!! It worked out of the box for the current release version of helix! Thank you SO much for this!
In case anyone wants to try, here is the equivalent config for the latest master (the configuration changed a bit):
[[language]]
name = "sql"
scope = "source.sql"
file-types = ["sql"]
roots = ["dbt_project.yml"]
language-servers = ["dbt-language-server"]

[language-server.dbt-language-server]
command = "dbt-language-server"
args = ["--stdio"]
config = { pythonInfo = { path = 'python' }, lspMode = 'dbtProject', enableSnowflakeSyntaxCheck = true }

Closing, since you've totally nailed this issue :D

Thanks so much for this! For anyone else who might end up here, I was able to get this to work in neovim with
local lspconfig = require("lspconfig")
local configs = require('lspconfig.configs')

if not configs.dbtls then
    configs.dbtls = {
        default_config = {
            root_dir = lspconfig.util.root_pattern('dbt_project.yml'),
            cmd = { 'dbt-language-server', '--stdio' },
            filetypes = { "sql" },
            init_options = { pythonInfo = { path = 'python' }, lspMode = 'dbtProject', enableSnowflakeSyntaxCheck = true }
        },
    }
end

lspconfig.dbtls.setup{}

Thanks so much for this! For anyone else who might end up here, I was able to get this to work in neovim with
local lspconfig = require("lspconfig")
local configs = require('lspconfig.configs')

if not configs.dbtls then
    configs.dbtls = {
        default_config = {
            root_dir = lspconfig.util.root_pattern('dbt_project.yml'),
            cmd = { 'dbt-language-server', '--stdio' },
            filetypes = { "sql" },
            init_options = { pythonInfo = { path = 'python' }, lspMode = 'dbtProject', enableSnowflakeSyntaxCheck = true }
        },
    }
end

lspconfig.dbtls.setup{}

I'm having trouble getting it to work in Neovim. Are you still using dbtls? I tried both the newest version of dbtls and 0.30.2, both give the same error:
[ERROR][2024-03-20 10:05:48] .../vim/lsp/rpc.lua:789	"rpc"	"/home/bmvi/.nvm/versions/node/v18.17.1/bin/dbt-language-server"	"stderr"	"/home/bmvi/.nvm/versions/node/v18.17.1/lib/node_modules/@fivetrandevelopers/dbt-language-server/out/lsp_server/LspServerBase.js:27\n        throw new Error('Uncaught exception. Server will be restarted.');\n        ^\n\nError: Uncaught exception. Server will be restarted.\n    at LspServer.onUncaughtException (/home/bmvi/.nvm/versions/node/v18.17.1/lib/node_modules/@fivetrandevelopers/dbt-language-server/src/lsp_server/LspServerBase.ts:28:11)\n    at process.emit (node:events:514:28)\n    at process._fatalException (node:internal/process/execution:149:25)\n    at processPromiseRejections (node:internal/process/promises:288:13)\n    at processTicksAndRejections (node:internal/process/task_queues:96:32)\n    at runNextTicks (node:internal/process/task_queues:64:3)\n    at processImmediate (node:internal/timers:447:9)\n\nNode.js v18.17.1\n"

I'm using Neovim version 0.10.0~ubuntu1+git202403182109-e3bd04f2a-971e32c87-79705122c~ubuntu22.04.1 obtained from https://ppa.launchpadcontent.net/neovim-ppa/unstable/ubuntu. Thanks for sharing your config, by the way.

Thanks so much for this! For anyone else who might end up here, I was able to get this to work in neovim with
local lspconfig = require("lspconfig")
local configs = require('lspconfig.configs')

if not configs.dbtls then
    configs.dbtls = {
        default_config = {
            root_dir = lspconfig.util.root_pattern('dbt_project.yml'),
            cmd = { 'dbt-language-server', '--stdio' },
            filetypes = { "sql" },
            init_options = { pythonInfo = { path = 'python' }, lspMode = 'dbtProject', enableSnowflakeSyntaxCheck = true }
        },
    }
end

lspconfig.dbtls.setup{}

I'm having trouble getting it to work in Neovim. Are you still using dbtls? I tried both the newest version of dbtls and 0.30.2, both give the same error:
[ERROR][2024-03-20 10:05:48] .../vim/lsp/rpc.lua:789	"rpc"	"/home/bmvi/.nvm/versions/node/v18.17.1/bin/dbt-language-server"	"stderr"	"/home/bmvi/.nvm/versions/node/v18.17.1/lib/node_modules/@fivetrandevelopers/dbt-language-server/out/lsp_server/LspServerBase.js:27\n        throw new Error('Uncaught exception. Server will be restarted.');\n        ^\n\nError: Uncaught exception. Server will be restarted.\n    at LspServer.onUncaughtException (/home/bmvi/.nvm/versions/node/v18.17.1/lib/node_modules/@fivetrandevelopers/dbt-language-server/src/lsp_server/LspServerBase.ts:28:11)\n    at process.emit (node:events:514:28)\n    at process._fatalException (node:internal/process/execution:149:25)\n    at processPromiseRejections (node:internal/process/promises:288:13)\n    at processTicksAndRejections (node:internal/process/task_queues:96:32)\n    at runNextTicks (node:internal/process/task_queues:64:3)\n    at processImmediate (node:internal/timers:447:9)\n\nNode.js v18.17.1\n"

I'm using Neovim version 0.10.0~ubuntu1+git202403182109-e3bd04f2a-971e32c87-79705122c~ubuntu22.04.1 obtained from https://ppa.launchpadcontent.net/neovim-ppa/unstable/ubuntu. Thanks for sharing your config, by the way.

Yes I am still using. I have node 18.16.0 and dbt-language-server 0.33.2 if that's helpful.


Could this be made working for RedHat (CentOS) and/or Ubuntu?

I am a no python person and find this tool is unique/complete among its peers in public domain. Thanks in advance!


Add arrow buttom left pattern

Hey i want to have patterns like this:
*       *
* *   *
* * *
* * * *
* * * * *

Do it in your favorite programing language, remember to ask for assign this issue.
If your favorit language folder doesn't exist, make it by your self and add file name it arrow-button-left
Don't forget to test and check your PR

Hey @Reynadi531 . I want to work on this issue

sure, go ahead

I have to write code only for n=5 or n can be greater than 5?

No problem, please comment that issue at the end of your solution

i want to have this

yes you can do it in your favorite language

i add the code :)

Give it to me

Hey there, I would like to contribute. I will do it in java. Pls assign.

Yeah everyone go ahead

Thanks for contribute everyone


Chrome extension build fails on macOS

Hey
Running bazel build chrome_ext fails with this error:
ERROR: /private/var/tmp/_bazel_user1/efaa21b110f06464002c5605311ac560/external/polymer_bundler_repo/BUILD.bazel:5:1: no such package '@vulcanize//': npm_install failed:  (Process terminated by signal 6) and referenced by '@polymer_bundler_repo//:vulcanize_bin'
ERROR: Analysis of target '//chrome_ext:chrome_ext' failed; build aborted: Loading failed
INFO: Elapsed time: 3.898s
FAILED: Build did NOT complete successfully (0 packages loaded)

macOS 10.13.3
node 9.4.0 (npm 5.6.0)
bazel 0.9.0
FYI, native_messaging is built successfully.
Update:
I built it on Ubuntu and installed the extension on Mac; Also, built the native messaging on Mac.
How should I configure the native package? (detanglenm binary).
The extension complains that the native package is missing.

@therealmik Thanks, but the problem is that the extension complains about the native part (it says the native part is not installed; I've tried different combinations of options, but no success;
For example I've installed the ".json" file in both user and system directories for NativeMessageing, and I've also copied the detanglenm to /usr/local/bin, so it can be called by others. I've also restarted Chrome to make sure it reloads everything)

Building chrome_ext will succeed if you use rules_nodejs-0.5.3 instead of 0.3.1


The Italian Job #2

Created package.nls.it.json. I hope, again, that it helps

This was meant to be part of the same pr, but oh well. I guess I have a little more learning to do with regards to GitHub pull requesting