Build from source on MacOS M1 failed

Describe the issue:
I followed instructions from https://nni.readthedocs.io/zh/stable/notes/build_from_source.html,
run command
conda create --name my_env
conda activate my_env
git clone https://github.com/microsoft/nni.git
cd nni
pip install --upgrade setuptools pip wheel
pip install jupyterlab==3.0.9
export NNI_RELEASE=2.0
python setup.py build_ts

I got error when running 'python setup.py build_ts', as shown below
running build_ts
# Downloading node.js from https://nodejs.org/dist/v16.14.2/node-v16.14.2-darwin-arm64.tar.xz
# Extracting node.js
# Downloading yarn from https://github.com/yarnpkg/yarn/releases/download/v1.22.10/yarn-v1.22.10.tar.gz
# Extracting yarn
# Building NNI manager
# yarn  (path: ts/nni_manager)
yarn install v1.22.10
[1/5] 🔍  Validating package.json...
[2/5] 🔍  Resolving packages...
[3/5] 🚚  Fetching packages...
[4/5] 🔗  Linking dependencies...
warning " ><EMAIL_ADDRESS>has incorrect peer dependency<EMAIL_ADDRESS>warning " ><EMAIL_ADDRESS>has incorrect peer dependency "<EMAIL_ADDRESS>warning " ><EMAIL_ADDRESS>has incorrect peer dependency<EMAIL_ADDRESS>|| ^6.0.0".
warning Workspaces can only be enabled in private projects.
[5/5] 🔨  Building fresh packages...
[1/3] ⢀ sqlite3
[2/3] ⢀ cpu-features
warning Error running install script for optional dependency: "/Users/shuffleofficial/nni/ts/nni_manager/node_modules/cpu-features: Command failed.
Exit code: 1
Command: node-gyp rebuild
Arguments:
Directory: /Users/shuffleofficial/nni/ts/nni_manager/node_modules/cpu-features
Output:
gyp info it worked if it ends with ok
gyp info using<EMAIL_ADDRESS>gyp info using<EMAIL_ADDRESS>| darwin | arm64
gyp info find Python using Python version 3.10.4 found at \"/Users/shuffleofficial/opt/anaconda3/envs/ds_wgan/bin/python3\"
gyp info spawn /Users/shuffleofficial/opt/anaconda3/envs/ds_wgan/bin/python3
gyp info spawn args [
gyp info spawn args   '/Users/shuffleofficial/.config/yarn/global/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'make',
gyp info spawn args   '-I',
gyp info spawn args   '/Users/shuffleofficial/nni/ts/nni_manager/node_modules/cpu-features/build/config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/Users/shuffleofficial/.config/yarn/global/node_modules/node-gyp/addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/Users/shuffleofficial/Library/Caches/node-gyp/16.14.2/include/node/common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=/Users/shuffleofficial/Library/Caches/node-gyp/16.14.2',
gyp info spawn args   '-Dnode_gyp_dir=/Users/shuffleofficial/.config/yarn/global/node_modules/node-gyp',
gyp info spawn args   '-Dnode_lib_file=/Users/shuffleofficial/Library/Caches/node-gyp/16.14.2/<(target_arch)/node.lib',
gyp info spawn args   '-Dmodule_root_dir=/Users/shuffleofficial/nni/ts/nni_manager/node_modules/cpu-features',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'build',
gyp info spawn args   '-Goutput_dir=.'
gyp info spawn args ]
gyp info spawn make
gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
  ACTION Configuring dependencies /Users/shuffleofficial/nni/ts/nni_manager/node_modules/cpu-features/deps/cpu_features/build/Makefile
-- The C compiler identification is AppleClang <IP_ADDRESS>00029
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Looking for dlfcn.h
-- Looking for dlfcn.h - found
-- Looking for getauxval
-- Looking for getauxval - not found
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/shuffleofficial/nni/ts/nni_manager/node_modules/cpu-features/deps/cpu_features/build
  TOUCH Release/obj.target/config_deps.stamp
  ACTION Building dependencies /Users/shuffleofficial/nni/ts/nni_manager/node_modules/cpu-features/deps/cpu_features/build/libcpu_features.a
[ 11%] Building C object CMakeFiles/utils.dir/src/filesystem.c.o
[ 22%] Building C object CMakeFiles/utils.dir/src/stack_line_reader.c.o
[ 33%] Building C object CMakeFiles/utils.dir/src/string_view.c.o
[ 33%] Built target utils
[ 44%] Building C object CMakeFiles/unix_based_hardware_detection.dir/src/hwcaps.c.o
[ 55%] Building C object CMakeFiles/unix_based_hardware_detection.dir/src/unix_features_aggregator.c.o
[ 55%] Built target unix_based_hardware_detection
[ 66%] Building C object CMakeFiles/cpu_features.dir/src/cpuinfo_arm.c.o
In file included from /Users/shuffleofficial/nni/ts/nni_manager/node_modules/cpu-features/deps/cpu_features/src/cpuinfo_arm.c:15:
/Users/shuffleofficial/nni/ts/nni_manager/node_modules/cpu-features/deps/cpu_features/include/cpuinfo_arm.h:118:2: error: \"Including cpuinfo_arm.h from a non-arm target.\"
#error \"Including cpuinfo_arm.h from a non-arm target.\"
 ^
1 error generated.
make[3]: *** [CMakeFiles/cpu_features.dir/src/cpuinfo_arm.c.o] Error 1
make[2]: *** [CMakeFiles/cpu_features.dir/all] Error 2
make[1]: *** [all] Error 2
make: *** [/Users/shuffleofficial/nni/ts/nni_manager/node_modules/cpu-features/deps/cpu_features/build/libcpu_features.a] Error 2
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/Users/shuffleofficial/.config/yarn/global/node_modules/node-gyp/lib/build.js:201:23)
gyp ERR! stack     at ChildProcess.emit (node:events:526:28)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (node:internal/child_process:291:12)
gyp ERR! System Darwin 21.6.0
gyp ERR! command \"/Users/shuffleofficial/nni/nni_node/node\" \"/Users/shuffleofficial/.yarn/bin/node-gyp\" \"rebuild\"
gyp ERR! cwd /Users/shuffleofficial/nni/ts/nni_manager/node_modules/cpu-features
gyp ERR! node -v v16.14.2
gyp ERR! node-gyp -v v9.1.0
✨  Done in 39.75s.
# yarn build (path: ts/nni_manager)
yarn run v1.22.10
$ tsc
/bin/sh: tsc: command not found
error Command failed with exit code 127.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Traceback (most recent call last):
  File "/Users/shuffleofficial/nni/setup.py", line 300, in <module>
    _setup()
  File "/Users/shuffleofficial/nni/setup.py", line 93, in _setup
    setuptools.setup(
  File "/Users/shuffleofficial/opt/anaconda3/envs/ds_wgan/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup
    return distutils.core.setup(**attrs)
  File "/Users/shuffleofficial/opt/anaconda3/envs/ds_wgan/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
    return run_commands(dist)
  File "/Users/shuffleofficial/opt/anaconda3/envs/ds_wgan/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
    dist.run_commands()
  File "/Users/shuffleofficial/opt/anaconda3/envs/ds_wgan/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 968, in run_commands
    self.run_command(cmd)
  File "/Users/shuffleofficial/opt/anaconda3/envs/ds_wgan/lib/python3.10/site-packages/setuptools/dist.py", line 1217, in run_command
    super().run_command(command)
  File "/Users/shuffleofficial/opt/anaconda3/envs/ds_wgan/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
    cmd_obj.run()
  File "/Users/shuffleofficial/nni/setup.py", line 224, in run
    setup_ts.build(release)
  File "/Users/shuffleofficial/nni/setup_ts.py", line 53, in build
    compile_ts(release)
  File "/Users/shuffleofficial/nni/setup_ts.py", line 173, in compile_ts
    _yarn('ts/nni_manager', 'build')
  File "/Users/shuffleofficial/nni/setup_ts.py", line 272, in _yarn
    subprocess.run([str(_yarn_path), *args], cwd=path, check=True, env=_yarn_env)
  File "/Users/shuffleofficial/opt/anaconda3/envs/ds_wgan/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/Users/shuffleofficial/nni/toolchain/yarn/bin/yarn', 'build']' returned non-zero exit status 127.

Environment:

NNI version: 2.0
Training service (local|remote|pai|aml|etc): local
Client OS: MacOS 12.6
Server OS (for remote mode only):
Python version: Python 3.10.4 (main, Mar 31 2022, 03:37:37) [Clang 12.0.0 ] on darwin
PyTorch/TensorFlow version: Not applicable
Is conda/virtualenv/venv used?: conda
Is running in Docker?: No

How to reproduce it?: On a macos with M1 chip, run the terminal command above.

hi @DDDOH  Could you use nni v2.9?

hi @DDDOH  Could you use nni v2.9?

With export NNI_RELEASE=2.9?

Please try following command in ts/nni_manager folder:
/Users/shuffleofficial/nni/toolchain/yarn/bin/yarn tsc

It should find tsc in ts/nni_manager/node_modules/.bin/tsc, which is a symlink to ts/nni_manager/node_modules/typescript/bin/tsc.
If tsc is not there, please remove node_modules folder and run /Users/shuffleofficial/nni/toolchain/yarn/bin/yarn.
If the file is there but cannot be found, then there might be something wrong with yarn.
In this case, upgrade yarn_version in setup_ts.py to 1.22.19 and try again. (Remember to clean first)

To further debug it, you can add a line "debug": "echo $PATH", to package.json line 6 ("scripts" section)
Then you can run yarn debug to check searching paths.
It should include /Users/shuffleofficial/nni/ts/nni_manager/node_modules/.bin

https://github.com/microsoft/nni/issues/5142#issuecomment-1260335580
I upgrade yarn_version in setup_ts.py to 1.22.19, and run python setup.py clean,
then python setup.py build_ts  (and failed again with error message exactly the same as before).

Please try following command in ts/nni_manager folder:
/Users/shuffleofficial/nni/toolchain/yarn/bin/yarn tsc

gets
yarn run v1.22.19
error Couldn't find a package.json file in "/Users/shuffleofficial/nni"
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

And there is no tsc in the ts/nni_manager/node_modules/.bin/ folder:

I then run /Users/shuffleofficial/nni/toolchain/yarn/bin/yarn, there's still not a tsc in the ts/nni_manager/node_modules/.bin/ folder.
I'm not familiar with node and javascript staff, so I actually don't quite figure out which package.json you are referring to. If more information is needed, please let me know.
But I finally found one way to run nni on a M1 macbook: create a x86 conda environment, then pip install nni, and unsurprisingly this method is really slow but is useful for debugging.

#5142 (comment) I upgrade yarn_version in setup_ts.py to 1.22.19, and run python setup.py clean, then python setup.py build_ts  (and failed again with error message exactly the same as before).

Please try following command in ts/nni_manager folder:
/Users/shuffleofficial/nni/toolchain/yarn/bin/yarn tsc

gets
yarn run v1.22.19
error Couldn't find a package.json file in "/Users/shuffleofficial/nni"
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

And there is no tsc in the ts/nni_manager/node_modules/.bin/ folder:
I then run /Users/shuffleofficial/nni/toolchain/yarn/bin/yarn, there's still not a tsc in the ts/nni_manager/node_modules/.bin/ folder.
I'm not familiar with node and javascript staff, so I actually don't quite figure out which package.json you are referring to. If more information is needed, please let me know.
But I finally found one way to run nni on a M1 macbook: create a x86 conda environment, then pip install nni, and unsurprisingly this method is really slow but is useful for debugging.

@cruiseliu - more suggestions?

Please run yarn in ts/nni_manager directory (not nni directory), and then check:

Whether there is a typescript directory in node_modules.
Whether there is<EMAIL_ADDRESS>(or other version) entry yarn.lock file.
Whether there is "typescript": "^4.3.2" in package.json file.

If the first answer is yes, please try manually make the symlink and run yarn build to compile.
If the first answer is no and the second is yes, then it's a yarnpkg bug and we should report it to their issue tracker.
Otherwise, please git checkout master and try again.

hi @DDDOH Did you still have the install issue? You could follow the @cruiseliu suggestions.


Wrong request's 'Host' Header Value Due to Duplicate Server Names in' X-Forwarded-Server' Header

When you send a request to a Flask server through Apache using a reverse proxy and include the X-Forwarded-Server header with values like "servername1, servername1", a problem occurs. On the Flask server's side, when trying to read the request headers, the 'Host' header appears incorrect, showing the value as "servername1, servername1" instead of the expected "servername1". The issue is that the X-Forwarded-Server header value is affecting the Host value, even though these headers should be separate. This situation is likely to happen when multiple reverse proxies are used in Apache before reaching the Flask server.
For example, if you examine the Apache configurations below, the Flask server listens on port 9966 and can be accessed via "restV2". This setup leads to duplicate server names in the X-Forwarded-Server header.
Apache Configurations:
<Location /restInternal>
    ProxyPass http://<IP_ADDRESS>:9966 retry=1 Keepalive=On timeout=300
    ProxyPassReverse http://<IP_ADDRESS>:9966
</Location>

<Location /restV2>
    ProxyPass http://<IP_ADDRESS>/restInternal retry=1 Keepalive=On timeout=300
    ProxyPassReverse http://<IP_ADDRESS>:restInternal
</Location>

This issue arises due to the interaction between Apache, the reverse proxy, and the Flask server
Environment:

Python version: 3.6.8
Flask version: 2.3.2
Apache version: 2.4.6

I've found the root cause, it's not related to the flask
so this is not a flask bug


Devel

We don't need to merge it right now, but I thought we could discuss the following points:

I'm using numpy.arrays as a convenience for myself, but since there is no complicated matrix operations we could easily change them to some other data type and remove that dependency.

To generate random variates, I use the scipy module. Because of the need for a Dirichlet random variable, I need a recent version (>= 0.15.0). If this is too strong a requirement, we could draw from gammas or betas, which are both part of any version of scipy.

As it is set-up right now, the MCMC algorithm is called using the default parameters (i.e. thining parameter, length of burn-in period and number of iterations are fixed). Perhaps we should add something to the parser to allow the user to change these parameters.

Let me know what you think.

I'm using numpy.arrays as a convenience for myself, but since there is no complicated matrix operations we could easily change them to some other data type and remove that dependency.

It might be nice to remove the dependency, but during optimization it may become even more important to use Cython objects (since we're already stuck with Cython from pysam), so rather than make the code less convenient now and possibly still have to change it again later, it's probably better to just keep NumPy for now.

I need a recent version (>= 0.15.0). If this is too strong a requirement, we could draw from gammas or betas, which are both part of any version of scipy.

We already have the same problem with pysam (pip version is up to date but Ubuntu repository version is too old), so I don't think this makes the problem any worse. It might be helpful to implement version checks though.

Perhaps we should add something to the parser to allow the user to change these parameters.

Definitely.

I've changed the implementation.

I'm not using scipy at all anymore, but using the numpy random number generator.

Lists are being used now instead of numpy arrays.

I'm also removing zeros at the beginning of the computation. It doesn't have an impact on the estimation process, but since there's quite a lot of zeros, it has a tremendous impact on performance.

Great - it now runs in a reasonable amount of time and gets very similar results to the R package. But not exactly the same:
> rowSums(uniform)
ERCC-00009:889 ERCC-00009:890 ERCC-00009:891 ERCC-00009:892 ERCC-00009:893
            37              5             17              7              1
ERCC-00009:894 ERCC-00009:904 ERCC-00009:944
             2              1              1
> rowSums(raw) * p.true
ERCC-00009:889 ERCC-00009:890 ERCC-00009:891 ERCC-00009:892 ERCC-00009:893
     38.373342       5.414261      16.813019       6.717017       1.000000
ERCC-00009:894 ERCC-00009:904 ERCC-00009:944
      1.725803       1.000000       1.000000

(where uniform is from the Python version and p.true is from the R version)
It looks like this is due to random variation in the Python output, as I get slightly different counts each time I run it, but the R version seems quite robust. Does the Python version need to run a numerical algorithm for more iterations or something like that?

I think the R version looks more robust because you're taking the median of
a single quantity, whereas the Python version takes a median for each tag
with non-zero count, then rounds them before taking the sum. If I tweaked
the R algorithm so that it computes the same medians as python, I also get
some variation.
2015-10-12 9:39 GMT-04:00 jwfoley<EMAIL_ADDRESS>
Great - it now runs in a reasonable amount of time and gets very similar
results to the R package. But not exactly the same:

rowSums(uniform)
ERCC-00009:889 ERCC-00009:890 ERCC-00009:891 ERCC-00009:892 ERCC-00009:893
37              5             17              7              1
ERCC-00009:894 ERCC-00009:904 ERCC-00009:944
2              1              1
rowSums(raw) * p.true
ERCC-00009:889 ERCC-00009:890 ERCC-00009:891 ERCC-00009:892 ERCC-00009:893
38.373342       5.414261      16.813019       6.717017       1.000000
ERCC-00009:894 ERCC-00009:904 ERCC-00009:944
1.725803       1.000000       1.000000

(where uniform is from the Python version and p.true is from the R
version)
It looks like this is due to random variation in the Python output, as I
get slightly different counts each time I run it, but the R version seems
quite robust. Does the Python version need to run a numerical algorithm for
more iterations or something like that?
—
Reply to this email directly or view it on GitHub
https://github.com/jwfoley/umi-bayes/pull/2#issuecomment-147402758.

Is there a reason to prefer the way with the multiple medians? Cet. par. it would be nice to minimize the random variation in rerunning the program.

It's simply a consequence of the change in focus between the two
algorithms. Recall that in the R package, we were interested in estimating
the proportion of non-duplicates. In the Python implementation, we are
interested in the same proportion, but for each tag.
However, now that we are estimating up to 1,000 parameters, we should
probably increase the default number of iterations. But this can wait until
after the optimisation phase is over, I think.
2015-10-12 10:18 GMT-04:00 jwfoley<EMAIL_ADDRESS>
Is there a reason to prefer the way with the multiple medians? Cet. par.
it would be nice to minimize the random variation in rerunning the program.
—
Reply to this email directly or view it on GitHub
https://github.com/jwfoley/umi-bayes/pull/2#issuecomment-147411138.

Do I understand correctly that the difference is the R version estimates the "true" sum of all UMI hit counts, while the Python version estimates the true hit count for each UMI separately?
If so, although we do need the latter numbers in order to choose which reads to mark as duplicates, I think it would still make more sense to estimate the sum first and then distribute the appropriate number of duplicates across all the UMIs, according to some rule that keeps the non-duplicate counts proportional to the duplicate counts but also retains at least one hit for each observed UMI. Downstream analysis is just going to look at the sum, so it's more important to get the sum right than work out which specific reads are the true duplicates.
Or maybe these two approaches will converge anyway with more iterations?

I think that from a methodological perspective, the Python approach makes more sense: we are essentially estimating the "true hit count" using the expected value of the corresponding random variable. I'm not sure that a distributive approach, given the sum of true counts, would have the same theoretical appeal.
With more iterations, each approach will be more stable, but we have to remember that what we are estimating is not really a count, but rather a proportion, and therefore there will always be some rounding up to do. The R version rounds only once, whereas the Python version round multiple times, which explains the discrepancy between the two approaches in how they estimate the sum of true counts. But it seems to me that the difference is negligible.


chore(deps): bump trivy to v0.38.1

bump trivy to v0.38.1

@simar7 Would you take a look and cut a new version?


Restore previous status of dragbars

When clicking on the placeholder title, the opened state of the plugins is toggled

@FinalAngel could you review this?

@yakky LGTM

@astagi when you have an half-opened placeholder, the first click fully opens all the plugins: is this the desired behavior?

@yakky I didn't change the default behaviour, just restored the right status.

Thanks @astagi

yw


std::thread support

Right now, using std::thread::spawn leads to panic. I'm aware that wasm32-wasi doesn't have thread support, but wasm32-wasi-preview1-threads does. Is there any way to use threads on the browser?
WebAssembly.memory and web workers allow parallelization on the web. Does jco make the use out of them right now? If not, where should I start to contribute?

Hi there, thanks for asking! wasm32-wasi-preview1-threads adds experimental support for threads to the WASI Preview 1 target. Jco, Wasm Components, and the Component Model all target WASI Preview 2, which does not yet have support for threads.
The current work on thread support for WASI Preview 2 is tracked in the wasi-threads repo. This builds on the Wasm Core Threads proposal. Right now it's still too early to try and implement wasi-threads since the spec is still very much in progress. We'll want to wait for the spec authors to green light the spec as ready to implement before we try and implement anything.
So unfortunately right now there's nothing we can do here. But my hope is that after the Preview 2 release threads become a priority, and we'll be able to add experimental support for it as soon as the spec authors believe we should.

It's a sad thing that wasi-threads is not even reaching phase 2 after 1 year... but I get the point. Thanks!

See https://github.com/WebAssembly/wasi-threads/issues/48#issuecomment-1646138912 for more context

Please excuse me continuing the discussion.
Reading the link above, it's now clear that wasi-thread and wasm-core-thread are being proposed separately. Also, wasi-thread seems to have concerns that it might be deprecated by wasm-core-thread.
wasm-core-thread(If I'm right) has reached phase 4 a few weeks ago, while wasi-thread is still at phase 1, worrying about its incompatibility with the component model

https://github.com/WebAssembly/threads/issues/198#issuecomment-1778877964

Is jco open to using wasm-core-threads(Of course, after Rust supports it), or is it purely focusing on wasi-thread?

If jco is it purely focusing on wasi-thread, would we wait for wasi-thread proposal to reach phase 4 as well?

Part of the acceptance criteria for any WASI proposal is that it has multiple implementations. I don't expect us to want to wait for the proposal to reach phase 4 before implementing, but instead implement it sooner in order to help the proposal progress through the stages.
I haven't been in the room for the WASI Threads presentations and discussion, but my understanding is that the current plan is what I shared earlier: support for threads needs to land in Wasm Core first. And then thread spawning support for WASI and Wasm Components can be implemented in terms of that. I wouldn't worry too much about specific stages and timelines; for several members of the ByteCode Alliance this is one of the highest priority items after the Preview 2 release. That means work on it is likely going to be funded, and we might see it progress relatively fast.

The importance of thread spawning to our users is very well understood, and it is being prioritized above many other worthy tasks right now.
wasm-core-thread is just a spec proposal about the behavior of threads, it does not cover the behavior of creating threads. @abrown, @sbc100, and @conrad-watt are exploring a proposal for core wasm covering thread creation here https://github.com/abrown/thread-spawn .
We expect that anything that happens with wasi-thread, if turns out to still be necessary at all, will only happen after the core wasm thread spawning proposal has reached some amount of stability.
WebAssembly specification can be a slow process because it takes a lot of work and coordination across many different companies. In particular, Web engines are very conservative in what they ship because once they ship a feature they can never undo it or break it without breaking the Web itself, something they work very hard to never do. But as Yosh says, many of the companies involved have made thread spawning a business priority and are funding experts to work on it, and the Bytecode Alliance is exploring ways to ship thread spawn before Web engines do.
One thing we do need to do better is communicate to the current and potential users of wasi-threads that the proposal shouldn't be expected to make progress until the core spec work has matured, and that users probably don't want to depend on it unless they are prepared to deal with the pain of moving to the new way of doing things once that is figured out.


Bug - links with relative paths

For example, in this rule https://www.ssw.com.au/rules/management---is-your-client-clear-on-how-you-manage-projects

...the link at the very end is relative (https://www.w3schools.com/html/html_filepaths.asp)
...and is linking to https://www.ssw.com.au/rules/_layouts/15/FIXUPREDIRECT.ASPX?WebId=3dfc0e07-e23a-4cbb-aac2-e778b71166a2&TermSetId=07da3ddf-0924-4cd2-a6d4-a4809ae20160&TermId=8d7b1fab-5923-4e07-8da2-ba906053c056 which obviously leads to an error:

Please see if you can find an automatic solution for theses (FYI - I don't think there any many many many relative links, but hard to tell)

Hi @tiagov8
You can use CodeAuditor to scan for all the broken links that are similar to this one.
Here is a scan that was ran the other day to reveal broken links

Hi @tiagov8 ,
Work has been done on the extractor to replace these links by the correct link. With the fix on friendly URL, the next extraction should fix this issue.
@tombui99 , the links will have to be tested by CodeAuditor after the next extraction


OSC escape sequences received out-of-order in 3rd-party terminals

Windows Terminal version
1.19.11213.0
Windows build number
10.0.22631.0
Other Software
No response
Steps to reproduce
ANSI compatibility has been pretty good in ConPTY. We're coming up on some niche cases.
In a 3rd-party terminal, e.g. alacritty, produce some text with an OSC escape sequence interleaved between some text.
PowerShell 7.4.2
PS> echo "hello `e]2;new title`a world"
hello  world

Visually, things look pretty OK. The correct text is echo'd and the title is changed.

But, here is the exact data that alacritty read out of ConPTY:

hello  world
PS> ESC]0;new titleBEL

Observe that when the text was produced in the shell, the OSC command to set title was between "hello" and "world". However, when the text was read out of ConPTY by the terminal, the ordering is changed.
Expected Behavior
I expect the ordering of text, including the ANSI escape sequences, to be preserved.
Actual Behavior
ANSI escape sequences are being parsed and re-serialized in a "lossy" way. While this may be necessary when a Windows Terminal is involved, it isn't when the terminal application only receives ANSI sequences. That is, there is no need to try to parse the ANSI. ConPTY does still need to convert console API calls coming from Windows command-line apps, but plain ANSI text doesn't need any conversion.

This is essentially a duplicate of #8698, and is probably not generically solvable without passing through the ANSI sequences directly, as you're proposing. That's tracked in #1173.


Conduct public fund raise

Description

At this point we would be raising external capital to manage officially.
Specifics

[ ] research/set target amount
[ ] contact network for general guidance/specific introductions
[ ] contact Repool team for guidance/recommendations
[ ] raise funds

holy shit it's real, it's a GitHub issue and everything

Closing to consolidate into separate issues.


Added String topic in DSA

Related Issue #920
[Cite any related issue(s) this pull request addresses. If none, simply state "None”]
Description
[Please include a brief description of the changes or features added]
Type of PR

[x] Documentation update

Screenshots / Videos (if applicable)

[Attach any relevant screenshots or videos demonstrating the changes]
Checklist

[x] I have performed a self-review of my code.
[x] I have read and followed the Contribution Guidelines.
[x] I have tested the changes thoroughly before submitting this pull request.
[x] I have provided relevant issue numbers, screenshots, and videos after making the changes.
[x] I have commented my code, particularly in hard-to-understand areas.

Additional Context
[Include any additional information or context that might be helpful for reviewers.]
Resources for Guidance
Here are some resources that may be helpful as you contribute to CodeHarborHub:

Code Harbor Hub Community Features
Docusaurus Documentation
React.js Documentation
Markdown Guide
MDX Documentation
Mermaid Documentation

Resources for DSA Features on Our Site:

Tabs: Switch Languages
Line Numbering
Math Equations

hello @Ajay-Dhangar
please review this PR

https://github.com/CodeHarborHub/codeharborhub.github.io/assets/137484958/123a3f8f-e828-476a-ae3f-fe9874c20898

hello @Ajay-Dhangar
Is this documentation all right?


Update linitng docs for run_lint_checks

Discussion Reference for the issue.
PR in which file was changed
This particular problem requires us to update all references of pre_commit_linter to run_lint_checks throughout the docs.

Done, thanks!

Completed via https://github.com/oppia/oppia-web-developer-docs/pull/324


Three-way, and faster, comparisons

The current comparison operation looks like this:
static func <(lhs: Self, rhs: Self) -> Bool {
    lhs.high < rhs.high || lhs.high == rhs.high && lhs.low < rhs.low
}

It's short and sweet, but it often takes redundant steps:
| 3, 4, 5, 6 | 3, 4, 5, 0 | 3, 4, 5, 9 |
| 3, 4, 5, 6 | 3, 4, 5, 6 | 3, 4, 5, 6 |
|------------|------------|------------|
| 6 <  6     | 0 <  6     | 9 <  6     |
| 6 == 6     |            | 9 == 6     |
| 5 <  5     |            | 5 == 5     |
| 5 == 5     |            | 9 == 6     |
| 6 == 6     |            |            |
| 4 <  4     |            |            |
| 4 == 4     |            |            |
| 3 <  3     |            |            |

This can be improved upon with a three-way operation:
// presented sans withUnsafePyramidOfDoom(_:)
static func compare(_ lhs: Self, to rhs: Self) -> Int {
    var index = lhs.count
    assert(!index.isZero)

    backwards: do {
        index &-= 1
        let lhsWord  = Digit(bitPattern: lhs[index])
        let rhsWord  = Digit(bitPattern: rhs[index])
        if  lhsWord != rhsWord { return lhsWord < rhsWord ? -1 : 1 }
    }

    backwards: while !index.isZero {
        index &-= 1
        let lhsWord  = lhs[index]
        let rhsWord  = rhs[index]
        if  lhsWord != rhsWord { return lhsWord < rhsWord ? -1 : 1 }
    }

    return Int.zero
}

For large integers, it scales better and yields more information:
| 3, 4, 5, 6 | 3, 4, 5, 0 | 3, 4, 5, 6 |
| 3, 4, 5, 6 | 3, 4, 5, 6 | 3, 4, 5, 6 |
|------------|------------|------------|
| 6 == 6     | 0 == 6     | 9 == 6     |
| 5 == 5     | 0 <  6     | 9 <  6     |
| 4 == 4     |            |            |
| 3 == 3     |            |            |

The current division algorithm performs up to two comparisons per quotient digit (#72):
1. Test Case '-[ANKFullWidthKitBenchmarks. Int256BenchmarksOnDivision testDividingFullWidth]' passed (0.097 seconds).
2. Test Case '-[ANKFullWidthKitBenchmarks.UInt256BenchmarksOnDivision testDividingFullWidth]' passed (0.088 seconds).

That awkward moment when this profiles better:
func compared(to other: Self) -> Int {
    let high = self.high.compared(to: other.high)
    if !high.isZero { return high }
    return/**/ self.low .compared(to: other.low )
}

| 3, 4, 5, 6 | 3, 4, 5, 0 | 3, 4, 5, 6 |
| 3, 4, 5, 6 | 3, 4, 5, 6 | 3, 4, 5, 6 |
|------------|------------|------------|
| 6 <  6     | 0 <  6     | 9 <  6     |
| 6 == 6     |            | 9 == 6     |
| 5 <  5     |            |            |
| 5 == 5     |            |            |
| 4 <  4     |            |            |
| 4 == 4     |            |            |
| 3 <  3     |            |            |
| 3 == 3     |            |            |


[Question] Manually updatable progress bars

Hi, first let me just say this lib looks awesome. I am currently using something else to render a histogram like this to command line and was trying in vain to animate it.

It looks to me like I could use something like the multi progress bar example, but looking at the code it was not immediately obvious to my how to do this. Each bin would be a "task" that I could then update.
I suppose I'd have to create subclass of BarColumn to change the looks and somehow plug it into the Progress class?
Could you point me in the right direction?

You could render this with the Progress class, by having a task for each data point, but I suspect it would be simpler for you to generate a Table with a row for labels and a row for bars.
Start by creating a grid (table with no lines):
my_table = Table.grid()
my_table.title = "Samples"

Then add rows consisting of the label and a Bar.
for i, x in enumerate(data):
    my_table.add_row(str(i), Bar(size=100, end=y))

As for animation, the simplest solution would be to clear the screen and redraw. Something along these lines:
while True:
    with console:
        console.clear()
        console.print(my_table)
    update_data()

See examples/table_movie.py for an example of this kind of animation.


Uncaught Error: Cannot find module '../node_modules/keyboard-layout/build/Release/keyboard-layout...

Open Atom
Klick a button
???
Profit(or as here, this error)

Atom: 1.19.5 x64
Electron: 1.6.9
OS: EDO
Thrown From: Atom Core
Stack Trace
Uncaught Error: Cannot find module '../node_modules/keyboard-layout/build/Release/keyboard-layout-manager.node'
At module.js:472

Error: Cannot find module '../node_modules/keyboard-layout/build/Release/keyboard-layout-manager.node'
    at Module._resolveFilename (module.js:470:15)
    at Module._resolveFilename (~/AppData/Local/atom/app-1.19.5/resources/electron.asar/common/reset-search-paths.js:35:12)
    at Function.get_Module._resolveFilename (~/AppData/Local/atom/app-1.19.5/resources/app/src/module-cache.js:354:58)
    at Module.require (~/AppData/Local/atom/app-1.19.5/resources/app/static/index.js:40:43)
    at require (internal/module.js:20:19)
    at customRequire (~/AppData/Local/atom/app-1.19.5/resources/app/static/<embedded>:96:26)
    at get_KeyboardLayoutManager (~/AppData/Local/atom/app-1.19.5/resources/app/node_modules/keyboard-layout/lib/keyboard-layout.js:7:65)
    at get_manager (~/AppData/Local/atom/app-1.19.5/resources/app/node_modules/keyboard-layout/lib/keyboard-layout.js:16:42)
    at getCurrentKeyboardLayout (~/AppData/Local/atom/app-1.19.5/resources/app/node_modules/keyboard-layout/lib/keyboard-layout.js:32:17)
    at Object.getCurrentKeymap (~/AppData/Local/atom/app-1.19.5/resources/app/node_modules/keyboard-layout/lib/keyboard-layout.js:22:31)
    at exports.keystrokeForKeyboardEvent (~/AppData/Local/atom/app-1.19.5/resources/app/node_modules/atom-keymap/lib/helpers.js:244:66)
    at KeymapManager.module.exports.KeymapManager.keystrokeForKeyboardEvent (~/AppData/Local/atom/app-1.19.5/resources/app/node_modules/atom-keymap/lib/keymap-manager.js:456:20)
    at KeymapManager.module.exports.KeymapManager.handleKeyboardEvent (~/AppData/Local/atom/app-1.19.5/resources/app/node_modules/atom-keymap/lib/keymap-manager.js:336:30)
    at WindowEventHandler.module.exports.WindowEventHandler.handleDocumentKeyEvent (~/AppData/Local/atom/app-1.19.5/resources/app/src/window-event-handler.js:100:42)
    at HTMLDocument.<anonymous> (~/AppData/Local/atom/app-1.19.5/resources/app/src/window-event-handler.js:3:65)

Commands

Non-Core Packages
atom-material-syntax 1.0.6
atom-material-ui 2.0.4
language-pug 0.0.21
source-preview-pug 0.2.0

We've seen a handful of similar errors and I'll close as a duplicate of https://github.com/atom/atom/issues/14461.  Can you try reinstalling and let us know if you still get the error?


New Subcommand for Generating Shell Completions

I'm proposing a new subcommand that can generate shell completions. The clap crate is already being used for actually building the mirrord CLI, and they already build a crate clap_complete to make generating shell completions simple for a few popular shells (bash, fish, zsh, powershell, elvish). This would make discoverability of flags and subcommands much simpler to newcomers and as the number of options grow.
It looks like some subcommands are hidden (like extract and verify) - I don't know if clap_complete supports not generating completions for them, or if that would be an issue if they were.
It would look like:
$ mirrord completions bash
### long output of bash completion to stdout truncated here

## could directly source
$ source <(mirrord completions bash)

$ mirrord exec <TAB><TAB>
-t                             -c                             --agent-namespace              --skip-processes               --no-outgoing                  --help
-n                             -e                             --agent-log-level              --agent-ttl                    --no-tcp-outgoing              <BINARY>
-a                             -p                             --agent-image                  --agent-startup-timeout        --no-udp-outgoing              [BINARY_ARGS]...
-l                             -f                             --fs-mode                      --accept-invalid-certificates  --no-telemetry
-i                             -h                             --override-env-vars-exclude    --ephemeral-container          --disable-version-check
-x                             --target                       --override-env-vars-include    --steal                        --config-file
-s                             --target-namespace             --no-remote-dns                --pause                        --context

$ mirrord exec --fs-mode <TAB><TAB>
write                 read                  local                 local-with-overrides

I know Homebrew has some nice functionality for calling these scripts and putting the generated output/files to sane places to source from when mirrord is installed, not sure about other package managers.
Happy to do the implementation myself, it should be fairly straightforward.

Hey @bradschwartz, sounds good! Would be great if you could implement yourself, we're here or on Discord if any help is needed


framework_tests_libraries flakes` on date_picker_test

(part of https://github.com/flutter/flutter/issues/97036)
These data is based Q4 2021 flakes that happened from 10/01/2021 to 01/13/2022

Mac framework_tests_libraries: 1 flake on date_picker_test

01:22 +2352 ~14: /opt/s/w/ir/x/w/flutter/packages/flutter/test/cupertino/date_picker_test.dart: Date picker DatePicker golden tests
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following _Exception was thrown while running async test code:
Exception: Skia Gold received an unapproved image in post-submit
testing. Golden file images in flutter/flutter are triaged
in pre-submit during code review for the given PR.

Visit https://flutter-gold.skia.org/ to view and approve
the image(s), or revert the associated change. For more
information, visit the wiki:
https://github.com/flutter/flutter/wiki/Writing-a-golden-file-test-for-package:flutter

Debug information for Gold:
stdout: Given image with hash 82149302f6a7f444ca82b263480cc947 for test
cupertino.date_picker_test.datetime.initial
Expectation for test: 0eeb47eef65dca3d8a64fb6e65f2ca4d (positive)
Expectation for test: 6799e6f0fc81d60d1fe72afab17eddd5 (positive)
Expectation for test: 68b87a3d1557212339b24f7023c42b19 (positive)
Expectation for test: b3d8b516195bb7ffebca5d7e7efe4efa (positive)
Expectation for test: 48d89017a2272a34408b3cde8ffe6e9a (positive)
Expectation for test: 5aefe0b58dab72c0e8f8c9b41bd67cc9 (positive)
Expectation for test: c45dd1ceae206abca194143fb6ad97e2 (positive)
Expectation for test: 428d3a53ecb3918a7e8ec237c7ea17d0 (positive)
Expectation for test: f507b258a4e73141e3f0d8d74502ddf0 (positive)
Expectation for test: 647463642fd9a38edcd0419da4d94188 (positive)
Expectation for test: 6ee9fed2051d189bfaebcf461ae0e82c (positive)
Expectation for test: 90597edb22dc9acb483e78b29c290279 (positive)
Expectation for test: b674772b19a6d38f15353f4148f52702 (positive)
Expectation for test: c9f08d07320a045d1784c7de4d427a70 (positive)
Expectation for test: e5508d2f1a43a33f584f7a15637c2f40 (positive)
Expectation for test: 0765b31270baab1c4842b04ebb2cc89e (positive)
Expectation for test: 382fa8b67631fa1fb565a14393c90dbb (positive)
Expectation for test: 4058c311b3b9d72640cd3c82e96aac63 (positive)
Expectation for test: 47fc581696de17734c8d65d0d58336f8 (positive)
Expectation for test: 6d9acdfc0671f2663dc751a2dc809f39 (positive)
Expectation for test: 2700c605c481a597fe819b0193feba55 (positive)
Expectation for test: 5a9f6c19a01b886a5e9413963e27dc97 (positive)
Expectation for test: 5d9edb9f588a4ddde7e8c91a8f005d6f (positive)
Expectation for test: a60e7326b58775862b7a35213af289aa (positive)
Expectation for test: b359f8a491bd05bb39e64492329ca0dc (positive)
Expectation for test: 22aea9cc5bbaa0da7a7af0f6ef99dc77 (positive)
Expectation for test: 3214cb4ad0273830294315e09143f421 (positive)
Expectation for test: 4fd0d7d787d69bd27533fecdf7705948 (positive)
Expectation for test: 87599f3dec5b56dc110f1b63dc747182 (positive)
Expectation for test: a7ba699858f119f9641142bf9e503796 (positive)
Untriaged or negative image:
https://flutter-gold.skia.org/detail?test=cupertino.date_picker_test.datetime.initial&digest=82149302f6a7f444ca82b263480cc947

stderr: Test: cupertino.date_picker_test.datetime.initial FAIL

When the exception was thrown, this was the stack:
#0      SkiaGoldClient.imgtestAdd (package:flutter_goldens_client/skia_client.dart:215:7)
<asynchronous suspension>
<asynchronous suspension>
(elided one frame from package:stack_trace)
════════════════════════════════════════════════════════════════════════════════════════════════════

01:22 +2353 ~14: /opt/s/w/ir/x/w/flutter/packages/flutter/test/cupertino/date_picker_test.dart: Date picker DatePicker golden tests
01:22 +2354 ~14: /opt/s/w/ir/x/w/flutter/packages/flutter/test/cupertino/date_picker_test.dart: Date picker DatePicker golden tests
01:22 +2355 ~14: /opt/s/w/ir/x/w/flutter/packages/flutter/test/cupertino/date_picker_test.dart: Date picker DatePicker golden tests
01:22 +2356 ~14: /opt/s/w/ir/x/w/flutter/packages/flutter/test/cupertino/date_picker_test.dart: Date picker DatePicker golden tests
01:23 +2356 ~14: /opt/s/w/ir/x/w/flutter/packages/flutter/test/cupertino/date_picker_test.dart: Date picker DatePicker golden tests
01:23 +2357 ~14: /opt/s/w/ir/x/w/flutter/packages/flutter/test/cupertino/date_picker_test.dart: Date picker DatePicker golden tests
01:23 +2358 ~14: /opt/s/w/ir/x/w/flutter/packages/flutter/test/cupertino/date_picker_test.dart: Date picker DatePicker golden tests
01:23 +2359 ~14: /opt/s/w/ir/x/w/flutter/packages/flutter/test/cupertino/date_picker_test.dart: Date picker DatePicker golden tests
01:23 +2359 ~14: /opt/s/w/ir/x/w/flutter/packages/flutter/test/cupertino/date_picker_test.dart: Date picker DatePicker golden tests
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following _Exception was thrown while running async test code:
Exception: Skia Gold received an unapproved image in post-submit
testing. Golden file images in flutter/flutter are triaged
in pre-submit during code review for the given PR.

Visit https://flutter-gold.skia.org/ to view and approve
the image(s), or revert the associated change. For more
information, visit the wiki:
https://github.com/flutter/flutter/wiki/Writing-a-golden-file-test-for-package:flutter

Debug information for Gold:
stdout: Given image with hash e50886a850bd0214e9f5a239f8847804 for test
cupertino.date_picker_test.datetime.drag
Expectation for test: 2918601bd09078ea60132c2e46a806e4 (positive)
Expectation for test: 2dc15070c7c1c613682778bc34b2637d (positive)
Expectation for test: 43f8ab889961f48c890a2814e16dbb0a (positive)
Expectation for test: 70b636ef4fbbe8f85bdaca623f5bf770 (positive)
Expectation for test: 9545ce0ee130e0602e8f1f9da7fadd5e (positive)
Expectation for test: 0b9795b218a8e367b552dcea55c8d589 (positive)
Expectation for test: 104b81d5634aec660fd1300e4dc8ac63 (positive)
Expectation for test: 65560016d0c3df90512748506389ed31 (positive)
Expectation for test: 007c679a276885ad84777935ac0b63ce (positive)
Expectation for test: 1f1881fe0b2e88e416e3d5a39f47cc7c (positive)
Expectation for test: 7a09093e7f71963f9e02357c09e70ba0 (positive)
Expectation for test: f0a6cea41931cb63d141a7eb67b7f8ca (positive)
Expectation for test: 50ab8e3db4a8a7d3f552a244888a70b7 (positive)
Expectation for test: 5e1fee10924cee463667fb3f5f08f171 (positive)
Expectation for test: 3f2279d0d4b2c9fdec6b50d358a41428 (positive)
Expectation for test: 5ad1651f225018116b59ad898cc15b2d (positive)
Expectation for test: 7d04fc1ef547a8e092495dab4294b4cd (positive)
Expectation for test: aa748136c70cefdda646df5be0ae189d (positive)
Expectation for test: bc9e3f104b8fb41df08dd3bb385718b0 (positive)
Expectation for test: 27a2ee6b082b643d5ff5cf5f61d020a5 (positive)
Expectation for test: 35ab982bfdc166abe415e1a3c099a3b2 (positive)
Expectation for test: cfd121c87912497fe3f60e7ca94bfd41 (positive)
Expectation for test: 24d0b1b915250fc26be3aca836f44bc5 (positive)
Expectation for test: 4f8e9a0c36540790c44b82ce5a654056 (positive)
Expectation for test: f34fa9b2e9aed2f1611d01507803ba31 (positive)
Expectation for test: 466c0c67233adf307f5a1a0626480a28 (positive)
Expectation for test: 6a039a7979a4b4fa369367bc289f2cea (positive)
Expectation for test: c3312ad1479f50caf3754b42a42740c5 (positive)
Expectation for test: d1a97b66a2ba2c3ddf300836554ae7f9 (positive)
Expectation for test: 19c0169ad42f45513aa7a054ef54e894 (positive)
Expectation for test: 351eca00baac019ad7b80529a7545d65 (positive)
Untriaged or negative image:
https://flutter-gold.skia.org/detail?test=cupertino.date_picker_test.datetime.drag&digest=e50886a850bd0214e9f5a239f8847804

stderr: Test: cupertino.date_picker_test.datetime.drag FAIL

When the exception was thrown, this was the stack:
#0      SkiaGoldClient.imgtestAdd (package:flutter_goldens_client/skia_client.dart:215:7)
<asynchronous suspension>
<asynchronous suspension>
(elided one frame from package:stack_trace)
════════════════════════════════════════════════════════════════════════════════════════════════════
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following message was thrown:
Multiple exceptions (4) were detected during the running of the current test, and at least one was
unexpected.
════════════════════════════════════════════════════════════════════════════════════════════════════

01:23 +2359 ~14 -1: /opt/s/w/ir/x/w/flutter/packages/flutter/test/cupertino/date_picker_test.dart: Date picker DatePicker golden tests [E]
  Test failed. See exception logs above.
  The test description was: DatePicker golden tests

One similar flake on Windows framework_tests_libraries

/cc @Piinks could you help take a look? seems gold related. One possible related issue: https://github.com/flutter/flutter/issues/94729

This was from December. This is not a flake, it is a signal that someone merged a PR without triaging an image change in pre-submit as the error message says. I can see from the Flutter Gold triage log, that on that same day the images in question were updated: https://flutter-gold.skia.org/triagelog
I don't think this is a flake, nor an active issue at this time. The test failed as expected to notify of an image change that was not handled in pre-submit.

Thanks @Piinks for the clarification!
Closing as the issue has been fixed.


CentOS7: Failed to start gunicorn.service: Unit not found.

using master:HEAD on CentOS7 having declared:
class { 'python':
   version    => 'system',
   pip        => 'present',
   dev        => 'absent',
   virtualenv => 'absent',
   gunicorn   => 'present',
   manage_gunicorn => true,
 }

and:
python::gunicorn { 'puppet_reports':
   ensure      => present,
   mode        => 'wsgi',
   dir         => $dir,
   bind        => "<IP_ADDRESS>:${port}",
   accesslog   => "${log_dir}/access.log",
   errorlog    => "${log_dir}/error.log",
   appmodule   => 'puppet_reports:app',
   timeout     => 30,
   owner       => $owner,
   require     => Vcsrepo[$dir],
 }

I get:
Error: Could not start Service[gunicorn]: Execution of '/bin/systemctl start gunicorn' returned 5: Failed to start gunicorn.service: Unit not found.
Wrapped exception:
Execution of '/bin/systemctl start gunicorn' returned 5: Failed to start gunicorn.service: Unit not found.
Error: /Stage[main]/Python::Config/Service[gunicorn]/ensure: change from stopped to running failed: Could not start Service[gunicorn]: Execution of '/bin/systemctl start gunicorn' returned 5: Failed to start gunicorn.service: Unit not found.
Notice: /Stage[main]/Python/Anchor[python::end]: Dependency Service[gunicorn] has failures: true
Warning: /Stage[main]/Python/Anchor[python::end]: Skipping because of failed dependencies

I'm having the same problem. Has there been any work on this issue? Did you end up finding a solution @droopy4096 ?

Anything new on this?

not here...

Underlying problem is the current python-gunicorn rpm for CentOS/Redhat 7 doesn't deploy a systemd unit file named 'gunicorn.service,' causing declaration Service[gunicorn] to fail.  Also, the /etc/gunicorn.d directory is only used in Debian/Ubuntu packaging, ignored in CentOS/Redhat.
The systemd-friendly resolution would be to modify python::gunicorn to deploy a systemd template unit file on those OSes that require systemd (like CentOS/Redhat 7) and then launch an instance of the template.
So, python::gunicorn{ 'foo_app': ...} could write a systemd unit file from ERB to<EMAIL_ADDRESS>like this:
[Unit]
Description=Gunicorn instance to serve %I
After=network.target

[Service]
User=<%= @owner] %>
Group=<%= @group] %>
WorkingDirectory=<%= @dir %>
ExecStart=<%= @virtualenv %>/bin/gunicorn --workers <%= @workers %> --bind <%= @bind %> <%= @appmodule %>
Restart=always

[Install]
WantedBy=multi-user.target

Then launch the service like this:
# systemctl start<EMAIL_ADDRESS>And finally declare a service resource:
service { 'gunicorn@foo_app':
  ensure     => running,
  enable     => true,
}

The python::config class would also need updating not declare the resource Service[gunicorn], in such cases.  There still would be no service named just 'gunicorn.'
This module might be useful for farming out sysytemd unit file management:
https://fedoramagazine.org/systemd-template-unit-files/
Please note I don't yet have this suggestion working for me verbatim.  Just collecting what research I've gathered.

Debian, starting from version stretch (9) does not support gunicorn.d too.
This means, the /etc/gunicorn.d approach does not work since July 2020 on Debian stable.
https://metadata.ftp-master.debian.org/changelogs//main/g/gunicorn/gunicorn_19.6.0-10+deb9u1_NEWS


[INFRA] Do not require all conversations resovled

What changes were proposed in this pull request?
Some comments are used to highlight the key change or something else to help recall in future.
How was this patch tested?
N/A

cc @weiting-chen @yaooqinn thank you


YouTube-DL: Feature Request of 'Video Quality' option in 'Output Template'.

Checklist

[x] I'm reporting a feature request
[x] I've verified that I'm running youtube-dl version 2019.07.16
[ ] I've searched the bugtracker for similar feature requests including closed ones

Description

WRITE DESCRIPTION HERE
Greetings!
I  have an Issue/Feature Request for (explicit) 'Video Quality' in (of my) videos; same video, one is '1080p' and the other is '2160p' with same title. This one is having '16:9' ratio (HD, 1.77777..8). But, not always! There are many other ratios such as '1.85', '2.20','2.35' or '2.40'. And some are having higher ratios as well.
For example, a normal video with '16:9' ratio:
...
[ffmpeg] Merging formats into "D:\Downloads\Train Ride to Tibet on the Worlds Highest Railway (u20181031; 641; 2160p30)YT 'We're the Russos'.mp4"
[debug] ffmpeg command line: ...
Deleting original file D:\Downloads\Train Ride to Tibet on the Worlds Highest Railway (u20181031; 641; 2160p30)YT 'We're the Russos'.f313.webm (pass -k to keep)
....
And another video NOT having '16:9' ratio:
...
[ffmpeg] Merging formats into "D:\Downloads\The Fantastic Four - Trailer (2005) (u20110130; 146; 808p24)YT 'QualityIsNumber1'.mp4"
Deleting original file D:\Downloads\The Fantastic Four - Trailer (2005) (u20110130; 146; 808p24)YT 'QualityIsNumber1'.f137.mp4 (pass -k to keep)
...
The last one is supposed to be '1080p24', I am giving Output Template as ' -o "D:\Downloads%(title)s (u%(upload_date)s; %(duration)s; %(height)sp%(fps)s)YT '%(uploader)s'.%(ext)s" -- ' since, YT-DL doesn't have option for Video Quality (Output Template).
{ Also, you can note that the duration is in 'second' (146;) instead of 'h,m,s' (0,2,26;) If you want only the function/procedure, (like durationHMS) may be I can help. }
Thanks for (your) Patience.
ATB! Regards,

Greetings!
I just don't understand that 'dstftw' did as 'incomplete' and 'Closed'.
I made 'A Feature Request' for 'Video Quality' such as 1080p or 2160p, only when they are in 4:3 or 16:9 aspect ratios (AR) and showed two references for the same, as, 4:3 and 16:9 and not in those AR. So, I don't think the term 'Incomplete' apply.
ATB! Regards,

Your explanation is total mess I don't even want to bother deciphering.
If you want something to be implemented you must clearly explain what you're requesting, how to obtain such information with concrete examples URLs.

(P.S.: Please , see 'Stats for Nerds' option on Right click. ATB!)


Add json_serializable_dataclass decorator

The structured nature of @dataclasses plays very nicely with JSON serialization. This decorator automatically adds the requisite _json_dict_ method. You still have to register with the "resolver" for loading.

Automerge cancelled: A status check is failing.


Dependency failed on package which requires systemd-libs

I tryed to build joycond-git and got this error: --   Package 'libudev', required by 'virtual:world', not found
I think there is the systemd-libs package missing which is a part of base-devel in arch.
Is this problem related to aurutils? When i build it by hand everything works

Do you use Artix? When yes, I can't help you with that.

No i am on standard arch.
It just looks like the libudev is missing in the build environment for some reason.

Then where is that "virtual:world" thing coming from?

That's pkgconf output.
morganamilo@Octavia ~ % pkgconf --libs nonexist
Package nonexist was not found in the pkg-config search path.
Perhaps you should add the directory containing `nonexist.pc'
to the PKG_CONFIG_PATH environment variable
Package 'nonexist', required by 'virtual:world', not found

Ideally you send the full output and not just one line.

thats the output i get
...
-- Checking for module 'libevdev'
--   Found libevdev, version 1.11.0
-- Checking for module 'libudev'
--   Package 'libudev', required by 'virtual:world', not found
CMake Error at /usr/share/cmake-3.20/Modules/FindPkgConfig.cmake:556 (message):
A required package was not found
Call Stack (most recent call first):
/usr/share/cmake-3.20/Modules/FindPkgConfig.cmake:778 (_pkg_check_modules_internal)
CMakeLists.txt:11 (pkg_check_modules)
-- Configuring incomplete, errors occurred!
See also "/build/joycond-git/src/joycond/CMakeFiles/CMakeOutput.log".
==> ERROR: A failure occurred in build().
Aborting...
...

@rkunschke that's a packaging issue, the package maintainer is at fault here, not aurutils.
You need to add systemd-libs or libudev.so to the depends array, and also add systemd to the makedepends array for pkgconfig file.


ordering top-k using argpartition()

While writing the recommend_k_items() method for the rbm I noticed a small problem related to the argpartition() function. Basically, the returned k-items are not sorted in order, so a smaller number may appear earlier in the list than a bigger one.
This seems to happen with real numbers that are very close, e.g. 4.979 vs 4.981 but it may affect other situations as well. The solution is provided below, more details in
https://docs.scipy.org/doc/numpy/reference/generated/numpy.argpartition.html#numpy.argpartition
What is affected by this bug?

recommend_k_items() method

This is used to select the top-k items in SAR
How do we replicate the issue?
consider the following vector
a = np.random.rand(10000)*100
I want the labels of the first 10 elements in decreasing order (biggest --> small), i.e. top_k=10

id1 = np.argpartition(-a, top_k)[:top_k]
returns: array([8255, 1269, 1778, 9788, 8726, 5621, 5660, 9619, 8709, 6207], dtype=int64)

Now we can check the actual values
a[id1] = array([99.97332566, 99.9455154 , 99.92515612, 99.94902373, 99.98342226,        99.95773838, 99.93918414, 99.99000961, 99.92482682, 99.92437996])
In the above example, 99.92 is positioned earlier than 99.94 or 99.98. In SAR, the function is used as
id1=  np.argpartition(a, -top_k)[-top_k:] , giving:
id1 = array([6207, 8709, 1778, 5660, 8726, 9788, 5621, 1269, 9619, 8255],dtype=int64)
again, we can look at the actual values and find the same problem, just with a different order:
a[id1]= array([99.92437996, 99.92482682, 99.92515612, 99.93918414, 99.98342226,        99.94902373, 99.95773838, 99.9455154 , 99.99000961, 99.97332566])
Expected behavior (i.e. solution)
Solution 1: use full sorting algo like argsort
id2 = np.argsort(-a)[:top_k], giving:
id2 = array([9619, 8726, 8255, 5621, 9788, 1269, 5660, 1778, 8709, 6207], dtype=int64)
with values correctly ordered:
a[id2] = array([99.99000961, 99.98342226, 99.97332566, 99.95773838, 99.94902373,        99.9455154 , 99.93918414, 99.92515612, 99.92482682, 99.92437996])
this solution is slower than np.argpartition().
Solution 2 (preferred): use top_k as a list
id3 = np.argpartition(-a, range(top_k) )[:10], giving the same result of argsort()
id3= array([9619, 8726, 8255, 5621, 9788, 1269, 5660, 1778, 8709, 6207], dtype=int64)
benchmark: For a small array as the one considered in the example above the time difference of the two
algos is negligible. However, for the movielens dataset, the user-affinity matrix is (6040, 3952), in this case
np.argsort() -->  1.2492165565490723
np.argpartition() --> 0.29970526695251465
Therefore, solution 2 seems the preferred one

There is an even faster solution with numpy:
np.argpartition(a, range(-top_k, 0))[:,-1:-top_k-1:-1]
It can be way faster than np.argpartition(-a, range(top_k) )[:top_k] in some situations because there is no need to compute -a
Here is a small benchmark:

There is an even faster solution with numpy:
np.argpartition(a, range(-top_k, 0))[:,-1:-top_k-1:-1]
It can be way faster than np.argpartition(-a, range(top_k) )[:top_k] in some situations because there is no need to compute -a
Here is a small benchmark:

This is not what we do though, we don't use range, see https://github.com/microsoft/recommenders/blob/efaa3d7742183dee0846877e2dc64977098e1977/reco_utils/common/python_utils.py#L97


Declare prop-types as external

Hi, I noticed that you're bundling in the prop-types library into the package which increases bundle size and seems unnecessary?

It's only related to umd. Does bundle size matter?


Remove Text/InlineSpan Deprecations [ToDo Audit - Painting Library]

Description
This PR closes out some old deprecation todos from the painting library.
Checklist
Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]). This will ensure a smooth and quick review process.

[x] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[x] I signed the CLA.
[x] I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
[x] I read the Tree Hygiene wiki page, which explains my responsibilities.
[x] I updated/added relevant documentation (doc comments with ///).
[x] All existing and new tests are passing.
[x] The analyzer (flutter analyze --flutter-repo) does not report any problems on my PR.
[x] I am willing to follow-up on review comments in a timely manner.

Breaking Change
Did any tests fail when you ran them? Please read Handling breaking changes.

[x] No, no existing tests failed, so this is not a breaking change.
[ ] Yes, this is a breaking change. If not, delete the remainder of this section.

[ ] I wrote a design doc: https://flutter.dev/go/template Replace this with a link to your design doc's short link
[ ] I got input from the developer relations team, specifically from: Replace with the names of who gave advice
[ ] I wrote a migration guide: https://flutter.dev/go/breaking-changes-template Replace this with a link to a pull request that adds the migration guide to https://flutter.dev/docs/release/breaking-changes

Waiting to land after verifying the deprecation policy.


invite andrewthad to this repo

@goolord can you do this?

I have been invited.

have you accepted the invitation?


POW is wasted resources that could be used to power the service instead

The current system is centralised and wide open to being overtaken by a more advanced and decentralised implementation. It also uses wasteful PoW mining instead of allowing users to contribute those same resources to improving the speed and stability of the network.
Expected behaviour
Content loads faster than YouTube
The app should leverage encrypted, peer to peer bandwidth and hosting and reward users for contributing "proof-of-bandwidth" and/or "proof-of-capacity" resources to the network in LBRY credits, instead of wasting those same resources doing PoW mining.
Actual behaviour
Videos from the app are slow to arrive as they are hosted on centralised servers
The LBRY token comes from PoW mining which is wasteful of electricity
There is no incentive to contribute resources/bandwidth to the network

Which parts do you think are centralized?
The LBRY P2P network includes anyone who downloaded the file, not only our servers.
The long term plans include data markets / incentivized hosting.

Awesome!
(yes, bitcoin is absolutely a waste of resources, but it's a product of it's age. Newer cryptocurrencies have different proof methods that are less resource intensive)

We have no plans to move away from PoW at the moment. If PoS tech improves in the future, we will consider it for the LBRY blockchain.


Validating UUID without getting a 500 error code (the Spec validator isNotFound method is called before regex)

This POST request to store an user gives me a 500 error code.
Notice the language id have invalid x as the first character.
UUIDs are generated on database level (postgres), not programatically
{
	"data": {
		"type": "users",
		"attributes": {
			"name": "Hello Wo2rld",
			"email"<EMAIL_ADDRESS>			"password": "qwerty123"
		},
		"relationships": {
			"language": {
				"data": [
					{
						"type": "languages",
						"id": "x2e5086ef-55ba-452d-af87-5b47d261ca27"
					}
				]
			}
		}
	}
}

The problem is we have a spec validation running there, and it validates whenever a resource exists in database which is fine, but its bold to bet that end user entered the ID in correct format.
The 500 error is too serious for that

The spec validator just calls the exists() method on the adapter. So you can fix this by overloading the exists method on the languages adapter, i.e. something like:
public function exists(string $resourceId): bool
{
    if (1 === preg_match('/.../', $resourceId)) {
        return parent::exists($resourceId);
    }

    return false;
}

The spec validator just calls the exists() method on the adapter. So you can fix this by overloading the exists method on the languages adapter, i.e. something like:
public function exists(string $resourceId): bool
{
    if (1 === preg_match('/.../', $resourceId)) {
        return parent::exists($resourceId);
    }

    return false;
}

Thanks. Working like a charm!


fix log probs mismatch

What does this PR do ?
This PR fixes the log probs mismatch issue (https://github.com/NVIDIA/NeMo-Aligner/issues/227) between policy and reference policy in DPO/RPO/variants.
This PR solves this issue with two changes:

Refactor the code to compute reference policy log probs so that it is exactly the same as the code to compute policy log probs. This removes any issue related to MBS, GBS, DP, TP mismatches.

With this change, the reference policy log probs match the policy log probs exactly at the first iteration.

The problem with the fixes in this PR is that it negates the entire purpose of the previous design: using a higher MBS for the reference log prob calculations in order to speed it up, because the reference log probs are calculated via no_grad. As I understand it, this new PR just makes the reference logprobs run with the same GBS/MBS as the actor policy, thus not taking advantage of the potential speedup from using a higher MBS for reference logprobs. Under the old code, if you set MBS == log_prob_forward_micro_batch_size did you still have the differences?

The problem with the fixes in this PR is that it negates the entire purpose of the previous design: using a higher MBS for the reference log prob calculations in order to speed it up, because the reference log probs are calculated via no_grad. As I understand it, this new PR just makes the reference logprobs run with the same GBS/MBS as the actor policy, thus not taking advantage of the potential speedup from using a higher MBS for reference logprobs. Under the old code, if you set MBS == log_prob_forward_micro_batch_size did you still have the differences?

@gshennvm and I tried multiple settings in the old code and the issue persists when TP = 4
While I understand that setting a larger log_prob_forward_micro_batch_size enables potential speed up. In practice, we have been setting it to 1 or 2, which does not enjoy the benefit.
The current problem is that, we don't know if the log probs difference is due to pure randomness (I personally doubt so) or some major bug. Therefore, it is just safe to use this new change so that we can eliminate any potential issue.
Besides the code refactoring, a critical change is this https://github.com/NVIDIA/NeMo-Aligner/pull/228/files#diff-42df318806389e8adb3b04283f2bd1c33ef2a241f212cdedcc4b0a82c959e49dR423, which I believe is indeed a bug.

I think I was wrong about this line. It is not a bug in the original implementation.
https://github.com/NVIDIA/NeMo-Aligner/blob/8d8ef38bc7b190d9a3d0b5face76147b9e0ae863/nemo_aligner/models/nlp/gpt/megatron_gpt_dpo_model.py#L345
In the original implementation, the global batch is first concatenated into an array like array([log p(y_1^c), ... log p(y_k^c), log p(y_1^r), ..., log p(y_k^r)]) (link).  Then it is split into num_microbatches and the log probs are computed. Then the logprobs_list is concatenated. It is not a bug.
In the refactored implementation in the PR, I didn't concat the chosen responses and rejected responses together like the original implementation. Instead, the batch is directly passed in without any modification. In this case, each item of the logprobs_list is something like array([log p(y_1^c), log p(y_1^r)]). Therefore, I need to split them first into chosen & rejected and then separately concat them together.
Overall, both implementations have no bugs. The refactored implementation in the PR enables that the log probs mismatch issue is not existent. In the original implementation, I am not sure if the mismatch is due to randomness or some parallelism issue in nemo.

I'm also a bit confused (like @trias702): @gshennvm is saying that we use collect_non_loss_data=True but @shengyangs is saying that collect_non_loss_data = False...

Sorry should have made it clearer. I was saying that in loss_func, the collect_non_loss_data = False. Therefore, we don't need to have this argument (no_loss_data) in loss_func. It is only needed in logprobs_func.

The problem with the fixes in this PR is that it negates the entire purpose of the previous design: using a higher MBS for the reference log prob calculations in order to speed it up, because the reference log probs are calculated via no_grad. As I understand it, this new PR just makes the reference logprobs run with the same GBS/MBS as the actor policy, thus not taking advantage of the potential speedup from using a higher MBS for reference logprobs. Under the old code, if you set MBS == log_prob_forward_micro_batch_size did you still have the differences?

@gshennvm and I tried multiple settings in the old code and the issue persists when TP = 4
While I understand that setting a larger log_prob_forward_micro_batch_size enables potential speed up. In practice, we have been setting it to 1 or 2, which does not enjoy the benefit.
The current problem is that, we don't know if the log probs difference is due to pure randomness (I personally doubt so) or some major bug. Therefore, it is just safe to use this new change so that we can eliminate any potential issue.

I routinely set my log_prob_forward_micro_batch_size to 4 and 8 depending on the TP/DP topology of my run and I do notice a speedup. It seems like you have perhaps not been setting yours correctly to take advantage of the TP/DP differences?
Also, does this difference between reference and actor logprobs even matter? In your corrected code, have you done an A/B test with the old code on the same data, and does the fix actually improve MT-Bench performance?

Thanks everyone for the comments.
I just pushed a new commits which

enables log_prob_forward_micro_batch_size which is 2 * micro_batch_size by default.
some minor changes
add inference_only=validation_step in from_parallel_logits_to_logprobs as pointed out by @odelalleau

With these changes, the new code can enjoy the speed up by setting log_prob_forward_micro_batch_size larger. Specifically, for both the original implementation and the new implementation, I tried different log_prob_forward_micro_batch_size and print out the first iterations of log probs. It shows that

In the original implementation, the policy log probs and the reference log probs are different.
In the new implementation, the policy log probs and the reference log probs are always the same for  log_prob_forward_micro_batch_size=2,4,8.

Therefore, I still recommend to merge this PR because

It can enjoy the same speed up as the original implementation by setting log_prob_forward_micro_batch_size.
We are rest assured that the policy log probs and the reference log probs are no problem no matter what the underlying nemo parallelism becomes.
@trias702 I will do an ablation study with MT bench in a training job comparing these two different implementations.

Original Implementation. log_prob_forward_micro_batch_size=2

Original Implementation. log_prob_forward_micro_batch_size=4

New Implementation. log_prob_forward_micro_batch_size=2

New Implementation. log_prob_forward_micro_batch_size=4

New Implementation. log_prob_forward_micro_batch_size=8

@trias702 I did an apple-to-apple comparison by running RPO on top of 15b_8T_ct3_daring-anteater_lr3e-6 (MT Bench 6.96).
With the same setting sft_loss_weight = 1e-5,

The old implementation. The best model I got is MT bench 6.91.
The new implementation. The best model I got is MT bench 7.04.

When setting sft_loss_weight=0., the new implementation gives MT bench 7.24. I didn't run the old implementation in this setting.
Overall, I think this verifies the effectiveness of the new implementation.

@trias702 I did an apple-to-apple comparison by running RPO on top of 15b_8T_ct3_daring-anteater_lr3e-6 (MT Bench 6.96).
With the same setting sft_loss_weight = 1e-5,
* The old implementation. The best model I got is MT bench 6.91.

* The new implementation. The best model I got is MT bench 7.04.

When setting sft_loss_weight=0., the new implementation gives MT bench 7.24. I didn't run the old implementation in this setting.
Overall, I think this verifies the effectiveness of the new implementation.

Did you train and generate with non-deterministic mode enabled for TransformerEngine? That alone can account for 0.2 - 0.3 swings in MT-Bench, so the 6.91 -> 7.04 may just be noise

@trias702 I did an apple-to-apple comparison by running RPO on top of 15b_8T_ct3_daring-anteater_lr3e-6 (MT Bench 6.96).
With the same setting sft_loss_weight = 1e-5,
* The old implementation. The best model I got is MT bench 6.91.

* The new implementation. The best model I got is MT bench 7.04.

When setting sft_loss_weight=0., the new implementation gives MT bench 7.24. I didn't run the old implementation in this setting.
Overall, I think this verifies the effectiveness of the new implementation.

Did you train and generate with non-deterministic mode enabled for TransformerEngine? That alone can account for 0.2 - 0.3 swings in MT-Bench, so the 6.91 -> 7.04 may just be noise

I didn't set the deterministic model for TE. It is possible that the improvement is from evaluation noise or training noise. This is interesting to me because it shows that the change causes no harm, with the additional benefit of matching the log probs.

@trias702 I did an apple-to-apple comparison by running RPO on top of 15b_8T_ct3_daring-anteater_lr3e-6 (MT Bench 6.96).
With the same setting sft_loss_weight = 1e-5,
* The old implementation. The best model I got is MT bench 6.91.

* The new implementation. The best model I got is MT bench 7.04.

When setting sft_loss_weight=0., the new implementation gives MT bench 7.24. I didn't run the old implementation in this setting.
Overall, I think this verifies the effectiveness of the new implementation.

Did you train and generate with non-deterministic mode enabled for TransformerEngine? That alone can account for 0.2 - 0.3 swings in MT-Bench, so the 6.91 -> 7.04 may just be noise

I didn't set the deterministic model for TE. It is possible that the improvement is from evaluation noise or training noise. This is interesting to me because it shows that the change causes no harm, with the additional benefit of matching the log probs.

I disagree, I'm not sure this ablation study tells us anything, as we have seen multiple times that TE non-deterministic mode (especially compounded twice for training and MT-Bench generation) can cause huge swings in mtb scores. I once ran a monte carlo study and the swings can be as large as 0.4.
I am willing to go on a limb to believe that these PR changes are benign, but you cannot conduct proper scientific ablation studies when using non-deterministic code, because how will you ever know if any changes to measurement variable are due to the ablation or the non-deterministic noise? We simply cannot say with any certainty if the 6.91 -> 7.04 are caused by the PR or TE non-determinism.

I think I was wrong about this line. It is not a bug in the original implementation.
https://github.com/NVIDIA/NeMo-Aligner/blob/8d8ef38bc7b190d9a3d0b5face76147b9e0ae863/nemo_aligner/models/nlp/gpt/megatron_gpt_dpo_model.py#L345
In the original implementation, the global batch is first concatenated into an array like array([log p(y_1^c), ... log p(y_k^c), log p(y_1^r), ..., log p(y_k^r)]) (link). Then it is split into num_microbatches and the log probs are computed. Then the logprobs_list is concatenated. It is not a bug.
In the refactored implementation in the PR, I didn't concat the chosen responses and rejected responses together like the original implementation. Instead, the batch is directly passed in without any modification. In this case, each item of the logprobs_list is something like array([log p(y_1^c), log p(y_1^r)]). Therefore, I need to split them first into chosen & rejected and then separately concat them together.
Overall, both implementations have no bugs. The refactored implementation in the PR enables that the log probs mismatch issue is not existent. In the original implementation, I am not sure if the mismatch is due to randomness or some parallelism issue in nemo.

ah I see -- if i understand right this does a reordering so the chosen and rejected of the same sample stays in the same batch for logprob computation

Did you train and generate with non-deterministic mode enabled for TransformerEngine? That alone can account for 0.2 - 0.3 swings in MT-Bench, so the 6.91 -> 7.04 may just be noise

@trias702  I generated responses with deterministic mode for both with sft_loss_weight=0. The original implementation gets 7.12. The new implementation gets 7.23.

ah I see -- if i understand right this does a reordering so the chosen and rejected of the same sample stays in the same batch for logprob computation

@gshennvm  That is true. Someone this resolves the log probs mismatch issue.
Overall, I think we should merge this PR, for

Log probs mismatch is not existent, even if log_probs_forward_mbs is big.
You can set log_probs_forward_mbs big to have speed up.
Performance-wise, a little improvement in MT bench.

I think I was wrong about this line. It is not a bug in the original implementation.
https://github.com/NVIDIA/NeMo-Aligner/blob/8d8ef38bc7b190d9a3d0b5face76147b9e0ae863/nemo_aligner/models/nlp/gpt/megatron_gpt_dpo_model.py#L345
In the original implementation, the global batch is first concatenated into an array like array([log p(y_1^c), ... log p(y_k^c), log p(y_1^r), ..., log p(y_k^r)]) (link). Then it is split into num_microbatches and the log probs are computed. Then the logprobs_list is concatenated. It is not a bug.
In the refactored implementation in the PR, I didn't concat the chosen responses and rejected responses together like the original implementation. Instead, the batch is directly passed in without any modification. In this case, each item of the logprobs_list is something like array([log p(y_1^c), log p(y_1^r)]). Therefore, I need to split them first into chosen & rejected and then separately concat them together.
Overall, both implementations have no bugs. The refactored implementation in the PR enables that the log probs mismatch issue is not existent. In the original implementation, I am not sure if the mismatch is due to randomness or some parallelism issue in nemo.

ah I see -- if i understand right this does a reordering so the chosen and rejected of the same sample stays in the same batch for logprob computation

Can we please verify this? At the moment this PR makes a lot of changes which aren't really necessary or even advantageous. I think we need to lock down what is the individual line(s) of code which fixes the problem, and then just change that. I am not a fan of folding the logprob function into fwd_bwd as has been done here, if instead we can just change the array order in the fwd_only logprobs function instead

Did you train and generate with non-deterministic mode enabled for TransformerEngine? That alone can account for 0.2 - 0.3 swings in MT-Bench, so the 6.91 -> 7.04 may just be noise

@trias702 I generated responses with deterministic mode for both with sft_loss_weight=0. The original implementation gets 7.12. The new implementation gets 7.23.

ah I see -- if i understand right this does a reordering so the chosen and rejected of the same sample stays in the same batch for logprob computation

@gshennvm That is true. Somehow this resolves the log probs mismatch issue.
Overall, I think we should merge this PR, for
* Log probs mismatch is not existent, even if `log_probs_forward_mbs` is big.

* You can set `log_probs_forward_mbs` big to have speed up.

* Performance-wise, a little improvement in MT bench.

But this is still not correct, since you did the training with non-deterministic mode, so there is random noise in your system, and therefore we cannot isolate what is causing the positive change.
Please re-train with pure deterministic mode, and then generate with deterministic mode too, so we can all know for sure what is happening.

But this is still not correct, since you did the training with non-deterministic mode, so there is random noise in your system, and therefore we cannot isolate what is causing the positive change.
Please re-train with pure deterministic mode, and then generate with deterministic mode too, so we can all know for sure what is happening

Just checked that the models have been trained with deterministic mode.
&& export NVTE_ALLOW_NONDETERMINISTIC_ALGO=0 \
&& export NVTE_MASKED_SOFTMAX_FUSION=0 \

But this is still not correct, since you did the training with non-deterministic mode, so there is random noise in your system, and therefore we cannot isolate what is causing the positive change.
Please re-train with pure deterministic mode, and then generate with deterministic mode too, so we can all know for sure what is happening

Just checked that the models have been trained with deterministic mode.
&& export NVTE_ALLOW_NONDETERMINISTIC_ALGO=0 \
&& export NVTE_MASKED_SOFTMAX_FUSION=0 \

I'm confused, before you said you only did deterministic mode for generation, but now you are saying you used deterministic mode for generation and training both?

I'm confused, before you said you only did deterministic mode for generation, but now you are saying you used deterministic mode for generation and training both?

There were some misunderstand between us. Here is the clarification:

All the training were run with deterministic mode.
Previous numbers were evaluated with non-deterministic mode.
** sft_loss_weight=1e-5: original implementation: 6.91
** sft_loss_weight=1e-5: new implementation: 7.04
** sft_loss_weight=0: new implementation: 7.24
New numbers were evaluated with deterministic mode.
** sft_loss_weight=0: original implementation: 7.12
** sft_loss_weight=0: new implementation: 7.23

@gshennvm @trias702 If no more questions, can you please approve and we can merge the PR?

Which version of megatron-lm are you using? I found the same problem and want to use your code to quick fix this problem. But I found a little problem in finding the megatron/core/num_microbatches_calculator dependency.

The function get_ref_policy_log_prob fails when pipeline_parallel_size > 1

Which version of Megatron-LM are you using? I found the same problem and want to use your code to fix it quickly. However, I had a little problem finding the megatron.core.num_microbatches_calculator dependency.

I used megatron_core-0.7.0


Fixes for messages:

Fix the English for the ‘min’ and ‘max’ block tooltips
Some additional cleanup for unused variables

If you haven't pushed #615 to gerrit yet, please review this one instead @jisqyv @halatmit

@josmas I just closed #615, it was not set to Gerrit.

@josmas LGTM

MERGED


[Error] Finetuning: CPUAdam param is on cuda:0 and must be 'cpu'

Problem:
When I run a finetuning on llama2 with a custom dataset, I get the following error:
AssertionError: CPUAdam param is on cuda:0 and must be 'cpu', make sure you enabled 'offload_optimizer': 'cpu' in your ZeRO config.
Code:
from xturing.datasets.instruction_dataset import InstructionDataset
from xturing.models import BaseModel

instruction_dataset = InstructionDataset("./train")
# Initializes the model
model = BaseModel.create("llama2")
# Finetuned the model
model.finetune(dataset=instruction_dataset)
# Once the model has been finetuned, you can start doing inferences
output = model.generate(texts=["Why LLM models are becoming so important?"])
print("Generated output by the model: {}".format(output))
# Save the model
model.save("./llama_weights")

Logs:
Traceback (most recent call last):
  File "/workspace/xturingexample/llama2_train.py", line 9, in <module>
    model.finetune(dataset=instruction_dataset)
  File "/usr/local/lib/python3.10/dist-packages/xturing/models/causal.py", line 119, in finetune
    trainer.fit()
  File "/usr/local/lib/python3.10/dist-packages/xturing/trainers/lightning_trainer.py", line 187, in fit
    self.trainer.fit(self.lightning_model)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 545, in fit
    call._call_and_handle_interrupt(
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/call.py", line 44, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 581, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 990, in _run
    results = self._run_stage()
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 1036, in _run_stage
    self.fit_loop.run()
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/fit_loop.py", line 202, in run
    self.advance()
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/fit_loop.py", line 359, in advance
    self.epoch_loop.run(self._data_fetcher)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/training_epoch_loop.py", line 136, in run
    self.advance(data_fetcher)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/training_epoch_loop.py", line 240, in advance
    batch_output = self.automatic_optimization.run(trainer.optimizers[0], batch_idx, kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/optimization/automatic.py", line 187, in run
    self._optimizer_step(batch_idx, closure)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/loops/optimization/automatic.py", line 265, in _optimizer_step
    call._call_lightning_module_hook(
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/call.py", line 157, in _call_lightning_module_hook
    output = fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/core/module.py", line 1282, in optimizer_step
    optimizer.step(closure=optimizer_closure)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/core/optimizer.py", line 151, in step
    step_output = self._strategy.optimizer_step(self._optimizer, closure, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/strategies/strategy.py", line 230, in optimizer_step
    return self.precision_plugin.optimizer_step(optimizer, model=model, closure=closure, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/plugins/precision/precision_plugin.py", line 117, in optimizer_step
    return optimizer.step(closure=closure, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/optim/lr_scheduler.py", line 68, in wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/optim/optimizer.py", line 373, in wrapper
    out = func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/deepspeed/ops/adam/cpu_adam.py", line 150, in step
    assert p.device == device, f"CPUAdam param is on {p.device} and must be 'cpu', make " \
AssertionError: CPUAdam param is on cuda:0 and must be 'cpu', make sure you enabled 'offload_optimizer': 'cpu' in your ZeRO config.

Hi, @Koliham
Can you please share what is your machine setup, how many GPUs are you using?


error when do 'make'

0. Before Your ASK

Try to find an answer by reading a FAQ.

Ⅰ. Issue Description
error in nginx Make process
Ⅱ. Describe what happened
try nginx 1.21.0 and 1.24.0
cd nginx file
'configure' -- ok
'make': some errors happens
example:
src/event/ngx_event_openssl.c: In function ‘ngx_ssl_load_certificate_key’:
src/event/ngx_event_openssl.c:729:9: error: ‘ENGINE_by_id’ is deprecated: Since OpenSSL 3.0 [-Werror=deprecated-declarations]
729 |         engine = ENGINE_by_id((char *) p);
|         ^~~~~~
In file included from src/event/ngx_event_openssl.h:22,
from src/core/ngx_core.h:84,
from src/event/ngx_event_openssl.c:9:
/usr/include/openssl/engine.h:336:31: note: declared here
336 | OSSL_DEPRECATEDIN_3_0 ENGINE *ENGINE_by_id(const char *id);
|                               ^~~~~~~~~~~~
...
cc1: all warnings being treated as errors
make[1]: *** [objs/Makefile:888: objs/src/event/ngx_event_openssl.o] Error 1
make[1]: Leaving directory '/home/ec2-user/nginx-1.21.0'
make: *** [Makefile:10: build] Error 2
Ⅲ. Describe what you expected to happen
how to solve this problem
Ⅳ. How to reproduce it (as minimally and precisely as possible)

wget https://github.com/chobits/ngx_http_proxy_connect_module/archive/refs/heads/master.tar.gz
sudo wget http://nginx.org/download/nginx-1.21.0.tar.gz
tar
cd nginx
./configure --user=www --group=www --prefix=/usr/local/nginx --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-threads --add-module=/home/ec2-user/ngx_http_proxy_connect_module-master
make

Ⅴ. Anything else we need to know?

If applicable, add nginx  debug log.

Ⅵ. Environment:

Tengine/Nginx/OpenResty version (use sbin/nginx -v):
Which patch do you use?

and i do the patch and make again, stiil have the problem

hi
Two problem I found here:

It seemed that you didnt patch nginx source with proper patch provied by proxy_connect module. see here for how to patch & build this module: https://github.com/chobits/ngx_http_proxy_connect_module#install

The conpilation error seems a SSL compilation error, you can disable this module to check whether it still exists.

BTW, new triggered testing for 1.24.0 has passed, you can check it here: https://github.com/chobits/ngx_http_proxy_connect_module/actions/runs/4686133835/jobs/8303899129?pr=260

Thanks, I'll check it out.
Xiaochen Wang @.***> 于2023年4月13日周四 14:27写道：

hi
Two problem I found here:

It seemed that you didnt patch nginx source with proper patch provied
by proxy_connect module. see here for how to patch & build this module:
https://github.com/chobits/ngx_http_proxy_connect_module#install
2.
The conpilation error seems a SSL compilation error, you can disable
this module to check whether it still exists.

BTW, new triggered testing for 1.24.0 has passed, you can check it here:
https://github.com/chobits/ngx_http_proxy_connect_module/actions/runs/4686133835/jobs/8303899129?pr=260
—
Reply to this email directly, view it on GitHub
https://github.com/chobits/ngx_http_proxy_connect_module/issues/259#issuecomment-1506417051,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/A6U3X3PILP4BATU3VNJY3TTXA6MDTANCNFSM6AAAAAAW3DXV24
.
You are receiving this because you authored the thread.Message ID:
@.***>


Problem with the way subtheme option is passed around

Reported by Dunhamzzz | June 1st, 2011 @ 10:21 AM
There is a slight problem with the way that the subTheme option is passed around, in particular this is causing a problem when using the comments plugin with AJAx implementations.
At the minute, the subThemes option is passed into the $params array of the commentWidgetHelper, allowing you to easily select a theme with this line:
<?php echo $commentWidget->display(array('subtheme' => 'products'));?>

This is fine, a problem however rises when implementing AJAX, as the ajax.ctp element that you are supposed to call as per the documentation calls $commentWidget->display(); without the possibility of adding any arguments.
My suggestion would be to make subtheme an option you pass with $commentWidget->options() method and simply amend the display() method to look in the helpers globalvars array instead of the passed $params array in the helper.
Hope this helps, with this working the comments plugin is extremely powerful!

add diplayOptions params into ajax.ctp


Fix bind-mounts only partially removed

When calling delete on a bind-mount volume, the config file was bing
removed, but it was not actually being removed from the volume index.
Found this when working on #10308

One small remark on style, but LGTM on the intent.

@icecrime rebased and updated with the suggested style.

Thanks! LGTM

LGTM


Add a pod mutating webhook to auto inject the pod network resources

Thanks for contributing!
Notice:

[x] unite test or E2E test
[x] do not forget essential code comment and log
[x] document for the PR
[x] release note label
"release/none"
"release/bug"
"release/feature"
[x] read about  Contribution notice: https://spidernet-io.github.io/spiderpool/latest/develop/contributing/

What issue(s) does this PR fix:

Fixes https://github.com/spidernet-io/spiderpool/issues/4177
Special notes for your reviewer:

如果 multusconfig 配置中没有 rdma resource， 是否也能 正常工作 ？


Remove obsolete comma

@marcvaneijk : Thanks for your contribution! The author, @zroiy, has been notified to review your proposed change.

#sign-off

@cynthn: I'm sorry - only the author of this article, @zroiy, can sign off on your changes. But we do have an exception process - if you are on the Microsoft content or product team for this product area, you can ask the PR review team to review and merge it by sending mail to the techdocprs alias.


Bulk data export returns as an array, rather than just a sequence of resources

Bulk data export format should be:
{ resource }
{ resource }
{ resource }
...

But it currently returns as:
[{ resource }{resource }{resource}…]

Please confirm that this issue is resolved.


Generate typed-router.d.ts in a specific generated folder

Currently typed-router.d.ts is generated directly in the root directory, which might not be elegant for a generated typing file that should be added to .gitignore. Some other frameworks choose to create directory like .nuxt and .svelte-kit to place generated files. Maybe unplugin-vue-router can similarly put the typings in location such as .router/typed-router.d.ts as the default behavior.

This is already doable with the dts option. I prefer to keep the default generation at the root to be visible and let people decide where they want to put it.


Question: Source For The Room Caps

Iv been trying to find a source for the Room Point caps to answer these two questions.
1: Is the point cap per room actually 20,000?
2: Is the point cap per room raid wide or individual wide?

https://secure.runescape.com/m=news/tombs-of-amascut-drop-mechanics--osmumtens-fang?oldschool=1

You cannot gain more than 20,000 Room points as an individual in a single room.
You cannot accumulate more than 64,000 Total points as an individual in the entire raid.

This is also backed up by the wiki calculator which had input from jmods when it was being created: https://oldschool.runescape.wiki/w/Module:Tombs_of_Amascut_loot
It won't have the cap on it, but it does show the "puzzle" rooms being treated the same as boss rooms in terms of point mechanics.


Pre-request script doesn't get executed

Expected Behavior
Pre-request script should be executed
Actual Behavior
I checked using console.log, the script inside pre-request script doesn't get executed at all
Reproduction Steps
https://github.com/Kong/insomnia/assets/145996165/7093ea5e-f1de-4eee-8c0d-459717313a39
Is there an existing issue for this?

[X] I have searched the issue tracker for this problem.

Additional Information
No response
Insomnia Version
v9.2.0
What operating system are you using?
Ubuntu
Operating System Version
Linux Mint 21.3
Installation method
tar.gz
Last Known Working Insomnia version

Hello @ywidiastuti , the console.log doesn't print to the dev tool, could you check outputs in the "timeline" tab in the right bar?

Thank you for the information @ihexxa . I found them 👍


Tray icon appears when exiting the quake shell

Windows Terminal version
1.19.2682.0
Windows build number
10.0.22635.0
Other Software
No response
Steps to reproduce

Ensure the two notification area options are disabled
Launch Windows Terminal
Win`
Either

Execute exit
CtrlShiftW

Expected Behavior
No tray icon.
Actual Behavior
A wild tray icon appears!

There are two other phenomen associated with closing the quake window's shell.  In WT, press Win+backtick.  The quake window opens, nicely spanning the top of one monitor, its accent-color border right at the left, top, and right screen edges.  Exit the shell in the quake window and in the original WT, press Win+backtick again.  This time ...

The quake window is wider; its accent border off the right and left screen edges, and
It has a tab bar (but no tabs) at the top.

Here's the original WT with the left-top of the monitor (showing the second quake window) in the background.

Yes, I'm on Windows 10.  It could be the same issue; I didn't read carefully.  I saw something about "borderless" in there.  In my experiment, the quake window has a border (accent color) is both cases but it's on the monitor in the first case and off the monitor in the second case.


[GLUTEN-1141][FOLLOWUP][VL]Fix q31 join key not found

What changes were proposed in this pull request?
Same problem as https://github.com/oap-project/gluten/issues/1141

We can ignore the comparison of exprId when only one attribute name match is found in output.
How was this patch tested?
tpcds q31 3g

cc @zhouyuan

@YannByron Counld you help test this patch in your cluster?


Cannot load a saved model using "keras.models.load_model"

I am able to save a model after training in .h5 format using save() method without a problem, but facing problems while loading the model.
tensorflow == 2.4.1
(working on GCP)
**Error : **
raise ValueError('Unknown ' + printable_module_name + ': ' + class_name)
ValueError: Unknown layer: ClassToken

In order to diagnose, please provide code which reproduces the problem.