node.py: Fix node:watch_log_for to handle multiple expressions matchi…

…ng same line
Original impl did not handle doing something like:
watch_log_for([".*Prepare.kossa.", ".*Prepare.mu."])
and the log producing either
[...] Prepare kossa, elk, ninja
[...] Prepare mu, tuta
or
[...] Prepare kossa, elk, mu
i.e. expressions that match same or different lines, depending on
timing.
This will run each match on every line instead, which seems more
correct.
See /scylladb/scylla#6067

@bhalevy Yes, you are right. This fundamentally changes the semantics of wait_log_for(). In particular it will be impossible to use a single call for this function when we wait for multiple appearances of the same line like we do in hintedhandoff_removenode_test().
In particular it I don't see how it would be possible to check such an event after this patch.
What we see in the patch description is exactly what you expect when you want to continue after all two patterns have been seen.
If we want to match all patterns to all log lines then we need to return the mark where each match has been detected. - This way we would be able to solve the problem I described above.

@elcallio @bhalevy Yes, you are right. This fundamentally changes the semantics of wait_log_for(). In particular it will be impossible to use a single call for this function when we wait for multiple appearances of the same line like we do in hintedhandoff_removenode_test().
In particular it I don't see how it would be possible to check such an event after this patch.

So, basically the inverse of what my perceived use case was?
The obvious answer would be "loop repeat wait_log_for, but then the obvious retort is "you can implement the same tactics for the match-same-line use case".
Perhaps simply add a default-no parameter that determines match mode?

@elcallio @bhalevy Yes, you are right. This fundamentally changes the semantics of wait_log_for(). In particular it will be impossible to use a single call for this function when we wait for multiple appearances of the same line like we do in hintedhandoff_removenode_test().
In particular it I don't see how it would be possible to check such an event after this patch.

So, basically the inverse of what my perceived use case was?
The obvious answer would be "loop repeat wait_log_for, but then the obvious retort is "you can implement the same tactics for the match-same-line use case".
Perhaps simply add a default-no parameter that determines match mode?

@elcallio The semantics of wait_log_for() today is that if is going to actually wait till specified lines appear in the log.
What you suggested before can be achieved by grep_log() from node.py and the whole reason for existence of this wait_log_for() is to avoid these loops.

Yeah, I conclude modifying this is just gonna be error prone and cause conflict. I solved it in test instead. A little more expensive, but good enough.


python/gpu/test_rtc.py test fails

For bugs or installation issues, please provide the following information.
The more information you provide, the more likely people will be able to help you.
Environment info
Operating System: ubuntu (AWS DL Ubuntu AMI)
Compiler: gcc version 4.8.4
Package used (Python/R/Scala/Julia): python
MXNet version:
Or if installed from source:
MXNet commit hash (git rev-parse HEAD): a23608fcd07e348ce4818b5cde4a95feb5068b3b
If you are using python package, please provide
Python version and distribution: Python 2.7.6
If you are using R package, please provide
R sessionInfo():
Error Message:
Please paste the full error message, including stack trace.
python ./tests/python/gpu/test_rtc.py       [1/1209]
[23:20:47] /home/ubuntu/mxnet-most-recent-2/dmlc-core/include/dmlc/./logging.h:300: [23:20:47] src/common/mxrtc.cc:46: Check failed: err = cuModuleLoadDataEx(&module, ptx_, 0, 0, 0) == CUDA_SUCCESS (3 vs. 0) CudaError: 3

Stack trace returned 10 entries:
[bt] (0) /home/ubuntu/mxnet-most-recent-2/python/mxnet/../../lib/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7fb5f4b5bb2c]
[bt] (1) /home/ubuntu/mxnet-most-recent-2/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet5MXRtc4pushERKSt6vectorINS_7NDArrayESaIS2_EES6_jjjjjj+0x5ab) [0x7fb5f53f610b]
[bt] (2) /home/ubuntu/mxnet-most-recent-2/python/mxnet/../../lib/libmxnet.so(MXRtcPush+0x7d0) [0x7fb5f53b5390]
[bt] (3) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) [0x7fb64fa59adc]
[bt] (4) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x1fc) [0x7fb64fa5940c]
[bt] (5) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(_ctypes_callproc+0x48e) [0x7fb64fc705fe]
[bt] (6) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(+0x15f9e) [0x7fb64fc71f9e]
[bt] (7) python(PyEval_EvalFrameEx+0x98d) [0x5244dd]
[bt] (8) python(PyEval_EvalCodeEx+0x2b1) [0x555551]
[bt] (9) python(PyEval_EvalFrameEx+0x1a10) [0x525560]

Traceback (most recent call last):
  File "./tests/python/gpu/test_rtc.py", line 15, in <module>
    rtc.push([x], [y], (1, 1, 1), (10,1,1))
  File "/home/ubuntu/mxnet-most-recent-2/python/mxnet/rtc.py", line 91, in push
    mx_uint(block_dims[2])))
  File "/home/ubuntu/mxnet-most-recent-2/python/mxnet/base.py", line 78, in check_call
    raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [23:20:47] src/common/mxrtc.cc:46: Check failed: err = cuModuleLoadDataEx(&module, ptx_, 0, 0, 0) == CUDA_SUCCESS (3 vs. 0) CudaError: 3

Stack trace returned 10 entries:
[bt] (0) /home/ubuntu/mxnet-most-recent-2/python/mxnet/../../lib/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7fb5f4b5bb2c]
[bt] (1) /home/ubuntu/mxnet-most-recent-2/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet5MXRtc4pushERKSt6vectorINS_7NDArrayESaIS2_EES6_jjjjjj+0x5ab) [0x7fb5f53f610b]
[bt] (2) /home/ubuntu/mxnet-most-recent-2/python/mxnet/../../lib/libmxnet.so(MXRtcPush+0x7d0) [0x7fb5f53b5390]
[bt] (3) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) [0x7fb64fa59adc]
[bt] (4) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x1fc) [0x7fb64fa5940c]
[bt] (5) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(_ctypes_callproc+0x48e) [0x7fb64fc705fe]
[bt] (6) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(+0x15f9e) [0x7fb64fc71f9e]
[bt] (7) python(PyEval_EvalFrameEx+0x98d) [0x5244dd]
[bt] (8) python(PyEval_EvalCodeEx+0x2b1) [0x555551]
[bt] (9) python(PyEval_EvalFrameEx+0x1a10) [0x525560]

Minimum reproducible example
if you are using your own code, please provide a short script that reproduces the error.
Steps to reproduce
or if you are running standard examples, please provide the commands you have run that lead to the error.

python ./tests/python/gpu/test_rtc.py

What have you tried to solve it?

Haven't attempted it yet.

This issue is closed due to lack of activity in the last 90 days. Feel free to ping me to reopen if this is still an active issue. Thanks!


Add search functionality to start page

As a user, I want to be able to pick-up where I left once I start going through dFlow.
It is currently possible to enter a topic ID, but this mechanism is not easy to use.
A search functionality similar to the one in ToB is necessary to make this process more user-friendly and easy to use.

This was resolved by #155


Real Time Capture Question

Hi,
I'm using the Arkime container, and I have a question about its capabilities. Does the Arkime container support real-time packet capture, or is it limited to ingesting pre-saved pcap files as described in the README?
Thank you!

Hi @jagonza1ez,
I never tried to capture the network traffic from within the Arkime container. Currently my only use case for Arkime is to put already captured .pcap files into it. Therefore, I added the phrase This feature is not tested yet! to the README.
Nevertheless the Arkime capture process should be started if you set the env variable CAPTURE=on and ARKIME_INTERFACE to your network device name, e.g. eth0.
To access your host's network device from inside the container it should be possible to forward it with the --device parameter (https://docs.docker.com/reference/cli/docker/container/run/#device). Also the --privileged flag could be needed here.
Maybe you can try to run the Arkime container with the following parameter:
docker run \
    -e OS_HOST=opensearch \
    -e OS_PORT=9200 \
    -e CAPTURE=on \
    -e ARKIME_INTERFACE=capturenet0 \
    -v <config_volume>:/data/config \
    -v <pcap_volume>:/data/pcap \
    -v <log_volume>:/data/logs \
    --device=/dev/eth0:/dev/capturenet0 \
    --privileged \
    mammo0/docker-arkime:<tag>

Please tell me, if this works. Then I can update my README. Thank you.


Cannot install gnumake man-pages to profile

GNU Make does not install man pages into the active profile.
Steps to reproduce
If, for example installing GNU Make via.
nix-env -iA nixos.gnumake
nix-env -iA nixos.gnumake.man
nix-env reports successful install, I even have the man-pages in my /nix/store
/nix/store/gx70gyfwdkqf5vlicm9dwaqyklxqzzax-gnumake-4.2.1-man
but these pages are not in my ~/.nix-profile
configuration.nix is default:
programs.man.enable = true;
Have I installed these incorrectly, or require an additional config line?

Hey, this is kind of old. I'm experiencing the same behaviour with nixos.gnumake.info and I can't see them linked in my ~/.nix-profile/share/info. I've opened a thread on the forums. Could you please share what is the reason you closed this?


Add custom HTTP headers option

I think it could be a helpful feature, specially for URLs that behave differently depending on headers content (I'm thinking about LinkedIn at this moment, but not only).
If you this feature would be interesting, I can create a pull request.

I also have this requirement, because some servers require certain headers which are reliably sent by browsers (Accept in the case of the one I'm struggling with at the moment) and may respond with a 400 or 404 if those headers aren't present.
I also ran into the LinkedIn issue @miguelcalderon refers to, where it seems to respond to BLC with a non-standard HTTP 999 status.

I guess that custom headers should be applied to all requests to URLs with the base url of siteURL?

I would have said apply it to all requests regardless... but I am not totally sure I grok what siteUrl does.  I use BLC via https://github.com/LukasHechenberger/broken-link-checker-local.
After some digging with the site that was failing for me (crates.io), I've realised that it actually returns a 200 OK for every request, regardless of path, and then 'boots' an Ember application which displays a 'not found' error if the path can't be routed. Wonderful 🤦 .  But I think custom headers would address the restrictions imposed by LinkedIn, and probably others.

I think that sending headers to all URLs will create unexpected issues.

Yeah this would be a great feature.
A method in the config object might be better than a complex configuration file.
new SiteChecker(options, {
  headers (request) {
    if (/foo/.test(request.url))
      request.headers.some-header = 'special value'
  }
})

Although it looks as though this syntax of passing handlers in as the second argument in new SiteChecker(options, handlers) (as shown on npmjs) has been deprecated? the readme here on github has event based handlers, which couldn't be used to update the request like this.


Formatting

Add capital letters to some descriptions, add a full stop at the end of each description, add "back to top" button under Hackathons.

Editor's line wrap made it seem like two separate entries. Fixed it.


Add redirect async

Closes https://github.com/SneaksAndData/adapta/issues/238
This PR adds final strokes for full asyncio-safe logging support with DataDog log handler by enabling async redirects from non-python processes. Thus we consider "multiple loggers" feature implemented - for async programs only, though. However since we aim at using asyncio-only programs, I consider this good enough :)
This change is required for Nexus to be compatible with frameworks that use earlier versions of adapta themselves.
Scope
Implemented:

Added redirect_async method that is designed purely for async programs and uses an async context. Also added this to LoggerInterface
Added regular redirect context-managed method to LoggerInterface
Added support for redirect via sync and async loggers, with async logger pretending to be sync (backwards-compat option)

Additional changes:

Added proper unit tests for output rediretion using libc printf
Bumped pylint from 2 to 3
Bumped minor from 2.7 to 2.8

Checklist

[x] GitHub issue exists for this change.
[x] Unit tests added and they pass.
[x] Pylint 10.0/10.0 without bloating .pylintrc with exceptions.
[x] Self-review.

Oh great tests fail on GH. Will investigate


"ls *" -> ls: Application Data: Path Error.

$ echo *
*
<MARKEDONE:~>[1]
$ ls *
ls: Application Data: Path Error.

"%USERPROFILE%\Application Data" was broken symbolic link.
So,the Commit 7f433ca73981d05c79ad1cd468214232924ec5af is enough modifying to close this issue.


[check_ceph_osd] says ok when an OSD is Down+out

Hi,
Thanks for this plugin, I have one question about check_ceph_osd, it would be nice if the below output generate a warning. An OSD is down but the status of OSD's are OK so we dont get a warning that an OSD is out. The OSD was faulty and was taken out by ceph itself.
OSD OK
Up OSDs: osd.1 osd.4 osd.7 osd.10
Down+In OSDs:
Down+Out OSDs: osd.13

I think using the option --out should do what you want, and throw a WARNING for a down/out OSD.


Channels display add for MPC

Related to this issue: https://github.com/MediaArea/MediaInfoLib/issues/535#issuecomment-308850445
Taken from FFmpeg code source:
https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/mpc.c#L94
Thank you to @JeromeMartinez for his advice :wink:

Closes https://github.com/MediaArea/MediaInfoLib/issues/535


KeyNotFoundException: The given key was not present in the dictionary.

Hi,
I might not be using the right terms to describe what I want to achieve but please bear with me on the following description:
This is from he example code ColyseusClient.cs
room.Listen ("players/:id/:axis", this.OnPlayerMove);
room.Listen ("players/:id", this.OnPlayerChange);
room.Listen ("messages/:number", this.OnMessageAdded);
room.Listen("test_messages/:odd_numbers", this.OnMessageAdded);
Where do I check to find the key axis this set in players object?
Where do I check to find the key number for messages?
How do I add new keys?
For example,
constructor () {
super();
this.setState({
  players: {},
  messages: [],
  test_messages: []
});

}
In test_messages I would like a key to be name "odd_numbers" instead of numbers, how to change it? Or documentation explaining this would help.

Hi @shophk,
I've updated the example project. Maybe it's a bit clearer now how it works.
Let me know if you still have any questions. Feel free to re-open this issue if you need to.


Add available dates

#195

Thanks, looks good. Please make sure the PR is up to date with the latest master so we are able to merge this PR.

This can probably be closed in favor of #247

Agreed.


generate_buildpack_bump_pr workflow: push branch to fork

What
https://www.pivotaltracker.com/story/show/185492866
Since https://github.blog/changelog/2023-06-13-fix-to-improve-security-around-creation-of-pull-requests-in-public-repos/ we've been unable to create PRs from internal branches using unprivileged accounts. Instead push our branches to the bot user's fork, mimicking the pattern a random member of the public would use if they were to submit a PR to paas-cf.
How to review
See #3325, created by a test run of this @ https://github.com/alphagov/paas-cf/actions/runs/5645902521

🚨⚠ Please do not merge this pull request via the GitHub UI ⚠🚨

No, because it clones from the original repo and therefore bases its work on its' default branch.


linking with 'cc' cannot find -lpythonXX

Hi and first, thank you for this project, it's been extremely helpful!
I'm hoping you can help me out with the following error i get using the maturin-action. I've got a stripped down project that I'm using as a proof of concept to see. It only contains the Using Python from Rust example from pyo3, and pyproject.toml from maturin new -b bin command.
I continue to receive the following error when building for linux (building MacOS universal2 wheels also has problems, windows wheels build fine) I also don't have this problem building wheels on the desktop (both on linux X86_64 and MacOS arm64)
error: linking with `cc` failed: exit status: 1
...
/opt/rh/devtoolset-10/root/usr/libexec/gcc/x86_64-redhat-linux/10/ld: cannot find -lpython3.XX # where XX is the python version

full logs here https://github.com/dpgraham4401/quil/actions/runs/3345795050
I figure this is user error, is there something in pyo3/maturins/maturin-actions documentation that I'm missing or is building wheels this way not a supported feature?
The full workflow file can be found here, this is the linux job based on the examples in maturin-action's README.
jobs:
  linux:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        target: [ x86_64 ]
        py-version: ['3.9', '3.10']
    steps:
      - uses: actions/checkout@v3

      - uses: actions/setup-python@v4
        with:
          python-version: ${{ matrix.py-version }}

      - name: build wheels
        uses: messense/maturin-action@v1
        with:
          target: ${{ matrix.target }}
          manylinux: auto
          maturin-version: latest
          args: --release -i ${{ matrix.py-version }} --out dist

I think the problem is that no Python is in PATH in quay.io/pypa/manylinux2014_x86_64:latest image.

Perhaps you can try container: ghcr.io/pyo3/maturin.
https://github.com/PyO3/maturin/blob/main/Dockerfile

I tried using the container: ghcr.io/pyo3/maturin but I got an error trying to use the same commands as before here. I also tried just running the containers from my local workstation, but I don't have that log info at the moment. I should have some more time for this tomorrow.
Thanks!

That error should be fixed by https://github.com/messense/maturin-action/commit/326dfcfb49111ade8850836fe5afdca76c4c92a0, please try again with uses: messense/maturin-action@main.

similar problem unfortunately,
I tried a couple variants with uses: messense/maturin-action@main since I wasn't sure what what you meant.

maturin-action@main with contianer: ghcr.io/pyo3/maturin
maturin-action@main with manylinux: auto
one where I removed the `-i ${{ matrix.py-version }} flag to see if that was causing problems, but I receive the Python interpreter version (3.6) is lower than PyO3's minimum supported version (3.7)

Can you test uses: messense/maturin-action@main plus manylinux: auto and container: ghcr.io/pyo3/maturin ?

(similar results](https://github.com/dpgraham4401/quil/actions/runs/3355484228/jobs/5559778998#step:4:204) :/

It's kinda expected to have issues since this action is built for building Python extension module (which does not link to libpython) while what you are trying to build is a binary that
must link to libpython.

I'm hoping the manylinux containers have the necessary Cpython libraries to link to (would need to look closer), and it is just a matter of adjusting the RUSTFLAGS. But I'm not particularly familiar with the subject.
I may mess around with the manylinux container and see if I can manually link the necessary library (if present) and i'll let you know if I find anything.
At least we fixed the use case for the ghcr.io/pyo3/maturin container (yay!).

📦 Including license file "LICENCE"
🐍 Found CPython 3.10 at /opt/python/cp310-cp310/bin/python3.10
📡 Using build options bindings from pyproject.toml
   Compiling pyo3-build-config v0.17.2
   Compiling pyo3-macros-backend v0.17.2
   Compiling pyo3-ffi v0.17.2
   Compiling pyo3 v0.17.2
error: failed to run custom build command for `pyo3 v0.17.2`

Caused by:
  process didn't exit successfully: `/io/quil/target/debug/build/pyo3-446e9ed31194a563/build-script-build` (exit status: 1)
  --- stdout
  cargo:rustc-cfg=Py_3_6
  cargo:rustc-cfg=Py_3_7
  cargo:rustc-cfg=Py_3_8
  cargo:rustc-cfg=Py_3_9
  cargo:rustc-cfg=Py_3_10

  --- stderr
  error: The `auto-initialize` feature is enabled, but your python installation only supports embedding the Python interpreter statically. If you are attempting to run tests, or a binary which is okay to link dynamically, install a Python distribution which ships with the Python shared library.

  Embedding the Python interpreter statically does not yet have first-class support in PyO3. If you are sure you intend to do this, disable the `auto-initialize` feature.

  For more information, see https://pyo3.rs/v0.17.2/building_and_distribution.html#embedding-python-in-rust
warning: build failed, waiting for other jobs to finish...
  error: could not find native static library `python3.10`, perhaps an -L flag is missing?

error: aborting due to previous error

error: could not compile `pyo3-ffi` due to 2 previous errors
💥 maturin failed
  Caused by: Failed to build a native library through cargo
  Caused by: Cargo build finished with "exit status: 101": `"cargo" "rustc" "--message-format" "json" "--bin" "quil"`

manylinux docker images will not work because they deliberately does not include the Python shared library, see https://github.com/pypa/manylinux/issues/255#issuecomment-450761634

Thanks for pointing that out.
I was thinking I could use the manylinux images as a base image and add the static library but I think that defeats the purpose of the manylinux container.
Thanks for your patience, I think we can close this issue.


Testing fails on Debian

Using version 3.0.1 + fix for select, here is the output from the tests:
[==========] Running 22 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 22 tests from xsimd
[ RUN      ] xsimd.sse_float_trigonometric
[       OK ] xsimd.sse_float_trigonometric (9 ms)
[ RUN      ] xsimd.sse_double_trigonometric
[       OK ] xsimd.sse_double_trigonometric (9 ms)
[ RUN      ] xsimd.sse_float_rounding
/<<PKGBUILDDIR>>/test/xsimd_rounding_test.cpp:34: Failure
Value of: res
  Actual: false
Expected: true
[  FAILED  ] xsimd.sse_float_rounding (0 ms)
[ RUN      ] xsimd.sse_double_rounding
/<<PKGBUILDDIR>>/test/xsimd_rounding_test.cpp:41: Failure
Value of: res
  Actual: false
Expected: true
[  FAILED  ] xsimd.sse_double_rounding (0 ms)
[ RUN      ] xsimd.sse_float_power
[       OK ] xsimd.sse_float_power (6 ms)
[ RUN      ] xsimd.sse_double_power
[       OK ] xsimd.sse_double_power (4 ms)
[ RUN      ] xsimd.sse_float_hyperbolic
[       OK ] xsimd.sse_float_hyperbolic (8 ms)
[ RUN      ] xsimd.sse_double_hyperbolic
[       OK ] xsimd.sse_double_hyperbolic (5 ms)
[ RUN      ] xsimd.sse_float_fp_manipulation
[       OK ] xsimd.sse_float_fp_manipulation (0 ms)
[ RUN      ] xsimd.sse_double_fp_manipulation
[       OK ] xsimd.sse_double_fp_manipulation (0 ms)
[ RUN      ] xsimd.sse_float_exponential
/<<PKGBUILDDIR>>/test/xsimd_exponential_test.cpp:35: Failure
Value of: res
  Actual: false
Expected: true
[  FAILED  ] xsimd.sse_float_exponential (10 ms)
[ RUN      ] xsimd.sse_double_exponential
/<<PKGBUILDDIR>>/test/xsimd_exponential_test.cpp:42: Failure
Value of: res
  Actual: false
Expected: true
[  FAILED  ] xsimd.sse_double_exponential (7 ms)
[ RUN      ] xsimd.sse_float_error_gamma
/<<PKGBUILDDIR>>/test/xsimd_error_gamma_test.cpp:35: Failure
Value of: res
  Actual: false
Expected: true
[  FAILED  ] xsimd.sse_float_error_gamma (23 ms)
[ RUN      ] xsimd.sse_double_error_gamma
/<<PKGBUILDDIR>>/test/xsimd_error_gamma_test.cpp:42: Failure
Value of: res
  Actual: false
Expected: true
[  FAILED  ] xsimd.sse_double_error_gamma (13 ms)
[ RUN      ] xsimd.sse_float_basic_math
[       OK ] xsimd.sse_float_basic_math (0 ms)
[ RUN      ] xsimd.sse_double_basic_math
[       OK ] xsimd.sse_double_basic_math (0 ms)
[ RUN      ] xsimd.sse_float_basic
[       OK ] xsimd.sse_float_basic (0 ms)
[ RUN      ] xsimd.sse_double_basic
[       OK ] xsimd.sse_double_basic (0 ms)
[ RUN      ] xsimd.sse_int32_basic
[       OK ] xsimd.sse_int32_basic (0 ms)
[ RUN      ] xsimd.sse_int64_basic
[       OK ] xsimd.sse_int64_basic (0 ms)
[ RUN      ] xsimd.sse_conversion
[       OK ] xsimd.sse_conversion (0 ms)
[ RUN      ] xsimd.sse_cast
[       OK ] xsimd.sse_cast (0 ms)
[----------] 22 tests from xsimd (94 ms total)

[----------] Global test environment tear-down
[==========] 22 tests from 1 test case ran. (94 ms total)
[  PASSED  ] 16 tests.
[  FAILED  ] 6 tests, listed below:
[  FAILED  ] xsimd.sse_float_rounding
[  FAILED  ] xsimd.sse_double_rounding
[  FAILED  ] xsimd.sse_float_exponential
[  FAILED  ] xsimd.sse_double_exponential
[  FAILED  ] xsimd.sse_float_error_gamma
[  FAILED  ] xsimd.sse_double_error_gamma

 6 FAILED TESTS

Which version of gcc are you using ?
I will update tests so they provide more uselfull outputs.

Which version of gcc are you using ?

6.4.0

@JohanMabille Were you able to reproduce this?

@ghisvail sorry I'm very busy with xeus. I don't have gcc 6 on my machines, so I hope travis will catch the error. I should have more time to deal with this problem this week end.

@ghisvail I've added gcc 6 in the matrix test, unfortunately travis uses gcc 6.2 and tests are ok with it. I've added more detailed to the console output when tests fail (on master), could you give me the output you get with GCC 6.4 ?
Also quick question, do you patch the cmake so it uses different compilation flags ? If so could you post them here ?
Last question, is it easy for you to test with gcc 6.2 ? If so can you confirm it works on your side ?
Many thanks for your help on this!

I'll run a test build on debomatic and forward you the build log, which should answer most of your questions. Regarding gcc, I can only test with the current version on unstable (6.4.0 at the moment, 7.x soon).

Cannot reproduce on Debian unstable on master with gcc version 6.4.0 20170820 (Debian 6.4.0-4)

@lorenzhs thanks for testing!
It looks that the differences may come from a specific instruction set. I'm adding a log (both in file and console) that prints the instruction set used.

I have not tried since, and things probably got moving quite a bit in testing / unstable.

I was testing on an i7-4790T (AVX2) machine, for the record.

Closing this since the problem has not been reproduced.


add a space to separate two words

I think we should add a space here, otherwise the two words will join together.
And replace the host string with a constant, otherwise when I need to modify the host, I need to modify several files.

LGTM. Merged to trunk.


Android Hang / Cross Permissions

Please fill out either the bug or feature request section and remove whatever section you are not using.
Bug
Version Number of Plugin: 1.2.1
Device Tested On: Several Android devices
Simulator Tested On:
Expected Behavior
Cross permission will not lock the main thread
Actual Behavior
Cross permission locks the main thread.
Steps to reproduce the Behavior
This problem only replicates in a signed production artifact on android. I am unable to replicate this when publishing directly from visual studio or when pushing a debug artifact. I am not sure what the cause of this can be. The behavior is the app locks up for several seconds ( a minute or two) when requesting permission for the first time.
Here are some logs.

05-13 13:00:38.449 16678-16692/? I/PlayCommon: [14352] com.google.android.play.a.g.a(368): Successfully uploaded logs.
05-13 13:00:38.576 2139-2139/? I/mono-stdout: LogController::Time::PermissionService::CheckAll::31453
05-13 13:00:38.576 2139-2139/? I/mono-stdout: LogController::Time::PermissionService::Check Sms::0
05-13 13:00:38.580 8993-9017/? E/nvaudio_hw: nvaudio_out_set_parameters: Could not find the usecase (0) in the list
05-13 13:00:38.584 2139-2139/? I/mono-stdout: LogController::Time::PermissionService::Check Photos::7
05-13 13:00:38.584 2139-2139/? I/mono-stdout: LogController::Time::PermissionService::Check Camera::0
05-13 13:00:38.598 683-694/? I/ActivityManager: START u0 {act=android.content.pm.action.REQUEST_PERMISSIONS pkg=com.android.packageinstaller cmp=com.android.packageinstaller/.permission.ui.GrantPermissionsActivity (has extras)} from uid 10994 on display 0
05-13 13:00:38.620 2139-2139/? I/mono-stdout: A task was canceled.
05-13 13:00:38.620 2139-2139/? I/mono-stdout: LogController::Time::PermissionService::Check Microphone::35
05-13 13:00:38.623 683-1226/? I/ActivityManager: START u0 {act=android.content.pm.action.REQUEST_PERMISSIONS pkg=com.android.packageinstaller cmp=com.android.packageinstaller/.permission.ui.GrantPermissionsActivity (has extras)} from uid 10994 on display 0
05-13 13:00:38.630 2139-2139/? I/mono-stdout: A task was canceled.
05-13 13:00:38.630 2139-2139/? I/mono-stdout: LogController::Time::PermissionService::Check Microphone::9
05-13 13:00:38.634 683-1293/? I/ActivityManager: START u0 {act=android.content.pm.action.REQUEST_PERMISSIONS pkg=com.android.packageinstaller cmp=com.android.packageinstaller/.permission.ui.GrantPermissionsActivity (has extras)} from uid 10994 on display 0
05-13 13:00:38.646 2139-2139/? I/mono-stdout: A task was canceled.
05-13 13:00:38.646 2139-2139/? I/mono-stdout: LogController::Time::PermissionService::Check Contacts::16
05-13 13:00:38.652 683-1195/? I/ActivityManager: START u0 {act=android.content.pm.action.REQUEST_PERMISSIONS pkg=com.android.packageinstaller cmp=com.android.packageinstaller/.permission.ui.GrantPermissionsActivity (has extras)} from uid 10994 on display 0
05-13 13:00:38.658 2139-2139/? I/mono-stdout: A task was canceled.
05-13 13:00:38.658 2139-2139/? I/mono-stdout: LogController::Time::PermissionService::Check Contacts::12
05-13 13:00:38.664 683-898/? I/ActivityManager: START u0 {act=android.content.pm.action.REQUEST_PERMISSIONS pkg=com.android.packageinstaller cmp=com.android.packageinstaller/.permission.ui.GrantPermissionsActivity (has extras)} from uid 10994 on display 0
05-13 13:00:38.668 2139-2139/? I/mono-stdout: A task was canceled.
05-13 13:00:38.668 2139-2139/? I/mono-stdout: LogController::Time::PermissionService::Check Storage::9
05-13 13:00:38.671 683-1195/? I/ActivityManager: START u0 {act=android.content.pm.action.REQUEST_PERMISSIONS pkg=com.android.packageinstaller cmp=com.android.packageinstaller/.permission.ui.GrantPermissionsActivity (has extras)} from uid 10994 on display 0
05-13 13:00:38.674 2139-2139/? I/mono-stdout: A task was canceled.
05-13 13:00:38.675 2139-2139/? I/mono-stdout: LogController::Time::PermissionService::Check Storage::6
05-13 13:00:38.678 683-694/? I/ActivityManager: START u0 {act=android.content.pm.action.REQUEST_PERMISSIONS pkg=com.android.packageinstaller cmp=com.android.packageinstaller/.permission.ui.GrantPermissionsActivity (has extras)} from uid 10994 on display 0
05-13 13:00:38.680 2139-2139/? I/mono-stdout: A task was canceled.
05-13 13:00:38.680 2139-2139/? I/mono-stdout: LogController::Time::PermissionService::Check Speech::5
05-13 13:00:38.683 683-1291/? I/ActivityManager: START u0 {act=android.content.pm.action.REQUEST_PERMISSIONS pkg=com.android.packageinstaller cmp=com.android.packageinstaller/.permission.ui.GrantPermissionsActivity (has extras)} from uid 10994 on display 0
05-13 13:00:38.686 2139-2139/? I/mono-stdout: A task was canceled.
05-13 13:00:38.687 2139-2139/? I/mono-stdout: LogController::Time::PermissionService::Check Speech::6
05-13 13:00:38.689 683-1266/? I/ActivityManager: START u0 {act=android.content.pm.action.REQUEST_PERMISSIONS pkg=com.android.packageinstaller cmp=com.android.packageinstaller/.permission.ui.GrantPermissionsActivity (has extras)} from uid 10994 on display 0
05-13 13:00:38.691 2139-2139/? I/mono-stdout: A task was canceled.
05-13 13:00:38.691 2139-2139/? I/mono-stdout: LogController::Time::PermissionService::Check Phone::4
05-13 13:00:38.702 683-898/? I/ActivityManager: START u0 {act=android.content.pm.action.REQUEST_PERMISSIONS pkg=com.android.packageinstaller cmp=com.android.packageinstaller/.permission.ui.GrantPermissionsActivity (has extras)} from uid 10994 on display 0
05-13 13:00:38.708 2139-2139/? I/mono-stdout: A task was canceled.
05-13 13:00:38.708 2139-2139/? I/mono-stdout: LogController::Time::PermissionService::Check Phone::16
05-13 13:00:38.717 683-694/? I/ActivityManager: START u0 {act=android.content.pm.action.REQUEST_PERMISSIONS pkg=com.android.packageinstaller cmp=com.android.packageinstaller/.permission.ui.GrantPermissionsActivity (has extras)} from uid 10994 on display 0
05-13 13:00:38.727 2139-2139/? I/mono-stdout: A task was canceled.
05-13 13:00:38.951 2139-2139/? I/art: Starting a blocking GC Explicit
05-13 13:00:38.967 2139-2139/? I/art: Explicit concurrent mark sweep GC freed 25677(1209KB) AllocSpace objects, 5(432KB) LOS objects, 32% free, 8MB/12MB, paused 314us total 13.765ms
05-13 13:00:38.983 683-692/? I/art: Background sticky concurrent mark sweep GC freed 16150(1007KB) AllocSpace objects, 5(192KB) LOS objects, 5% free, 29MB/31MB, paused 7.618ms total 28.692ms
05-13 13:00:39.224 683-1245/? I/ActivityManager: Process de.codenauts.hockeyapp (pid 31343) has died

of special interest are these lines.
05-13 13:00:38.668 2139-2139/? I/mono-stdout: A task was canceled.
Here is our code

 public static async Task<bool> Check(Permission permission, bool ask = true, bool warn = true)
        {
            LogController.Time("PermissionService::Check " + permission);

            try
            {
                var val = await CrossPermissions.Current.CheckPermissionStatusAsync(permission);

                if (val != PermissionStatus.Granted && ask)
                {
                    await CrossPermissions.Current.RequestPermissionsAsync(permission);
                }

                val = await CrossPermissions.Current.CheckPermissionStatusAsync(permission);

                if (val != PermissionStatus.Granted && val != PermissionStatus.Unknown && warn)
                {
                    //Notification.Notify(ToastNotificationType.Error, AgLanguage.Error, string.Format(AgLanguage.PermissionError, permission));
                    await UserDialogs.Instance.AlertAsync(AgLanguage.ReConfirmPermissionText, AgLanguage.AgriSync, "Next");
                    await Check(permission, true);
                }
                return val == PermissionStatus.Granted;
            }
            catch (Exception ex)
            {
                LogController.LogError(ex);
            }

            return false;

        }

Did you add this code?
https://github.com/jamesmontemagno/PermissionsPlugin#android-specific-in-your-baseactivity-or-mainactivity-for-xamarinforms-add-this-code

Yes, I have the handler and the Compile / Targets set as in that document.
@jamesmontemagno

My recommendation is to change it to this:
public static async Task<bool> Check(Permission permission, bool ask = true, bool warn = true)
        {
            LogController.Time("PermissionService::Check " + permission);

            try
            {
                var val = await CrossPermissions.Current.CheckPermissionStatusAsync(permission);

                if (val != PermissionStatus.Granted && ask)
                {
                    var status = await CrossPermissions.Current.RequestPermissionsAsync(permission);
                    val = status[permission]
                    if(val == PermissionStatus.Granted)
                        return true;
                }

                if (val != PermissionStatus.Granted && val != PermissionStatus.Unknown && warn)
                {
                    //Notification.Notify(ToastNotificationType.Error, AgLanguage.Error, string.Format(AgLanguage.PermissionError, permission));
                    await UserDialogs.Instance.AlertAsync(AgLanguage.ReConfirmPermissionText, AgLanguage.AgriSync, "Next");
                    await Check(permission, true);
                }
                return val == PermissionStatus.Granted;
            }
            catch (Exception ex)
            {
                LogController.LogError(ex);
            }

            return false;

        }

I also worry of some odd edge case that you might end in a stack overflow too, not sure why you check again later on.

In addition to this, below is an exception with app crashed on Google pixel - android 7.0.
Java.Interop.JniEnvironmentRecursive entry to executePendingTransactions
Raw
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw():0
Java.Interop.JniEnvironment.InstanceMethods.CallNonvirtualVoidMethod(JniObjectReference instance, JniObjectReference type, JniMethodInfo method, JniArgumentValue* args)<4bbf2573138b4315b15bceffb7e9feb1>:0
Java.Interop.JniPeerMembers.JniInstanceMethods.InvokeVirtualVoidMethod(string encodedMember, IJavaPeerable self, JniArgumentValue* parameters)<4bbf2573138b4315b15bceffb7e9feb1>:0
Android.App.Activity.OnResume()<94ae0a556c074fa195090da65c701c11>:0
Xamarin.Forms.Platform.Android.FormsAppCompatActivity.OnResume()<15c9790557774c809371f43133cc42c4>:0
AgriSync.Droid.MainActivity.OnResume()<68022563e05540e5bdb596c38aa42a8c>:0
Android.App.Activity.n_OnResume(IntPtr jnienv, IntPtr native__this)<94ae0a556c074fa195090da65c701c11>:0
at (wrapper dynamic-method) System.Object:3695a3a0-b1f0-47f4-a027-b557e9399f50 (intptr,intptr)
--- End of managed Java.Lang.IllegalStateException stack trace ---
java.lang.IllegalStateException: Recursive entry to executePendingTransactions
at android.support.v4.app.FragmentManagerImpl.execPendingActions(FragmentManager.java:1589)
at android.support.v4.app.FragmentController.execPendingActions(FragmentController.java:339)
at android.support.v4.app.FragmentActivity.onResume(FragmentActivity.java:484)
md50bbf98d479584333dba7cd5a20f65b03.MainActivity.n_onResume(Native Method)
at md50bbf98d479584333dba7cd5a20f65b03.MainActivity.onResume(MainActivity.java:42)
at android.app.Instrumentation.callActivityOnResume(Instrumentation.java:1269)
at android.app.Activity.performResume(Activity.java:6786)
at android.app.ActivityThread.performResumeActivity(ActivityThread.java:3407)
at android.app.ActivityThread.handleResumeActivity(ActivityThread.java:3470)
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1528)
at android.os.Handler.dispatchMessage(Handler.java:102)
at android.os.Looper.loop(Looper.java:154)
at android.app.ActivityThread.main(ActivityThread.java:6121)
java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:889)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:779)

You are probably attempting to call into things before they have completed and it is upset. Make sure you call the base first i guess before calling up to it.
In normal usage this is no problem at all from any of my 10 apps and 20+ samples.

Resolved the issue.
The issue is we were checking permission and based on that state navigating. My guess is that the plugin was holding onto a reference of the view that called it and using it after the async process completes. Solved it with an arbitrary wait after the check call.


Tilemap editor draws lines

Godot version: 3.1 beta 1

OS/device including version: Windows 10 x64

Issue description:

When putting a tile in the tilemap editor, the editor draws a line of tiles.
https://ibb.co/p4ZY6S4
Steps to reproduce:
Open a window of a different program on the side of Godot Editor. With the Tilemap Editor opened and a tile selected, click that window, then click your canvas to put the tile.
Minimal reproduction project:

I think the problem might be comming from this piece of code:
https://github.com/godotengine/godot/blob/7c8fbd892c827abae464fad2c427ca2bd17ff3f8/editor/plugins/tile_map_editor_plugin.cpp#L1193-L1212
Can't reproduce unless I hold the mouse pressed as I Alt-Tab. Maybe it behaves differently on Windows.
I guess we should just switch away from TOOL_PAINTING if the window is unfocused.

Still valid in 05395cb


Renaming Dialogs

Describe the bug
The Framework Composer seems to have issues building the project if you rename the top level dialog. This problem also exists for renaming child dialogs, but doing so only disrupts the UX if it is accessed during the conversation flow. I used the CoreWithLanguage template, made a few child dialogs, then attempted to change the name of CoreWithLanguage to MetricIntent. It did not build. When I then changed it back to CoreWithLanguage to revert the bug, the original name also failed to build. The words in the error message flip flop, suggesting there is something sluggish happening behind the scenes. Closing and reopening the project did not fix things.
Version
Version: 2.1.1
Electron: 8.2.4
Chrome: 80.0.3987.165
NodeJS: 12.13.0
V8: 8.0.426.27-electron.0
OS
Windows 10
To Reproduce
Steps to reproduce the behavior:

Create a CoreWithLanguage bot template (or any template I guess)
Edit the name of the dialog in the properties pane
Build the project
See error. Build failed.

Expected behavior
I expected the Framework Composer to acknowledge the name change and propagate it automatically through the rest of the bot code and the LUIS backend.
Additional context
Output trace (personal information censored):
Error occurred building the bot
Error: Resource MetricIntent.dialog not found. at ConfigurationResourceExporer.loadType (C:\Users\****\******************\CoreWithLanguage\CoreWithLanguage\node_modules\botbuilder-dialogs-declarative\lib\resources\resourceExplorer.js:195:19) at DialogExpressionConverter.convert (C:\Users\****\******************\CoreWithLanguage\CoreWithLanguage\node_modules\botbuilder-dialogs-adaptive\lib\converters\dialogExpressionConverter.js:34:55) at C:\Users\****\******************\CoreWithLanguage\CoreWithLanguage\node_modules\botbuilder-dialogs-declarative\lib\defaultLoader.js:37:48 at Array.reduce () at DefaultLoader.load (C:\Users\****\******************\CoreWithLanguage\CoreWithLanguage\node_modules\botbuilder-dialogs-declarative\lib\defaultLoader.js:31:39) at ConfigurationResourceExporer.load (C:\Users\****\******************\CoreWithLanguage\CoreWithLanguage\node_modules\botbuilder-dialogs-declarative\lib\resources\resourceExplorer.js:240:23) at Array. (C:\Users\****\******************\CoreWithLanguage\CoreWithLanguage\node_modules\botbuilder-dialogs-declarative\lib\resources\resourceExplorer.js:214:25) at JSON.parse () at ConfigurationResourceExporer.loadType (C:\Users\****\******************\CoreWithLanguage\CoreWithLanguage\node_modules\botbuilder-dialogs-declarative\lib\resources\resourceExplorer.js:203:36) at new CoreBot (C:\Users\****\******************\CoreWithLanguage\CoreWithLanguage\node_modules\botbuilder-dialogs-adaptive-runtime\lib\coreBot.js:22:45) npm ERR! code ELIFECYCLE npm ERR! errno 1 npm ERR! CoreWithLanguage@ dev: cross-env NODE_ENV=dev node index.js "--port" "3981" "--urls" "http://<IP_ADDRESS>:3981" "--luis:endpointKey" "********************************" "--SkillHostEndpoint" "http://<IP_ADDRESS>:3981/api/skills" npm ERR! Exit status 1 npm ERR! npm ERR! Failed at the CoreWithLanguage@ dev script. npm ERR! This is probably not a problem with npm. There is likely additional logging output above. npm ERR!  C:\Users\*****\AppData\Roaming\npm-cache_logs\2021-10-11T18_38_36_577Z-debug.log

Hi @TylerDKnight Any updates or further information you can share to help us reproduce or understand what might be going on?


CSV Sender Self-Validation Website Tool, no data in production

User Story:
As a potential new CSV Sender, I would like the ability to work at my own pace to create CSV files from my system and validate the CSV data before engaging in a formal Onboarding process.
Hypotheses:
Potential Senders would prefer to use a self-validation / pre-onboarding validation tool to insure they can comply with our standard before actually engaging with us.  This will reduce the time spent on basic initial data validation between the Sender and RS; thus reducing the overall time & effort required to GO LIVE.  Note, NIST offers something like this for HL7.
Assumptions:
The current CSV schema now catches a very high percentage of user errors upfront in the JSON messages, so it may not be a huge deal to create a single OKTA Login for testing which uses a single OKTA group in RS Staging that allows users to upload a CSV file to Staging and receive the JSON error responses.  Basic on the error messages, they can make corrections at their own pace and attempt the upload again at anytime.  No involvement from RS personnel.  Note, RS must discard all data sent because they will inevitably sent files with PHI.

The self-validation and process improvements for CSV Uploader has been moved to SimpleReport and the Experience Team.  Closing this ticket.


feat(UpdateCampaign): add API for updates

Allows editing of an existing update campaign.
Closes #278

Pull Request Test Coverage Report for Build 0ad0c3cad897917ff9bf101a28c6fa8996c78ebb-PR-305

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+82.1%) to 82.051%

Totals

Change from base Build<PHONE_NUMBER>:
82.1%

Covered Lines:
1152

Relevant Lines:
1404

💛  - Coveralls

I guess we can close this, since this will need to be reimplemented using Ash


Possible bug in lr scheduler

Hi, thank you for releasing the code for your paper, it's been a great help for our own research. When finetuning the pretrained models using your code we noticed that the lr scheduler was behaving unexpectedly, and after some search we believe the reason for that lies in this code snippet:
https://github.com/microsoft/ClimaX/blob/efd6de4b7e3f0e552693ade5c90d250b6ef0b4b6/src/climax/global_forecast/module.py#L223-L230
It seems to me that the interval should be set as "epoch" and not "step" in line 230, as the milestones for the scheduler are also given in epochs.

Hi, thank you for your interest in ClimaX. The performance we got in the paper was by using a step-level scheduler, and to do that correctly we had to set the warmup_epochs and max_epochs to the number of warmup steps and max steps, respectively. To use an epoch-level scheduler then yes, you should modify line 230 to "interval": "epoch"

did your problem get resolved @J-L-O?

To clarify, we could have better variable names, but the scheduler implementation is correct.

Ah I see, that makes sense. Thank you for taking the time to answer!


Migrations: Implement adapter method that copies values between two fields in all documents in a collection

In this branch, I implemented an adapter method named copy_value_from_field_to_field_in_each_document, which migrator authors can use to copy all values in one field of a document into another field of that same document (creating the latter field if it doesn't already exist), for every document in a given collection.
Previously, migrator authors could do this by iterating over documents in Python. This new adapter method delegates the iteration to MongoDB.
Fixes #2259

@eecavanna Did you want to add implementing this method in to https://github.com/microbiomedata/nmdc-schema/blob/main/nmdc_schema/migrators/partials/migrator_from_11_0_3_to_11_1_0/migrator_from_11_0_3_to_11_1_0_part_2.py ?  If so, I think doing that in this PR makes sense.

Thanks for reviewing this! That level of review is helpful for me (e.g. in case I made a mistake somewhere while implementing this in haste). Rather than merge this in, I'd prefer to merge the third PR I opened this afternoon, since it includes these changes. That PR is: #2265

On second thought, I implemented that third PR in a way where this can be merged in independently. I'll go ahead and merge this in.


DOC: fix docstring errors in in stats.wilcoxon

Reference issue
Fix #12985
What does this implement/fix?
Fixed some docstring errors in stats.wilcoxon.
I changed single backticks to double backticks for arguments and to double quotations for strings.

CircleCI service is degraded and being fixed so didn't fire but looks good. Thanks @AtsushiSakai


100% CPU usage on Windows 10

Environment

OS: Windows 10
Node version: v16.13.0
gtop version: 1.1.2

Description
gtop is constantly using 100% of my CPU time, making it unusable:

What is more, it doesn't show itself on the process list.
This could be a part of issue #12.

I admit that the NodeJS environment and underlying tech for gtop is not so performant. see #12
100% usage is still too high. Try to use the latest NodeJS version in a better machine. You can alternatively use an implementation in a more performant language such as Rust (ytop) & Golang (gotop).


result.minimum -> result.minimizer

propagate upstream renaming

Do we need to wait for a version to propagate or something?

You are right, I was using Optim master locally.

Could you ping this PR when there is a new Optim.jl release? We'll also want to update the REQUIRE file accordingly.

Do you still want to support julia 0.3? If not, this can work with both master and current release of Optim.

No, I'd be happy to drop support for Julia 0.3.

Okay this is ready!

I guess #20 will soon make this obsolete, but might as well have something working with Optim master until that merges...

I hadn't seen this, but just for future reference Optim.minimizer(results) would have been a bit safer, as it has been in the package for a few tags, we just waited to change the fields, because some packages (like this) might still be calling .minimum directly.


form schema returns incorrect types

When using the API call to get the form schema (https://odkcentral.docs.apiary.io/#reference/forms-and-submissions/'-individual-form/getting-form-schema-fields) or using ruODK form_schema(), the type returned is not correct. For example, select1 fields are marked as "string".
See below, for example:

See:  https://forum.getodk.org/t/incorrect-type-returned-by-api-form-schema-fields/31044
See: https://github.com/ropensci/ruODK/issues/101

In XForms, selects are considered a view-only concern. The underlying type is indeed string. To get that a field is a select, you would need to look at the form definition body.

Thanks for going to the forum, that’s generally a better place to start conversations. I added more details af https://forum.getodk.org/t/incorrect-type-returned-by-api-form-schema-fields/31044/2

In XForms, selects are considered a view-only concern. The underlying type is indeed string. To get that a field is a select, you would need to look at the form definition body.

Tks for the clarification.


[backport] Fix cufft callback manager error in Windows

Backport of #4377

[automatic post] Jenkins, test this please.

pfnCI failure:
https://github.com/cupy/cupy/blob/2aa98be0ae11c13694a038d04a19272f49cfb3f6/.pfnci/script.sh#L159-L165

I think that for pfcn is expected to fail. Thanks!

Jenkins CI test (for commit 6e90696d89b000e8cdc886b06a62b3f9ea908bce, target branch hotfix-v9.0.0a3) succeeded!

Jenkins CI test (for commit 6e90696d89b000e8cdc886b06a62b3f9ea908bce, target branch hotfix-v9.0.0a3) succeeded!


Transition to LFX Meeting Management

Please share any additional details on this topic
The goal is to shift meetings to using LFX Meeting Management, which enables a smoother experience, easier management, and better data tracking.
Detail what actions or feedback you would like from the TAC
Support in the transition
How much time do you need for this topic?
5 minutes or less

Update on transition:
Projects moved over:
CI Working Group
Digital Production Example Library (DPEL)
OpenAssetIO
OpenFX
OpenImageIO
Open Review Initiative
OpenVDB
Rez
USD Working Group
In Progress:
OpenColorIO
TODO:
OpenCue
OpenEXR
MaterialX
Open Shading Language (OSL)
OpenTimelineIO
RAW to ACES Utility
D&I Working Group
Rust WG

@bcipriano and @cary-ilm - please reach out so we can make this move

We are now 100% complete! Thank you to all the projects for your support.


No HCL object could be decoded

terrascan version: 0.1.0 (development mode)
Python version: 3.6.1
Operating System: OS X 10.13.6

Description
Trying to run tests against terraform templates with terrascan set up in development
What I Did
terrascan -l . -t all

This is one of many similar tracebacks
ERROR: test_aws_security_group_rule_open (terrascan.checks.security_group.TestSecurityGroups)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/zachary.zeid/github/terrascan/terrascan/.eggs/terraform_validate-2.5.0-py3.6.egg/terraform_validate/terraform_validate.py", line 464, in parse_terraform_directory
    hcl.loads(new_terraform)
  File "/Users/zachary.zeid/github/terrascan/terrascan/.eggs/pyhcl-0.3.10-py3.6.egg/hcl/api.py", line 61, in loads
    if isHcl(s):
  File "/Users/zachary.zeid/github/terrascan/terrascan/.eggs/pyhcl-0.3.10-py3.6.egg/hcl/api.py", line 39, in isHcl
    raise ValueError("No HCL object could be decoded")
ValueError: No HCL object could be decoded

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/zachary.zeid/github/terrascan/terrascan/terrascan/checks/security_group.py", line 21, in setUp
    self.v = terraform_validate.Validator(self.path)
  File "/Users/zachary.zeid/github/terrascan/terrascan/.eggs/terraform_validate-2.5.0-py3.6.egg/terraform_validate/terraform_validate.py", line 431, in __init__
    self.terraform_config = self.parse_terraform_directory(path)
  File "/Users/zachary.zeid/github/terrascan/terrascan/.eggs/terraform_validate-2.5.0-py3.6.egg/terraform_validate/terraform_validate.py", line 466, in parse_terraform_directory
    raise TerraformSyntaxException("Invalid terraform configuration in {0}\n{1}".format(os.path.join(directory,file),e))
terraform_validate.terraform_validate.TerraformSyntaxException: Invalid terraform configuration in /Users/zachary.zeid/github/terrascan/terrascan/variables.tf
No HCL object could be decoded

----------------------------------------------------------------------
Ran 5 tests in 0.036s

FAILED (errors=5)  ```

I found the resolution to this issue, as my variables.tf file was empty.  However, this error needs to have better error handling.


Sails keeps running on the background even if I have stoppped it

Hi,
I just installed my app on AWS EC2. And this is crazy. The app just keeps running after I have taken down the server.
Port : 80
Sails : 0.10.5
If I need to stop the app from running I have to reboot the server.
( Is anyone using EC2 Ubuntu server on port 80 to run a SailsJS app ? )
Please let me know about this

Is that solved?

Has this been resolved?


Rename and republish both the repo and the package

TL;DR Summary
The @dustin-ruetz/devdeps package name is pretty long and it doesn't convey that the development dependencies are specifically for web projects, so rename the package to the shorter and more accurate web-devdeps.

Package
Before
After

Name
@dustin-ruetz/devdeps
web-devdeps

Type
User-scoped namespace
Unscoped

Tasks
github.com

[x] Rename the repo.
[x] Rename GitHub token variable name.
[x] Re-onboard the renamed repo to Renovate.

Didn't need to do this since Renovate automatically detected the renamed repo.

[x] Delete the previous @dustin-ruetz/devdeps releases and tags.
[ ] Delete the previous auto-generated release comments on past PRs.

npmjs.com

[x] Rename NPM token variable name.
[x] Delete the @dustin-ruetz/devdeps NPM package.

1Password item

[x] Rename GitHub and NPM token variable names.

Local repo

[x] Rename the directory locally.
[x] Remove the old origin links, then add the new ones.
[x] Update all github.com/dustin-ruetz/devdeps links.
In the .github/workflows/release.yaml file:

[x] Rename "Human-readable 1Password CLI secret reference" item name.
[x] Rename GitHub and NPM token variable names.

[x] CHANGELOG.md file: Delete all version entries, but leave the h1 heading as-is.
[x] package-lock.json file: change name and reset version to "0.0.0" (note that there are two instances for each of these fields).
[x] package.json file: change name and reset version to "0.0.0". Remove the "publishConfig" object.
[x] Remove all instances of the @dustin-ruetz user-scoped namespace.
[x] Search codebase to review all instances of dustin-ruetz/devdeps.
[x] Rename all instances of the devdeps package name to web-devdeps.
[x] Ensure that the local Git tags in the repo are deleted.
[ ] After releasing v1 of the new web-devdeps package, simplify the CHANGELOG.md file to just the "initial release of web-devdeps package" feature.

:tada: This issue has been resolved in version 1.0.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Runoff for 1/20°

We need to fix the regridding / mask of the 1/20° runoff.
At our 20/1/23 meeting @angus-g volunteered to do this. However we need to wait until the 1/20° bathymetry has been redone, so we have a final land/sea mask.
@angus-g also volunteered to share his script and write some documentation so that we can repeat this process for the 1/40° config. @angus-g to chat to @micaeljtoliveira about adding this script to domain-tools.

We also need the coupler mask. And to add this to domain-tools with documentation also.

Generating the mask is now documented in the wiki

So fast!!
On Fri, 20 Jan 2023 at 15:29, Angus Gibson @.***> wrote:

Generating the mask is now documented in the wiki
https://github.com/COSIMA/mom6-panan/wiki/Preparing-inputs-for-a-new-configuration#generating-the-exchange-grid
—
Reply to this email directly, view it on GitHub
https://github.com/COSIMA/mom6-panan/issues/18#issuecomment-1397910512,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACA44U27ESGRS62TXYNHAZLWTIIDPANCNFSM6AAAAAAUBDDNBY
.
You are receiving this because you authored the thread.Message ID:
@.***>


Solver adding 15CP to base with no buffs selected

Required Information
Class: Leatherworker
Level: 70
Craftsmanship: 813
Control: 818
CP: 372
Recipe Name: Orphan Donations
Recipe Level: 60
Solver Seed: (Look at the top of the Execution Log on the Solver page)
1563306435885
Expected Behaviour
Have 372 CP for solver
(Example: Standard Synthesis progress is 42)
Actual Behaviour
When starting solver it adds 15 CP to my actual  372 CP making it 387. This is temporarily fixed by subtracting 15 CP from the crafter attributes
(Example: Standard Synthesis progress is 38)
Steps To Reproduce
No foods selected all attribute buffs set to zero
(Example: Buffs, Food, Synthesis Condition, Simulator/Solver Options if different from default)

Hello, maybe you have check this option on your Leatherworker's attributes page ?
 . This option adding exactly 15 CP according to "Stroke of Genius" buff when you are Specialist

Didn't want to start a new thread but the titanium nugget is listed at 40/40 durability when it is actually 35/35 in game. Of course any macro makes the synth go boom. Thanks.

I would look as soon as I have time. But from my point of view, it is advisable to create a new topic each time you mention a problem or suggestion about the application that does not match the thread's original topic. This makes it easier to track and develop a corresponding patch / change and make each thread more readable.

First time.posting sorry :(

And it very well could be the specialist buff.

Didn't want to start a new thread but the titanium nugget is listed at 40/40 durability when it is actually 35/35 in game. Of course any macro makes the synth go boom. Thanks.

I' have created a new issue referenced to your message : see #288


Create snakeWaterGun.py

Created a Snake Water Gun or Rock Paper Scissor game using Python.

Merging! GTG!


Feature: Environment variables should be sent to admininstrator program's

Description
If I start a .bat file in C:\users as administrator it default to system32.
Steps To Reproduce
No response
Expected behavior
It should have C:\users as its start location.
Files Version
1.9.17
Windows Version
10.0.19043.1151
Relevant Assets
debug.log
debug_fulltrust.log

Does Explorer behaves any different?

Apparently not. Didn't realise it was a windows issue.

This is not to say it can't be done in Files, reopening this as a feature request but don't be surprised if this proves technically impossible to implement ;)

@gave92 can we solve this by setting the working directory when we start the process?

You can try but this hints it won't work (and why).

https://superuser.com/questions/1067901/how-to-open-command-prompt-in-a-specific-folder-as-administrator
Looks like you cant run the bat directly bu you need to do like (untested):
cmd /c cd /d "~dp0" && call my_bat.bat


Revert "Repair max slew and max capacitance violations at all resizer stages (#1024)"

This reverts commit 2f276c62c30661f1ffd7b2c0cc721bb62ce2306e.
Amongst other things, repair_design undoes all the hold violation fixes
previously done.

@donn @maliberty I only just realised repair_design calls Resizer::resizeToTargetSlew() which calls Resizer::findTargetCell() which then considers all buffers equivalent (buf, clkbuf, delay buffers etc). So calling it later in the flow is bad, because it might convert delay buffers back to normal buffers (undoing hold violation fixups).
However I'm not sure why repair_design wants to substitute one type of buffer for another at all. It does try to avoid inserting delay buffers, but it only works sometimes (I see lots of delay buffers after a repair_design):
            // Library may have "delay" buffers/inverters that are
            // functionally buffers/inverters but have additional
            // intrinsic delay. Accept worse target load matching if
            // delay is reduced to avoid using them.
            ? ((delay < best_delay
                && dist < best_dist * 1.1)
               || (dist < best_dist
                   && delay < best_delay * 1.1))
            : dist < best_dist

@tspyrou FYI on rsz issue


DIANN changes in the current pipeline

Description of feature
Dear @daichengxin @vdemichev @jpfeuffer:
I have started testing the DIANN pipeline in quantms. The main problem I found is that still the pipeline is really slow. However, after talking to @vdemichev some other issues should be solved. First two ideas about what do we want to solve between DIANN and quantms:
quantms + diann should be able to:

reanalyze DIA-based data in PRIDE Archive annotated in SDRFs and Uniprot FASTA files.
export the results into standard file formats including mzTab + mzML (still pending issue here #119 that we will continue discussing)
use the capabilities of quantms to run DIANN distributed and cloud as much as possible.

The current pipeline/parallelization does the following :

Generate the "generic config files" for the analysis including the PTMs, Enzyme rules, etc. For that, we use the following command in one node:

prepare_diann_parameters.py generate \
    --enzyme "Trypsin" \
    --fix_mod "" \
    --var_mod "Oxidation (M),Carbamidomethyl (C)" \
    --precursor_tolerence 10 \
    --precursor_tolerence_unit ppm \
    --fragment_tolerence 0.05 \
    --fragment_tolerence_unit Da \
    > GENERATE_DIANN_CFG.log

This actually generates a file (diann_config.cfg) like containing the following information:
--dir ./mzMLs --cut K*,R*,!*P --var-mod Oxidation,15.994915,M --var-mod Carbamidomethyl,57.021464,C --mass-acc 10 --mass-acc-ms1 20 --matrices --report-lib-info

Each raw file in the SDRF is converted to mzML; or the mzML that was converted outside the pipeline (ABSiex case) is used for the analysis.

 In addition, @vdemichev as commented me that generating a spectral library by raw file will give you wrong results in the next step (step 4) when searching all the data together and perform the quantification (q-value wrong).

The quantification step the pipeline uses all the mzMLs and libraries generated in the individual steps to perform the quantification and statistical assessment:

diann   `cat diann_config.cfg` \
        --lib N294-1_lib.tsv--lib N294-2_lib.tsv--lib N295-1_lib.tsv--lib N295-2_lib.tsv--lib N296-1_lib.tsv--lib N299-1_lib.tsv--lib N296-2_lib.tsv--lib N299-2_lib.tsv \
        --relaxed-prot-inf \
        --fasta Homo-sapiens-uniprot-reviewed-isoforms-contaminants-decoy-202105.fasta \
         \
         \
         \
         \
        --threads 25 \
        --missed-cleavages 2 \
        --min-pep-len 6 \
        --max-pep-len 40 \
        --min-pr-charge 2 \
        --max-pr-charge 4 \
        --var-mods 3 \
        --matrix-spec-q 0.01 \
         \
        --reannotate \
         \
        --out diann_report.tsv \
        --verbose 3 \
        > diann.log

Discussing with @vdemichev about performance, he mentioned that this approach will give you wrong statistical results.
@vdemichev

can you suggest which changes can be done to parallelize DIANN?
can you suggest which parameters can improve the performance of all the steps?

Lets discuss the ideas in this thread in details.

Hi All, below on some thoughts how to structure the workflow.
Step 1. Generate an in silico predicted spectral library from a FASTA sequence database (in Uniprot format). This step does not require any raw files. The slowdown observed was because it was carried out for each raw file separately, whereas it should only be carried out once per FASTA.
For most DIA experiments, it's better to use the default DIA-NN settings: max 1 missed cleavage, peptide length range 7-30, no variable modifications. Anything else will lead to a slowdown and, potentially, slightly worse performance. Yes, if there's really huge amount of methionine oxidation, then including M(ox) as variable mod (and setting max var mods to some reasonable number, e.g. 1 or 2) might be ultimately beneficial, but I am yet to see a dataset where it would provide a significant advantage. Protein N-terminal acetylation is typically fine to include but it will also yield to a minor slowdown due to PTM scoring overhead.
Result: *.predicted.speclib file containing an in silico library.
Step 2. Preliminary analysis of individual raw files. Now, one could of course do everything automatically as part of MBR analysis, but, as I understand, the goal is to make things as parallelisable as possible, so let's discuss how to do this 'manually'. Thus, step 2 consists of analysing each run separately with the in silico library generating in step 1. The mass accuracies and the scan window settings in DIA-NN should be either fixed or left automatic. If mass accuracies are automatic, please also supply the --quick-mass-acc command. Specify a folder with --temp where .quant files will be saved to.
Result: *.quant file for each raw file that contains IDs and quant info; log file.
Step 3. Assemble an empirical spectral library from the .quant files. For this all .quant files must be accessible for the given instance of DIA-NN. No access to the raw data is required, if DIA-NN is supplied with the --rt-profiling command, which I strongly recommend. This corresponds to "IDs, RT & IM profiling" mode of library generation, which I would recommend for this workflow. To execute this step, run DIA-NN with the --use-quant command, listing all raw files with --f though (but they don't need to be accessible, these file names are only used to infer .quant files file names). The location of .quant files can be specified with --temp, please see the docs.
Result: *.tsv spectral library.
Step 4. Final analysis of individual raw files. Could now use the empirical spectral library to analyse all files in one go (much faster than the preliminary step), but let's consider again how to do things manually with analysis steps for each run being separate. As in step 2, need to process each run with the empirical spectral library. Now, mass accuracies & scan window must be fixed here. In the log file produced by the Step 2, there's a line like "Averaged recommended settings for this experiment: Mass accuracy = 11ppm, MS1 accuracy = 15ppm, Scan window = 7". Need to extract the settings from this line and pass to DIA-NN during this search step. Specify a folder with --temp where .quant files will be saved to, make sure it's a different folder from the one used previously, so that old .quant files (preliminary analysis) are not overwritten.
Result: *.quant file for each raw file that contains IDs and quant info.
Step 4. Summarise the info. For this need the .tsv empirical spectral library, the FASTA database to annotate it and the newly created during step 4 .quant files. Run DIA-NN specifying the raw file names with --f (but they again don't need to be accessible, these file names are only used to infer .quant files file names) and with --use-quant, with --temp specifying the .quant files location. Here indicate --relaxed-prot-inf (for FragPipe-like protein inference) and --pg-level 2 (implicit grouping by gene names). --pg-level 0 is also OK (implicit grouping by sequence IDs). If using --pg-level 2 and having multiple species in the database, can add --species-genes (adds a suffix to gene names indicating the species - works provided the database is Uniprot-formatted).
Result: DIA-NN output files.

Some notes:

For dia-PASEF it is usually beneficial to set the precursor charge range to 2-3.
dia-PASEF files must be in .d (more compact) or .dia (faster, pre-converted by DIA-NN) format; conversion to .mzML is impossible here (there are tools for that, but DIA-NN will not understand such .mzML files).
Low RAM & high speed mode enabled by --min-corr 2.0 --corr-diff 1.0 --time-corr-only might prove useful, it typically leads to no or very minimal drop in ID numbers.
High-precision quant mode enabled by --no-ifs-removal is what I would use, also if no peak broadening occured in the given experiment, can use --peak-center (Robust LC quant mode). This is only relevant for Step 4. Peak broadening can be detected based on DIA-NN's *.stats.tsv output file.
Please let me see the scripts with settings once ready.

@daichengxin Lets start to implement this approach for the next release of the pipeline. I will add some lines here to clarify the pipeline steps:

Hi All, below are some thoughts how to structure the workflow.
Step 1. Generate an in silico predicted spectral library from a FASTA sequence database (in Uniprot format). This step does not require any raw files. The slowdown observed was because it was carried out for each raw file separately, whereas it should only be carried out once per FASTA.
For most DIA experiments, it's better to use the default DIA-NN settings: max 1 missed cleavage, peptide length range 7-30, no variable modifications. Anything else will lead to a slowdown and, potentially, slightly worse performance. Yes, if there's really huge amount of methionine oxidation, then including M(ox) as variable mod (and setting max var mods to some reasonable number, e.g. 1 or 2) might be ultimately beneficial, but I am yet to see a dataset where it would provide a significant advantage. Protein N-terminal acetylation is typically fine to include but it will also yield to a minor slowdown due to PTM scoring overhead.
Result: *.predicted.speclib file containing an in silico library.

1rst step library generation from FASTA:
Default parameters:

max miss cleavages = 1
min peptide length =  7
max peptide length = 30
no variable modifications
variable modifications like M(ox) and N-term(ac) should be provided by SDRF
max number of PTMs = 2

Step 2. Preliminary analysis of individual raw files. Now, one could of course do everything automatically as part of MBR analysis, but, as I understand, the goal is to make things as parallelisable as possible, so let's discuss how to do this 'manually'. Thus, step 2 consists of analysing each run separately with the in silico library generated in step 1. The mass accuracies and the scan window settings in DIA-NN should be either fixed or left automatic. If mass accuracies are automatic, please also supply the --quick-mass-acc command. Specify a folder with --temp where .quant files will be saved to.
Result: *.quant file for each raw file that contains IDs and quant info; log file.
Step 3. Assemble an empirical spectral library from the .quant files. For this all .quant files must be accessible for the given instance of DIA-NN. No access to the raw data is required, if DIA-NN is supplied with the --rt-profiling command, which I strongly recommend. This corresponds to "IDs, RT & IM profiling" mode of library generation, which I would recommend for this workflow. To execute this step, run DIA-NN with the --use-quant command, listing all raw files with --f though (but they don't need to be accessible, these file names are only used to infer .quant files file names). The location of .quant files can be specified with --temp, please see the docs.
Result: *.tsv spectral library.
Step 4. Final analysis of individual raw files. Could now use the empirical spectral library to analyse all files in one go (much faster than the preliminary step), but let's consider again how to do things manually with analysis steps for each run being separate. As in step 2, need to process each run with the empirical spectral library. Now, mass accuracies & scan window must be fixed here. In the log file produced by the Step 2, there's a line like "Averaged recommended settings for this experiment: Mass accuracy = 11ppm, MS1 accuracy = 15ppm, Scan window = 7". Need to extract the settings from this line and pass to DIA-NN during this search step. Specify a folder with --temp where .quant files will be saved to, make sure it's a different folder from the one used previously, so that old .quant files (preliminary analysis) are not overwritten.
Result: *.quant file for each raw file that contains IDs and quant info.
Step 5. Summarise the info. For this need the .tsv empirical spectral library, the FASTA database to annotate it and the newly created during step 4 .quant files. Run DIA-NN specifying the raw file names with --f (but they again don't need to be accessible, these file names are only used to infer .quant files file names) and with --use-quant, with --temp specifying the .quant files location. Here indicate --relaxed-prot-inf (for FragPipe-like protein inference) and --pg-level 2 (implicit grouping by gene names). --pg-level 0 is also OK (implicit grouping by sequence IDs). If using --pg-level 2 and having multiple species in the database, can add --species-genes (adds a suffix to gene names indicating the species - works provided the database is Uniprot-formatted).
Result: DIA-NN output files.

@vdemichev in this original analysis we should remove some parameters related with inference, etc. because this is done in the last step 5.? Any other parameter related with quantitation, inference, etc. that will speed this step.
Indeed, protein inference only happens at Step 5. Quiantification - at steps 4 and 5. These algorithms however don't take any CPU time.

@vdemichev when we extract this from the log files "Averaged recommended settings for this experiment: Mass accuracy = 11ppm, MS1 accuracy = 15ppm, Scan window = 7" in Step 2, Then run each independent file against the "summarize" library with their corresponding settings extracted from the log?
This log line will be displayed during Step 3 (sorry, I initially incorrectly wrote that it's Step 2; in fact it is Step 3, now also corrected it above) and will provide recommended averages, based on summarising the analysis carried out at Step 2. So these settings will be the same for all runs during Step 4.

max miss cleavages min peptide length max peptide length max_mods. These parameters are shared with msgf  and comet, Do I need to modify the default values in nextflow.config?
The current modification parameters come directly from the SDRF file. no variable modifications and variable modifications like M(ox) and N-term(ac) should be provided by SDRF meaning I only keep these two as variable modifications and filter out other variable modifiers?

@daichengxin

[x] reuse the same variables. We may need to advise the users to reduce the max_mods in DIA analysis.
[x] as you said, continue using the SDRF for the modifications as we are doing now. I may need to annotate in a better way the SDRFs.

Sept 1 step finished in #169

Also in regular closed search, I would not advise searching for more 2-3 max_mods right?

The decision should actually be made between 2 and 3. For diann the default should be 2, but in DDA the most common used is 3 😏

Ok but I think after generating the library only once, this might not be the bottleneck anymore.. let's see

The larger the search space (the library), also the slower the main analysis. 3 variable modificaitons is quite a lot.

Because the pipeline share the variable with the DDA (and still DDA is more popular 😝) we will put as default 3, but we will let that clear in the documentation and probably also in the pipeline. @vdemichev just to be clear the user can pass that variable, we are talking here about the default value.

Hi @vdemichev, In step 3: Assemble an empirical spectral library from the .quant files. Does the --lib  parameter specify the library generated by the first step or the second step ?

@daichengxin, --use-quant should notmally be used with exactly the same settings as the initial analysis. Meaning that --lib specifies the initial output library. If there are any difficulties, please let me know & attach full logs, including the log of DIA-NN producing the .quant files

Means to specify the silico-library generated in the first step? Full commands as follows in step 3 now:
diann  --f ${(mzMLs as List).join(' --f ')} \\
            --lib ${lib} \\
            ${min_pr_mz} \\
            ${max_pr_mz} \\
            ${min_fr_mz} \\
            ${max_fr_mz} \\
            --threads ${task.cpus} \\
            --out-lib empirical_library.tsv \\
            --missed-cleavages $params.allowed_missed_cleavages \\
            --min-pep-len $params.min_peptide_length \\
            --max-pep-len $params.max_peptide_length \\
            --min-pr-charge $params.min_precursor_charge \\
            --max-pr-charge $params.max_precursor_charge \\
            --var-mods $params.max_mods \\
            --cut {enzyme} \\
            --var_ptm { from sdrf } \\
            --fixed_ptm { from sdrf} \\
            --verbose $params.diann_debug \\
            --rt-profiling \\
            --temp ./quant/ \\
            --use-quant \\
            --predictor \\
            |& tee assemble_empirical_library.log

the log file does not have "Averaged recommended settings for this experiment: Mass accuracy = 11ppm, MS1 accuracy = 15ppm, Scan window = 7" info in Step 3. Full logs as follows:

assemble_empirical_library.log

assemble_empirical_library.log

Means to specify the silico-library generated in the first step?

Yes.

'Averaged recommended settings' will be reported only if DIA-NN is run with either --individual-mass-acc or --individual-windows, and these options are only relevant if mass accuracies and the scan window are not fixed. I amended the description of Step 4 to clarly indicate this.
The --predictor command should not be used in Step 3. You already have an in silico predicted library, no point in carrying out prediction yet another time.
Not sure what are 'var_ptm' and 'fixed_ptm', but I guess this is parsed somehow before being sent to DIA-NN?

Thanks a lot. I just updated steps 2 and 3: add --individual-mass-acc and --individual-windows . Then remove --predictor in step 3.  But the average recommendation settings are produced at step 2 and not at step 3. Finally I uploaded step 4:
Step 4:
    mass_acc = params.mass_acc_ms2
    scan_window = params.scan_window
    ms1_accuracy = params.mass_acc_ms1
    time_corr_only = params.time_corr_only ? "--time-corr-only" : ""

    if (params.mass_acc_automatic | params.scan_window_automatic){
        mass_acc = "\$(cat ${diann_log} | grep \"Averaged recommended settings\" | cut -d ' ' -f 11 | tr -cd \"[0-9]\")"
        scan_window = "\$(cat ${diann_log} | grep \"Averaged recommended settings\" | cut -d ' ' -f 19 | tr -cd \"[0-9]\")"
        ms1_accuracy = "\$(cat ${diann_log} | grep \"Averaged recommended settings\" | cut -d ' ' -f 15 | tr -cd \"[0-9]\")"
    }

    """
    diann  --lib ${library} \\
            --f ${mzML} \\
            ${min_pr_mz} \\
            ${max_pr_mz} \\
            ${min_fr_mz} \\
            ${max_fr_mz} \\
            --threads ${task.cpus} \\
            --missed-cleavages $params.allowed_missed_cleavages \\
            --min-pep-len $params.min_peptide_length \\
            --max-pep-len $params.max_peptide_length \\
            --min-pr-charge $params.min_precursor_charge \\
            --max-pr-charge $params.max_precursor_charge \\
            --var-mods $params.max_mods \\
            --verbose $params.diann_debug \\
            --temp ./ \\
           --cut {enzyme} \\
           --var_ptm {variable modifications from sdrf input by users} \\
           --fixed_ptm {fixed modifications from sdrf input by users} \\
            --min-corr $params.min_corr \\
            --corr-diff $params.corr_diff \\
            --mass-acc \$(echo ${mass_acc}) \\
            --mass-acc-ms1 \$(echo ${ms1_accuracy}) \\
            --window \$(echo ${scan_window}) \\
            ${time_corr_only} \\

@daichengxin there's something wrong, average recommended settings cannot be produced during Step 2, only Step 3. Can you please share the logs?

log in step 2
report.log.txt
log in step 3
report.log.txt

Need in Step 3 one of the two commands that indicate that the mass accs or scan window need to be determined separately for different runs, please see above.

So is my current second step command correct? It also produced average recommended settings.

Indicated this in the main description now.

Nice. Thanks a lot @vdemichev. I am working.

Yes, Step 2 is correct, except it just produces recommended settings for a single run, which is not something of interest

Hi @vdemichev . This is full command in step 4. Could you check it? The recommended settings are extracted from log in step3 and passed to step4. Thanks a lot in advance!
if (params.mass_acc_automatic | params.scan_window_automatic){
      mass_acc = "\$(cat ${diann_log} | grep \"Averaged recommended settings\" | cut -d ' ' -f 11 | tr -cd \"[0-9]\")"
      scan_window = "\$(cat ${diann_log} | grep \"Averaged recommended settings\" | cut -d ' ' -f 19 | tr -cd \"[0-9]\")"
      ms1_accuracy = "\$(cat ${diann_log} | grep \"Averaged recommended settings\" | cut -d ' ' -f 15 | tr -cd \"[0-9]\")"
  }

  """
  diann  --lib ${library} \\
          --f ${mzML} \\
          ${min_pr_mz} \\
          ${max_pr_mz} \\
          ${min_fr_mz} \\
          ${max_fr_mz} \\
          --threads ${task.cpus} \\
          --missed-cleavages $params.allowed_missed_cleavages \\
          --min-pep-len $params.min_peptide_length \\
          --max-pep-len $params.max_peptide_length \\
          --min-pr-charge $params.min_precursor_charge \\
          --max-pr-charge $params.max_precursor_charge \\
          --var-mods $params.max_mods \\
          --verbose $params.diann_debug \\
          --temp ./ \\
          --min-corr $params.min_corr \\
          --corr-diff $params.corr_diff \\
          --mass-acc \$(echo ${mass_acc}) \\
          --mass-acc-ms1 \$(echo ${ms1_accuracy}) \\
          --window \$(echo ${scan_window}) \\
          --no-ifs-removal \\
          --cut {enzyme from sdrf input by user} \\
          --var-mod {var_mods from sdrf input by user} \\
          --fixed-mod { fixed_mods from sdrf input by user} \\
          ${time_corr_only} \\

Full command in step 5:
    diann  --lib ${empirical_library} \\
            --fasta ${fasta} \\
            --f ${(mzMLs as List).join(' --f ')} \\
            ${min_pr_mz} \\
            ${max_pr_mz} \\
            ${min_fr_mz} \\
            ${max_fr_mz} \\
            --threads ${task.cpus} \\
            --missed-cleavages $params.allowed_missed_cleavages \\
            --min-pep-len $params.min_peptide_length \\
            --max-pep-len $params.max_peptide_length \\
            --min-pr-charge $params.min_precursor_charge \\
            --max-pr-charge $params.max_precursor_charge \\
            --var-mods $params.max_mods \\
            --verbose $params.diann_debug \\
            ${scan_window} \\
            --min-corr $params.min_corr \\
            --corr-diff $params.corr_diff \\
            ${mass_acc} \\
            ${time_corr_only} \\
            --cut ${enzyme from sdrf input bu user} \\
            --var-mod ${var mods from sdrf input by user} \\
            --fixed-mod ${ fixed mods from sdrf input by user} \\
            --temp ./quant/ \\
            --relaxed-prot-inf \\
            --pg-level $params.pg_level \\
            ${species_genes} \\
            --use-quant \\
            --out diann_report.tsv \\

Are mass-acc mass-acc-ms1 and window parameters still needed here? and from the recommended settings

Hi @daichengxin, Step 4 I would recommend to run without --min-corr and --corr-diff and without --time-corr-only. Also, during Step 5 these have no effect. --cut, --var-mod and --fixed-mod as well as all other digest settings have no effect during Steps 4 and 5. Suring Step 5 I would add --matrices. Otherwise looks good. Please let me see the full DIA-NN logs from all Steps 1-5 once the pipeline is set up.

ok. --min-pr-mz,  --max-pr-mz, --min-fr-mz, --max-fr-mz, --min-pep-len, --max-pep-len, --min-pr-charge and --max-pr-charge should also have no effect during step4 and 5? @vdemichev

Hi @vdemichev  This full DIA-NN logs from all Steps 1-5.
step1 log:
lib.log.txt
step2 log:
report.log.txt
step3 log:
report.log.txt
step4 log:
report.log.txt
step5 log:
diann_report.log.txt

Seems good! Some comments:
Step 1:

Min peptide length set to 15 - very strange, I guess for testing purposes only?
Min fragment m/z set to 500 - bad for most DIA datasets
Steps 2 to 5: digest settings have no effect.
Step 4: can add --no-main-report, as don't need a report; can also turn off protein inference, as it's not necessary at this stage.

Yes, min 15 is because we need CI/CD to be running fast, it is the same case for min fragment m/z set to 500. Can you check the other comments @daichengxin
Thanks a lot @vdemichev for your support

Thanks a lot @vdemichev . There are updated logs from step2 to 5. Could you check again?
step 2:
report.log.txt
step 3:
report.log.txt
step 4:
report.log.txt
step 5:
diann_report.log.txt

Looks good!

@Thanks to @vdemichev and @daichengxin for working in the first iteration of DIANN with quantms. Here the first full dataset reanalyzed with DIANN http://ftp.pride.ebi.ac.uk/pride/data/proteomes/differential-expression/PXD004684-disease/
The volcano plot result of MSstats can be found here.

Some comments:

Homo-sapiens-uniprot-reviewed-isoforms-contaminants-decoy-202105.fasta - the FASTA database supplied to DIA-NN should never include decoys; same applies to spectral libraries
Step 4 can work fine without FASTA database here, but the subsequent protein inference (Step 5) might perform better if the FASTA database is specified (if it is, please make sure protein inference is enabled).  I now amended the description of Step 4.
32 million precursors are A LOT. I would seriously consider not to specify all those variable mods & restrict missed cleavages to 1. It is fundamentally wrong to treat DIA data the same way it;s customary to deal with DDA data, and if software can do something, does not mean it's beneficial.

Hi @vdemichev . This is the log for step 4, Could you check again ?
report.log.txt

With or without? It looks like he suggested without if I am not mistakenm

@daichengxin Step 4 log seems good!
@ypriverol FDR thresholds during steps 2-4 should be set to 1%. During Step 5, can use any threshold. Filtering stricter than 1% might significantly reduce IDs - it's always better to analyse at 1%, and then the user can filter the output for their specific workflow. --matrix-spec-q and --matrix-qvalue affect matrices only, not the main report. I suggest not using these, as default settings for matrices seem the best for most datasets, and those users who want more stringent can filter the main report. But good to include matrices too, for those who don't use R/Python.

@vdemichev correct me If I'm wrong:

we will apply in Step 2,3,4 Precursor FDR default at 1%, corresponding parameter (--matrix-qvalue = 0.01), the user will be able to set different values in case they want to get more ids, but by default with be 0.01.
configurable parameter in step 5:

--matrix-spec-q: we will pass this default value 1% (this is 0.01) to run protein filter, if the users wants to define their own value to get less highlight reliable data, they can relax this parameter.  In reanalysis because we can't check in details each dataset, we may use this parameter as default 0.01 for high reliable results.
--matrix-qvalue: Configurable also in the Step 5 for the user if they want to get more ids they may use a less stringent FDR let say 0.05.

We will not use in any step the  --matrix-tr-qvalue or --matrix-ch-qvalue.
Please correct me if I understand the parameter's combination well.

Hi @ypriverol, matrix q-value setting only controls the q-value filtering of matrix output, nothing else. So of no relevance for Steps 1-4. For Steps 2-4, please only use --qvalue 0.01 (but 0.01 is the default value, so can also just omit this).
--matrix-spec-q: configures run-specific protein FDR for matrix output. Other software simply does not have this setting at all, filtering on run-specific protein FDR is questionable (many advise against it) and reduces data completeness. So for most applications it's better not to filter, and DIA-NN by default does not filter. Note that global protein FDR is always applied to matrices regardless of this setting.
--matrix-qvalue - again, better to keep 0.01 (default) for most applications. Whoever wants stricter filtering can always filter the main report manually. 0.05 is too relaxed for most applications. As matrices cannot be filtered further, better to stick to a reasonable value which is 0.01.

Hi @vdemichev . Is it possible to manually filter the value of the column Q.Value in main report ?

Hi @vdemichev . Is it reasonable to manually filter the value of the column Q.Value in main report  so that downstream analysis?

Yes, if the --qvalue was set to value higher than 0.01 (this is OK only for Step 5), then the output report must be filtered at Q.Value <= 0.01. Applying more strict filtering than 0.01 will result in more missing values & likely worse downstream analysis. I suggest not to do that, as tghis strict filtering will inevitably result in mahy people looking at the data and thinking, falsely, that the ID numbers are low and missing value rate is high...


Compiling benchmarks for HIP and CUDA kernels simultaneously fails

When HAS_CUDA and HAS_HIP are both set, benchmark/utils/formats.hpp transitively includes both cuda_runtime.h and hipsparse.h, which causes some issues with redefinition of symbols.
Excerpt of the error messages:
In file included from /opt/rocm/include/hip/hcc_detail/channel_descriptor.h:28:0,
                 from /opt/rocm/include/hip/hcc_detail/hip_texture_types.h:38,
                 from /opt/rocm/include/hip/hcc_detail/hip_runtime_api.h:45,
                 from /opt/rocm/include/hip/hip_runtime_api.h:329,
                 from /opt/rocm/include/hipsparse.h:41,
                 from /home/tobi/ginkgo/benchmark/utils/hip_linops.hip.hpp:40,
                 from /home/tobi/ginkgo/benchmark/utils/formats.hpp:51,
                 from /home/tobi/ginkgo/benchmark/solver/solver.cpp:45:
/opt/rocm/include/hip/hcc_detail/hip_vector_types.h:839:1: error: conflicting declaration ‘using uchar1 = struct HIP_vector_type<unsigned char, 1>’
 __MAKE_VECTOR_TYPE__(uchar, unsigned char);
 ^
In file included from /usr/local/cuda-10.1/targets/x86_64-linux/include/driver_types.h:59:0,
                 from /usr/local/cuda-10.1/targets/x86_64-linux/include/builtin_types.h:59,
                 from /usr/local/cuda-10.1/targets/x86_64-linux/include/cuda_runtime.h:91,
                 from /home/tobi/ginkgo/benchmark/utils/cuda_linops.hpp:40,
                 from /home/tobi/ginkgo/benchmark/utils/formats.hpp:48,
                 from /home/tobi/ginkgo/benchmark/solver/solver.cpp:45:
/usr/local/cuda-10.1/targets/x86_64-linux/include/vector_types.h:362:42: note: previous declaration as ‘typedef struct uchar1 uchar1’
 typedef __device_builtin__ struct uchar1 uchar1;
                                          ^~~~~~

For completeness, here is a part of the include tree as output by c++ -H ...

We definitely can.  Putting something like the following somewhere in benchmark/CMakeLists.txt should do the trick (untested):
if (GINKGO_BUILD_CUDA AND GINKGO_BUILD_HIP AND GINKGO_HIP_PLATFORM MATCHES "hcc")
  message(FATAL_ERROR "Building the benchmarks with both HIP AMD and CUDA support "
      "at the same time is currently not supported. Use `export HIP_PLATFORM=nvcc` instead.")
endif()

FATAL_ERROR can be replaced by WARNING depending on what we want.

The issue seems more complicated than expected: Even when the build was initially configured with HIP_PLATFORM=nvcc, when executing make from another shell without this environment variable, we revert back to the hcc platform. I'm not entirely sure whether this is caused by Ginkgo or some part of HIP, but nevertheless we should take care of this before merging #395

We cannot do anything about this. This is how HIP works. Use /opt/rocm/hip/bin/hipconfig --platform. Without the environment variable and without a CUDA GPU, this will return hcc. Only and only when you do export HIP_PLATFORM=hcc is this properly defined.
I feel they should have at least made a global configuration file somewhere, which eventually the environment variable could take precedence over, but that isn't the case.

And I might add that you cannot set this from CMake, because of the problem you said. Consider inside CMake you set this environement variable.

During CMake time, the variable will be properly set and the proper configuration will be created,
When you call make, you call this from another terminal, therefore the setting inside CMake does not exist, and you will consider the wrong architecture when building Ginkgo.

I think it is environment variable problem.
When you use export HIP_PLATFORM=nvcc, its lifetime is only in this shell.
You can add this in .bashrc to make the environment always same, or let cmake can add GKO_HIP_PLATFORM in compile time (I do not know cmake works or not).
I prefer the first way because execution maybe needs this variable.


No module named honssh.config

Hi,
Im a complete Linux noob and as part of a uni project need to setup HonSSH, I believe I've installed everything correctly and when starting to start honssh I keep getting the subject error.

Been following this guide (but on host rather than virtual env):
https://www.linkedin.com/pulse/analyse-your-attackers-vectors-ssh-honey-pot-robert-putt
There is a honssh.cfg file so not really sure what to do. Is this the file its referring to or is there another file in a different location?

Thanks for answering, I've done that - pythonpath="/root/honssh" and still get the same error. Any other ideas?
I've looked at those pages and they're not really helpful for newbies like myself that have very limited Linux knowledge, I can essentially install stuff and navigate the files, but would have no idea how to install and actually start HonSSH.

Hi @stuartt44
Is your host listening publicly on the internet? I'd be more than happy to SSH in and take a poke at this for you as I am guessing time is of the essence due to it being a university project.
Best Regards,
Rob

Hi @stuartt44
And I have the same problem about the PYTHONPATH.
export PYTHONPATH="$PYTHONPATH:/folder/to/honssh" and export PYTHONPATH=/path/to/honssh are not work for me, too. (idk why...)
Anyway, python scripts are easy to debug.
Try to fix it
In honssh.tac, add sys.path.append('/path/to/honssh') before load honssh.config module.
And then, No module named GeoIP occurred in output_handler.py.
Since I installed python-geoip by using pip, the module name should be geoip.
So, edit output_handler.py. import GeoIP ->import geoip
It seems no bugs right now...(still studying)
Hope you can fix the problem and sorry for my poor English.