cellbender -> pytorch -> disk full, but it is not!

Description of the bug
Ok, I get a "disk full" error. With large single-cell datasets this does not sound completely unexpected, but it is also why I am in our HPC environment, which from what "df -h" keeps telling me has many hundreds of terabytes at my disposal. The only exception is $(HOME) with just some 35GB.  /tmp seems to be fine, too (not memory mapped). A disk /scratch has a local 1TB ssd waiting that nobody knows about. The work directory is on that huge /data one.
For better compatibility with singularity, @nictru kindly suggested adding
singularity.runOptions = '--no-mount tmp --writable-tmpfs'

to nextflow.config. Is that in conflict with that pytorch magic?
I have an environment setting for Singularity
$ env|grep -i singu
SINGULARITY_BIND=/data

which I can possibly extend to my home directory and then have the singularity.runOptions changed back to the defaults?
Command used and terminal output
(base) [sm718@haumea1 4392e73f083be38edd2f0f80698751]$ cat .command.err
cellbender:remove-background: Command:
cellbender remove-background --input JM_24h_unified.h5ad --output JM_24h_cellbender.h5
cellbender:remove-background: CellBender 0.3.0
cellbender:remove-background: (Workflow hash b4603011ef)
cellbender:remove-background: 2024-07-04 11:11:36
cellbender:remove-background: Running remove-background
cellbender:remove-background: Loading data from JM_24h_unified.h5ad
cellbender:remove-background: AnnData format
cellbender:remove-background: Found `.layers['counts']`. Using for count data.
cellbender:remove-background: Only 6834 barcodes were found.
This suggests the matrix was prefiltered.
CellBender requires a raw, unfiltered [Barcodes, Genes] matrix.
cellbender:remove-background: WARNING: Only 6834 barcodes in the input file. Ensure this is a raw (unfiltered) file with all barcodes, including the empty droplets.
cellbender:remove-background: Features in dataset: 89916 NA
cellbender:remove-background: Trimming features for inference.
cellbender:remove-background: 55800 features have nonzero counts.
cellbender:remove-background: Prior on counts for cells is 22156
cellbender:remove-background: Prior on counts for empty droplets is 10829
cellbender:remove-background: Excluding 3138 features that are estimated to have <= 0.1 background counts in cells.
cellbender:remove-background: Including 52662 features in the analysis.
cellbender:remove-background: Trimming barcodes for inference.
cellbender:remove-background: Excluding barcodes with counts below 5414
cellbender:remove-background: Using 464 probable cell barcodes, plus an additional 2673 barcodes, and 2981 empty droplets.
cellbender:remove-background: Largest surely-empty droplet has 11967 UMI counts.
cellbender:remove-background: Attempting to unpack tarball "ckpt.tar.gz" to /tmp/tmp_ymlxgn8
cellbender:remove-background: No saved checkpoint.
cellbender:remove-background: No checkpoint loaded.
cellbender:remove-background: Running inference...
cellbender:remove-background: [epoch 001]  average training loss: 23449.0117
cellbender:remove-background: [epoch 002]  average training loss: 22924.1592  (34.2 seconds per epoch)
cellbender:remove-background: Will checkpoint every 13 epochs
cellbender:remove-background: [epoch 003]  average training loss: 22145.3729
cellbender:remove-background: [epoch 004]  average training loss: 21626.0684
cellbender:remove-background: [epoch 005]  average training loss: 21037.9555
cellbender:remove-background: [epoch 005] average test loss: 22359.1018
cellbender:remove-background: [epoch 006]  average training loss: 20722.9589
cellbender:remove-background: [epoch 007]  average training loss: 20325.7407
cellbender:remove-background: [epoch 008]  average training loss: 20033.0595
cellbender:remove-background: [epoch 009]  average training loss: 19803.3412
cellbender:remove-background: [epoch 010]  average training loss: 19571.5222
cellbender:remove-background: [epoch 010] average test loss: 20960.7742
cellbender:remove-background: [epoch 011]  average training loss: 19405.0574
cellbender:remove-background: [epoch 012]  average training loss: 19188.8284
cellbender:remove-background: [epoch 013]  average training loss: 19022.1505
cellbender:remove-background: Saving a checkpoint...
cellbender:remove-background: Could not save checkpoint
cellbender:remove-background: Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torch/serialization.py", line 379, in save
    _save(obj, opened_zipfile, pickle_module, pickle_protocol)
  File "/opt/conda/lib/python3.7/site-packages/torch/serialization.py", line 604, in _save
    zip_file.write_record(name, storage.data_ptr(), num_bytes)
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/cellbender/remove_background/checkpoint.py", line 115, in save_checkpoint
    torch.save(model_obj, filebase + '_model.torch')
  File "/opt/conda/lib/python3.7/site-packages/torch/serialization.py", line 380, in save
    return
  File "/opt/conda/lib/python3.7/site-packages/torch/serialization.py", line 259, in __exit__
    self.file_like.write_end_of_file()
RuntimeError: [enforce fail at inline_container.cc:319] . unexpected pos 5469568 vs 5469456

terminate called after throwing an instance of 'c10::Error'
  what():  [enforce fail at inline_container.cc:319] . unexpected pos 5469568 vs 5469456
frame #0: c10::ThrowEnforceNotMet(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void const*) + 0x64 (0x2b02ea7e7cf4 in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x2bb7329 (0x2b02e41c0329 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #2: mz_zip_writer_add_mem_ex_v2 + 0x603 (0x2b02e41ba573 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #3: caffe2::serialize::PyTorchStreamWriter::writeRecord(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void const*, unsigned long, bool) + 0x113 (0x2b02e41c2e23 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #4: caffe2::serialize::PyTorchStreamWriter::writeEndOfFile() + 0x49b (0x2b02e41c378b in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #5: caffe2::serialize::PyTorchStreamWriter::~PyTorchStreamWriter() + 0x17e (0x2b02e41c3a0e in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #6: <unknown function> + 0x6b81fb (0x2b02e0eec1fb in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #7: <unknown function> + 0x346b02 (0x2b02e0b7ab02 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #8: <unknown function> + 0x3478ff (0x2b02e0b7b8ff in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #9: <unknown function> + 0xe5698 (0x56063f15c698 in /opt/conda/bin/python3.7)
frame #10: <unknown function> + 0xe6038 (0x56063f15d038 in /opt/conda/bin/python3.7)
frame #11: <unknown function> + 0xe604e (0x56063f15d04e in /opt/conda/bin/python3.7)
frame #12: _PyEval_EvalFrameDefault + 0x5bcf (0x56063f2341af in /opt/conda/bin/python3.7)
frame #13: _PyEval_EvalCodeWithName + 0x33d (0x56063f174ccd in /opt/conda/bin/python3.7)
frame #14: _PyFunction_FastCallKeywords + 0x320 (0x56063f1a6570 in /opt/conda/bin/python3.7)
frame #15: <unknown function> + 0x191de8 (0x56063f208de8 in /opt/conda/bin/python3.7)
frame #16: _PyEval_EvalFrameDefault + 0x16bb (0x56063f22fc9b in /opt/conda/bin/python3.7)

Relevant files
No response
System information
  N E X T F L O W
  version 24.04.2 build 5914
  created 29-05-2024 06:19 UTC (08:19 MESZ)

HPC
slurm
Singularity
CentOS Linux
-r dev

Could you attach the .nextflow.log of this run?

Ah I was able to replicate the issue, no need for the log. Will add the fix to #55 once I have it

I have not been able to find out why exactly the No space left on device occurs, but was able to relocate the TMPDIR to the process workdir, so that we have more control over what is happening.


Vim9: cannot refer to function-local variable in command run by assert_fails()

Describe the bug
In Vim9 script, we cannot refer to a function-local variable in a command run by assert_fails().
To Reproduce
Run this shell command:
vim -Nu NONE -S <(cat <<'EOF'
    vim9script
    def Test_dict_deepcopy()
        v:errors = []
        var d: dict<any> = {'1': 1, '2': 2}
        var l = [4, d, 6]
        extend(d, {'3': l})
        assert_fails('deepcopy(d, 1)', 'E698')
        echom v:errors
    enddef
    Test_dict_deepcopy()
EOF
)

v:errors contains this error message:
['command line..script /proc/16106/fd/11[10]..function <SNR>1_Test_dict_deepcopy line 5: Expected ''E698'' but got ''E121: Undefined variable: d'': deepcopy(d, 1)']

Expected behavior
v:errors is empty.
Environment

Vim version: 8.2 Included patches: 1-1588
OS: Ubuntu 16.04.7 LTS
Terminal: xterm(359)

Additional context
If the same code is refactored in Vim script legacy, then v:errors is empty.
vim9script
func Test_dict_deepcopy()
    let v:errors = []
    let d = {'1': 1, '2': 2}
    let l = [4, d, 6]
    call extend(d, {'3': l})
    call assert_fails('call deepcopy(d, 1)', 'E698')
    echom v:errors
endfunc
Test_dict_deepcopy()

assert_fails() runs the command in the current context, where "d" is not define, it's on the stack.
This works:
try
deepcopy(d, 1)
catch
assert_match('E698:', v:exception)
endtry
But it doesn't check that there was an exception.
For Ex commands that evaluate an expression we already have code to not call the normal handling, but compile it, so that the variables on the stack can be found.
We don't make an exception for builtin functions yet, but this is one place where it makes sense.

It would be simpler (and better) to add a new command:
assertfail
eval unknown
endassertfail E99:
Advantage is that the failing command doesn't need to be put in a string (with escaping)
It doesn't work for an unknown command, that can't be compiled.  But for that the existing assert_fails() can be used.
The argument to endassertfail should be a pattern, so we can use "E99:.*number expected".
Usual Ex command abbreviation:
assert
eval unknown
endassert E99:

Closing because the solution is always the same.  Whenever we have some legacy code which raises an error in Vim9 because some function-local variable can't be found, we need to move it in the script-local scope.
Applied to the previous example, it gives:
vim9script
var d: dict<any>
def Test_dict_deepcopy()
    v:errors = []
    d = {'1': 1, '2': 2}
    var l = [4, d, 6]
    d = extendnew(d, {'3': l})
    assert_fails('deepcopy(d, 1)', 'E698')
    echom v:errors
enddef
Test_dict_deepcopy()

However, for some reason, deepcopy() no longer raises E698.  Maybe because of extendnew()... Anyway, that's a different issue.


Load metadata when running Load on Dora

What changes are proposed in this pull request?
Load metadata as well as data when running Load on Dora
Why are the changes needed?
User still need to know the metadata(ex. length) when openFile
Does this PR introduce any user facing changes?
na

alluxio-bot, merge this please

alluxio-bot, merge this please


RetrieveServiceContent() not found

every time i tried to get Data out of a VHost i get the Error
PHP Fatal error:  Uncaught Error: Call to undefined method Vmwarephp\SoapClient::RetrieveServiceContent()

You need to inform the path to the vmwarephp composer, mine was like this.
 "autoload": {         "psr-4": {             "App\\": "app/",             "Database\\Factories\\": "database/factories/",             "Database\\Seeders\\": "database/seeders/"         },         "classmap": [             "vendor/vmwarephp/vmwarephp/library"         ]     },
then you need to update composer to work.
composer update
https://github.com/vadimcomanescu/vmwarephp/issues/8


Expose the iframe content window when demo-iframe-renderer is in use

Use case: a Vaadin.Router demo using the baseUrl feature. For the particular demo, after the iframe document is loaded, I want to make a client-side navigation in the content window to some base URL instead of the default /.
Currently, accessing the content window for a demo snippet’s iframe is hard. Also, it is hard to get the timing right to invoke thins after load. The iframeSrc property does not help: it changes the iframes’s src, and this results in a full document reload.
Example workaround:
const baseUrlDemoSnippet = this.$['some-demo'];
const origShowDemo = baseUrlDemoSnippet._showDemo;
baseUrlDemoSnippet._showDemo = function() {
  origShowDemo.apply(this, arguments);
  const iframeDemoRenderer = baseUrlDemoSnippet.$.demo.firstElementChild;
  iframeDemoRenderer.__iframeReady.then(() => {
    iframeDemoRenderer.url = '/ui/';
    iframeDemoRenderer._postIframeMessage({type: 'set-url', url: '/ui/'});
  });
};

This repository is no longer maintained and will be archived.


Tablet submission

Code edited and pushed
for Jonathan last comment "Double the width of the clinic name column" am facing some issues while changing the column width because bootstrap table are giving auto size for columns depend on the text inside.(still working on it)


test: Adding tests to achieve 100% test coverage for blas/ext/base/dnansumkbn2

Resolves #3074
Description
This pull request:

Adds test cases to achieve 100% test coverage for the file:- blas/ext/base/dnansumkbn2

Related Issues
This pull request:

resolves #3074

Questions
No.
Other
No.
Checklist

[x] Read, understood, and followed the contributing guidelines

@stdlib-js/reviewers

@Planeshifter sure I'll look into the falling checks and implement the necessary changes.

@Planeshifter I'm finding it difficult to set up an environment and run those test cases could you help me a bit?

@Neerajpathak07 Follow this guide https://github.com/stdlib-js/stdlib/blob/develop/docs/development.md


Running nvm install-latest-npm on current node reverts npm to LTS node npm version

Operating system and version:
Windows 10 Ver 2004
Build 1904.1.572
WSL2 - Ubuntu 20.04
nvm debug output:

nvm --version: v0.37.0
$SHELL: /usr/bin/zsh
$SHLVL: 1
whoami: 'bc1121'
${HOME}: /home/bc1121
${NVM_DIR}: '${HOME}/.nvm'
${PATH}: ${NVM_DIR}/versions/node/v14.15.1/bin:/home/linuxbrew/.linuxbrew/bin:/home/linuxbrew/.linuxbrew/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/mnt/c/Windows/system32:/mnt/c/Windows:/mnt/c/Windows/System32/Wbem:/mnt/c/Windows/System32/WindowsPowerShell/v1.0/:/mnt/c/Windows/System32/OpenSSH/:/mnt/c/Program Files (x86)/Intel/Intel(R) Management Engine Components/DAL:/mnt/c/Program Files/Intel/Intel(R) Management Engine Components/DAL:/mnt/c/Program Files (x86)/NVIDIA Corporation/PhysX/Common:/mnt/c/Program Files/NVIDIA Corporation/NVIDIA NvDLISR:/mnt/c/HashiCorp/Vagrant/bin:/mnt/c/Users/BrentCarter/AppData/Roaming/nvm:/mnt/c/Program Files/nodejs:/mnt/c/windows/system32:/mnt/c/windows:/mnt/c/windows/System32/Wbem:/mnt/c/windows/System32/WindowsPowerShell/v1.0/:/mnt/c/windows/System32/OpenSSH/:/mnt/c/Program Files/PowerShell/7/:/mnt/c/Program Files/Git/cmd:/mnt/c/ProgramData/chocolatey/bin:/mnt/c/Program Files/Docker/Docker/resources/bin:/mnt/c/ProgramData/DockerDesktop/version-bin:/mnt/c/Users/BrentCarter/AppData/Local/Microsoft/WindowsApps:/mnt/c/Users/BrentCarter/AppData/Local/Programs/Microsoft VS Code/bin:/mnt/c/Users/BrentCarter/AppData/Roaming/nvm:/mnt/c/Program Files/nodejs:/mnt/c/Program Files/JetBrains/PhpStorm 2020.2.1/bin:/mnt/c/Program Files (x86)/GitHub CLI/
$PREFIX: ''
${NPM_CONFIG_PREFIX}: ''
$NVM_NODEJS_ORG_MIRROR: ''
$NVM_IOJS_ORG_MIRROR: ''
shell version: 'zsh 5.8 (x86_64-ubuntu-linux-gnu)'
uname -a: 'Linux 4.19.104-microsoft-standard #1 SMP Wed Feb 19 06:37:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux'
checksum binary: 'sha256sum'
OS version: Ubuntu 20.04.1 LTS
curl: /usr/bin/curl, curl 7.68.0 (x86_64-pc-linux-gnu) libcurl/7.68.0 OpenSSL/1.1.1f zlib/1.2.11 brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.2.0) libssh/0.9.3/openssl/zlib nghttp2/1.40.0 librtmp/2.3
wget: /usr/bin/wget, GNU Wget 1.20.3 built on linux-gnu.
git: /usr/bin/git, git version 2.29.2
ls: cannot access 'grep:': No such file or directory
grep: grep: aliased to grep --color=auto --exclude-dir={.bzr,CVS,.git,.hg,.svn,.idea,.tox} (grep --color=auto --exclude-dir={.bzr,CVS,.git,.hg,.svn,.idea,.tox}), grep (GNU grep) 3.4
awk: /usr/bin/awk, GNU Awk 5.0.1, API: 2.0 (GNU MPFR 4.0.2, GNU MP 6.2.0)
sed: /usr/bin/sed, sed (GNU sed) 4.7
cut: /usr/bin/cut, cut (GNU coreutils) 8.30
basename: /usr/bin/basename, basename (GNU coreutils) 8.30
rm: /usr/bin/rm, rm (GNU coreutils) 8.30
mkdir: /usr/bin/mkdir, mkdir (GNU coreutils) 8.30
xargs: /usr/bin/xargs, xargs (GNU findutils) 4.7.0
nvm current: v14.15.1
which node: ${NVM_DIR}/versions/node/v14.15.1/bin/node
which iojs: iojs not found
which npm: ${NVM_DIR}/versions/node/v14.15.1/bin/npm
npm config get prefix: ${NVM_DIR}/versions/node/v14.15.1
npm root -g: ${NVM_DIR}/versions/node/v14.15.1/lib/node_modules

nvm ls output:

->     v14.15.1
        v15.2.1
default -> node (-> v15.2.1)
node -> stable (-> v15.2.1) (default)
stable -> 15.2 (-> v15.2.1) (default)
iojs -> N/A (default)
unstable -> N/A (default)
lts/* -> lts/fermium (-> v14.15.1)
lts/argon -> v4.9.1 (-> N/A)
lts/boron -> v6.17.1 (-> N/A)
lts/carbon -> v8.17.0 (-> N/A)
lts/dubnium -> v10.23.0 (-> N/A)
lts/erbium -> v12.19.1 (-> N/A)
lts/fermium -> v14.15.1

How did you install nvm?

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.37.0/install.sh | bash
manually pasted following source to .zshrc
export NVM_DIR="$([ -z "${XDG_CONFIG_HOME-}" ] && printf %s "${HOME}/.nvm" || printf %s "${XDG_CONFIG_HOME}/nvm")"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh" # This loads nvm
What steps did you perform?
nvm install node
nvm install --lts
What happened?
Ran nvm install-latest-npm on fresh --lts install of node
resulted in no change to npm version

 ~  npm --version
6.14.8
 ~  nvm install-latest-npm
Attempting to upgrade to the latest working version of npm...
* Installing latest `npm`; if this does not work on your node version, please report a bug!
/home/bc1121/.nvm/versions/node/v14.15.1/bin/npm -> /home/bc1121/.nvm/versions/node/v14.15.1/lib/node_modules/npm/bin/npm-cli.js
/home/bc1121/.nvm/versions/node/v14.15.1/bin/npx -> /home/bc1121/.nvm/versions/node/v14.15.1/lib/node_modules/npm/bin/npx-cli.js
+<EMAIL_ADDRESS>updated 1 package in 5.412s
* npm upgraded to: v6.14.8
 ~  npm --version
6.14.8

Ran nvm install-latest-npm on fresh install of current node
resulted in downgrade to latest --lts matched version of npm instead of latest current matched version of npm

~  nvm use node
Now using node v15.2.1 (npm v7.0.8)
 ~  npm --version
7.0.8
 ~  nvm install-latest-npm
Attempting to upgrade to the latest working version of npm...
* Installing latest `npm`; if this does not work on your node version, please report a bug!

removed 51 packages, changed 196 packages, and audited 434 packages in 3s

2 packages are looking for funding
  run `npm fund` for details

2 low severity vulnerabilities

To address all issues, run:
  npm audit fix

Run `npm audit` for details.
* npm upgraded to: v6.14.8
 ~  npm --version
6.14.8
~  node --version
v15.2.1

Uninstalled current node and reinstalled current node to fix

~  nvm use --lts
Now using node v14.15.1 (npm v6.14.8)
 ~  nvm uninstall node
Uninstalled node v15.2.1
 ~  nvm install node
Downloading and installing node v15.2.1...
Local cache found: ${NVM_DIR}/.cache/bin/node-v15.2.1-linux-x64/node-v15.2.1-linux-x64.tar.xz
Checksums match! Using existing downloaded archive ${NVM_DIR}/.cache/bin/node-v15.2.1-linux-x64/node-v15.2.1-linux-x64.tar.xz
Now using node v15.2.1 (npm v7.0.8)

What did you expect to happen?
I expected the matched npm version of current node to stay the same or be upgraded instead of downgraded
Is there anything in any of your profile files that modifies the PATH?

Maybe brew? See below for anything I missed
.zprofile

echo "Starting .zprofile"

eval $(/home/linuxbrew/.linuxbrew/bin/brew shellenv)

# Using Linux PHPStorm inside WSL2
# https://www.ddev.com/ddev-local/ddev-local-and-phpstorm-debugging-with-wsl2/
export DISPLAY=$(awk '/^nameserver/ {print $2; exit;}' </etc/resolv.conf):0.0

echo "Completed .zprofile"

.zshrc

echo "Starting .zshrc"

if type brew &>/dev/null; then
  FPATH=$(brew --prefix)/share/zsh/site-functions:$FPATH
  autoload -Uz compinit
  compinit
fi

# Path to your oh-my-zsh installation.
export ZSH="/home/bc1121/.oh-my-zsh"
ZSH_THEME="agnoster"
plugins=(git)
source $ZSH/oh-my-zsh.sh

alias zshconfig="code ~/.zshrc"
alias ohmyzsh="code ~/.oh-my-zsh"

# bc - https://github.com/ohmyzsh/ohmyzsh/wiki/Themes
DEFAULT_USER=bc1121
prompt_context(){}

source $HOME/.keychain/$HOSTNAME-sh

# 2020-11-17
# https://github.com/nvm-sh/nvm
export NVM_DIR="$([ -z "${XDG_CONFIG_HOME-}" ] && printf %s "${HOME}/.nvm" || printf %s "${XDG_CONFIG_HOME}/nvm")"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" # This loads nvm

echo "Completed .zshrc"

Oops, my fault. Nothing to see here. No issue. Just an uneducated assumption on my part.
I noticed the same exact behavior using...
npm install -g npm@latest
    and
npm install -g npm@next

I found my answers here...
https://www.npmjs.com/package/npm?activeTab=versions
At first, I thought the latest and next version of npm depended on the version of node currently in use on a system, but now I see that next and latest aren't dependant on the version of node in use.

You’re exactly right - until npm 7 is actually “latest”, this will be the behavior. On node 10+, you can safely run npm install -g npm@7 in the meantime.


Is it possible to directly provide train.pkl and test.pkl?

Thank you for your work!
A few months ago, I downloaded the Scannet-GSReg dataset, but I have since lost the train.pkl and test.pkl files. Since downloading the entire dataset takes a long time, I would like to ask if it would be possible to directly provide these two files? Thank you!

All right, these two files are available in OneDrive: https://1drv.ms/f/c/31078419ab45666e/EvF8hMSxY1ZKpng7yYK-WzMB8RuFNZsx3OksoKYgqAVi7g?e=tBqma5


feat: proof of concept for custom entry

Implements # <issue number>

closing as this was used to complete #2142


Add BDArmory Sorting from Kerbal Stuff

This pull request was automatically generated by Kerbal Stuff on behalf of jenyaza01, to add BDArmory Sorting to CKAN.
Please direct questions about this pull request to jenyaza01.

Link is 404. This mod is added via #2210


perf: update ModeDashboardQueryExtractor to discovery api

Summary of Changes
Update ModeDashboardQueryExtractor to use Mode discovery api. Test added.
Documentation
ModeDashboardQueryExtractor config example in README
CheckList
Make sure you have checked all steps below to ensure a timely review.

[x] PR title addresses the issue accurately and concisely, including a title prefix.
[x] PR includes a summary of changes.
[x] My commits follow the guidelines from "How to write a good git commit message"

@dorianj Will this "First-time contributors need a maintainer to approve running workflows." go away once we change the default branch to main? Any plan when to make main as the default branch?

@allisonsuarez @dkunitsk @feng-tao PTAL

I update the default branch to main.

@youngyjd for all the mode ingestion change from legacy API to new discovery API, do we need to update the readme?

@feng-tao I updated the readme for every extractor that I have changed. Any more places that I need to check?
We reached an agreement with the Mode team about changes that they need to make on their discovery api:
1. Mode team adds last_run_status, last_run_timestamp, total_views_count, and space_token to the report discovery api
2. Provide a filter option for the Discovery API Reports and Report stats endpoints that will take a space token and return just the reports for that space
3. Nice to have but not required: include username in membership discovery api

For those extractors that cannot be fully switched to discovery api now before Mode team finishes their work, I will have a temporary fix for them.

@youngyjd that's great!


Support vlocity datapacks natively

sfpowerscripts will support vlocity datapacks as if its another data package. A package will be automatically created when a data package contains with a manifest (a vlocity yaml) along with multiple datapacks, and the reverse mechanism applied as well during installation.
In addition, during prepare stage, do any additional pre-requisites that need to be done before these data packages can be installed into a blank scratch org

Early release in alpha stage in #625

Moving this to October 21, as we are still awaiting feedback from the last release

Closing this issue, JSON linter will be added as a separate feature and will be added in subsequent releases


Update golangci-lint to v1.29.0

Update golangci-lint from v1.27.0 to v1.29.0 and fix some new linter issues.

Coverage remained the same at 35.0% when pulling a4619243317a54288486eafc8d68e9fe1fe331aa on fix-golangci-lint into 5d1459b11104df24c746509845aa23ee49b8e740 on master.

Coverage remained the same at 35.0% when pulling a4619243317a54288486eafc8d68e9fe1fe331aa on fix-golangci-lint into 5d1459b11104df24c746509845aa23ee49b8e740 on master.


Move validation logic out of python frontend and into the backend

Currently we have a lot of validation logic in Collection.py. This is nice in that we validate things before sending them over the wire, but does not abstract well for different clients (like javascript) and different protocols (other than REST).
Because of this, we re-implemented some (but not all) in the JS client. https://github.com/chroma-core/chroma/blob/14c26f5c222e3bb72e811a4403cbc644846629d6/clients/js/src/index.ts
We want to at a minimum dupe the validation logic into the backend, and perhaps also remove it from Collection.py - the "frontend" client.

@levand im guessing this is not a part of the refactor because we are keeping the API the same, correct?

We are keeping the API the same and there are also some performance and operational benefits to having as much validation as we can up front.
Cost is, as you note, code duplication.

difficulty: would require copying logic down from frontend into backend - fairly straightforward
concerns: performance in the distributed system --> validators that require round trips

Update issue - we should do client and server side validation


Cleanup if discovered mqtt switch can't be added

Description:
If a discovered MQTT switch fails to be added, make sure the discovery hash is cleaned up from the ALREADY_DISCOVERED list.
If this is not done, devices can't be discovered on the same topic until Hass is restarted.
Checklist:

[x] The code change is tested and works locally.
[x] Local tests pass with tox. Your PR cannot be merged unless tests pass
[x] There is no commented out code in this PR.

If the code does not interact with devices:

[x] Tests have been added to verify that the new code works.

@OttoWinter No entity is being removed, it's errors preventing an entity from being created that are caught. If an error is caught, no entity had been created, and the discovery hash should no longer be in the most of known discovery hashes.

@fabaff Can you have a look as well?

We could have mqtt discovery listen to a signal and do async_dispatcher_send from the component instead of clearing the list. Do you think it would be better?


column minWidth

Hello,
When I define a column, the minWidth is not taken into account. The column always has the max width, even if I don't have enough space. I have a column I defined minWidth 200 and maxWidth 500, with no data : it always takes 500 width, and that creates an horizontal scroll because my screen is too small.
For info, I tried the DetailsList from FluentUI and it works.
Thanks !

Hi @David-Rousselle
Looking into this


authorize endpoints now require reCAPTCHA for generating new access/refresh tokens

It seems that for some users (or at least for me), trying to POST to /oauth2/v3/authorize using the documented approaches here results in the user being forwarded back to the sign-in page with this error message:

Recaptcha is required.

This differs from the SVG-based CAPTCHA login flow that others have documented previously.
It doesn't seem like there's a workaround to this unless you use a webdriver and track redirects.
The documentation needs to be updated to reflect this if anyone else is experiencing the same behavior.

My findings regarding this topic.
reCaptcha initial loading checks are:
Exact headers with exact user-agent
minTLS 1.2
keep-alive:true
reCaptcha login submission.
Exact headers
5 cookies
you will receive 4 after initial loading and the last one after loading i18n text files
In the bottom of the login page is a random script tag what needs to be executed in a browser before submit the login information.
This script will submit information "sensor_data" like timestamp cookies and useragent

Site key is key to solving the problem and then outsourcing site key to captcha solving service using recaptcha :
https://github.com/fkhera/powerwallCloud/commit/d731951c746d44b25dc3905bacb0f8aba7e1d87aOn

But the user have to pay for this? Why don't let the user login manually and copy paste the auth_code to receive the auth_token. This login is once a lifetime action for new installations.
This is the way how it is done  in the ioBroker Adapter to have full control over tesla and powerwalls.

Or just kinda brute-force it, like I'm doing: https://github.com/timdorr/tesla-api/discussions/501#discussioncomment-1829937

Site key is key to solving the problem and then outsourcing site key to captcha solving service using recaptcha :
fkhera/powerwallCloud@d731951

Wow.... their workers are only paid ~$1.01/21000sec or ~$0.17/hr. I wonder how many are just ML bots.

Always get the 403 as well - any solution for that?

Always get the 403 as well - any solution for that?

@MobAbi Did you solve the 403 error for step 2? Any experience could share?

I haven't had any issue getting refresh tokens, but it seems that you need to regenerate your access token every few months. (I think I've had to do it twice, but been using this API for over a year or so)
I'm not able to get through the current issue where either reCaptcha or other user-agent issues are preventing me from getting the initial token. Even if it takes some manual work to get the access token then automate the refresh token, has anybody found any way to workaround this? (This is mostly a bump and subscribe to any updates. 😅)


Inquiry: Will this library be updated to Angular V14

With the addition of typed forms in Angular 14, will this library be updated to support this version?

Hello @dfulgham,
Sure, will let you know once it is done.

@dfulgham Besides the addition of untyped form classes, is there any things you might find necessary to adapt ?

@dfulgham Please update to v1.4.0 and let me know if something is not OK with it

Currently I'm using angular v14.1.0 So give me type error in ClassValidatorFormControl.
Now : export class ClassValidatorFormControl<T = any> extends **FormControl **
Solved: export class ClassValidatorFormControl<T = any> extends FormControl<T | any>;
There is formcontrol type error so please mearge that.


Web-Socket Pre-Upgrade Handler

This is a question/feature request if there's no answer.
Is there a way to do things before connection (e.g. during the upgrade phase)? For example, I'd like to return a 400 if a certain query parameter is unspecified, or do some authentication (returning a 401 if it fails), as well as determine the requested WebSocket protocol.
Thanks!

Hi @travigd. If I remember correctly, you can use the AccessManager for this. At least for the 401.

@travigd, 401 can be handled by the AccessManager, but AFAIK most people use wsBefore (handling auth only during the upgrade isn't always sufficient, since connections are stateful).
I'm trying to clean up a bit, so I'll close this, but feel free to keep commenting.

Thanks for the response but that's still only half of my question. :^)
Is there any way to intercept the upgrade request and do things before the upgrade is completed (like return a 400)?

:^) :^) :^)
The HTTP Servlet ignores WebSocket upgrade requests, but the WsServlet runs the current AccessManager to check if the upgrade is okay. This means you get a 401 if the request doesn't get through, but you can't send back something arbitrarily, like a 400.
To be more clear: No, it's not possible to manually intercept upgrade requests and send back customized responses.


Add Create and Upsert CTL Dataset YAML Routes [#1767]

👉 Note, this PR is against unified-fides-resources not main
Closes #1767
Code Changes

[x] Add new endpoint POST {{host}}/yml/dataset.

[x] This endpoint is similar to POST {{host}}/yml/dataset except your request body should be yaml.
[x] Content-type should be application/x-yaml
[x] Only one dataset can be submitted through this endpoint

[x] Add new endpoint POST {{host}}/yml/dataset/upsert

[x] This endpoint is similar to POST {{host}}/yml/dataset/upsert except your request body should be yaml.
[x] Content-type should be application/x-yaml
[x] Multiple datasets can be submitted through this endpoint.  If a fides_key matches an existing fides_key, we update the matching resource, otherwise, we create a new record.

Steps to Confirm

[ ] See new endpoints in committed postman collection.  There's no UI for these endpoints but you can test them out via API calls.

Pre-Merge Checklist

[ ] All CI Pipelines Succeeded
Documentation:

[ ] documentation complete, PR opened in fidesdocs
[ ] documentation issue created in fidesdocs

[ ] Issue Requirements are Met
[ ] Relevant Follow-Up Issues Created
[ ] Update CHANGELOG.md. <- We're going to have to update the CHANGELOG anyway before merging unified-fides-resources

Description Of Changes
The old DatasetConfig yaml endpoint PATCH {{host}}/yml/connection/{{connection_key}}/dataset will likely be deprecated soon, as it is upserting CTL Datasets before linking the CTL Dataset to a DatasetConfig.  We'd like to the CTL Dataset to be modified directly first without this happening magically behind the scenes.
Anyway, what was nice about this existing endpoint is that you could upload yaml directly instead of having to convert a yaml file to JSON first. We're porting this over to the CTL Dataset endpoints in case it's useful.

@pattisdr I think I'm just missing the context here, but what was the primary reasoning for wanting to enable loading datasets via yaml? I think theoretically it is an extra step, but considering the CLI abstracts this, is this an additional maintenance burden that feels necessary? We've had no requests from customers to add this functionality that I'm aware of

I found this functionality useful on the ops side and I was maintaining parity with an API endpoint that existed ops side that we are planning on deprecating.  If that's not a strong enough case, can you instead explain what is available on the ctl side?   Can you only create datasets via json or is there a yaml option?

I found this functionality useful on the ops side and I was maintaining parity with an API endpoint that existed ops side that we are planning on deprecating. If that's not a strong enough case, can you instead explain what is available on the ctl side? Can you only create datasets via json or is there a yaml option?

I thought I had opened an issue semi-related to this but I can't find it, so maybe I didn't 🤔
On the ctl side, we only support JSON in the API endpoints, and the primary reason for this is that we always expect the user to interact with the webserver via the CLI. So the CLI ingests the YAML, does the parsing and validity checks, and then sends a JSON object. This is a difference between ops and ctl, so in the end maybe this doesn't/shouldn't apply to ops, but I think having two different user experiences detracts from the synergy we're trying to establish.
I had planned on opening an issue (I still should) that enables the same thing experience for ops. Right now, as you know, a user has to make API calls, most likely by setting up scripts of some kind, to configure ops programmatically. My idea was that the CLI should also handle YAML versions of the ops objects so that the users aren't needing to make API calls directly, and instead can store their ops configuration as YAML objects and with a single fides push configure everything. On second thought I'm not sure if that's a good idea considering there would be some secrets there potentially and we wouldn't want users storing them in version control.
That being said, this is a huge tangent and not directly related to this PR, so I don't see a blocker here! It just triggered my brain to remember that I had planned on establishing that functionality, but there's no harm in supporting YAML as well, it just didn't seem strictly necessary when I glanced at it

thanks for the background @ThomasLaPiana, given your eventual plans to have the CLI handle yaml versions of ops objects, would you prefer if we just closed?

thanks for the background @ThomasLaPiana, given your eventual plans to have the CLI handle yaml versions of ops objects, would you prefer if we just closed? I'm now leaning towards closing.

If it is beneficial to you in the short-term to have this functionality, I'd say merging is the right move for sure. However if you're only doing this to update/expand existing functionality, it is worth considering that eventually I'd like to remove this in favor of handling the ops objects via the CLI

OK thanks Thomas, I'm just going to close.


How do I check if the Kafka connection is loss and to reconnect

Description
For any open Producer or Consumer, is there way to check if the Kafka connection is down and to issue a reconnect.
How to reproduce
Checklist
Please provide the following information:

[ ] Confluent.Kafka nuget version:
[ ] Apache Kafka version:
[ ] Client configuration:
[ ] Operating system:
[ ] Provide logs (with "debug" : "..." as necessary in configuration)
[ ] Provide broker log excerpts
[ ] Critical issue

the client will automatically recover (reconnect) from all errors not marked as fatal (note: the IsFatal property on ErrorEvent is new in 1.0-beta). The only fatal errors currently are related to the idempotent producer (not yet in a release).
You can get information about whether there are connectivity issues via the OnError event, but you don't need to take action. Documenting all these cases is on our todo list.

I like it, nothing to do.


Vaccination endpoint does not return valid data again.

Yesterday you fixed the vaccination endpoint and it worked. This morning it seems, that there is an error again. Could you please check this?

sorry for that! Yesterday in the evening i put another little patch , and there is something going worng.
Afther marlon is approving the new PR all should work fine!


carpool db schema 9/26/16

I extracted the current DB schema today 9/26/16.
As the schema is still evolving, I'll extract it daily through cron.

I think it would make sense to use a .sql extension here. I'll file a ticket to rename.


Update ExchangeSenderMemberOf to include non mail-enabled security gr…

Just changed the description to highlight that non mail-enabled security groups can also be added while creating policy for exchange

Docs Build status updates of commit 0e1a437:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

exchange/exchange-ps/exchange/New-DlpCompliancePolicy.md
:white_check_mark:Succeeded
View (exchange-ps)

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit 1ac6f50:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

exchange/exchange-ps/exchange/New-DlpCompliancePolicy.md
:white_check_mark:Succeeded
View (exchange-ps)

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

@tanaybaldua, I believe it's possible to add a non-mail-enabled security group to the policy. But, the parameter Type value for ExchangeSenderMemberOf and ExchangeSenderMemberOfException is RecipientIdParameter[], which means more than one recipient. And, there's no way a non-mail-enabled security group is a recipient.
Comments, please.

@chrisda : Non Mail enabled security groups can be added for exclusion of certain policies and thus the write up should have it. the policy works to include/exclude these security groups. an admin may want to create a policy that wants users to send a reason for override - but this would be applicable to non mail enabled security groups and thus they want to exclude them. Hope this helps.

Docs Build status updates of commit 1905ea1:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

exchange/exchange-ps/exchange/New-DlpCompliancePolicy.md
:white_check_mark:Succeeded
View (exchange-ps)

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


Make other pages as per the theme.

Theme Copied From: https://modern.seowptheme.com/

hello @Priyanshu85 ,
which page you want to make please specify and assign the task to me.

Hey @sahillede940 ,
https://modern.seowptheme.com/
This is the website which I cloned using react. Take any page ex. About and as  many you can and clone it. Also don't forget to make Routes.

@Priyanshu85 I can start working on the contact section...

@Priyanshu85 But the pages on your project. So your telling us to clone from the specified theme and add it to your project? Or make a new page related to theme from scratch?

Clone a Page like about and add it in my repo and then pass the Route to it using React Router Dom

@Priyanshu85 I will be working on About page. By saying Route are you suggesting to make React-Router? or just add path in href?

Use React Router Dom bruh!

@Priyanshu85 I can start working on the contact section...

Sure...Just make sure to make route

@Priyanshu85 I will be working on About page. By saying Route are you suggesting to make React-Router? or just add path in href?

Use React Router Dom bruh!

thats what i was asking about. I did routing with react router dom also linked header and footer links. Should I make PR.

@Priyanshu85 I will be working on About page. By saying Route are you suggesting to make React-Router? or just add path in href?

Use React Router Dom bruh!

thats what i was asking about. I did routing with react router dom also linked header and footer links. Should I make PR.

Ya Sure if your  About Page is Ready!


fix: allow yss motion to work with dot-repeat

Allows lines surrounded with yss to be dot-repeatable. Currently dot repeating will only surround to the end of the line, since the ^ motion is processed before the <Plug> call. This PR also keeps the current count functionality.
Current behavior (when last action was e.g. yssB):
hi |there -> . -> hi BthereB
With this PR:
hi |there -> . -> Bhi thereB
Cursor move is also respected.

Not sure why this other test is failing, it should be unaffected. I'll look into it more


Resolve library home correctly when installed as haxelib

Currently when installed through haxelib the value of jsStubPath resolves to [...]haxelib/modular-js/js which is missing the haxelib version part of the path.
This fixes resolution of that path by starting at the classpath [...]/src going up one level to ../js

Oh yes, that's much better.


semgrep rules?

I wrote some simple semgrep rules to help catch common mistakes (failing to defer try.Handle et al) and misuses (try.E without a function-local handler). They have some exceptions that are peculiar to our codebase, but we'd like to share a general purpose rules file, I could extract one.

Sure, wanna drop them in a readme or something?


2358 nations showing as ‘na’ on organisation search results

Changes in this PR
This seemed to be being caused by the distinctOn function in the allLive function that was initially added to prevent duplicate orgs showing.
Screenshots of UI changes
Before

After

This makes my very nervous as I can see commit 0b7735c1 specifically added this in 😅
If you've interrogated the ways the duplicates might occur - multiple professions to an org? those professions having multiple versions? - I'm good for this to go in. And definitely check on staging/live once this reaches those
A possible "while you're here" thing (feel free to say this is scope creep) - would it make sense to pass in ProfessionVersionStatus.Live to the versionsWithJoins here?

Yeah I agree, I'm going to spend some time this afternoon trying to get duplicates showing locally, I was hoping there would be a failing test if they do show but I don't think there is.


About cross chain address validation.

Description
In some cases that need cross chain verification, original contract address is needed to make sure the action is exactly as expected.
Scenario
Like #1885 , related to cross chain transfer from chain C1 to chain C2, actual token contract address of chain C1 should be validated and stored on chan  C2.
Solution
API
Genesis Smart Contract
ValidateSystemContractAddress(address, hashName)
Validate system contract address existence inside chain.
Contract SDK
GetZeroSmartContractAddress(chainId)
Return genesis contract address of specific chain.
How to validate address of chain C1 on chain C2
Step 1.  Validate itself locally
Transaction T  on chain C1  invoking
ValidateSystemContractAddress(address, hashName)of genesis contract to validate token address  A existence.
Step2. Validate address A on Chain C2
Do cross chain verify  for transaction T.
If it succeeds,  token address  A from chain   C1 can be trusted on chain C2.
What is more, in this processing, to be sure that transaction T exactly happened on right place,  genesis contract  address A0 from chain C1 can be checked by GetZeroSmartContractAddress(chainId) on Chain C2 .

This solves 1 and 3 in #1868 , however, we will need to take care of Guanglei's 2nd concern as well.

This solves 1 and 3 in #1868 , however, we will need to take care of Guanglei's 2nd concern as well.

Check #1885 . TokenInfo identity can be protected among chains.

I understand, I thought this is mainly for cross chain transfer. I have no concern with this solution if it is purely to manage address validation in cross chain.
For cross chain transfer, I checked #1885 and I am still not clear how we will handle the token supply amount for a specific token. To transfer, we burn first (supply.sub(x)) then on the other chain we only add the amount on the balance of receiving address. Currently it is difficult to trace the number of tokens in circulation.

I understand, I thought this is mainly for cross chain transfer. I have no concern with this solution if it is purely to manage address validation in cross chain.
For cross chain transfer, I checked #1885 and I am still not clear how we will handle the token supply amount for a specific token. To transfer, we burn first (supply.sub(x)) then on the other chain we only add the amount on the balance of receiving address. Currently it is difficult to trace the number of tokens in circulation.

The code is not refactored yet. Receiver chain supply would be increased when cross chain received and total supply in TokenInfo won't be changed anytime.

Got it, thank you.

Close as code updated.


test-branch/execute-job/wrong-comment-on-pr/d9af847d-0f21-44fb-8eae-a412bfb002ce

PR for testing ff5518d00e9be96753e5e82a3c9a94268adb0640

comment 1


Laravel 9 issue while try to install dingo

Q
A

Bug?
no

New Feature?
no

Framework
Laravel

Framework version
9

Package version
3.0.8

PHP version
8

Expected Behaviour
Problem 1
- Root composer.json requires api-ecosystem-for-laravel/dingo-api 3.0.8 -> satisfiable by api-ecosystem-for-laravel/dingo-api[v3.0.8].
- api-ecosystem-for-laravel/dingo-api v3.0.8 requires illuminate/routing ^7.0|^8.0 -> found illuminate/routing[v7.0.0, ..., 7.x-dev, v8.0.0, ..., 8.x-dev] but these were not loaded, likely because it conflicts with another require.
Installation failed, reverting ./composer.json and ./composer.lock to their original content.

Hi, this is already addressed in api-ecosystem-for-laravel/dingo-api


Create a test notebook comparing alternate methods for downloading the NWM data

Context
Dealing with NWM data
To understand the directions we are exploring to better serve the National Water Model data out to users, it is helpful to give some context. Importantly, this is an exploration and is experimental. So, we probably are making mistakes. If you notice one, please email us<EMAIL_ADDRESS>The Xarray/Zarr/Kerchunk options
In 2017, Ryan Abernathy (University of Columbia; professor of data science and advocate of open, reproducible science)  started this thread years ago seeking a

path forward to a truly scalable [cloud] data store for xarray

I don't have all the pieces of the intervening development, but in 2022, the conversation was still clearly underway. Significant progress had been made. Critically, the xarray library (with many contributions from Dr. Abernathey) had matured to include significant improvments to multi-file access and remote-data access. Also, developments with cloud-friendly formats from other domains have matured and expanded. While the underlying question seems to be the same, there may be a light (or perhaps several lights) at the end of the tunnel, as can be seen in this discussion stemming from a question about the best (subjective) method for hosting cloud optimized data.
Two excerpts from that discussion seem to capture most of the important gist:
From @martindurant on that thread

In brief: converting to zarr gets you the goodness of cloud-friendly access, and lets you choose the compression and chunking, but duplicates the data, with the associated cost. kerchunk is designed to get you cloud-optimised access to the original data (no duplication, just a small index file), but you are limited to the original chunking/compression. kerchunking will not be possible for all data sets.

And also from @rabernat

I’m leading strongly toward the following recommendation: the primary data store on the cloud should be the original netCDF files + a kerchunk index. From this, it should be really easy to create copies / views / transformations of the data that are optimized for other use cases. For example, you could easily use rechunker to transform [a subset of] the data to support a different access pattern (e.g. timeseries analysis). Or you could put xpublish on top of the dataset to provide on-the-fly computation of derived quantities.

So that seems to suggest options Zarr and Kerchunk.
TODO: find a better recipe-based link for Zarr...
But what about Parquet?
This conversation and this discourse thread explore that option and seem to indicate that for lesser dimensional data, Parquet's columnar format will perform very well.
And Database methods?
There are at least two database-backed methods that we have seen have some success. One drives the backend for the interactive plots shown at water.noaa.gov/map. The other was associated with the evaluation method used to adjudicate the recent USBR/CEATI streamflow rodeo. (See some comments from a participant are here, the news release from the judges, and the topcoder leaderboard.
The Rodeo-backend method was essentially a datacube concept and there are several threads to explore there.
So what is the problem?
Very specifically aligned (but not affiliated) with the purposes of the CIROH, the Azavea group did a project exploring methods for faster access to the NWM datasets in the cloud. A presentation of their work made to the national water modeling group is linked
here (last accessed 2022-08-19)
Basically, they laid things out like this
National Water Model Reanalysis Dataset

A 26 year (1993 - 2018) retrospective simulation that provides historical context to current real-time conditions
Stored as NetCDF files, 4 products for each day, 2 for every hour, 2 every four hours
Each file represents the state of the entire country at that time slice

National Water Model Queries

Most real-world queries are concerned with particular regions (e.g. HUCs) that span a range of time
Currently, this requires downloading a very large amount of data (over the range of time) and then discarding most of it (all except for the particular region), and transform it, which is slow, inefficient, and expensive!
Ideally users should be able to specify the attributes, space, and time ranges, and get just that data, in a readily usable format

Real-time output
While the Azavea group was specifically exploring the retrospective dataset, Similar statements are true about the real-time outputs
Proposal
Goal:
Ultimately the goal is to create a notebook or collection of notebooks to
compare Zarr, Parquet, Kerchunk, Postgres (NWC method), Postgres (Opendatacube method) to access multiple hydrographs from the NWM output.
For each of those, there is a preprocessing step of some kind to prepare the data, then a query step. The query step could take a lot of forms, and the nature of the query (scope, frequency, etc.) might influence what is the best method or details of the method.
Queries to test:
Like before, we need to be able to compare to observed flows, where there are gages – so the NWIS data series is still relevant.
Pick the analysis and assimilation time series (Need to ask Corey which hour; start with tm00) for an arbitrary time-window
Pick a single current forecast trace (e.g., short_range, t00z, f001-f018; or medium_range, t12z, f001-f208)
And plot it.
Pick a single historic forecast trace (see above) and plot it against and observation time series.
Pick a particular historic time-window and for all valid forecasts in the time window plot the envelope of forecasts against the analysis_assim or against observed (if that is available.)
A horizon-based skill metric requires slicing at a particular forecast range across multiple forecasts – like for hurricanes: http://www.hurricanescience.org/science/forecast/models/modelskill/
Assemble the AnA_No_DA time series for a window on a segment and compare to the regular AnA. (I believe the No_DA is only run once per day… The comparison to any other value on that day is still valid, it’s just updated and posted less frequently.)
Updates and actions
No response

@rabernat @martindurant -- you are tagged in this issue because of your background. We welcome your input, but aren't asking for it yet, just trying to provide attribution.

@CoreyKrewson-NOAA @karnesh @arpita0911patel
Put this together to frame discussion.

I am happy to advise on parquet, zarr and kerchunk. However, I should begin by saying that benchmarking is hard, especially for very different formats like this. There will be datasets that suit a columnar format like parquet, and others that are naturally multi-dimensional. Furthermore, the specific encodings versus the content of the data, and the expected access patterns versus the chosen chunking structure. You might very well find that the choice of options available within any particular format make a bigger difference than the choice of format.

I'm unfamiliar with Azavea's original plans for parquet. Those original plans may have changed.
I'd also add that the proposed queries seem to skew toward times series at one (or a few) channel features. This makes sense because it is the most common use-case (or rather, the most commonly requested capability). However, it seems to leave the question of scalability unanswered. In my experience, the most common queries have been:

Long time series of simulated streamflow (ANA_NoDA) for an arbitrary channel reach. Reaches are not necessarily limited to USGS gage locations, since some municipalities and academic inquiries manage their own gauge networks, but know which NWM reaches correspond to their gauges.
Multiple forecast streamflow time series (18 hours to 10 days) over a specified period for an arbitrary channel reach.
Multiple simulated streamflow time series from a particular spatially contiguous region affected by a significant flooding event. Spatial scales can vary from a small town or US county to a third of the continent following wide scale tropical events.
Same as 3, but forecast streamflow across multiple issue times.

Evaluations have tended to involve either a comparison of (a) simulated vs observed, (a) forecast vs observed, or (c) forecast vs simulated.
As @jameshalgren mentioned, there are multiple dimensions to the data including reference time (issue time), valid time, feature ID, model configuration (short range, medium range), and variable (streamflow, velocity, etc).

Some (very) useful snippets, based on kerchunk, posted in the PR comments here:
https://github.com/jmunroe/national-water-model/pull/1

Closing this issue and moving to https://github.com/AlabamaWaterInstitute/data_access_examples/issues/7

@danames -- here are a few thoughts on the queries that a NWM datastore might need to support.


Link to .NET (Services) SDK is wrong

In the "References" section on the left side of this page 9https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-csharp-csharp-twin-getstarted), the link to .NET (Services) goes to the same page as the link to .NET (Devices) - and the documentation is only for the C# Device SDK (everything is in the 'Microsoft.Azure.Devices' Namespace). I expected to find Services documentation in a 'Microsoft.Azure.Services' Namespace.
Where is the documentation for C# Services SDK??

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: f15f149b-4059-d877-f390-156852d88d1d
Version Independent ID: 3f633622-d0b0-be26-f115-9595a6ee4cde
Content: Get started with Azure IoT Hub device twins (.NET/.NET)
Content Source: articles/iot-hub/iot-hub-csharp-csharp-twin-getstarted.md
Service: iot-hub
GitHub Login: @robinsh
Microsoft Alias: robinsh

@TurtleBeach Thanks for reaching out!  I clicked on the links and one takes me to two different pages one is Microsoft.Azure.Devices for (Services) and the second one to Microsoft.Azure.Devices.Client (for client applications running on the IoT device).
In contrast if I click on Java(Service )  it takes me to com.microsoft.azure.sdk.iot.service but comparing the two pages I see the same functionality available.

Hi @TurtleBeach were you able to look into the links more closely to see if they were useful?

@TurtleBeach any updates?

@Alberto-Vega-MSFT : Apologies for the tardy reply. Too many alligators last week. You are correct, they are two different pages. I looked at the titles, saw Devices on both pages, and assumed they were the same. That said, I do find it strange (i.e., confusing) that a link to a "services" SDK (contrasted with a link to a "Devices" SDK) goes to to a page named Microsoft.Azure.Devices.
Jack

Do you have a suggestion on how this can be better?

Hi @TurtleBeach If you have any updates I will assign the issue to the content author so she can evaluate and address your feedback

Since we have not heard back from you we will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.


Batch inference

Hi, serizba :)
a very nice project of yours, finally a nice way to use the Tensorflow API. Using Keras/Tensorflow serving I'm used to do batch interference. I was wondering if it is possible to do multiple inference also with your code.
Is set_data with a shape as a second argument the correct path for me to find the solution?
    int predictions = 2;
    std::vector<int32_t> data_a(predictions * 24);
    std::iota(data_a.begin(), data_a.end(), 1);
   #data_a.resize(24 * predictions);

    std::vector<float> data_b(predictions * 3);
    std::iota(data_b.begin(), data_b.end(), 1);
    #data_b.resize(3* predictions);

    const std::vector<std::int64_t> data_a_shape = { predictions,24};
    const std::vector<std::int64_t> data_b_shape = { predictions, 3 }

	input_a->set_data(data_a, data_a_shape);
	input_b->set_data(data_b, data_b_shape);

	m.run({ input_a, input_b }, output);

Just to add some extra information.
Example network has to input parameters A with 24 parameters and B with 3 parameters.
for prediction=1 it works fine

Hi!
Sorry to answer so late. I am sorry, but I am not sure what do you mean with multiple inference. Could you explain a bit more what do you want to achieve?
Thanks

I proceed to close the issue due to lack of activity


Testing Additions

This PR adds some basic tests with the hypothesis library, as well as a pytest suite for accuracy against an unsymmetric beam in biaxial bending from Peery. Results pass within 0.5% of theoretical (possibly less... that was good enough for me). I also added a new docs page explaining the tests, showing some code examples, and showing the latest results from the pytest log file.

I'm fine with merging this as-is, but I am also willing to keep expanding it, if anyone has other suggestions. I mentioned the other tests on the docs page, but didn't show anything about them or save a report from running them.

I have not had a chance to look at this yet, so my comments below are not directly related to your PR.  They are just some observations of mine.
It is best practice to not store the documentation build in the repository, only the docs source.  This is especially true of binary data (e.g. images) which cannot be diff'd with git.  This is even more true if you re-create the images/build regularly, like with every push to master or every library release.  Other projects have various ways of dealing with this, usually pushing the documentation build artifacts directly to some 3rd party service.
I took a closer look around at the pyvista repository last night.  What they do in their ci file is automatically delete their dummy examples and docs repositories and then push the latest contents to them.  I like this idea as it keeps only source in your main repository.  Further, their documentation website is actually hosted by github pages and is directly served from that pyvista-docs repository.  For anyone unaware, you can use custom domain names to server github pages, see here for some details.  I know that section-properties currently uses readthedocs, but I bet we could do something similar where we push the documentation to a second "docs" repository and then feed that to readthedocs.

@Spectre5 , I think the docs change you're talking about should probably be its own PR. I don't really understand the downsides to storing documentation the way Robbie originally set it up. Feel free to start a discussion on docs for this, in our fancy new discussions area.

Definitely fair point, the discussion would be a better place to make those comments.  I'll put it there

As for the tests themselves, test_Perry.py runs fine and passes for me, but I did get an error on test_hyp.py:
sectionproperties/tests/test_hyp.py:45:
----------------------------------------
d = 0.01, b = 0.01

    @given(
        d=st.floats(min_value=.01, max_value=10),
        b=st.floats(min_value=.01, max_value=10)
    )
    def test_rectangle_mesh_size(d, b):
        '''
        This function verifies that the ValueError is raised when mesh size is
        set too small.
        '''
        with pytest.raises(ValueError):
            xsect = sections.RectangularSection(d, b)
>           mesh = xsect.create_mesh([0.1/251])
E           Failed: DID NOT RAISE <class 'ValueError'>

sectionproperties/tests/test_hyp.py:55: Failed
----------------------------------------
Falsifying example: test_rectangle_mesh_size(
    d=0.01, b=0.01,
)

So in this PR, you remove the entire docs/build directory directory that robbievanleeuwen currently has in the project. Was that intentional?

That was not intentional... I didn't notice that. I don't know why it didn't sync up. I'll try again soon.

I did get an error on test_hyp.py

I need to re-run this on my local branch and check. I may have included this with an error check in the geometry.create_mesh() method. If we don't want to include that error handling, then I need to remove this test. If we do, I need to make sure I included the right file in this PR.
Some stuff for me to check later today after work! Thanks for taking a look!

On more note, although I pointed out that you've removed the entire build directory with the documentation, I actually support doing that (see my first comment on this PR)!  But I figured that you didn't do it intentionally.  If someone wants to get the docs, they can just clone the repo and build it themselves or go to readthedocs to see it, so there is no reason to store the docs build in the repository and it is considered poor practice.  Note that the way readthedocs works, is that they essentially clone your repository and then build the documentation, so it doesn't use the "build" directory that is committed here.
This brings up another point --- @robbievanleeuwen, can you enable the docs to be built on pull requests too, please.  It should be an option in the readthedocs settings.  I've seen other projects do it and it makes a lot of sense so that we can see what the documentation would look like on new pull requests before actually merging it!

I will revisit this, and add some more unit tests as available. Need to upgrade everything to v2, and get acquainted with the changes to the codebase first. Closing this PR. Will make a new one when ready.

@Czarified much appreciated! Sorry there was a bit of a lull at the start of last year, I think this will be a really great addition 🚀


Initiate UDP connection from MAVSDK

The feature
This PR adds a way for MAVSDK to initiate a UDP connection, as follows:

udp://:14540 stays the same, it means that MAVSDK listens for heartbeats on 14540
udp://<IP_ADDRESS>:14540 explicitly states that MAVSDK should be have as a server (like above) and listen for heartbeats on 14540
udp://<IP_ADDRESS>:14540 says that MAVSDK should send heartbeats to <IP_ADDRESS>:14540, and therefore initiate the connection. This is the new feature.

It should not break any workflow, because if somebody used the format udp://<IP_ADDRESS>:14540 before, that was probably not what they wanted to do (the behavior then was filtering the network interface, but nobody ever does that with MAVSDK as far as I know).
The fixes
While implementing integration tests to test the feature, I encountered two bugs:

When using set_udp_remote (in order to initiate the MAVLink connection), we would create a new system with our system_id and comp_id. Which makes no sense: we are initiating the connection to a system that is not us, but we don't know its sysid/compid. So I set it to 0/0 now.
In order to consider a system as discovered, we had this if (!_connected && _uuid_initialized) test. But _uuid_initialized is only ever true when the system represents an autopilot. If MAVSDK itself represents an autopilot, this just fails and systems are never discovered.

Those are small changes, but I must say I have no clue what that will imply more generally. I don't really get why we had the _uuid_initialized condition in the first place, and I am not completely sure how starting a system with 0/0 is handled. I am pretty sure it will fail in some use-cases here (e.g. what happens if you initiate the connection to two remote systems, since both will create a 0/0 system?), but fixing that seems to require more fundamental changes in the discovery mechanism, so I'm hoping this is not needed yet!

Also, we can then add the server mode to the TCP connection.

This is what I currently see for the FtpTest:
cmake --build build -j8 && build/src/integration_tests/integration_tests_runner --gtest_filter="FtpTest.*"
ninja: no work to do.
Note: Google Test filter = FtpTest.*
[==========] Running 4 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 4 tests from FtpTest
[ RUN      ] FtpTest.ListDirectory
[08:32:05|Info ] MAVSDK version: v0.39.0-15-g228ceb76 (mavsdk_impl.cpp:21)
unknown file: Failure
C++ exception with description "vector::_M_range_check: __n (which is 0) >= this->size() (which is 0)" thrown in the test body.
[  FAILED  ] FtpTest.ListDirectory (15 ms)
[ RUN      ] FtpTest.DownloadFile
[08:32:05|Info ] MAVSDK version: v0.39.0-15-g228ceb76 (mavsdk_impl.cpp:21)
[08:32:05|Error] bind error: Address already in use (udp_connection.cpp:86)
../src/integration_tests/ftp.cpp:188: Failure
Expected equality of these values:
  ret
    Which is: Bind error
  ConnectionResult::Success
    Which is: Success
[  FAILED  ] FtpTest.DownloadFile (10 ms)
[ RUN      ] FtpTest.UploadFiles
[08:32:05|Info ] MAVSDK version: v0.39.0-15-g228ceb76 (mavsdk_impl.cpp:21)
[08:32:05|Error] bind error: Address already in use (udp_connection.cpp:86)
../src/integration_tests/ftp.cpp:220: Failure
Expected equality of these values:
  ret
    Which is: Bind error
  ConnectionResult::Success
    Which is: Success
[  FAILED  ] FtpTest.UploadFiles (11 ms)
[ RUN      ] FtpTest.TestServer
[08:32:05|Info ] MAVSDK version: v0.39.0-15-g228ceb76 (mavsdk_impl.cpp:21)
unknown file: Failure
C++ exception with description "vector::_M_range_check: __n (which is 0) >= this->size() (which is 0)" thrown in the test body.
[  FAILED  ] FtpTest.TestServer (0 ms)
[----------] 4 tests from FtpTest (36 ms total)

[----------] Global test environment tear-down
[==========] 4 tests from 1 test suite ran. (36 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 4 tests, listed below:
[  FAILED  ] FtpTest.ListDirectory
[  FAILED  ] FtpTest.DownloadFile
[  FAILED  ] FtpTest.UploadFiles
[  FAILED  ] FtpTest.TestServer

@julianoes @JonasVautherin what are we missing here to bring it in?

@TSC21 fixing the Ftp test.

@TSC21 fixing the Ftp test.

Is that something I can help with? @JonasVautherin


Unable to create static Python 3 on CentOS 6.10 with CMake 3.12.4

I see the following issue with attempting to make a static Python build using CMake 3.12.4:
51%] Building C object CMakeBuild/libpython/CMakeFiles/_freeze_importlib.dir/home/vagrant/scratch/Python-3.6.4/Modules/tkappinit.c.o
[ 51%] Building C object CMakeBuild/libpython/CMakeFiles/_freeze_importlib.dir/home/vagrant/scratch/Python-3.6.4/Modules/_tkinter.c.o
[ 51%] Building C object CMakeBuild/libpython/CMakeFiles/_freeze_importlib.dir/home/vagrant/scratch/Python-3.6.4/Modules/zlibmodule.c.o
[ 51%] Linking C executable _freeze_importlib
/usr/local/bin/ld: CMakeFiles/_freeze_importlib.dir/home/vagrant/scratch/Python-3.6.4/Modules/_cursesmodule.c.o: undefined reference to symbol 'halfdelay'
/usr/local/bin/ld: /lib64/libtinfo.so.5: error adding symbols: DSO missing from command line
collect2: ld returned 1 exit status
make[2]: *** [CMakeBuild/libpython/_freeze_importlib] Error 1
make[1]: *** [CMakeBuild/libpython/CMakeFiles/_freeze_importlib.dir/all] Error 2
make: *** [all] Error 2
It looks like -ltinfo is not properly being added to LIBS.  The version of ncurses that comes with Centos 6.10 requires the tinfo library to be passed, but I cannot find the proper place to define this.
Please see "https://github.com/hishamhm/htop/issues/198" for more information on why tinfo is required with some ncurses compiles.
Recompiling ncurses and putting it into /usr/local/ should help, but cmake keeps finding the original ncurses first and using those libraries.

Thanks for the report :+1:

tinfo is required with some ncurses compiles.

The tinfo library is looked up here:
https://github.com/python-cmake-buildsystem/python-cmake-buildsystem/blob/c3b8d532242efed85fb3cd77b0e39e424b6db475/cmake/ConfigureChecks.cmake#L41
Then the tinfo library is linked in the curses module here:
https://github.com/python-cmake-buildsystem/python-cmake-buildsystem/blob/c3b8d532242efed85fb3cd77b0e39e424b6db475/cmake/extensions/CMakeLists.txt#L575-L588
If you inspect the CMakeCache.txt file found in the build tree, what is the value of associated with the TINFO_LIBRARY cache entry ?

Thank you for your speedy reply!
TINFO_LIBRARY is not in CMakeCache.txt.
[vagrant@localhost python-build]$ grep TINFO CMakeCache.txt
HAVE_SIGWAITINFO:INTERNAL=1
I believe cmake is not properly identifying the need for TINFO, but I'm no CMake expert!  Is there anything else I could paste here to help troubleshoot?
I was thinking it's something funny in 3.12.4's FindCurses.cmake file but I couldn't puzzle out what...

You could try to turn the option USE_SYSTEM_Curses to ON
How could I best reproduce the issue ? Would using the centos:centos6.10 docker image be a good approach ?


Relative links by default in v4?

What do you say to turning on relative links by default in v4? I'm not sure when it isn't awesome to have relative links.

I'm doing some issue-gardening 🌿🌷🌾 and came across this issue.
Since 4.0 is already released I'd suggest closing this issue.


Refactor, add Doc Strings, and add Type Hints for code readability

Refactor, add Doc Strings, and add Type Hints for code readability.


Documentation added

Hello, I did some changes to the high-level API of this brilliant package and propose to merge these into the main project.

reworked the documentation of all the high-level functions
added the possibility to provide a format string when writing the string representation of a float
combined some functions by using Union
reworked receive function to accept non single character delimiters

I am of cause open for discussions.

Hello, and thank you for your contributions! I am traveling at the moment, but will merge after testing 801b9a3.

Let me know if everything is working.


Add dockerhub step to release guide

This is working now, so we can add it to the docs.
✨ RENDERED ✨

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

R: @Abacn


Building pdfjs-dist version 2.16.105 failed

Details
Failed to build<EMAIL_ADDRESS>because the bundle contains modern ES features such as private class fields (# syntax).
Only a minor version was bumped but got a breaking change in our build process.
Error:
ERROR in ./node_modules/pdfjs-dist/build/pdf.js 1413:9
Module parse failed: Unexpected character '#' (1413:9)
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
|
| class PDFDocumentLoadingTask {
>   static #docId = 0;
|
|   constructor() {

Configuration:

PDF.js version: 2.16.105
Webpack: v4.46.0
babel: v7.6.0

Steps to reproduce the problem:

Create a simple webpack 4 + babel 7 configuration (without private class fields support)
install<EMAIL_ADDRESS>import the package and run build
the error should surface

What is the expected behavior? (add screenshot)
Build should work without transpiling the library through babel and without private fields support.
What went wrong? (add screenshot)
Build error from babel with a unexpected character

PDF.js version: 2.16.105

That version is no longer supported, please find the latest releases at https://mozilla.github.io/pdf.js/getting_started/#download

It sounds like you should use a legacy-build instead, since that one is already transpiled with Babel; note that we mentioned the legacy-build in both https://github.com/mozilla/pdf.js/wiki/Frequently-Asked-Questions#faq-support and https://github.com/mozilla/pdfjs-dist/blob/master/README.md


GSSoC: QR Code page UI and Customizations

Description
I would like to code the UI for QR code page and add customizations to the generate qr code feature.
Changes You need to make
Will add feature to change:

color
size
margin
format
etc.. for the qr code to be generated ....

Files to be changed:

client\src\components\pages\qr-codes.tsx
building new components

I have gone through the qr code api to check for parameters that I can add to the qr code...
Checkout the UI design I have built on figma for the QR code page... any further improvements are open for changes...
Screenshots

Checklist
Before submitting your pull request, ensure that you have completed the following tasks:

[x] Gone through the rules and guidelines of the project.
[x] When you are creating PR, your PR should title follows the specified format: [Feature/Enhancement/Bug Fix]: Brief description of the change.
[x] Attached OSSI or GSSoC in the heading of the Issue.
[x] I have checked this issue is not already covered in another issue.

Expected Time to Complete
around 2-4 days
Additional Notes
I will make the page mobile responsive as well and add most api features to customize the QR Code based on user needs....
I will also add a loading animation on qr code image when generate qr is clicked to provide a feedback to the user.
If you like the changes, please assign the Issue to me and add gssoc labels and appropriate level....

Have youi gone through the Current page ? @anisharma07
It looks like

yeah I know ...

Okay. Then I am assigning this to you...


bg-orange-500 - color not rendered but -400 etc are

Describe the bug
Trying to use bg-orange-500 does not render orange but -400 does and others. Seems there is an issues with -500
To Reproduce
Add this
      <Text className="bg-orange-500">hello</Text>

Expected behavior
Should show an orange background like -400 for example, i.e. this works
      <Text className="bg-orange-400">hello</Text>

Additional context
Using the latest version
    "nativewind": "4.0.36",

Same happening here, same version.
In my case adding and removing a class + restarting the metro server made it work. It's inconsistent, it doesn't happen every time.
Also for context, I have a simple fresh install, composed of expo 50.0.13 and expo-router 3.4.8, using the new app directory, and tailwind 3.4.1.

Yes, you are right. I removed the class, restarted it (bun run ios), and then re-added it, and it worked.
I think the -500 was the issue, but I also got it to happen with -600 also.
Once I get the issue, just restarting the server (bun run ios) will not work. I have to remove the class, restart and then add the class back.

I also tried a prebuild --clean, and this did not fix it.
Finally, I was able to get it working by having a View with some value on these and just changing the gap-1 to gap-2 and back again and re-rendered correctly.
    <View className="gap-1">
      <Text className="color-sky-400 bg-red-500">hello</Text>
      <Text className="color-sky-400 bg-red-400">hello</Text>
      <Text className="color-sky-400 bg-red-500">hello</Text>
      <Text className="color-sky-400 bg-red-400">hello</Text>
      <Text className="color-sky-400 bg-red-400">hello</Text>

If it happens again, then I need to change the gap-1 to gap-2 and back again and then all is fine.
So it appears its not related directly to orange-500, from the above snippet, it suddenly stopped rendering anything and restarting the server does not help but updating a style in the view above seems to cause it to render correctly again.
Very strange issue.

Also I tried switching between development build and also Expo Go - same issue.
Tried changing iPhone simulator to another I..e iPhone SE, iPhone 15 / Pro etc.. same issue.
One thing interesting is that I moved to Android and I couldn't get it to happen, it seemed to work all the time.
So it seems (for me at least), its related to iOS only and it seems that it doesn't matter which simulator model I use.

We are closing all pre-v4.1 issues now that v4.1 is released. If the problem is still happening with the latest v4.1, please open a new issue and reference this one in the description.


PM2

Anyone opposed to deprecating forever in favor of PM2?

Let's do it!

:+1:

+1

Just to give us some idea of where we want it, @codydaig depending on how easy it is we may do it in 0.5.0 or a further release.

:+1: definitely

:+1:

Is anyone willing to take this up in a PR for 0.5.0?

Isn't that more of a deployment config rather than part of the framework?
It's not like we have something runner Forever (the only reference to it is Heruku's Procfile).
What did you have in mind with regards to the PM2 as part of 0.5.0?

@lirantal we have a forever task in grunt and gulp for development/production.

@ilanbiala I searched there but I couldn't find it. Where you possibly referring to the generator repo and not the main codebase? Otherwise give me a link.

@ilanbiala I thought we used nodemon for grunt and gulp tasks.

Sorry yeah nodemon, but anyway it's some kind of server reloading task. We're using forever in the Procfile.

So is this kind of enhancement even valid? nodemon is used when you do development so it can monitor if the server needs to be reloaded and such, it's not really used in production.
And we don't use forever, again that's on the procfile which relates to heroku - again not really project standard for mean.js users.
Usually production users setup pm2 and such as part of their servers setup/build/etc.
I don't see how we fit it into the project

@lirantal nodemon, forever, and pm2 should all be usable in production. Setting that up for users would probably be helpful.

IMO the current gruntfile with nodemon starting there with debug mode too is quite  bizarre as normally production nodejs apps won't be running like that.
I do agree that we want to enable users to run nodejs in production mode, and I'm supporting pm2 but there's more to it than just configuring a gruntfile - for example installing pm2.

We can't really account for the many scenarios that exist for deployments. Some users may want to use pm2, and others may want to use something else that suits their needs.
I think the general philosophy with this project has been to create a robust framework, that users can build upon. We shouldn't get too opinionated when things fall outside the scope of the workings of the application itself; which I think this definitely falls outside that scope.

I agree with you, it's what I've been trying to explain. But, I do think there's a place to adopt the good tools, just like we are attempting to slowly replace grunt for gulp, then moving to pm2 is the same movement going on.
And I think it doesn't end in adding pm2 as part of the gruntfile or whatever, it also means some documentation about how to run the app with pm2.

I agree that adding documentation on using pm2 with meanjs would be enough. I don't think it needs to be added as a core feature/tool. Especially, since the only place forever is being used is in the Procfile, which is used in some deployment environments; for instance Heroku (not sure if any others use it).

Closing this one as I don't think we really need to do anything else here.
PM2 documentation just as anything else is beyond the scope of the project IMO.

so what command do i have to give to run the app via pm2  ( in prod version i.e it should minify the stuff before it run )  ?


Deepseek VL?

On the model page for deepseek-vl-7b-chat you link to https://github.com/deepseek-ai/DeepSeek-VL but the repository does not exist?
It makes it impossible to actually try the model. Please fix!

Fixed


Refactor signup responses

What kind of a pull request is this?

QA / Code Review

Code Checklist

[x] The code follows dotkom code style
[x] The code passes the defined tests
[ ] I have added tests for the code I added
[ ] I have provided documentation for the code I added
[x] The code is ready to be merged

(Possible) Breaking Changes

[x] The changes are backwards compatible

Don't think so, but maybe we need more tests?
Description of changes
Trying to clean up signup logic a bit, don't know if it became any clearer :shrug:
Screenshot(s) if appropriate

Haha, I forgot this PR so ended up doing it again https://github.com/dotkom/onlineweb4/pull/2704


Combined Fleet: Second Fleet Condition is not updated on Battle Prediction

Title explains everything.

wip on #461

ah haven't read that one :laughing:

ah this is alright, most of the items being tracked on #461 have their own threads, and this one can be the dedicated one for the combined fleet indexing

sad that the "digital sutra" is already linkified on the main thread (too lazy too open with this slow connection :sob: )

lol I didn't even know what sutra was so I kept quiet :laughing:

@ReiFan49 Sorry for that... it's to keep the thread tidy enough to keep up with the progress.
If we could've make a collapsible spoiler boxes here in github, then we won't have to linkify it...

@LostLogia4 nevermind lol, what i really think is.. "move the sutra to this one" :laughing:
specific thread sounds OK while the general/large thread sounds NG.
ps: it's possible to request it to the github to make a spoiler boxes (iirc, because skimmed most of the MDing part)

"request it to the github"? You mean asking the github devs to introduce collapsible boxes?

after checking around, maybe linkify it for the best solution...

@Javran did multiple revisions done on prediction since then.
Will close this issue now, and probably just re-open in case there are still reports after v17 release


Raptor Page not working

**Devices are not showing in the list but i get the notification on discord that victim is connected

**

Hello and thank you for your work.
I dont find MailConfig.smali in the story file

Hello and thank you for your work.
I dont find MailConfig.smali in the story file

I don't understand, I thought he said the video was old


[signals] handle SIGINT same as SIGTERM

This commit handles SIGINT so that otbr-agent exits gracefully under Ctrl+C.

I think we intentionally not handles SIGINT so that it can better support gdb. We use Ctrl-C to interrupt the program and start debugging.

Thanks for information. That make sense to me.
Should be fine to leave SIGINT unhandled anyway.
Closeing this PR.


Include option to run brownfield optimisation

Currently, there is limited possibility to run the model with existing electricity generation and transmission infrastructure. Typically, only hydropower and some Nuclear infrastructure is considered in the hybrid-greenfield approach.
We have been discussing adding the feature of running a brownfield optimisation where:

Existing VREs (wind and solar) are included.
Existing transmission infrastructure exists and can also be expanded (which is different from running with  fx_trans == 'ON')

There are some methodological choices one needs decide upon to make this happen.

For the first point, one initial step is to find data on existing VREs. A few tools/databases exist:

powerplantmatching
Global Energy Monitor
Open Power System Data
DTU's wind power database (not publicly available at the moment)

From some initial analysis, powerplantmatching does not seem capable of cross-referencing large datasets of solar and wind power plants (see this issue and section 2.4 of Gotzens et al. (2019))

Since especially wind and solar units are comparatively small in terms of unit-wise capacity (ranging from very-low kW to low MW scale), but huge in terms of deployed numbers (e.g. more than 1.7 million single units only in Germany [24]), they would massively impede the matching process. Moreover, since most of the single solar panels and wind turbines do not have a specific name (except larger wind and solar parks), there is no data in the name column and, consequently, no input for the string comparator available. Therefore, all wind and solar units are being filtered as part of the data mending, enabling us to keep the entire process computationally manageable.

However, for a further usage of the derived power plant data in modelling approaches, PPM is able to concatenate given wind and solar units from the OPSD renewable data package [24] to the final dataset at the end of the matching process.

From an initial look, GEM, which has full coverage of all highRES countries does not look too bad, with some exceptions. OPSD data is not available for all countries, but could act as a complementary data source for those countries where it is available. On the other hand, the DTU database seems to have similar levels of installed capacity as OPSD, but covering all countries, although it falls short for some countries, such as Sweden, Norway and Finland.

Note that the DTU database is only in a preliminary version, and that this can be analysed fully once it's ready (early July 2024)

An alternative approach worth looking into is to distribute the country-level data (which should be available with good confidence) based on some heuristic.
PyPSA-Eur says the following:

Existing wind and solar capacities are retrieved from IRENA annual statistics and distributed among the nodes in a country proportional to capacity factor. (This will be updated to include capacity distributions closer to reality.)

Some testing of how the heuristic compares to real distributions should be explored.

An alternative approach worth looking into is to distribute the country-level data (which should be available with good confidence) based on some heuristic.
PyPSA-Eur says the following:

Existing wind and solar capacities are retrieved from IRENA annual statistics and distributed among the nodes in a country proportional to capacity factor. (This will be updated to include capacity distributions closer to reality.)

Some testing of how the heuristic compares to real distributions should be explored.

I had a look at the heuristic, and it looks "fine".
Some considerations:

As Oskar said, the heuristic distributes the country-level capacities into nodes in a country based on capacity factors. I followed the heuristic defined in Pypsa, and I defined it for our case:

I considered the same number of nodes as regions in a country (NUTS2) (I also did for NUTS3, but it is similar)
I defined the annual mean capacity factor per region (I selected 2020), and then I calculated the total capacity factor per country  (I summed the capacity factor per region in each country)
I divided the annual mean capacity factor per region of a country by the corresponding total capacity factor per country.
Finally, I have a factor (fraction) per region in a country

The heuristic was only defined for the nuts2 regions in our cutout (e.g. Iceland wasn't considered because the cutout doesn't contain that part)

I assumed Iceland with a factor of 1 (only one region in NUTS2)

The plot only focused on the onshore area.

I couldn't apply the heuristic to the offshore area because I didn't know how many nodes/regions I should consider.

The plot of the existing capacities applying the heuristic. Left side: heuristic, and right side: DTU database

And here is the difference between both. Red means DTU is higher, and the blue heuristic is higher (white means no difference)

The heuristic usually overestimates the installed capacities in each region (more blue)
It tries to prioritize the region with better capacity factors, but it assigns more capacity to regions where it is not possible to install (because of topography)

It could be improved by excluding some grid cells from the cutout (maybe?).


Add Overlay Quota support

This patch vendors in the latest containers/storage which include quota support for overlay storage on XFS file systems.
Also include updates to the crio.conf.5 man page to describe how to configure these quota options.

@mrunalp @nalind @rhvgoyal @runcom PTAL
And see if you can get quota setup in your environment.
We need to add tests for this, but the tests can only run on a specific storage setup, so we can not run them in general.

@cevich PTAL, also to see if you have ideas of how to do a test.

/test all

Failures in test:
not ok 6 pids limit
# (from function `start_crio' in file ./helpers.bash, line 224,
#  in test file ./cgroups.bats, line 13)
#   `PIDS_LIMIT=1234 start_crio' failed
# 7:pids:/
# time="2017-09-26T18:06:42-04:00" level=error msg="error opening storage: overlay: the backing xfs filesystem is formatted without d_type support, which leads to incorrect behavior. Reformat the filesystem with ftype=1 to enable d_type support. Running without d_type is not supported."
# rm: cannot remove ‘/tmp/tmp.WKtccoA3m4/crio/overlay2’: Device or resource busy
not ok 7 ctr not found correct error message
# (from function `start_crio' in file ./helpers.bash, line 224,
#  in test file ./ctr.bats, line 10)
#   `start_crio' failed
# time="2017-09-26T18:06:43-04:00" level=error msg="error opening storage: overlay: the backing xfs filesystem is formatted without d_type support, which leads to incorrect behavior. Reformat the filesystem with ftype=1 to enable d_type support. Running without d_type is not supported."
# rm: cannot remove ‘/tmp/tmp.bp3NtaoHd7/crio/overlay2’: Device or resource busy
not ok 8 ctr termination reason Completed

@mrunalp So the question is whether we want to force users to run with d_type?  All freshly installed RHEL7.4 systems should have this enabled by default.  (I believe, @rhvgoyal?)  So can we up the tests machines to be RHEL7.4?  Or we can turn this error to a warning, but quota will not work on these systems.

@skuznets is looking into this

/test all

/test integration_fedora

/test all

/test ami

/test ami

/test e2e

/test ami

/test ami
/test e2e

/test ami
/test e2e

/test ami

/test ami_rhel

/test ami_rhel

bad code in the ami packaging step in the playbook I used, updated
/test ami

okay, so looked at https://ci.openshift.redhat.com/jenkins/job/test_pull_request_crio_ami_rhel/22/s3/ The crio.service failed to come up. However, this is expected because we would need the ami to boot up with 954 for the kernel cmdline flags to take effect. So, I think we will have to first get in 954 (drop that commit from here) and then this PR would be able to boot off ami created with 954 and pass the tests.
-- Logs begin at Fri 2017-09-15 19:46:50 EDT, end at Thu 2017-09-28 20:06:13 EDT. --
Sep 28 20:06:02 ip-172-18-8-38.ec2.internal systemd[1]: Starting Open Container Initiative Daemon...
Sep 28 20:06:03 ip-172-18-8-38.ec2.internal crio[21492]: time="2017-09-28 20:06:03.055229394-04:00" level=warning msg="Using pre-4.0.0 kernel for overlay, mount failures may require kernel update"
Sep 28 20:06:03 ip-172-18-8-38.ec2.internal crio[21492]: time="2017-09-28 20:06:03.057001932-04:00" level=fatal msg="overlay: the backing xfs filesystem is formatted without d_type support, which leads to incorrect behavior. Reformat the filesystem with ftype=1 to enable d_type support. Running without d_type is not supported."
Sep 28 20:06:03 ip-172-18-8-38.ec2.internal systemd[1]: crio.service: main process exited, code=exited, status=1/FAILURE
Sep 28 20:06:03 ip-172-18-8-38.ec2.internal systemd[1]: Failed to start Open Container Initiative Daemon.
Sep 28 20:06:03 ip-172-18-8-38.ec2.internal systemd[1]: Unit crio.service entered failed state.
Sep 28 20:06:03 ip-172-18-8-38.ec2.internal systemd[1]: crio.service failed.

@mrunalp removed your patch and repushed.

@rhatdan thanks! I am trying to verify that in a AWS VM manually.

/test ami

/test ami

/test e2e_fedora

/test all

/test e2e_rhel

@mrunalp Seems like e2e_rhel is hung,  I think this is ready to go.  Should we just merge?

In the case CRI-O doesn't come up or something else critical fails .. e2e takes 4+ hours for all tests to time out and fail

/test e2e_rhel

okay, the e2e rhel in the last run only had the guestbook failure so this is good. @rhatdan Did we ge the other tests to pass after you changed the error to warning?

@mrunalp Yes.  I believe so.

Since this now passes the tests other then the known failure.  I am going to merge.

@mrunalp Since this is merged, are we ready for RC3?

Looks like this broke the build. See #963.


Final Greet check

Congratulations, @rahulgavhar! 🎉 Thank you for creating your issue for "Your Repo Name". Your contribution is greatly appreciated and we look forward to working with you to resolve the issue. Keep up the great work! Should you have any queries or require guidance, do not hesitate to ask.


Is it possible to talk to the Windows CSP via PKCS11 too?

Is it possible to generically use a PKCS#11 library like yours and switch between using keys from an underlying HSM and keys from the Windows CSP? Do you know of a library doing so?
Best regards,
D.R.

There's quite a few options actually:

https://github.com/PeculiarVentures/pvpkcs11
https://github.com/risacher/p11-capi
c:\Program Files\Mozilla Firefox\osclientcerts.dll

Oh nice, thanks for the quick reply! I'll give them a try.


Add a section for definitions

I my topic tree I always have a variable called "serial":
topics:
  event.teltonika.{serial}.mobile.bytes_received:
    parameters:
      - $ref: "#/x-definitions/serial_parameter"
    subscribe:
      [...]

x-definitions:
  serial_parameter:
          name: "serial"
          description: "Serial number of the router. Serial number can be taken from the device or via the command: `uci get hwinfo.hwinfo.serial`."
          schema:
            type: "string"
            title: "Serial number"
            description: "Serial number, usually 8 digests long."
            example: "01234567"

I want to use the parameters defintion of serial more often, so I needed to place it somewhere else. I found out, that I have no chance to put it under any subtype of the components section. So I used a self defined x-definitions.
Now my question is: Am I missing something, is there a better way to do this task? If not, what about adding a place for such definitions?

In AsyncAPI 1.2.0 you can use components/parameters

Yes, that is exactly what I need. But the online editor does not support the API 1.2.0 yet, right?

No, the editor is not yet updated. I need to find some time to update all the tooling yet.
The editor is here, so if you need it soon you might want to hack it yourself. I'll take care of the deployment.