VSCode test discovery fails while looking for non-existent file or directory

Environment data

VS Code version: 1.52.1

Version: 1.52.1

Commit: ea3859d4ba2f3e577a159bc91e3074c5d85c0523

Date: 2020-12-16T16:30:02.420Z

Electron: 9.3.5

Chrome: 83.0.4103.122

Node.js: 12.14.1

V8: <IP_ADDRESS>-electron.0

OS: Darwin x64 20.3.0

Extension version (available under the Extensions sidebar): ms-python.python:v2021.1.502429796

OS and version: macos 11.2

Python version (& distribution if applicable, e.g. Anaconda): 3.9.0

Type of virtual environment used (N/A | venv | virtualenv | conda | ...): virtualenv

Relevant/affected Python packages and their versions: fastapi, pydantic, asyncio, pytest, httpx, uvicorn

Relevant/affected Python-related VS Code extensions and their versions: none

Value of the python.languageServer setting: Jedi

Expected behaviour
I expect that test discovery will not look for a non-existent file/module/package that is neither referenced nor specified in the code respository.
Actual behaviour
Note: unit tests continue to succeed from the command line from the same virtualenv from which VSCode was started (code .).
I changed the structure of the project, resulting in the removal of a package with the name of 'base' and moving some things around.
Prior to refactoring, all unit tests succeeded. After refactoring, test discovery in VSCode complained that 'base' was missing.
Also, when the testing icon is selected in the feature panel (view, search, source code, docker, test, etc.), the associated panel is blank where before it showed the unit test tree.
In order, here are the attempts that failed to fix the missing [file or directory] issue:

uninstall the python extension in the VSCode app, then reinstall the extension
uninstall the python extension, remove the python extension directory from ~/.vscode, and reinstall the python extension
remove the VSCode application from the Mac Applications folder, remove files and directories from a list I found online (see below), download VSCode again, and reinstall the python extension.

rm -fr ~/Library/Preferences/com.microsoft.VSCode.helper.plist
rm -fr ~/Library/Preferences/com.microsoft.VSCode.plist
rm -fr ~/Library/Caches/com.microsoft.VSCode
rm -fr ~/Library/Caches/com.microsoft.VSCode.ShipIt/
rm -fr ~/Library/Application\ Support/Code/
rm -fr ~/Library/Saved\ Application\ State/com.microsoft.VSCode.savedState/
rm -fr ~/.vscode/

After all of these attempts, test discovery still reported ERROR: file or directory not found: base.
Even after removing VSCode and the files located in the bowels of ~/Library, it seems that the name 'base' is still cached somewhere on the computer. The output line,
python /Users/electronhead/.vscode/extensions/ms-python.python-2021.1.502429796/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir /Users/electronhead/dev/pyrambium -s --cache-clear base
references base. Is this reference unrelated to the issue submitted?
Steps to reproduce:
started off with something like:

pyrambium/
  base/
        service/
              foo.py
              bar.py
        test/
              test_foo.py
              test_bar.py
              fixtures.py
  app/
        main.py
        routers/
              router1.py
              router2.py
  jupyter/
        notebook1.ipynb
        notebook2.ipynb

ending with something like:
pyrambium/
  src/
        service/
              foo.py
              bar.py
        test/
              test_foo.py
              test_bar.py
              fixtures.py
        app/
              main.py
        routers/
              router1.py
              router2.py
   jupyter/
        notebook1.ipynb
        notebook2.ipynb

Logs
python /Users/electronhead/.vscode/extensions/ms-python.python-2021.1.502429796/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir /Users/electronhead/dev/pyrambium -s --cache-clear base
Test Discovery failed:
Error: ============================= test session starts ==============================
platform darwin -- Python 3.9.0, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /Users/electronhead/dev/pyrambium
plugins: asyncio-0.14.0
collected 0 items

========================= no tests collected in 0.01s ==========================
ERROR: file or directory not found: base

Traceback (most recent call last):
  File "/Users/electronhead/.vscode/extensions/ms-python.python-2021.1.502429796/pythonFiles/testing_tools/run_adapter.py", line 22, in <module>
    main(tool, cmd, subargs, toolargs)
  File "/Users/electronhead/.vscode/extensions/ms-python.python-2021.1.502429796/pythonFiles/testing_tools/adapter/__main__.py", line 100, in main
    parents, result = run(toolargs, **subargs)
  File "/Users/electronhead/.vscode/extensions/ms-python.python-2021.1.502429796/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 44, in discover
    raise Exception("pytest discovery failed (exit code {})".format(ec))
Exception: pytest discovery failed (exit code 4)

Request
I would like to know how to completely eradicate VSCode from the mac so that a reinstalled VSCode will have no memory of 'base'. A reference to 'base' must be cached somewhere. The above list of 'rm' operations was not enough.

@electronhead, thanks for letting us know about this.  The Python extension doesn't do anything special to preserve this sort of testing-related data.  Would you mind providing the content of your settings.json file?  Thanks!

Hi,
Thanks for your prompt response and solution !!!
settings.json: Bingo!
{
"python.testing.pytestArgs": [
"base"
],
"python.testing.unittestEnabled": false,
"python.testing.nosetestsEnabled": false,
"python.testing.pytestEnabled": true
}
I don’t know how the reference got there, but its absence brought unit testing back to life in VSCode.
VSCode is awesome. I’m in my first three weeks of use, and it has been incredibly useful in my first attempt at a github-based Python project.
— Gary

On Feb 4, 2021, at 1:27 PM, Eric Snow<EMAIL_ADDRESS>wrote:
@electronhead https://github.com/electronhead, thanks for letting us know about this. The Python extension doesn't do anything special to preserve this sort of testing-related data. Would you mind providing the content of your settings.json file? Thanks!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/microsoft/vscode-python/issues/15314#issuecomment-773614427, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCCUGOEO6JLMSGW2ENK4Q3S5MGLVANCNFSM4XBZJIAA.

I'm seeing the same issue. When I check the python.testing.pytestArgs setting, it's blank.
python c:\Users\a6b8zzz\.vscode\extensions\ms-python.python-2021.1.502429796\pythonFiles\testing_tools\run_adapter.py discover pytest -- --rootdir d:\ff_orchestrator -s --cache-clear
Test Discovery failed:
Error: ============================= test session starts =============================
platform win32 -- Python 3.9.1, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: d:\ff_orchestrator
collected 0 items / 1 error

=================================== ERRORS ====================================
_________________ ERROR collecting test/ff_installer_test.py __________________
ImportError while importing test module 'd:\ff_orchestrator\test\ff_installer_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
C:\Program Files\Python39\lib\importlib\__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
d:\ff_orchestrator\tests\ff_installer_test.py:1: in <module>
    ???
E   ModuleNotFoundError: No module named 'ff_installer'
=========================== short test summary info ===========================
ERROR test/ff_installer_test.py
!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
==================== no tests collected, 1 error in 0.18s =====================

Traceback (most recent call last):
  File "c:\Users\a6b8zzz\.vscode\extensions\ms-python.python-2021.1.502429796\pythonFiles\testing_tools\run_adapter.py", line 22, in <module>
    main(tool, cmd, subargs, toolargs)
  File "c:\Users\a6b8zzz\.vscode\extensions\ms-python.python-2021.1.502429796\pythonFiles\testing_tools\adapter\__main__.py", line 100, in main
    parents, result = run(toolargs, **subargs)
  File "c:\Users\a6b8zzz\.vscode\extensions\ms-python.python-2021.1.502429796\pythonFiles\testing_tools\adapter\pytest\_discovery.py", line 44, in discover
    raise Exception("pytest discovery failed (exit code {})".format(ec))
Exception: pytest discovery failed (exit code 2)

I noticed it had cached the tests folder name. I had renamed it to test but I still see reference to previous folder name. Also the problem started happening after I change the name of my root folder.
C:\Program Files\Python39\lib\importlib_init_.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
d:\ff_orchestrator\tests\ff_installer_test.py:1: in
???

I was able to get it to work again by 1) renaming the root folder back to what it was when I enabled pytest using vscode settings and 2) renaming the tests subfolder back and  3) creating a init.py file in the tests subfolder. Seems very fragile.

@shah-prabal, if this continues to be a problem for you, please open a new issue.

@electronhead. that's great to hear.  Please let us know if you run into any other problems.

My project structure evolved again, thoroughly confusing vscode. So I downloaded the repository as a zip file into a new working directory, created a new repository, and pushed the downloaded files into the new repository.
Vscode really likes to hang on to stuff, so I basically found a way to start fresh using a new repository. Thankfully, at this stage, I’m the only coder, but this will change.
I understand that managing restructuring for multiple operating systems could be a challenge.
If I had time to work through the scenario in detail, I could maybe open an issue. I’ll go with how things work for now.
Thanks.
-- Gary

On Feb 8, 2021, at 13:38, Eric Snow<EMAIL_ADDRESS>wrote:

@shah-prabal, if this continues to be a problem for you, please open a new issue.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

If I had time to work through the scenario in detail, I could maybe open an issue. I’ll go with how things work for now.

Sounds good.


10.1164/rccm.201911-2199OC

Marie Deprez et al.
A Single-Cell Atlas of the Human Healthy Airways
https://www.atsjournals.org/doi/10.1164/rccm.201911-2199OC
GEO:
https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE143868
EGA:
https://ega-archive.org/studies/EGAS00001004082

I would like to work on this dataset for the workshop😊


Unhelpful error messages given when Pipfile.lock contains double requirements

I am using the Pipfile included below.  Running pipenv install with no Pipfile.lock succeeds, but the next time I run it, after Pipfile.lock has been generated, I get the following error:
Pipfile.lock out of date, updating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Installing dependencies from Pipfile.lock...
An error occured while installing!
Double requirement given: python-dateutil==2.2 (from -r /tmp/tmp2za0_8kx-requirements.txt (line 16)) (already in python_dateutil==2.6.0 (from -r /tmp/tmp2za0_8kx-requirements.txt (line 10)), name='python-dateutil')

I assume this is because two packages in my Pipfile are requiring different versions of python-dateutil.  I'm not sure if this is something that pipenv should handle, but in any case, the error message generated doesn't give me any information that I can use to solve the problem.
Is this something pipenv could handle better?
Pipfile:
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

[packages]
appdirs = "*"
psutil = "==2.1.1"
six = "*"
redis = "*"
pycrypto = "==2.6"
cachetools = "==1.0.0"
Cython = "*"
packaging = "*"
pep8 = "==1.6.2"
boto3 = "*"
botocore = "*"
freezegun = "==0.3.4"
python-dateutil = "==2.2"
boto = "<=3"
python-gflags = "==2.0"
scikit_image = "*"
pylint = "==1.4.4"
ndg_httpsclient = ">=0.4"
SPARQLWrapper = "==1.6.4"
numpy = "==1.8.1"
scipy = "*"
httpretty = "==0.8.10"
requests = "==2.6.0"
packaging = "*"

It appears this was actually caused by the Pipfile itself requiring python-dateutils.  Closing.


feat: handle list value for add_dict_to_node function

fix this issue:https://github.com/neoave/mrack/issues/291 :)

Hi, could you also add  a unit test that verifies that the use case in #291 works? That the resulting xml is as expected and that the beaker client accepts it.

Updated,Sorry for the late reply,I was on National holiday again,it's
outing season here:)
On Tue, Apr 30, 2024 at 10:24 PM Petr Vobornik @.***>
wrote:

Hi, could you also add a unit test that verifies that the use case in #291
https://github.com/neoave/mrack/issues/291 works? That the resulting
xml is as expected and that the beaker client accepts it.
—
Reply to this email directly, view it on GitHub
https://github.com/neoave/mrack/pull/292#issuecomment-2085468710, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AKFR23EVX44P4AWRUR47RS3Y76SQZAVCNFSM6AAAAABG55T44SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBVGQ3DQNZRGA
.
You are receiving this because you authored the thread.Message ID:
@.***>

Sure,updated:)
On Tue, May 7, 2024 at 7:09 PM Petr Vobornik @.***>
wrote:

@.**** commented on this pull request.
the code LGTM, but please fix the lint issues highlighted by CI:

title of the test commit
flake8 check
https://dev.azure.com/neoave/mrack/_build/results?buildId=1153&view=logs&jobId=e4354d11-29aa-59c2-ce04-0d9aced3ea05&j=e4354d11-29aa-59c2-ce04-0d9aced3ea05&t=37a28340-749f-5255-d3d8-b83a7b4239ca

—
Reply to this email directly, view it on GitHub
https://github.com/neoave/mrack/pull/292#pullrequestreview-2042880909,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AKFR23B47KZXIFDRHDXMOETZBCY7VAVCNFSM6AAAAABG55T44SVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDANBSHA4DAOJQHE
.
You are receiving this because you authored the thread.Message ID:
@.***>

Thanks:)
On Tue, May 14, 2024 at 4:11 PM Petr Vobornik @.***>
wrote:

Merged #292 https://github.com/neoave/mrack/pull/292 into main.
—
Reply to this email directly, view it on GitHub
https://github.com/neoave/mrack/pull/292#event-12797745529, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AKFR23GATN2YK433N5ERFPTZCHAEVAVCNFSM6AAAAABG55T44SVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJSG44TONZUGU2TEOI
.
You are receiving this because you authored the thread.Message ID:
@.***>


fix thread pool deadlock for now

 scopedvec Test terminiert manchmal nicht

Pull Request Test Coverage Report for Build 961415668

5 of 5 (100.0%)  changed or added relevant lines in 1 file are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage decreased (-0.02%) to 80.364%

Files with Coverage Reduction
New Missed Lines
%

operators/src/util/input/string_or_number.rs
1
80.0%

Totals

Change from base Build 957405564:
-0.02%

Covered Lines:
17288

Relevant Lines:
21512

💛  - Coveralls


Add MacOS neovim app activation

This might be a looser matcher than we want, but works for me. Worth reviewing to ensure it's not too loose. In particular the other matchers seem to require VIM MODE, but that didn't appear in my window title 🤔

I guess we need to unify this:

can we just match on app.exe? that is the normal way in community so ideal if possible
if not we can rely on the win.title containing nvim
if not we can rely on the win.title containing VIM MODE (for instance on Windows if Talon detects it as app.exe: conhost.exe)

Update: we definitely can't just rely on the window title containing nvim, as I was doing before, because that broke my vscode when I opened a file that contained nvim in its name 😅. I updated the matcher to look for VIM MODE as well, like we're doing for Linux, but that does mean that installing https://github.com/hands-free-vim/talon.nvim becomes a hard requirement

deprecated by https://github.com/hands-free-vim/neovim-talon/pull/24


Update CI to Node 16

receive-pr-and-deploy
Node.js 12 actions are deprecated. For more information see: https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/. Please update the following actions to use Node.js 16: actions/checkout, actions/github-script, actions/checkout

receive-pr-and-deploy Node.js 12 actions are deprecated. For more information see: https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/. Please update the following actions to use Node.js 16: actions/checkout, actions/github-script, actions/checkout

I think we should upgrade to 18.x because 16.x went into maintenance.
https://github.com/nodejs/release#release-schedule

build-and-test
Node.js 12 actions are deprecated. For more information see: https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/. Please update the following actions to use Node.js 16: actions/checkout@v2, actions/setup-node@v2, actions/upload-artifact@v2

Install and Build Workflow: .github#L1
ubuntu-latest workflows will use ubuntu-22.04 soon. For more details, see https://github.com/actions/runner-images/issues/6399

I found the problem will take care of that

done https://github.com/ubiquity/ubiquity-dollar/actions/runs/3679952590
https://github.com/ubiquity/ubiquity-dollar/actions/runs/3679951103
there is only a warning for new latest ubuntu version that will be v22 and we should be fine as it impacts ruby, php and python version.

on a precautionary basis I tried it with ubuntu 22.04 and it works great :+1:  https://github.com/ubiquity/ubiquity-dollar/actions/runs/3680087591 https://github.com/ubiquity/ubiquity-dollar/actions/runs/3680087453

Lets just replace it with ubuntu-22.04 and merge your PR.

@zgorizzo69 https://etherscan.io/tx/0x36a4c9a4cd8debc539e2bf2a229149612fd284d6861374c56dff40172454db9e

I just merged #311


normal cdf function gives probability of 0.0 for very low probabilities

I have noticed that for very low probability the normal cdf function gives a probability of 0.0 when actual probability are in the order of 10^(-30)

I can't reproduce this.  If I evaluate the CDF of a normal distribution with mean 0 and standard deviation 1 at x = -35.0, I get 1.124910706417e-268.
Can you provide code that gives a minimal working example that shows the unexpected result?

let z = 8.78491962525629;
let n = Normal::new(0.0, 1.0).unwrap();
let p_g = 1.0 - n.cdf(z);
This gives p_g = 0.0
In comparison, when I use the r_stats crate, documentation here I get:
let p_g2 = r_stats::normal_cdf(z, 0.0, 1.0, false, false);
I get
p_g2 = 0.0000000000000000007823818993898402

The expression 1 - cdf(x) is often called the complementary CDF or the survival function.  For values of x that are sufficiently large, actually computing it with the expression 1 - cdf(x) is not a good method, as you have observed.  The problem is that with x that large, cdf(x) is numerically 1.0.  So you end up with 1 - cdf(x) = 1 - 1 = 0.
Fortunately the sf method was added to the continuous distributions in version 0.16 of statrs, so if you are using the latest version, you can compute the desired quantity with the new method.
If you can't upgrade statrs, a simple fix is to use the symmetry of the normal distribution: sf(μ + x) = cdf(μ - x).


SyntaxError when executing python -c

Hello!
I have a weird problem here (not sure it's asdf-python issue though):
$ /usr/bin/python --version
$ ~/.asdf/shims/python --version

$ /usr/bin/python -c 'import sys; print("python")'
python

$ ~/.asdf/shims/python -c 'import sys; print("python")'
  File "<string>", line 1
    import
         ^
SyntaxError: invalid syntax

I would simply ignore this, but anaconda-mode in spacemacs is using -c to evaluate code, and throws an error every time I type something in a python file in spacemacs.

Hi, I will take a look at it!

This should now be fixed, could you try to run
asdf plugin-update python

and give it another try?

@tuvistavie thanks! will give it a try now.

@tuvistavie thank you, it's working after reinstalling python through asdf


Prevent crash when selecting NULL-valued attributes

There were multiple situations in which Lexbor would crash when a selected attribute had either no value or an empty value was given to the match function.
This PR prevents the following crashes:

Matching of class names or attributes with lxb_dom_elements_by_class_name() or lxb_dom_elements_by_attr() when the document contains elements with NULL-valued attributes
CSS class, ID, or attribute selectors when the document contains elements with NULL-valued attributes
lxb_dom_elements_by_class_name() and lxb_dom_elements_by_attr() when given an empty (non-NULL, but zero-length) or NULL value for comparison

If you pass an empty/NULL value to lxb_dom_elements_by_class_name(), it won't match anything, but at least it doesn't crash anymore. lxb_dom_elements_by_attr() will match attributes with either empty or NULL values.
Fixes #145
Tests were added.

For the time being, I created a custom manylinux Resiliparse build image from my own branch, which also includes the other unreleased fixes, so I can finally get rid of a few ugly workarounds and limitations.
https://github.com/chatnoir-eu/chatnoir-resiliparse/blob/develop/Dockerfile

I processed, extracted, and indexed 24 million documents within the last hour, so far no crashes! :+1:

@phoerious
Thanks for the PR!
I started watching it. Tomorrow I will continue.
I need to check the behavior against the specification.

Changes made.

@phoerious
Thanks for the patch, especially thanks for the added tests!


Expose scale, rotate and translate flag

Unintended merging to original upstream :)

Unintended merging to original upstream :)


Add OnError

Would be nice to see OnError<TData> as well
And consider to have OnError(Action<ICakeContext>) and OnError<TData>(Action<TData,ICakeContext>)

@devlead Could you please elaborate on this one?
As far as I understand, when the requested method overload is implemented, the sample above will transform to this:
public class BuildData
{
    public bool HasError { get; set; }
}

Setup(context=>new BuildData());

Task("Boom")
    .Does(()=>throw new Exception("Boom"))
    .OnError<BuildData>((exception, data) => {
            Error(exception);
            data.HasError = true;
        });

Teardown<BuildData>((teardownContext, data) =>
{
    if (data.HasError)
    {
        Error("There was errors.");
    }
    else
    {
        Information("There was no errors");
    }
});

RunTarget("Boom");

It means there must be a method OnError<TData>(this CakeTaskBuilder builder, Action<Exception, TData> errorHandler), which should end up setting ErrorHandler property of the CakeTask to some value of type Action<Exception>. But where do I get the data from? There's no global Context like in the sample above, nor there's a parameter of type ICakeContext.
I investigated the Does functionality which looks similar and has the necessary overload. However, it also has ICakeContext parameter and gets TData from there when it's needed.
I might be missing something obvious, but it doesn't match in my head right now... I would appreciate some hint. Thanks!

There's a global ICakeContext Context property available in non static methods and delegates in scripts.
So Context.Data.Get<BuildData>(); will work as long as your not in a class or static method/delegate.
But to sort a OnError<BuildData> it would require a little bit more work internally.

But to sort a OnError<BuildData> it would require a little bit more work internally.

Yep, that's clear. I'd like to take over this challenge and would appreciate some hint regarding what that work internally could be. Is there a similar logic in the project I can leverage to understand what should be changed?

@devlead I think I got your point. I would appreciate your review.


[Feature Request] Avatar Change Log

See the title. It'd be useful for when people change their avatar to NSFW and we can ban them for it.

We need to create a general log specficiation and then work on it.
Logs are difficult to perfectly create, as we don't want to save any user data but Discord is making it hard for us not to.

Why would we have to have a seperate log for checking avatars? Why not just check if the avatar is NSFW and do a punishment on it?

you could but that costs money with APIs yes

you could but that costs money with APIs yes

What? Additionally, there are bots that provide logging. I am even opposed to this one having it because its hard to innovate on logging

It's not hard to detect user avatar changes (on USER_UPDATE event), but it'll be hard to track down NSFW avatars without paying a fee to a service.


svg transparency support?

does svg renderer support transparency ?

You can use rgba values (https://www.w3schools.com/cssref/tryit.php?filename=trycss_color_hexa) in the darkColorHex and lightColorHex parameters of the SvgQrCode constructor. But please be aware that not all applications can handle SVGs with transparency defined this way. So ensure upfront that your target application is able to render transparency.

You can use rgba values (https://www.w3schools.com/cssref/tryit.php?filename=trycss_color_hexa) in the darkColorHex and lightColorHex parameters of the SvgQrCode constructor. But please be aware that not all applications can handle SVGs with transparency defined this way. So ensure upfront that your target application is able to render transparency.

Thanks i actually tried it before but ended up in only black rectangle. I must have made something wrong because now it works. I dont know why.
great library , keep up the good work 💪🏻


[P4_Symbolic] Add packet_synthesizer.

Keyword Check:
~/sonic-buildimage/src/sonic-p4rt/sonic-pins$ ~/tools/keyword_checks.sh .
Keyword check Passed.
Build Result:
/sonic/src/sonic-p4rt/sonic-pins$ bazel build $BAZEL_BUILD_OPTS ...
INFO: Analyzed 568 targets (153 packages loaded, 21484 targets configured).
INFO: Found 568 targets...
INFO: Elapsed time: 88.491s, Critical Path: 23.14s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
Test Result:
/sonic/src/sonic-p4rt/sonic-pins$ bazel test $BAZEL_BUILD_OPTS --cache_test_results=no ...
INFO: Analyzed 568 targets (0 packages loaded, 340 targets configured).
INFO: Found 384 targets and 184 test targets...
INFO: Elapsed time: 180.439s, Critical Path: 116.99s
INFO: 238 processes: 277 linux-sandbox, 18 local.
INFO: Build completed successfully, 238 total actions
//gutil:collections_test                                                 PASSED in 0.5s
//gutil:io_test                                                          PASSED in 0.5s
//gutil:proto_matchers_test                                              PASSED in 0.6s
//gutil:proto_ordering_test                                              PASSED in 0.6s
//gutil:proto_test                                                       PASSED in 0.5s
//gutil:status_matchers_test                                             PASSED in 0.5s
//gutil:test_artifact_writer_test                                        PASSED in 0.6s
//gutil:testing_test                                                     PASSED in 0.8s
//gutil:timer_test                                                       PASSED in 5.1s
//gutil:version_test                                                     PASSED in 1.4s
//sai_p4/instantiations/google:clos_stage_test                           PASSED in 0.5s
//sai_p4/instantiations/google:fabric_border_router_p4info_up_to_date_test PASSED in 0.0s
//sai_p4/instantiations/google:middleblock_p4info_up_to_date_test        PASSED in 0.1s
//sai_p4/instantiations/google:sai_nonstandard_platforms_build_test      PASSED in 0.1s
//sai_p4/instantiations/google:sai_nonstandard_platforms_cc_test         PASSED in 0.8s
//sai_p4/instantiations/google:sai_p4info_fetcher_test                   PASSED in 0.8s
//sai_p4/instantiations/google:sai_p4info_test                           PASSED in 1.1s
//sai_p4/instantiations/google:sai_pd_proto_test                         PASSED in 0.3s
//sai_p4/instantiations/google:sai_pd_util_test                          PASSED in 0.6s
//sai_p4/instantiations/google:tor_p4info_up_to_date_test                PASSED in 0.1s
//sai_p4/instantiations/google:union_p4info_up_to_date_test              PASSED in 0.1s
//sai_p4/instantiations/google:wbb_p4info_up_to_date_test                PASSED in 0.0s
//sai_p4/instantiations/google/test_tools:table_entry_generator_helper_test PASSED in 14.4s
//sai_p4/instantiations/google/test_tools:test_entries_test              PASSED in 0.9s
//sai_p4/instantiations/google/tests:p4_fuzzer_integration_test          PASSED in 3.8s
//sai_p4/tools:p4info_tools_test                                         PASSED in 0.7s
//sai_p4/tools:packetio_tools_test                                       PASSED in 0.9s
//thinkit:bazel_test_environment_test                                    PASSED in 0.7s
//thinkit:generic_testbed_test                                           PASSED in 0.9s
//thinkit:mock_control_device_test                                       PASSED in 0.6s
//thinkit:mock_generic_testbed_test                                      PASSED in 0.8s
//thinkit:mock_mirror_testbed_test                                       PASSED in 0.7s
//thinkit:mock_ssh_client_test                                           PASSED in 0.0s
//thinkit:mock_switch_test                                               PASSED in 0.7s
//thinkit:mock_test_environment_test                                     PASSED in 0.1s
//thinkit:switch_test                                                    PASSED in 0.9s
//sai_p4/instantiations/google/tests:p4_constraints_integration_test     PASSED in 0.9s
Stats over 5 runs: max = 0.9s, min = 0.7s, avg = 0.8s, dev = 0.1s
//sai_p4/instantiations/google/test_tools:table_entry_generator_test     PASSED in 44.6s
Stats over 50 runs: max = 44.6s, min = 0.8s, avg = 5.3s, dev = 12.8s
Executed 184 out of 184 tests: 184 tests pass.
INFO: Build completed successfully, 238 total actions

/azp run

Co-authored-by: kheradmandG<EMAIL_ADDRESS>


Cannot step into a function

Enter this code below:

#%%
import numpy as np
from sklearn.model_selection import train_test_split

import torch
from torch import nn
from torch import optim
from torch.utils.data import TensorDataset, DataLoader

from livelossplot import PlotLosses

#%%
base_string = "()()" + (32 - 4) * " "

def shuffle_string(s):
    indices = np.arange(len(s), dtype='uint8')
    np.random.shuffle(indices)
    return "".join(base_string[i] for i in indices)

def is_correct(seq):
    open_brackets = 0
    val = {"(": 1, " ": 0, ")": -1}
    for c in seq:
        open_brackets += val[c]
        if open_brackets < 0:
            return False
    return open_brackets == 0

char2id = {" ": 0, "(": 1, ")": 2}

def generate_pairs(size):
    X = np.zeros((size, 3, len(base_string)), dtype='float32')
    Y = np.zeros((size), dtype='int64')
    for i in range(size):
        s = shuffle_string(base_string)
        Y[i] = int(is_correct(s))
        for j, c in enumerate(s):
            X[i, char2id[c], j] = 1.
    return X, Y

def generate_train_test_pairs(size):
    X, Y = generate_pairs(size)
    return train_test_split(X, Y, test_size=0.25, random_state=42)

#%%

x = shuffle_string(base_string)
print(x, "- correct" if is_correct(x) else "- incorrect")

#%%
X_train, X_test, Y_train, Y_test = generate_train_test_pairs(1000)

trainloader = DataLoader(TensorDataset(torch.from_numpy(X_train), torch.from_numpy(Y_train)),
                         batch_size=32, shuffle=True)
testloader = DataLoader(TensorDataset(torch.from_numpy(X_test), torch.from_numpy(Y_test)),
                         batch_size=32, shuffle=False)

dataloaders = {
    "train": trainloader,
    "validation": testloader
}

#%%

class Recurrent(nn.Module):
    def __init__(self, rnn_size):
        super(Recurrent, self).__init__()
        self.gru = nn.GRU(input_size=3, hidden_size=rnn_size)
        self.fc = nn.Linear(rnn_size, 2)

    def forward(self, x):
        x = x.permute(2, 0, 1)
        output, hidden = self.gru(x)
        return self.fc(hidden.squeeze(0))

#%%
# train on cuda if available
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

def train_model(model, criterion, optimizer, num_epochs=10):
    liveloss = PlotLosses()
    model = model.to(device)

    for epoch in range(num_epochs):
        logs = {}
        for phase in ['train', 'validation']:
            if phase == 'train':
                model.train()
            else:
                model.eval()

            running_loss = 0.0
            running_corrects = 0

            for inputs, labels in dataloaders[phase]:
                inputs = inputs.to(device)
                labels = labels.to(device)

                outputs = model(inputs)
                loss = criterion(outputs, labels)

                if phase == 'train':
                    optimizer.zero_grad()
                    loss.backward()
                    optimizer.step()

                _, preds = torch.max(outputs, 1)
                running_loss += loss.detach() * inputs.size(0)
                running_corrects += torch.sum(preds == labels.data)

            epoch_loss = running_loss / len(dataloaders[phase].dataset)
            epoch_acc = running_corrects.float() / len(dataloaders[phase].dataset)

            prefix = ''
            if phase == 'validation':
                prefix = 'val_'

            logs[prefix + 'log loss'] = epoch_loss.item()
            logs[prefix + 'accuracy'] = epoch_acc.item()

        liveloss.update(logs)
        liveloss.draw()

#%%
model = Recurrent(8)
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.003)

#%%
train_model(model, criterion, optimizer, num_epochs=2)

Run everything above the last cell.
Debug that last cell
Step into the train_model
Expected:
train_model is in a python file
Actual:
train_model is an <ipython-hash.py> file.

Doesn't work for me:

Opens a VS tab for the same file
Break point is at the wrong place.

File is named sk.py
See screenshots.

Here are the experiments i belong to
User belongs to experiment group 'ShowPlayIcon - start'
User belongs to experiment group 'ShowExtensionSurveyPrompt - control'
User belongs to experiment group 'DebugAdapterFactory - experiment'
User belongs to experiment group 'PtvsdWheels37 - experiment'
User belongs to experiment group 'Reload - control'
User belongs to experiment group 'LS - control'
User belongs to experiment group 'UseTerminalToGetActivatedEnvVars - control'
User belongs to experiment group 'WebHostNotebook - control'
User belongs to experiment group 'LocalZMQKernel - control'
User belongs to experiment group 'CollectLSRequestTiming - control'
User belongs to experiment group 'CollectNodeLSRequestTiming - experiment'
User belongs to experiment group 'EnableIPyWidgets - experiment'

Currently using debug py

Wait are you talking about normal python debugging or debugging from the interactive window? You are not showing the same symptoms as the bug. The bug was stepping into a function defined in another cell.

Does the same thing for me with even a super simple example. Looks like the debugger is not even requesting the same file.

@karthiknadig @int19h how do we go backwards on the debugger? This feels like it's behaving incorrectly, but not sure if it's our fault or not.

You can use the OutOutFrom setting (at user scope), to opt out of PtvsdWheels37 - experiment to get the older debugger.

Yeah doing that makes it behave correctly. This seems to be a regression caused by debugpy.

I'm going to enter a new bug.

With PTVSD this should work


(number 0).is-int returns FALSE

(number 0).is-int must return TRUE
[] > is-int
  if. > @
    eq.
      ^
      0
    FALSE
    eq.
      div.
        ^
        ^
      1

@maxonfjvipon Indeed, it's a bug. Can you fix this? The budget is 30 minutes.

@Graur I think it's done. Could you please make a release?

@maxonfjvipon I think I don't have enough permission to do it.
@yegor256 please help us to release eo-math up to 0.2.1

@rultor release, tag is 0.2.1

@rultor release, tag is 0.2.1

@yegor256 OK, I will release it now. Please check the progress here

@rultor release, tag is 0.2.1

@yegor256 Done! FYI, the full log is here (took me 9min)

@maxonfjvipon I believe it is done

@Graur almost, just need to fix 2 tests that decorates nop. After this


sudo[5132]: sh: ps: command not found

I get this message, but I expect that this shouldn't have happened.
pkexec must be setuid root

ls -lah `which pkexec`
-r-s--x--x 1 root root 13K Dec 28 19:36 /var/setuid-wrappers/pkexec

Perhaps there is a bug in the program I am trying to get working, but some feedback would be nice.

I don't experience this issue anymore.

I can reproduce this issue (or perhaps feature) when I add --prefix PATH : "${procps}/bin" to some binary. The intention of procps was to have ps in PATH, but apparently that is not needed, because ps is normally already available, but I still get the following line in my logs then:
Jan 03 21:18:20 nixos sudo[5132]: sh: ps: command not found
Searching for 'ps' in source code is rather impossible with grep like tools, so, what can I do to find calls to 'ps' or to simply make it work (without identifying call sites)?

For procps maybe it's better to use --suffix, so that the setuid wrapper
takes precedence.
On Sat, Jan 3, 2015 at 10:23 PM, wwat<EMAIL_ADDRESS>wrote:

I can reproduce this issue (or perhaps feature) when I add --prefix PATH
: "${procps}/bin" to some binary. The intention of procps was to have ps
in PATH, but apparently that is not needed, because ps is normally
already available, but I still get the following line in my logs then:
Jan 03 21:18:20 nixos sudo[5132]: sh: ps: command not found
Searching for 'ps' in source code is rather impossible with grep like
tools, so, what can I do to find calls to 'ps' or to simply make it work
(without identifying call sites)?
—
Reply to this email directly or view it on GitHub
https://github.com/NixOS/nixpkgs/issues/5490#issuecomment-68610118.

--
NixOS Linux http://nixos.org

@lethalman and when I do that, I get  nixos sudo[8793]: pkexec must be setuid root, so it might be that the ps thing either solved something, or it reintroduced the same problem (which now means that I would be in an infinite loop).

@wwat: ensure that /var/setuid-wrappers is first in PATH. Then you'll get the (correct) setuid variant of pkexec.

@bjornfor Isn't that what @lethalman's suggestion comes down to?

@wwat: Yes. But it assumes that the setuid is already in path. My guess is that it isn't. That's why I suggest to explicitly put /var/setuid-wrappers in PATH for whatever command you're trying to run.

If you need it with setuid, then paths like ${procps}/bin are useless, both on nixos and non-nixos. You can either assume a reasonable $PATH gets passed from the caller, or e.g. use something like /var/setuid-wrappers:/usr/bin to cover both cases.

Is it safe to say that in general there should be no references to /nix/store/<path>/pkexec where pkexec is not suid in $out?

Yes, it seems so. pkexec requires setuid:
$ nix-build -A polkit
/nix/store/mda8yqp7wzi64kgrf3h8l6az1b41jikn-polkit-0.112
$ ./result/bin/pkexec
pkexec must be setuid root

The nix store cannot have setuid binaries (due to security + its multi-user capabilities), so setuid binaries are in /var/setuid-wrappers instead.

So, it would be possible to warn for references like this being created, i.e. make configuration time errors appear. (it would just be a matter of scanning the files under $out).

AFAICS, no explicit references are being made (so there is nothing to scan). Your issue happens at runtime because /var/setuid-wrappers isn't first in $PATH. Other programs aren't rebuilt when a setuid binary is written to /var/setuid-wrappers.

Ok, so you are saying that the polkit dependency add it to PATH, and that the /var/setuid-wrappers version is therefor never even found?

@wwat: Something like that. Run which -a pkexec. You should see something like this:
/var/setuid-wrappers/pkexec
/run/current-system/sw/bin/pkexec

Run both of them. THe first one works and the other will say "pkexec must be setuid root". This is why I earlier suggested you put /var/setuid-wrappers first in $PATH for whatever thing you're trying to run.

I am seeing this in my output:
patching script interpreter paths
/nix/store/caa209kmmvj489bsq7bqp0pl1193rlkp-zoneminder-1.28.0/bin/.zmsystemctl.pl-wrapped: interpreter directive changed from "/var/setuid-wrappers/pkexec /nix/store6qvjjaywj6qf8jn9splpiz9qap01w36w-perl-5.16.3/bin/perl" to "/nix/store/6hik1yf1ml2imw46mh0ma406391vsgj5-polkit-0.112/bin/pkexec /nix/store/
6qvjjaywj6qf8jn9splpiz9qap01w36w-perl-5.16.3/bin/perl "

Can anyone explain to me why that is happening? (I patched it such that it would use /var/setuid-wrappers/pkexec, but then there is something which patches it to the wrong value again. )

It's the patchShebangs function in the standard builder that does that. It replaces shebangs with absolute paths to nix store objects. But for this package it seems you have to stop it from doing that. Try setting dontPatchShebangs = true; in the derivation.  I didn't know that.

@bjornfor
Yes, it's being used as a shebang in zoneminder. Since shebangs are not portable, this can be considered a bug in Zoneminder too. Every single time you write #! you are essentially wasting bytes.
I figured out the dontPatchShebangs myself too in the meanwhile, but then I get:
 nixos sudo[26857]: Error creating textual authentication agent: Error opening current controlling terminal for the process (`/dev/tty'): No such device or address

I already configured polkit to allow everything and yet, it still whines (and that error message is also rather bad).

I don't know what you mean with "shebangs are not portable".
I don't know anything about polkit :-/

@bjornfor For a portable program you should always specify the interpreter in the caller.
a.sh
#!/<whatever>
echo hi

So, sh a.sh would be considered correct code, but ./a.sh is not.

I think the issue is solved: /var/setuid-wrappers/ must be in $PATH for the correct (setuid) pkexec binary to be run. The issue is also specific to the zoneminder package that has a script with a pkexec shebang (which can be patched out). Closing.


Set up Sentry?

It might make sense to set up sentry for exception logging. It's like 3 lines of code and would provide better visibility into any issues that arise.

+1
Is this what you're talking about?
https://docs.sentry.io/platforms/python/
Seems like the developer version is sufficient? I can try to sort out how
to add this in, but if it's straightforward and someone has experience, PRs
welcome too
On Fri, Jun 18, 2021 at 10:14 AM Cory Zue @.***> wrote:

It might make sense to set up sentry for exception logging. It's like 3
lines of code and would provide better visibility into any issues that
arise.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/dimagi/mpact/issues/57, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAXDWWBDREA5Y3C6URIG6LTTL55JANCNFSM465F4E2A
.


RFC: XZ compress HBRT image

The HBRT image is 3.4MB, which will take around 2 seconds to load from PNOR (if using the optimized read in skiboot). HostBoot loads the HBRT image in istep 21.1 (along with loading the payload and geting OCC going). If we xz compressed the HBRT image, it shrinks to 921kb, which is loaded (by the same fast skiboot code) in ~0.6 seconds.
Question: does hostboot need to load the hbrt image? Is there anything stopping us loading it in skiboot? Is there any initial setup done?.
before (run 1):
172.00640|ISTEP 21. 1
200.40760|htmgt|OCCs are now running in ACTIVE state
(total: 28.4s)
before (run 2):
168.26122|ISTEP 21. 1
195.30265|htmgt|OCCs are now running in ACTIVE state
(total: 27.0s)
after (run 1):
169.37556|ISTEP 21. 1
195.06400|htmgt|OCCs are now running in ACTIVE state
(total: 25.68s)
after(run 2):
169.90019|ISTEP 21. 1
193.87845|htmgt|OCCs are now running in ACTIVE state
(total: 23.98)
Saving: 1.3 - 4.4 seconds from boot. (average of 2.87s)
and that number seems in the correct ballpark considering the time pflash takes to read it.
More significantly, it frees up 2.4MB of PNOR.

This change is

Question: does hostboot need to load the hbrt image? Is there anything stopping us loading it in skiboot? Is there any initial setup done?.

We load it and point at it in the devtree because that was what we were told to do originally by the OPAL team.  Remember that we do this for both OpenPOWER and the enterprise systems.  In the latter case, OPAL needs to request the data from the FSP as a lid since you don't have access to the PNOR.  It would not be difficult to have divergent behavior on our side if OPAL can handle it though.
However, one more wrinkle in this story is Secureboot.  Hostboot has to load HBRT during IPL because we are the only ones would can verify the signature.  I think that forces us to keep the current design where HBRT is pushed into mainstore as part of the Hostboot IPL.

@e-liner - Can you take a look and pull this into gerrit if it looks okay?

@bofferdn, @e-liner  - The secureboot code added an extra layer to the PNOR VFS.  If we add additional sections to be XZ compressed, I wonder if we should try to integrate that feature directly into that new secureboot VFS layer.  With this proposed patch from Stewart we end up with effectively duplicated code.
On the performance aspect, I thought we were going to attempt to overlap PAYLOAD and HBRT loading with OCC start.  Did that ever happen?

No performance tweaks have been done to load pnor partitions in the background.

RE: Pushing decompression code into the VFS/PNOR code - I'm in total agreement with that idea as a better design.  The question is always about resources and schedule.  Putting this in as-is produces an immediate benefit (a few seconds) for very little work, while finding time to put the right solution in could take awhile.

This looks okay, but I'm going to want to test a few images on it. Plus we'll need an additional op-build change to compress the image if we're going this route.  I'm running short on time this week, but I can try to look at it tomorrow.

Patrick Williams<EMAIL_ADDRESS>writes:

@bofferdn, @e-liner  - The secureboot code added an extra layer to the
PNOR VFS.  If we add additional sections to be XZ compressed, I wonder
if we should try to integrate that feature directly into that new
secureboot VFS layer.  With this proposed patch from Stewart we end up
with effectively duplicated code.

Hence the RFC, because that's totally what should happen rather than
have inconsistent XZ calls scattered throughout hostboot.
This patch should totally not go in as-is, even though the error
handling (which I totally removed) is eerily similar to the error
handling already loading the HBRT image.
Ultimately, it looks like we don't even have to load the HBRT image
until opal_prd starts (there's no way I can think of how to do this on
P8 without breaking ABI though, but we could go that way for P9).
All of the partitions read in one go could be XZ compressed pretty
easily and save a bunch of boot time.

On the performance aspect, I thought we were going to attempt to
overlap PAYLOAD and HBRT loading with OCC start.  Did that ever
happen?

Tackling OCC start was the next thing to look at. ISTEP 21.1 was really
constrained by PNOR access and OCC start time when I delved into it,
with absolutely nothing else going on there except that, so everything
there (except loading the PAYLOAD itself) is a candidate for the PAYLOAD
to do in parallel with other tasks.
--
Stewart Smith
OPAL Architect, IBM.

Dan<EMAIL_ADDRESS>writes:

RE: Pushing decompression code into the VFS/PNOR code - I'm in total
agreement with that idea as a better design.  The question is always
about resources and schedule.  Putting this in as-is produces an
immediate benefit (a few seconds) for very little work, while finding
time to put the right solution in could take awhile.

I'm happy to do that work, may need some assistance with a few pointers
here and there.
After previous experiments, it seems like the best time to load the
PAYLOAD is during bringing up DRAM, as that's easiest the longest time
spent doing anything (especially with P8 and DDR4).. I know there's memory
constraints on larger systems (Brazos), but we probably have a bit more
leeway with the OpenPOWER kind of size... even if we only loaded a
subset of the PAYLOAD that early.
I know that if I implemented that it'd make all the IPL flow documents
out of date, but technical documentation that's up to date and current
just seems.... not quite right :)
--
Stewart Smith
OPAL Architect, IBM.

e-liner<EMAIL_ADDRESS>writes:

This looks okay, but I'm going to want to test a few images on
it. Plus we'll need an additional op-build change to compress the
image if we're going this route.  I'm running short on time this week,
but I can try to look at it tomorrow.

I think there's a few more partitions we could compress.
WINK and OCC come to mind, and they compress fairly well.
--
Stewart Smith
OPAL Architect, IBM.

Dan<EMAIL_ADDRESS>writes:

Question: does hostboot need to load the hbrt image? Is there anything stopping us loading it in skiboot? Is there any initial setup done?.

We load it and point at it in the devtree because that was what we
were told to do originally by the OPAL team.  Remember that we do this
for both OpenPOWER and the enterprise systems.  In the latter case,
OPAL needs to request the data from the FSP as a lid since you don't
have access to the PNOR.  It would not be difficult to have divergent
behavior on our side if OPAL can handle it though.

It was certainly the easy way to get things going, no problem to revisit
though :)

However, one more wrinkle in this story is Secureboot.  Hostboot has
to load HBRT during IPL because we are the only ones would can verify
the signature.  I think that forces us to keep the current design
where HBRT is pushed into mainstore as part of the Hostboot IPL.

We have to verify signatures in skiboot, so this shouldn't be too much
of an issue.
--
Stewart Smith
OPAL Architect, IBM.

The "other payload" requires Hostboot to load HBRT.  Their secureboot support will be part of the HBRT image so they end up with a (chicken-egg) otherwise.

Patrick Williams<EMAIL_ADDRESS>writes:

The "other payload" requires Hostboot to load HBRT.  Their secureboot
support will be part of the HBRT image so they end up with a
(chicken-egg) otherwise.

Okay, I thought that was a possibility.
Seeing as there's existing logic based on the payload type, I guess that
could be used.
Architecturally though, it does seem neater to have less knowledge of
the payload implementation and fewer if (payload=blah)....
--
Stewart Smith
OPAL Architect, IBM.

I know there's memory constraints on larger systems (Brazos), but we probably have a bit more leeway with the OpenPOWER kind of size

Actually, we are significantly more memory constrained in the OP systems because we have a lot more code loaded.  We are very close to the edge during the memory initialization steps, that is typically where we have seen OOM issues crop up.  If you make any changes, be sure to test on a Garrison box with full memory, that has been our worst case scenario in the past.

The secureboot code added an extra layer to the PNOR VFS. If we add additional sections to be XZ ?compressed, I wonder if we should try to integrate that feature directly into that new secureboot VFS layer. With this proposed patch from Stewart we end up with effectively duplicated code.

the secure provider already verifies the payload which is compressed, it just doesn't do the decompression after the fact.  I think what you are perhaps saying is .. after the verification, take the additional step of doing the decompression and expose the final, uncompressed image to the caller.  In that case getSectionInfo would have to be smart enough to give the uncompressed final size vs. the normal size.  Also right now the VA spaces for PNOR , temp, secure all mirror each other, but the secure space would have to be bigger to accommodate the larger section, so perhaps we'd have to migrate to a unique VA space per partition to allow for the expanded size.  Could probably be done with a new shim VA layer in between temp  + secure, so something like

compressed image in pnor
temp space reads in protected payload, does verification if secure boot enabled
if xz compressed, decompress into matching uncompressed VA space + free up all temp space pages
user  access to secure space VA range pulls in corresponding page of uncompressed, and free uncompressed page.  Now only secure space has that verified, uncompressed, pinned page

Would work great if partition is purely a read-only, protected payload.  If we need to support unprotected payloads or hash-page-table verification of anything, that introduces more complexity.  We're already mem constrained loading OCC early in the boot and we are prob. going to have to switch to hash page table verification for that, assuming we can't otherwise drive the memory footprint down (and it is pretty inefficient how it's being done).
There are also other modifications like making sure getSectionInfo for secure space is reporting the fully uncompressed size, etc.
And as mentioned, the plan was to load HBRT for P9 from Hostboot due to the chicken-and-egg issue of PHyp not having the verification code.

Dan<EMAIL_ADDRESS>writes:

I know there's memory constraints on larger systems (Brazos), but we probably have a bit more leeway with the OpenPOWER kind of size

Actually, we are significantly more memory constrained in the OP
systems because we have a lot more code loaded.  We are very close to
the edge during the memory initialization steps, that is typically
where we have seen OOM issues crop up.  If you make any changes, be
sure to test on a Garrison box with full memory, that has been our
worst case scenario in the past.

Fascinating.
I think we max out at 512GB the garrisons in our lab, so i'll see how I
go and ask for testing help/access if needed.
--
Stewart Smith
OPAL Architect, IBM.

Would work great if partition is purely a read-only, protected payload.

Even for this case, it only works to hide the decompression in the VMM if the partition is a 'load once and done' sort of thing, or we only access it after we have mainstore up.  We likely don't have the cache memory available to keep everything resident, and we can't swap out single pages without a way to decompress an arbitrary piece out of the middle.

Dan<EMAIL_ADDRESS>writes:

Would work great if partition is purely a read-only, protected payload.

Even for this case, it only works to hide the decompression in the VMM
if the partition is a 'load once and done' sort of thing, or we only
access it after we have mainstore up.  We likely don't have the cache
memory available to keep everything resident, and we can't swap out
single pages without a way to decompress an arbitrary piece out of the
middle.

Is there a HOWTO anywhere on looking at memory usage/maps while in cache?
--
Stewart Smith
OPAL Architect, IBM.


JavaScript and Java bindings now use libmraa.so instread of statically linking

This PR address #433 for JavaScript and Java, but no Python.
0041801 is the Java update and is a significant change; look at commit message for details.
@petreeftime, please get Java team to review this update.

Why is python not done? Any reason?
On 25 Feb 2016 21:48, "Henry Bruce"<EMAIL_ADDRESS>wrote:

This PR address #433 https://github.com/intel-iot-devkit/mraa/issues/433
for JavaScript and Java, but no Python.
0041801
https://github.com/intel-iot-devkit/mraa/commit/004180133bb93b75e922cd693e7dc5aaa5b15875
is the Java update and is a significant change; look at commit message for
details.
@petreeftime https://github.com/petreeftime, please get Java team to
review this update.
You can view, comment on, or merge this pull request online at:
https://github.com/intel-iot-devkit/mraa/pull/441
Commit Summary

javascript: Updated cmake so that mraa.node dynamically links to
libmraa
java: mraa jni library now dynamically links to libmraa
java: Allows Java GPIO ISR sample to exit so IsrExit() method can be
tested

File Changes

M api/mraa/gpio.h
https://github.com/intel-iot-devkit/mraa/pull/441/files#diff-0 (7)
M api/mraa/gpio.hpp
https://github.com/intel-iot-devkit/mraa/pull/441/files#diff-1 (7)
M examples/java/Isr.java
https://github.com/intel-iot-devkit/mraa/pull/441/files#diff-2 (18)
A include/java/mraajni.h
https://github.com/intel-iot-devkit/mraa/pull/441/files#diff-3 (44)
M src/CMakeLists.txt
https://github.com/intel-iot-devkit/mraa/pull/441/files#diff-4 (31)
M src/gpio/gpio.c
https://github.com/intel-iot-devkit/mraa/pull/441/files#diff-5 (56)
M src/java/CMakeLists.txt
https://github.com/intel-iot-devkit/mraa/pull/441/files#diff-6 (12)
M src/java/mraajava.i
https://github.com/intel-iot-devkit/mraa/pull/441/files#diff-7 (4)
A src/java/mraajni.c
https://github.com/intel-iot-devkit/mraa/pull/441/files#diff-8 (103)
M src/javascript/CMakeLists.txt
https://github.com/intel-iot-devkit/mraa/pull/441/files#diff-9 (4)

Patch Links:

https://github.com/intel-iot-devkit/mraa/pull/441.patch
https://github.com/intel-iot-devkit/mraa/pull/441.diff

—
Reply to this email directly or view it on GitHub
https://github.com/intel-iot-devkit/mraa/pull/441.

I've never written any python code before so wouldn't feel comfortable making the update.  I guess I've got to get to grips with python sometime and it might as well be now. Just don't expect a quick fix....

It should be really easy just requires the same thing as for nodejs. DW if
it doesn't work I'm happy to add it was just wondering if there was a
technical reason
On 25 Feb 2016 21:55, "Henry Bruce"<EMAIL_ADDRESS>wrote:

I've never written any python code before so wouldn't feel comfortable
making the update. I guess I've got to get to grips with python sometime
and it might as well be now. Just don't expect a quick fix....
—
Reply to this email directly or view it on GitHub
https://github.com/intel-iot-devkit/mraa/pull/441#issuecomment-189003515
.

OK. Will give it a go.

I got the Python binding working with dynamic linking but I realized a neater solution would be to use to adv_function table as an entry point for all binding specific code for the GPIO ISR. This would remove the binding language #ifdefs in gpio.c and make the code much neater. I'l work on this tomorrow for all bindings and hope to update PR by early next week.

this likely won't work in node.js since that requires it to be done from C++ at least in the current methodology.

I've left c++ code as is. It was the lack of Javascript native code in gpio.c that made the binding update so easy. No so luck with Java and Python. I'm testing with GPIO ISR samples for all languages to ensure my changes don't break anything.

I have pushed updates for Java and added Python support. I enhanced gpio ISR samples so I could test with a specified pin and exercise shutdown.

The patch works for Java. I checked with the FTDITest example, but made use of UPM for the button and led.

This is merged into the javacheck2 branch, please use that as the base for future commits. I still see some issues:

gpio raw mode is broken if we don't have a platform, setting lang_func needs to not depend on mraa_init() returning MRAA_SUCCESS or MRAA_ERROR_PLATFORM_ALREADY_INITIALISED for example MRAA_ERROR_PLATFORM_NOT_INITIALISED is perfectly valid. mraa_init() likely needs a small rework to return MRAA_SUCCESS to always create a MRAA_NULL_PLATFORM instead of just in USBPLAT usecase a little like done in the firmata branch.
This needs to be coverity clean, I added a check to the calloc on mraa_init for lang_func but I'm guessing it's not the only one... can someone run coverity or at least clang-analyzer and clean up the errors? Ping me if you need a hand there to run it in offline mode.

This is basically all merged now. Thanks!


A small stand-alone utility to inspect the contents of checkpoint files

Tried to run it I get ImportError: No module named 'distiller' when it tries to load the checkpoint. I think that since the checkpoints contain distiller objects, distiller should be explicitly imported.
Also - this is a generic utility, no? Maybe it should go in apputils? Or some dedicated scripts dir?

Checkpoints should not contain any Distiller class dependencies.  Are you trying some older checkpoint (there was a time when we had this dependency)?  Can you send it to me?
As far as its location: at the moment I don't want to open another directory.  apputils is a library, not an application directory, so I don't think it belongs there.

If a checkpoint is saved using apputils.save_checkpoint(), then it could include a scheduler object, for example. That's what happens in compress_classifier.py. Or am I missing something?

apputils.save_checkpoint() only stores dictionaries it gets from objects is serializes, such as CompressionScheduler instances.  In these dictionaries we only store Python and PyTorch types.
This is done carefully and intentionally in order to avoid creating a dependency on Distiller.  This is because these dependencies are very sensitive to the version of the code they are serializing (so sensitive that if you move file locations between Distiller versions, you won't be able to deserialize checkpoints).
In the past we had some dependencies so maybe you are using an old checkpoint file?
Maybe we still have a bug somewhere and this is manifested in your use-case (and checkpoint file)?  -- send it over ;-)

OK, dug a little deeper I think it's my bad - when I serialize the so-called quantizer_metadata, I store the quantizer's actual type (that is - the result of type(self), not as a string. That's a Distiller type and apparently to deserialize a type object, the type it refers to has to exist. So - I wasn't following our practice :)
So this will only happen when quantization was used. I'll fix it - up to you if you want to wait with this PR.


Feature: Hide image in editor

Is it possible to display the uploaded image as a link in the editor, instead of displaying the image in the editor.
(Similar to StackOverflow and Github, where upon uploading the image, only the link is shown in the editor).
In this example I would like to display the link of the photo instead of the photo.

Yea, you can render anything you want. Check out how createDOM works for LexicalNodes and decorate works for DecoratorNodes. These methods control what Lexical renders for the node. You can imagine an ImageNode like this:
export class ImageNode extends LexicalNode {
 ...
 createDOM() {
    const anchor = document.createElement('a');
   //...set href and such
   return anchor;
 }
...
}

Moving this is discussions as there's no underlying bug.


Update composer installation instructions

In my case (Ubuntu 20.04, Composer 2.2.9) the path was ~/.config/composer/vendor/bin instead of ~/.composer/vendor/bin

What about simply changing
export PATH=~/.composer/vendor/bin:$PATH

to
export PATH="$(composer global config bin-dir --absolute --quiet):$PATH"

?

@mlocati yes of course this would be much better. I also found that this path is gone after i log out from the server. If this is so, should we mention this in readme?

I also found that this path is gone after i log out from the server

Yep: here we just suggest to update the PATH environment variable for the current session.
To make it persistent, we could mention to add that line to the ~/.bashrc file (for people using bash)

To make it persistent, we could mention to add that line to the ~/.bashrc file (for people using bash)

In order to configure the PATH environment variable for any shell, not only for bash, we can add the line
export PATH="$(composer global config bin-dir --absolute --quiet):$PATH"

To the $HOME/.profile file (for the current user only), or to /etc/profile (for any user)

.profile and /etc/profile don't load by default in zsh and zsh is the default shell on macos. Zsh users should add the export to ~/.zshrc instead.


Using cbSDK to grab analogue data from a Blackrock NSP

I’m trying to develop a C++ code to grab analogue data from a Blackrock NSP using cbSDK. I can easily grab spikes, but I haven’t been able to get the wideband analogue data (30k). If someone knows how, please let me know?
Thank you in advance
Tom

Hi Tom, what have you tried so far?
Don't forget you have to first use Central to turn on the continuous data for each channel you want continuous data from. I've been tripped up on this a couple times.

I don't actually have the NSP for testing. I'm streaming previously recorded data using NPlay.
I made a class that initializes the connection to the Central and then gets the data. The wideband part I made is integrated in the two functions, connection_init_neuro() and get_neuro() (see below). A similar code works fine for extracting spikes (setting uEvents = 1 and uConts = 0 in the init function), but nothing comes through when I try to get wideband data.
int CBlackrock::connection_init_neuro(void)
{
cbSdkResult resTest2;
// SET CONFIG SETUP
UINT16 uBegChan   = 1;
UINT32 uBegMask   = 0;
UINT32 uBegVal    = 0;
UINT16 uEndChan   = 128;
UINT32 uEndMask   = 0;
UINT32 uEndVal    = 0;
bool   bDouble    = false;
bool   bAbsolute  = false;
UINT32 uWaveforms = 0;
UINT32 uConts     = 0;
UINT32 uEvents    = 0;
UINT32 uComments  = 0;
UINT32 uTrackings = 0;
UINT32 bWithinTrial = false;

resTest2 = cbSdkGetTrialConfig(m_blkrckInstance
	, &bWithinTrial
	, &uBegChan
	, &uBegMask
	, &uBegVal
	, &uEndChan
	, &uEndMask
	, &uEndVal
	, &bDouble
	, &uWaveforms
	, &uConts
	, &uEvents
	, &uComments
	, &uTrackings);

uEvents = 0;
uConts = cbSdk_CONTINUOUS_DATA_SAMPLES;
UINT32 bActive = 1; // 0 leave buffer intact, 1 clear the buffer
uBegChan = 0;
uEndChan = 0;
uBegVal = 0;
uEndVal = 0;
uBegMask = 0;
uEndMask = 0;
resTest2 = cbSdkSetTrialConfig(m_blkrckInstance
	, bActive
	, uBegChan
	, uBegMask
	, uBegVal
	, uEndChan
	, uEndMask
	, uEndVal
	, bDouble
	, uWaveforms
	, uConts
	, uEvents
	, uComments
	, uTrackings
	, bAbsolute); // Configure a data collection trial

if( resTest2 != CBSDKRESULT_SUCCESS )
{
	bool debug2 = true;
}

// SEND COMMENT
UINT8 charset = 1;
std::string commentMessage = "Neuro init";
resTest2 = cbSdkSetComment(m_blkrckInstance, 50000, charset, commentMessage.c_str());

//DEACTIVATING ALL THE CHANNELS
UINT32 bActive2 = 0; //activate channel 1, deactivating 0
UINT16 dchannel = 0;
resTest2 = cbSdkSetChannelMask(m_blkrckInstance, dchannel, bActive2);

//ACTIVATING ONLY THE CHANNELS TO BE USED
bActive2 = 1; //activate channel: 1, deactivating 0
for (UINT16 ch = 1; ch <= m_numNeuroChannels; ch++)  //i must start from 1 because dchannel =0, bactive= 0 means deactivate everything
{
	resTest2 = cbSdkSetChannelMask(m_blkrckInstance, ch, bActive2);
}

return 0;

}
void CBlackrock::get_neuro(const int* const CHANNELS2USE
, INT16** Signals)
{
cbSdkResult resTest2;
// INIT TRIAL (EVERY TIME)
resTest2 = cbSdkInitTrialData(m_blkrckInstance, NULL, &m_trialCont, NULL, NULL);

// BUFFER FOR FASTER UPDATING - smaller size (20 samples)
for (int chio=0; chio< m_trialCont.count; chio++)
{
	m_trialCont.num_samples[chio] = m_neuroBufferSize;
}

// GET CONFIG
bool bTrialDouble = false;
resTest2 = cbSdkGetTrialConfig(m_blkrckInstance, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &bTrialDouble);

// ALLOCATE MEMORY
int ch = 0;
double* pi = NULL;

if(bTrialDouble)
{
	for(ch=0; ch<m_trialCont.count; ch++)
	{
		m_trialCont.samples[ch] = new double[m_trialCont.num_samples[ch]];
	}
}
else
{
	for(ch=0; ch<m_trialCont.count; ch++)
	{
		m_trialCont.samples[ch] = new INT16[m_trialCont.num_samples[ch]];
	}
}

// GET DATA
bool bFlushBuffer = true;
UINT16 activeChannels = -1;
UINT32 num_samples = -1;
UINT32 time_rec = -1;
resTest2 = cbSdkGetTrialData(m_blkrckInstance, bFlushBuffer, NULL, &m_trialCont, NULL, NULL);

// WAIT UNTIL BUFFER UPDATED
double waitTime = 0;
if (m_trialCont.sample_rates[0] > 0)
{
	waitTime = double(m_emgBufferSize)/m_trialCont.sample_rates[0]*1000-2;
}

Sleep(waitTime);

// STORE IN GLOBAL FUNCTION
for (size_t chan = 0; chan<m_numNeuroChannels; chan++)
{
	INT16* test = (INT16 *)m_trialCont.samples[CHANNELS2USE[chan]];
	for (size_t j=0; j<m_trialCont.num_samples[0]; j++)
	{
		Signals[chan][j] = test[j];
	}
}

}

Can you edit your post and precede your code block with a "triple backtick C++" (```C++), then end it with a triple backtick? It'll make it much easier to read.
I use the Python wrapper mostly, so it's a little different, but I have a few suggestions.

Don't bother with the channel masking until after you've successfully retrieved data.

I think you can set all the relevant parameters to 0.

It might be instructive to look at the Pythonic way to get continuous data here, which is really calling functions defined here.

For every data fetch, it first calls cbSdkInitTrialData(nInstance, 0, trialcont, 0, 0), allocates the return buffer, then cbSdkGetTrialData(nInstance, reset, 0, trialcont, 0, 0). In my case I always have reset=True.

@tmile77 ,
I put together a labstreaminglayer app to stream data from the NSP. It isn't exactly the most concise way to demonstrate how to use CereLink, but it might be helpful. See here
I coded-in two different approaches:

Using a callback function

Data are returned in 'groups', where each group contains all channels that are sampled at the same rate (500, 1000, 2000, 10000, 30000, RAW).
For each group

cbSdkGetSampleGroupList to get the number of channels in the group
From there you can also get the info for each channel to do with as you please.

The callback function is called for each sample.

const cbPKT_GROUP *pPkt = reinterpret_cast< const cbPKT_GROUP* >(data);
Check the sample's group with pPkt->type
Do what you want with the data in pPkt->data. I simply add it to a buffer which I use elsewhere.

Using cbSdkGetTrialData

cbSdkGetTrialConfig; but this gives me zeros and null values in the out params.
call cbSdkSetTrialConfig with above out params, modified as needed.
Then, on every iteration of the main loop

Initialize a cbSdkTrialCont
Call cbSdkInitTrialData to find out how many samples per channel are waiting
Create a buffer for each channel and assign it to cbSdkTrialCont.samples[chanIx]
Call cbSdkGetTrialData with your new cbSdkTrialCont. Its .samples will be filled with the data.

The second approach isn't fully implemented because I didn't want to bother keeping a map of which channels belong to which group, and because the callback approach required less data reshaping.

Closing. Use #97


Fix parsing of qualified references starting with e as a decimal with an exponent

Currently, parsing the following valid SQLite SELECT statement yields an error:
SELECT * FROM foo WHERE foo.elem = 0;

This is happening because scanNumber is detecting the e in elem as part of an exponent in a float literal:
https://github.com/rqlite/sql/blob/768acfbd137ea63346566df98bc2dc16d9f049fd/scanner.go#L296
According to the SQLite numeric-literal parsing diagram, exponents must have digits in front of them, meaning 0e1, 0.0e1, 0.e1, and .0e1 are all supported, but .e1 is not. The current scanner sees .e1 as a valid float, which it isn't.
This PR modifies the scanner to return DOT if no digits are found before an exponent, which makes qualified references parse correctly. It also adds tests to make sure that they're parsed properly and that supported forms of exponents are still scanned correctly.

Thanks for this.


[AppMan] Update FlashPlayer and AIR to version 25

Guys please create http://www.flashdevelop.org/appman/frameworks_flexsdk_9-25.zip
http://www.adobe.com/support/flashplayer/debug_downloads.html

The file is now in place. Could you point the links also to that file and i will merge it then.

Done


Update .openpublishing.publish.config.json

Docs Build status updates of commit 509c999:
:clock10: Full build: the average full build time is 137 min(s) 44 sec(s), this is based on the last 314 build(s) for this repository.

Docs Build status updates of commit 509c999:
:clock10: Full build: the average full build time is 137 min(s) 44 sec(s), this is based on the last 314 build(s) for this repository.

Docs Build status updates of commit 509c999:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

.openpublishing.publish.config.json
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel


Make the left menu, consisting of avatar, and main buttons, collapsed by clicking on the button.

Is your feature request related to a problem? Please describe
The left menu cuts the 1/5 width from the page, and this area could be less disstractfull and more useful, e.g. for reading the articles in the centre
Describe the solution you'd like
Make the left menu, consisting of avatar, and main buttons, collapsed by clicking on the button.
Describe alternatives you've considered
The only useful alternative is to collapse the left menu
Additional context
No, thanks!

The sidebar exists to avoid the body text area moving to the left. This is the advantage of a 3-column layout: it allows the eye to look straight down the center line of the screen when reading.
So I disagree that putting away the sidebar improves the reading experience. The result is actually the opposite.

@cotes2020 after collapsing the left menu to the tiny bar it's possible to keep the same centered text as before or make it wider little bit. Thus, it still will help you to "look straight down the center line of the screen when reading."

Widening the content area will degrade the reading experience: your eyes will need to turn left and right to read a line of text.
You actually just like the two-column layout, but I'm not interested in that.


Errors with sample Prometheus queries

I am following https://github.com/kubecost/cost-model/blob/master/PROMETHEUS.md
prometheus is scraping the cost-model metrics but 2 queries from this link give error:
Query 1:
topk( 5,
  container_memory_allocation_bytes* on(instance) group_left() node_ram_hourly_cost  / 1024 / 1024 / 1024 * 730
  +
  container_cpu_allocation * on(instance) group_left() node_cpu_hourly_cost * 730
)

error:
Error executing query: found duplicate series for the match group {instance="<IP_ADDRESS>:9003"} on the right hand-side of the operation: [{__name__="node_ram_hourly_cost", endpoint="cost-model", exported_instance="ip-10-14-10-49.us-west-2.compute.internal", instance="<IP_ADDRESS>:9003", job="cost-model", namespace="cost-model", node="ip-10-14-10-49.us-west-2.compute.internal", pod="cost-model-6565bcd685-cgpv5", service="cost-model"}, {__name__="node_ram_hourly_cost", endpoint="cost-model", exported_instance="ip-10-14-10-107.us-west-2.compute.internal", instance="<IP_ADDRESS>:9003", job="cost-model", namespace="cost-model", node="ip-10-14-10-107.us-west-2.compute.internal", pod="cost-model-6565bcd685-cgpv5", service="cost-model"}];many-to-many matching not allowed: matching labels must be unique on one side

Query 2:
sum(
  container_memory_allocation_bytes{namespace="default"}
  *
  on(instance) group_left() node_ram_hourly_cost  / 1024 / 1024 / 1024
) by (namespace)

error:
Error executing query: found duplicate series for the match group {instance="<IP_ADDRESS>:9003"} on the right hand-side of the operation: [{__name__="node_ram_hourly_cost", endpoint="cost-model", exported_instance="ip-10-14-10-49.us-west-2.compute.internal", instance="<IP_ADDRESS>:9003", job="cost-model", namespace="cost-model", node="ip-10-14-10-49.us-west-2.compute.internal", pod="cost-model-6565bcd685-cgpv5", service="cost-model"}, {__name__="node_ram_hourly_cost", endpoint="cost-model", exported_instance="ip-10-14-10-107.us-west-2.compute.internal", instance="<IP_ADDRESS>:9003", job="cost-model", namespace="cost-model", node="ip-10-14-10-107.us-west-2.compute.internal", pod="cost-model-6565bcd685-cgpv5", service="cost-model"}];many-to-many matching not allowed: matching labels must be unique on one side

is there any change needed in the query?

@infa-ddeore this appears to be caused by an internal AWS node name change. You can handle with the following change:
sum(
  avg(container_memory_allocation_bytes{namespace="default"}) by (instance)
  *
  on(instance) group_left() avg(node_ram_hourly_cost) by (instance)  / 1024 / 1024 / 1024
) by (namespace)

Let me know if this works for you!

Hi @infa-ddeore, was that example query helpful?

I tried the trial version so I am good with it now but to answer your question the query results no data, it seems to be wrong, you are getting container_memory_allocation_bytes only from default namespace and only by instance so all other labels are removed, and the result is then list by namespace, nothing comes because there is no namespace label.
did you try this query in your env?

Glad to hear you are good!
This query works in one of our test environments. It is just meant to be a reference. You would need to change namespace from default if you do not have workloads in that namespace.

I don't think removing or changing namespace from default to something else will work.
I really wonder how the query is running in your env, query is clearly saying by (instance) means the the result is coming only with instance label and then how doing by namespace will work
but anyways, i am closing this bug, there isnt a working solution yet but i dont need that now.


High-Res Simulations Multiprocessing Issue

ADF run type
Model vs. Model
What happened?
When using ADF for high-res simulations, I seem to be running into some issues while parallelizing the computation using the multiprocessing pool for climatology generation (the mp.Pool step in scripts/averaging/create_climo_files.py). It seems that this bit of the code hangs for a while and then I repeatedly get the error: input in flex scanner failed. I'm wondering if this is some sort of memory or resource allocation issue with high-res?
@brianpm also suggested replacing the multiprocessing lines:
"""
Parallelize the computation using multiprocessing pool:
with mp.Pool(processes=number_of_cpu) as p:
result = p.starmap(process_variable, list_of_arguments)
"""
with serial processing:
"""
for i in list_of_arguments:
process_variable(*i)
"""
This caused the job to be killed upon processing the first variable.
ADF Hash you are using
fc809ba
What machine were you running the ADF on?
CISL machine
What python environment were you using?
ADF-provided Conda env
Extra info
No response

I wonder if this is a memory issue. I'm not sure what the best next step is, but I wonder if @nusbaume will have thoughts?

I think this is a fairly low-level error that I imagine could potentially be caused by several different things.  If possible could you try the following experiments, assuming you are running on a Casper compute node with the default ADF code:

Can you try a run with the num_procs config variable set to one?
If that test fails, then what happens if you reduce diag_var_list to just a single 3-D variable?  If that doesn't work, how about a single 2-D variable?
If none of those work, then what happens if you reduce the size of the timeseries files you are analyzing (e.g. make the difference between start_year and end_year smaller)?

Of course please let me know if need any help setting up any of those experiments.
Finally, if you want to point me to the ADF config file you are using then I can try and see if I can replicate the error myself, although I am mostly on PTO tomorrow and next week so it might be awhile before I get to it.  Thanks!

Thanks for these suggestions @nusbaume ! I'll try setting those up tomorrow-- and, of course, no worries if you are on PTO for a bit!

Sorry this slipped through the cracks! As a quick clarification, I am actually running on Derecho. With the first test, additional climo files were generated, but this still hung on the CVDP portion. However, with the second test and a few minor modifications (some issues with expected units), I was able to get past the portion that was hanging and generate plots as expected! So, thank you for these suggestions @nusbaume and @brianpm !

I wanted to document few more details in case anyone is working with high-res simulations and ADF in the future.
In order to generate climatology files for the 3D files, I ended up needing to regrid to 1-degree in order to avoid hanging on the following line: cam_climo_data = cam_ts_data.groupby('time.month').mean(dim='time'). This worked for our particular use case, but there is still the issue of hitting a limit on resources while using a single processor with these large files. @nusbaume noted that ultimately doing similar operations in the future will likely require implementing dask and ensuring flox is installed. This will be important for both ADF and CUPiD to ensure efficient operations.
#47 discusses implementing dask in ADF.


Add wagtail.search to INSTALLED_APPS

Hey, spent a day figuring out a bug with ImageChooserPanel
It was failing with this
python_1  | File "/usr/local/lib/python3.8/site-packages/wagtail/search/backends/database/postgres/postgres.py", line 63, in prepare_field
python_1  |     get_weight(field.boost),
python_1  | File "/usr/local/lib/python3.8/site-packages/wagtail/search/backends/database/postgres/weights.py", line 59, in get_weight
python_1  |     return weight

so I first created an issue in wagtails repo, but nothing there, I couldn't figure out what's happening, checked INSTALLED_APPS, added wagtail.search to it, now bug is gone!)

Thanks for bringing this up @itekhi! Solved in https://github.com/Frojd/Wagtail-Pipit/commit/1f95d1945c3384d5644574c4cf306c54ceae5dfa


Dataverse data access module

Aim
Including a draft dataverse data collector as part of gget tool boxes.
e.g.
gget dataverse -j dataverse.json -o path-to-dir

Tasks

[x] make a python module and CLI for data collection from dataverse
[x] collect data for a list of datasets defined as a JSON
[ ] build a JSON file for a list of datasets deposited in a given dataverse DOI at certain version
[ ] collect data by providing a dataverse DOI and version

History
As part of https://github.com/GilbertLabUCSF/CanDI/issues/34, related to #121, I draft a module to collect data from dataverse. Then I thought data collection from dataverse can be a more general functionality here as part of gget.
I mostly borrowed codes from gears/utils.py or TDC/load.py. However, there are other efforts such as pyDataverse and easyDataverse.
References

https://guides.dataverse.org/en/latest/api/dataaccess.html
https://github.com/mims-harvard/TDC/blob/main/tdc/utils/load.py

cc @amva13

Hi Abe, this is great work, thank you! I'm pretty busy right now but I'll try to start merging asap.
I just have two thoughts at the moment:

We'll need to add unittests for this module. Do you have any ideas on how they should look like?
Having a json file as input is a relatively complex input type. Is there perhaps a way around this to simplify the input to the module?

Hi Laura,

We'll need to add unittests for this module. Do you have any ideas on how they should look like?

I'll think about and add more commits asap.

Having a json file as input is a relatively complex input type. Is there perhaps a way around this to simplify the input to the module?

That's true. I think this can work with dataverse DOI and/or just fileId. However, I personally think a json file can be a good option to list all files related to a single project or topic. I'll try to add more commits in this regard.
I made this a draft PR while I'm adding more commits.

Oops sorry didn't mean to close this

Oops sorry didn't mean to close this

Oh no worries, I also didn't get chance to finish what we discussed before. I have the changes in a fork so I'll open another PR in the future. Thanks :)

No need, I'll reopen this one as soon as the unit tests are passing and I can make a new dev branch

I did it again lol sorry

https://github.com/mims-harvard/TDC/issues/314#issuecomment-2396435932


HHH-14608 Merge causes StackOverflow when JPA proxy compliance is enabled

https://hibernate.atlassian.net/browse/HHH-14608

IMO, the logic should remain in AbstractLazyInitializer so if/when a new LazyInitializer extends AbstractLazyInitializer (but not BasicLazyInitializer), it will work as expected with hibernate.jpa.compliance.proxy=true`.
With the proposed change, MapLazyInitializer is no longer affected by hibernate.jpa.compliance.proxy=true since it doesn't extend BasicLazyInitializer. Obviously, dynamic maps are not defined by JPA, so if an application uses them, they likely would not set hibernate.jpa.compliance.proxy=true.
My preference would be to add a method to AbstractLazyInitializer that does not have the initialization logic in it for Hibernate's use only; e.g., #getIdentifierInternal.

Thanks @gbadner for the suggestion, what do you think about this https://github.com/hibernate/hibernate-orm/pull/3997 ?

Superseded by https://github.com/hibernate/hibernate-orm/pull/3997.


Created a new layout for GitHub projects

Preview

Additions

Add repositories, type you repo name in the front matter as the following

repositories:
  - KingHowler/Arduino-Language-Support
  - KingHowler/Oscilloscope-Online
  - alshedivat/al-folio

Add icons to your title, to show which languages and tools were used

icons:
  - https://skillicons.dev/icons?i=processing
  - javascript/javascript-original.svg # Note: You don't have to specify the full link to devicons, only the foldername and filename
  - html5/html5-original.svg

Fixed layout issues for mobile phones
Preview

@george-gca kindly review this as well

I know this was previously discussed, but this is the updated version. I am using this for my projects section and it felt like a waste for it to not be a feature.
Next I am thinking about adding some default badges to include
for example I am using this in my about page :
Light Mode

Dark Mode

I was thinking about adding a little bit more like these so that they can make them easily? or should they do this themselves?

I believe they should add them themselves.
I am not sure about the layout of the icons in the desktop version, but tbh I don't currently have a counter-proposal. But I like the idea of this being a different layout available for the users. Can you create (or maybe modify) a project so it uses this layout, so it can be an example of usage?

I was thinking about adding a little bit more like these so that they can make them easily? or should they do this themselves?

I believe they should add them themselves.
I am not sure about the layout of the icons in the desktop version, but tbh I don't currently have a counter-proposal. Have you tested it in the description as smaller icons maybe? But I like the idea of this being a different layout available for the users. Can you create (or maybe modify) a project so it uses this layout, so it can be an example of usage?

Then should I center align the title and add the icons below it? Like in the mobile version?

I think the current desktop version doesn't look very professional, but the mobile one looks nice. I agree to not center align it in the desktop version. What about putting it after the description and before the repo card?

I think the current desktop version doesn't look very professional, but the mobile one looks nice. I agree to not center align it in the desktop version. What about putting it after the description and before the repo card?

That was how I did it initially but it looked like I was just dumping everything to the left side

This is the new design I have come up with
PC Version

Mobile Version

The icons and the repositories can be any number the user wills it to be
The {{ page.icon }} shown in the project card at the all projects page is displayed as an icon on the page

If this one is better, should I add the description before or after the icons?

I am also thinking about adding a divider right before the repositories, it will keep it separated from the center-aligned content

I am using the design on my website, you can take a look directly if you want to
https://kinghowler.github.io/projects/

Can you make it look like this?

I believe left aligned keeps the layout more similar to the other ones we are already using. If the user provides an image it could be shown in the left previous to the title like in this image, if not, it could be hidden and the title and everything would be left aligned.
Also I agree to adding a divider after the repo cards.

I'll try to make it look like that

Oh wow, I had my exams and completely forgot about this. I'll get working on this one.

@george-gca I've edited according to your reqiurements.
Preview

Requesting for a review. Also idk what the new 4 failed checks are, there were only 3 when I made the PR.

2 more things:

Can you make that the image ends with the icons? In your example the icons are positioned at half the image height. Like this:

Can you create or modify one of the projects so that it showcases this new template? You can link to this repository as the code example, and even add the information you showed in your previous comment so users get to know how to do this.

It's a bit different actually, It's not that the icons are placed at the middle of the image, the Entire Right section starts with the the top of the Image. I could add a simple padding-top to resolve it but I wanted to leave it so that we can use Multiline descriptions. If the description uses multiple line it will automatically shift the icons below.
Or I could have the right section be aligned to the bottom of the image.

Can't you instead make the image size relative to the right section, starting it and ending it with the section, but always scaling it? This way the image could be always squared and it would increase its size if the user added a large description.

Interesting, I'll look into it


Fix metric type

See https://github.com/GoogleCloudPlatform/python-docs-samples/issues/227

@broady

merged f1f3891ce1fd562e38b8f793a01a036f52ad9917