Tf.app.flags implicit parsing potentially causes crash with exception

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
Yes
OS Platform and Distribution:
Linux Fedora 30
TensorFlow installed from:
binary
TensorFlow version:
1.13.1
Python version:
3.7.3
CUDA version
10.0
GPU Model
NVIDIA TITAN V Black

Describe the current behavior
The -h, -help, -helpshort, and -helpfull arguments may cause exceptions to be triggered as opposed to displaying flags, default values, and programmer-provided information. This seems to occur when the tf.app.flags abseil wrapper attempts to use implicit parsing.
Describe the expected behavior
When any value of -h, -help, -helpshort, -helpfull are provided as arguments to scripts using tf.app.flags, the help list should appear.
Code to reproduce the issue
''' mwe.py: displays tensorflow --help argument issue '''
from tensorflow import app
app.flags.DEFINE_string('myflag', 'Default', 'Help output')
FLAGS = app.flags.FLAGS
global_string = "This string causes the abseil wrapper to begin processing the {0} flag".format(FLAGS.myflag)
def main(_):
  print("This string causes TF to race the abseil parsing process, which kills the help menu using string %s" % FLAGS.myflag)
if __name__ == '__main__':
  app.run(main)

Other info / logs
The below bash script uses the above code to display the issue. Notably, using abseil's app.flags will NOT cause the exception to occur with the same source code.
python mwe.py --help
sed -i 's/global_string/#global_string/g' mwe.py
python mwe.py --help

This is related to issue #28581. After elaborating on the issue and not receiving a response, I have opened this new issue. If further information is required, please let me know how I may assist you.

Can you please let us know what are the exceptions you are facing .Thanks.

Sure. Hopefully you can reproduce these results by running the included code above. But here's what I get:
python mwe.py
This string causes TF to race the abseil parsing process, which kills the help menu using string Default

python mwe.py -help

Traceback (most recent call last):
 File “mwe.py”, line 5, in <module>
   global_string = “This string causes the abseil wrapper to begin processing the {0} flag”.format(FLAGS.myflag)
 File “/home/.local/lib/python3.7/site-packages/tensorflow/python/platform/flags.py”, line 84, in __getattr__
   wrapped(_sys.argv)
 File “/home/.local/lib/python3.7/site-packages/absl/flags/_flagvalues.py”, line 633, in __call__
   name, value, suggestions=suggestions)
absl.flags._exceptions.UnrecognizedFlagError: Unknown command line flag ‘help’

I should also note that if you switch to just using abseil without the tensorflow wrapper (change from tensorflow import app to from absl import app, the following exception is raised regardless of whether or not a help flag is specified:
python mwe_absl.py [-help]

Traceback (most recent call last):
  File "mwe.py", line 5, in <module>
    global_string = "This string causes the abseil wrapper to begin processing the {0} flag".format(FLAGS.myflag)
  File "/home/.local/lib/python3.7/site-packages/absl/flags/_flagvalues.py", line 491, in __getattr__
    raise _exceptions.UnparsedFlagAccessError(error_message)
absl.flags._exceptions.UnparsedFlagAccessError: Trying to access flag --myflag before flags were parsed.

I was able to reproduce the code in Colab. Thanks.

Sorry, I just saw this. Will investigate

It also reproduces in latest nightly.
However, I don't think I'm a good owner for this, don't know where to fix. Unassigning myself.

Tested with TF 1.15.0rc1 it breaks with message,
UnrecognizedFlagError: Unknown command line flag 'f'

Hi There,
We are checking to see if you still need help on this, as you are using an older version of tensorflow which is officially considered end of life . We recommend that you upgrade to the latest 2.x version and let us know if the issue still persists in newer versions. Please open a new issue for any help you need against 2.x, and we will get you the right help.
This issue will be closed automatically 7 days from now. If you still need help with this issue, please provide us with more information.

I am no longer maintaining the particular codebase, but I was able to update my minimum working example as follows for 2.x compatibility and the issue still seems to occur. That said, it may be lower priority as it is legacy behavior now.
mwe_2x.py
import tensorflow as tf
from tensorflow.compat.v1 import app, flags
FLAGS = flags.FLAGS
flags.DEFINE_string('myflag', 'Default', 'Help output')
global_string = f"Use abseil wrapper to access {FLAGS.myflag}"
def main(argv):
  print(f"Race condition on {FLAGS.myflag}")
if __name__ == '__main__':
  app.run(main)

Test for error and resulting exception:
python3 mwe_2x.py --help

Output:
Traceback (most recent call last):
  File "mwe_2x.py", line 5, in <module>
    global_string = f"Use abseil wrapper to access {FLAGS.myflag}"
  File "/.../python3.8/site-packages/tensorflow/python/platform/flags.py", line 85, in __getattr__
    wrapped(_sys.argv)
  File "/.../python3.8/site-packages/absl/flags/_flagvalues.py", line 654, in __call__
    raise _exceptions.UnrecognizedFlagError(
absl.flags._exceptions.UnrecognizedFlagError: Unknown command line flag 'help'

Test to ensure error originates from Tensorflow, not Abseil (Abseil on its own properly creates exception):
sed "s/^import/#import/" mwe_2x.py > mwe_2x_test1.py;
sed -i "" -e "s/tensorflow.compat.v1/absl/" mwe_2x_test1.py;
python3 mwe_2x_test1.py --help

Output:
Traceback (most recent call last):
  File "mwe_2x_test1.py", line 6, in <module>
    global_string = f"Use abseil wrapper to access {FLAGS.myflag}"
  File "/.../python3.8/site-packages/absl/flags/_flagvalues.py", line 498, in __getattr__
    raise _exceptions.UnparsedFlagAccessError(error_message)
absl.flags._exceptions.UnparsedFlagAccessError: Trying to access flag --myflag before flags were parsed.

Note that Abseil properly catches the race condition even in --help is not given:
python3 mwe_test1.py

Output:
Traceback (most recent call last):
  File "mwe_2x_test1.py", line 5, in <module>
    global_string = f"Use abseil wrapper to access {FLAGS.myflag}"
  File "/.../python3.8/site-packages/absl/flags/_flagvalues.py", line 498, in __getattr__
    raise _exceptions.UnparsedFlagAccessError(error_message)
absl.flags._exceptions.UnparsedFlagAccessError: Trying to access flag --myflag before flags were parsed.

Test to run as expected, but without desired behavior:
sed "s/global_string/#global_string/" mwe_2x.py > mwe_2x_test2.py;
python3 mwe_2x.py --help

Output:

       USAGE: mwe_2x_test2.py [flags]
flags:

mwe_2x_test2.py:
  --myflag: Help output
    (default: 'Default')

Try --helpfull to get a list of all flags.

If it only comes from tf.compat, then I think we can close this. Thank you


Add ability to check issues against Virus Total

This pull requests adds the ability to check if there are already virus reports available on VirusTotal for the specified download url.
If there is no report available, it will add to the comment details as such.
This code requires that a secret variable is added to the Settings page of this repository called VIRUS_TOTAL_API_KEY, and needs to be added by someone with access to a VirusTotal api key and the repository settings page.

Example of the details comments with the information added can be seen here:
With viruses in the report: https://github.com/AdmiringWorm/chocolatey-package-requests/issues/59#issuecomment-785108306
With no viruses found: https://github.com/AdmiringWorm/chocolatey-package-requests/issues/47#issuecomment-785101983
With no report available: https://github.com/AdmiringWorm/chocolatey-package-requests/issues/60#issuecomment-785105291

@tunisiano187

~Maybe hybrid-analysis could be interesting, as VT is used by them + others ;-) https://www.hybrid-analysis.com/docs/api/v2~
max limit to 100m so no sorry

It looks like those limits only apply to direct uploads, though -- I just submitted a Chocolatey package to their API directly via the download URL (swig.<IP_ADDRESS>82020.nupkg, chosen pretty much at random), and they were able to scan it from there. I'm assuming the size limit wouldn't apply, going that route.
I assume incorrectly.
A bigger issue may be that they don't appear to support .nupkg files vey well, at current... you can see the results of the scan by looking up this hash in their API tool, if you have an API key:
33082cb5097d8535a98a80949109eb06391551b4f1eb24988635d6c7ea37f13c

They list the type as "Microsoft OOXML" (which is accurate), and the type strings as "docx" and "office" (which are not). The scan seems to be treating the package as a document, rather than an executable.
Ahhh, dangit. I just tried to point it at the URL for a larger .exe installer, and you're right -- the 100MB limit applies no matter how you submit the file to be scanned. So, doubly scratch that.

Hi @AdmiringWorm I wanted to take this for a spin on your branch, but it seems that you have switched branches

Hi @AdmiringWorm I wanted to take this for a spin on your branch, but it seems that you have switched branches

Yeah, I was working on something else and had the branch changed to the work in progress there.
I have switched back to the branch for this PR, so if you want to check it out then feel free to do so now.

This pull requests adds the ability to check if there are already virus reports available on VirusTotal for the specified download url.
If there is no report available, it will add to the comment details as such.
This code requires that a secret variable is added to the Settings page of this repository called VIRUS_TOTAL_API_KEY, and needs to be added by someone with access to a VirusTotal api key and the repository settings page.

Ola


[BUG] Hauler trying enable ipv6.

Environmental Info:
Rocky 9.4
Linux hauler 5.14.0-427.16.1.el9_4.x86_64 #1 SMP PREEMPT_DYNAMIC Wed May 8 17:48:14 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Hauler Version:
[root@hauler hauler]# hauler version
 __    __       ___       __    __   __       _______ .______
|  |  |  |     /   \     |  |  |  | |  |     |   ____||   _  \
|  |__|  |    /  ^  \    |  |  |  | |  |     |  |__   |  |_)  |
|   __   |   /  /_\  \   |  |  |  | |  |     |   __|  |      /
|  |  |  |  /  _____  \  |  `--'  | |  `----.|  |____ |  |\  \----.
|__|  |__| /__/     \__\  \______/  |_______||_______|| _| `._____|
hauler: Airgap Swiss Army Knife

GitVersion:    1.0.4
GitCommit:     05e67bc
GitTreeState:  clean
BuildDate:     2024-06-26T03:29:02Z
GoVersion:     go1.21.11
Compiler:      gc
Platform:      linux/amd64

Describe the Bug:
Hauler is logging to /var/log/messages the following:
Jul 25 00:30:59 hauler NetworkManager[788]: <warn>  [1721867459.5648] platform-linux: do-add-ip6-address[2: fe80::9436:bff:fedc:3d43]: failure 13 (Permission denied - ipv6: IPv6 is disabled on this device)
Jul 25 00:30:59 hauler NetworkManager[788]: <warn>  [1721867459.5648] l3cfg[a49669bcae08eef3,ifindex=2]: unable to configure IPv6 route: type unicast fe80::/64 dev 2 metric 1024 mss 0 rt-src ipv6ll
Jul 25 00:31:01 hauler NetworkManager[788]: <warn>  [1721867461.5670] ipv6ll[4b1e69a26f5728db,ifindex=2]: changed: no IPv6 link local address to retry after Duplicate Address Detection failures (back off)

Steps to Reproduce:
disable ipv6 on a rocky/rhel node and install hauler. Then run hauler store serve fileserver and hauler store serve registry

Expected Behavior:
NOT logging for ipv6.

Actual Behavior:
See errors above.

Additional Context:

how is ipv6 being disabled?

I disabled it with sysctl -w net.ipv6.conf.all.disable_ipv6=1.

@clemenko are you using the systemd unit proposed at https://github.com/hauler-dev/hauler/pull/181/files or something slightly different? please add the unit file that your are using as a comment or edit your bug report to include it as additional context.

Here is the systemd file
# /etc/systemd/system/hauler.service
[Unit]
Description=Hauler Serve %I Service

[Service]
Environment="HOME=/opt/hauler/"
ExecStart=/usr/local/bin/hauler store serve %i -s /opt/hauler/store
WorkingDirectory=/opt/hauler

[Install]
WantedBy=multi-user.target

basically hauler is assuming ipv6 is there and it is not.

Hey @clemenko, are you able to start with a fresh node of rocky/rhel and test this again and paste the output and commands ran on the node?

this appears to not be a hauler issue. sorry for the confusion.


Documentation to help support a custom data set

I am setting up an experiment to try DSP using a custom dataset. This consists of 2200 customer support conversations and 140 document articles, used in finding answers to customer questions. We curate this data and add which parts of conversations are useful and whether a conversation part is a preliminary follow up question from the support agent and the subsequent answer from the customer. This is very common, where the agent needs more information to answer the initial customer question. We also add a list of questions that are answered by an article to provide context.
What would be useful in the documentation is a walkthrough of how to support such a custom dataset. Some of the considerations would be:

How we condition our dataset to suit the Example inputs required by DSP. It is fairly clear that the preliminary questions and answers lend themselves to CoT approach, but it is not clear how this relates to the DSP Examples required.
Our final answers often have multiple steps to be followed by the customer. Also a lot of intermediate answers are quite long. All your examples have very short answers. It is not clear if our dataset suits DSP as a consequence of this difference.
We would need to train Colbert V2 on our data. It is not clear where we can do the training on a service supporting Colbert v2
What specification of PC we would need to run the Colbert V2 inference for RM. I have a 8 core 16GB RAM mini server available, but do not know if it would be adequate.
It would also be interesting to see if there is much difference in performance replacing RM with a simpler mechanism such a FAISS.

Thank you for the great work so far!

To answer my first question convqa_program mentioned in section 3.4 of the paper looks like it accommodates this requirement.

Hey @davidwynter ,
Please see here for indexing your own data for retrieval: https://github.com/stanford-futuredata/ColBERT#running-a-lightweight-colbertv2-server
Before you launch the server, use the ColBERT intro notebook (or the Overview in the ColBERT README) to index your collection
In principle, you shouldn't need to train a new retriever. You can usually use the one we provide (depending on licensing at least; it's trained on MS MARCO, the dataset itself is for non-commercial research use).


Sorting without touching timestamps

Is there a way to sort via Nova without changing the updated_at column?
Thanks!

You can create an observer that removes timestamps when sort_order is updated.
    public function updating(User $user)
    {
       if ($user->isDirty('sort_order')) {
           $user->timestamps = false;
       }
    }


fix: remove multi-source/domain sharding behavior

This PR removes support for configuring multiple sources and any remaining references to domain sharding. Domain sharding was deprecated/removed from imgix-rb in v3.0.0, which will have to be removed from this gem as well.

@jayeb Now that I think about it, perhaps we don't actually want to remove support for multiple sources so long as they don't rely on domain sharding to cycle through them? Or else users will be stuck with only configuring a single source on their app.

Agreed, let's keep multi-source support while removing sharding.


added university of edinburgh alumni email

When students graduate from the University of Edinburgh, their student email ending in .ac.ed.uk gets disconnected and they are given an alumni email ending in ed-alumni.net

@y00j Regrettably, when a student graduates from a university, the student is no longer eligible for a free educational subscription for JetBrains tools. Thus, I have to close this request without merging. Please ensure you are familiar with this repository rules located at the bottom of this page before submitting a new request. Thank you for the understanding!


[Event Request] Report 5885 Calc. Phys. Invt. Order (Bins) - OnAfterGetRecordCheckPhysInvtOrderBinFilter

Hi Team,
In Report 5885 we need possibility to have our additional checks . Having event in the trigger OnAfterGetRecord , would be a perfect solution :

[IntegrationEvent(true, false)]
local procedure OnAfterGetRecordCheckPhysInvtOrderBinFilter(PhysInvtOrderHeader: Record "Phys. Invt. Order Header"; Bin: Record Bin)

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability.


WIP: Converting some MD to RST

Converting some MD to RST

Adding WIP to title

@yangchen1 can you please ask Lindy for help to review using the "view" button to look at the RST on GitHub?

thanks @it33 . and will do @esethna

Closing this + YC and other PMs will resubmit in pieces, too many changes to merge.


Disable a rule and keeping the logging

(From: https://github.com/SpiderLabs/ModSecurity/issues/2249#issuecomment-579469632)
@SachinBadgujar

Additionally, how do you use SecRuleRemoveById with logging? Because if you disable the rule it will not log. I am having ModSecurity in the Blocking mode. I want to remove a rule, however, I want it to get logged in the log file for analysis. Do you know how is this achieved?

There are three loggings: debug, audit and error. What exactly you meant? please use a different issue to start that discussion.

closing as we are not having feedback from the user who originally reported it.


Upgrade otel 1 4 1

Description
Please include a summary of the change, motivation and context.

Testing
Please describe the tests that you ran to verify your changes. Please summarize what did you test and what needs to be tested e.g. deployed and tested helm chart locally.
Checklist:

[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] Any dependent changes have been merged and published in downstream modules

Documentation
Make sure that you have documented corresponding changes in this repository or hypertrace docs repo if required.

This PR obviates the need for #341 and helps us avoid the un-shading 😄


Update README.md

Description of the Change
Change sectioning of README.md. Macs can't run the binary file in atom-amd64.tar.gz, so I'm nesting this under the linux section, which is where I believe this was intended to be/
Release Notes
N/A
Sorry for not adding [ci skip] to the commit, I had already committed the change and then read the Contributing.md

View rendered README.md

Going to merge this, checked with @lee-dohm and CI failures should be unrelated since this is a documentation change.
Thanks @SConaway!


wireguard dkms failure on stemcell 621.76 (ok with 621.69)

With wireguard bosh release version 3, hit a dkms compilation failure
======================================

Selecting previously unselected package wireguard-tools.
(Reading database ... 61344 files and directories currently installed.)
Preparing to unpack .../wireguard-tools.deb ...
Unpacking wireguard-tools (1.0.20200319-0ppa1~16.04) ...
Setting up wireguard-tools (1.0.20200319-0ppa1~16.04) ...
Processing triggers for man-db (2.7.5-1) ...
Selecting previously unselected package wireguard-dkms.
(Reading database ... 61422 files and directories currently installed.)
Preparing to unpack .../wireguard-dkms.deb ...
Unpacking wireguard-dkms (0.0.20200318-0ppa1~16.04) ...
Setting up wireguard-dkms (0.0.20200318-0ppa1~16.04) ...
Selecting previously unselected package wireguard.
(Reading database ... 61568 files and directories currently installed.)
Preparing to unpack .../wireguard-deb/wireguard.deb ...
Unpacking wireguard (1.0.20200319-0ppa1~16.04) ...
Setting up wireguard (1.0.20200319-0ppa1~16.04) ...
~

Loading new wireguard-0.0.20200318 DKMS files...
First Installation: checking all kernels...
Building only for 4.15.0-106-generic
Building initial module for 4.15.0-106-generic
Error! Bad return status for module build on kernel: 4.15.0-106-generic (x86_64)
Consult /var/lib/dkms/wireguard/0.0.20200318/build/make.log for more information.
[#] ip link add wg0 type wireguard
RTNETLINK answers: Operation not supported
[#] ip link delete dev wg0
Cannot find device "wg0"
~
~
vpn/dba19fdd-7d5f-4fdb-9675-5f8655427dc9:/var/lib/dkms/wireguard/0.0.20200318/build# cat make.log
DKMS make.log for wireguard-0.0.20200318 for kernel 4.15.0-106-generic (x86_64)
Mon Aug  3 12:28:10 UTC 2020
make: Entering directory '/usr/src/linux-headers-4.15.0-106-generic'
  CC [M]  /var/lib/dkms/wireguard/0.0.20200318/build/main.o
  CC [M]  /var/lib/dkms/wireguard/0.0.20200318/build/noise.o
  CC [M]  /var/lib/dkms/wireguard/0.0.20200318/build/device.o
  CC [M]  /var/lib/dkms/wireguard/0.0.20200318/build/peer.o
  CC [M]  /var/lib/dkms/wireguard/0.0.20200318/build/timers.o
  CC [M]  /var/lib/dkms/wireguard/0.0.20200318/build/queueing.o
  CC [M]  /var/lib/dkms/wireguard/0.0.20200318/build/send.o
  CC [M]  /var/lib/dkms/wireguard/0.0.20200318/build/receive.o
  CC [M]  /var/lib/dkms/wireguard/0.0.20200318/build/socket.o
/var/lib/dkms/wireguard/0.0.20200318/build/socket.c: In function 'send6':
/var/lib/dkms/wireguard/0.0.20200318/build/socket.c:145:18: error: 'const struct ipv6_stub' has no member named 'ipv6_dst_lookup'
   dst = ipv6_stub->ipv6_dst_lookup_flow(sock_net(sock), sock, &fl,
                  ^
scripts/Makefile.build:330: recipe for target '/var/lib/dkms/wireguard/0.0.20200318/build/socket.o' failed
make[1]: *** [/var/lib/dkms/wireguard/0.0.20200318/build/socket.o] Error 1
Makefile:1577: recipe for target '_module_/var/lib/dkms/wireguard/0.0.20200318/build' failed
make: *** [_module_/var/lib/dkms/wireguard/0.0.20200318/build] Error 2
make: Leaving directory '/usr/src/linux-headers-4.15.0-106-generic'

references

https://forum.proxmox.com/threads/error-building-wireguard-module-on-latest-proxmox-update.65816/
https://launchpad.net/ubuntu/+source/wireguard-linux-compat

fixed for v4, bumping wireguard packages:
wireguard_1.0.20200513-1~16.04.1_all.deb
wireguard-dkms_1.0.20200611-1ubuntu1~16.04.1_all.deb
wireguard-tools_1.0.20200513-1~16.04.1_amd64.deb


Rewrite of Intro Section to reflect Miami

The section starts here:
https://github.com/Code-for-Miami/mia-councilmatic/blob/master/chicago/templates/chicago/index.html#L34

This is done.


Add Trabuco Hills High School

@michellehk Please provide us with:

the school official website URL
the school street address, including city and country
an URL of a page on the official website where the school offers an IT-related long-term (>1 year) course
a proof, which shows that the school recognizes the domain you are submitting as an official email domain for the students.

@michellehk You might miss my previous comment. Please react.

@michellehk I am sorry, this request remains unanswered for a while. I have to close it without merging. Please reopen and comment if you have an update regarding the aforementioned issue. Thank you for your understanding!


Fail to re-produce the speed-up of TeraSort with SparkRDMA

We have compare the performance of using Spark shuffle manager and pure TCP. The speed-up is 30% at most but not 2.63 times from the README.md.
Can the tester of this project provide more specified parameters to re-produce this performance? For example, how many partition should we use and should we restrict the executor memory or executor number?

Hi, the speedup depends on many factors (CPU number on machine, disks (HDD vs SSD vs NVME), Nic type (Connect X3  vs Connect X4 vs Connect X5 | Infiniband vs Roce). Some results published on the wiki page incl.configuration and hardware type.

Thanks @petro-rudenko and that is the detail we need.


Continuous integration with GitHub actions

I was trying to implement CI to ensure the component compiles for the various versions of ESP-IDF.
The problem is that the project is not set up in a way that allows for this:
ESP-IDF components should be in a subfolder (commonly main/, but it could also be M5GFX/). Without this, any attempt to compile the component with idf.py build leads to this error:

Called idf_component_register from a non-component directory.

Reference build: https://github.com/KenVanHoeylandt/M5GFX/actions/runs/8222729252/job/22484559726
.github/workflows/build.yml example:
name: Build
on: [push]
jobs:
  build-esp-idf-4.4:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repo
        uses: actions/checkout@v2
        with:
          submodules: recursive
      - name: build
        uses: espressif/esp-idf-ci-action@main
        with:
          esp_idf_version: v4.4
          target: esp32

To allow for idf.py build to work independently in this project, these steps are required:

[ ] Add a subfolder for this component (main/ or M5GFX/), move relevant files into this folder and add a CMakeLists.txt
[ ] Add include($ENV{IDF_PATH}/tools/cmake/project.cmake) to the root CMakeLists.txt

Considering that moving source files has a big impact on your currently unmerged branches, I figured I'd open this issue to discuss the proposed changes.

Hello, @KenVanHoeylandt
Sorry, it's my lack of understanding and I don't quite understand what you're asking.
Are you saying you want the main source code of the library to be in main instead of src?

Almost: I suggest to move the src folder into a main folder.
It doesn't have to be named "main" though, because ESP-IDF allows for renaming the module. "main" is just the default naming.
Having src in a folder allows us to create an ESP-IDF "module" (~ library).
I could make the CMakeList.txt files that would go along with it.
Once you have this module, you can compile it standalone, and thus we can avoid releases with compile errors as we can automate building.
Sorry for the slow response.


Graph generated upon comparing multiple NLU pipelines is not rendered correctly

Rasa version: 2.3.4
Rasa SDK version (if used & relevant): (not relevant;  using an NLU-only pipeline)
Rasa X version (if used & relevant): (not relevant for this issue)
Python version: 3.7.9
Operating system (windows, osx, ...): mac OS Big Sur 11.1
Issue:
See this forum topic for more info - https://forum.rasa.com/t/is-there-a-doc-on-how-to-interpret-pipeline-comparison-results/41165/2
I was looking to find a resource that would help me interpret the results of the pipeline comparison, but it appears that there are multiple graphs plotted over each other (as noted by @samsucik).
In addition, he notes that the graph also has lost information about the scale of its axes.  I'm not sure what the expected result should be.
Also tagging @dakshvar22.
Error (including full traceback):
See attached file in the forum - https://forum.rasa.com/uploads/short-url/koZaCVIBB4ozHLZdbBMbnJUt9IK.pdf
Command or request that led to error:
rasa test nlu --nlu data/nlu.yml --config available_config/spacy.yml available_config/mitie.yml

Content of configuration file (config.yml) (if relevant):
spacy.yml
language: "en"  # your two-letter language code

pipeline:
  - name: SpacyNLP
    model: "en_core_web_md"
    case_sensitive: False
  - name: SpacyTokenizer
  - name: SpacyFeaturizer
  - name: RegexFeaturizer
  - name: LexicalSyntacticFeaturizer
  - name: CountVectorsFeaturizer
  - name: CountVectorsFeaturizer
    analyzer: "char_wb"
    min_ngram: 1
    max_ngram: 4
  - name: DIETClassifier
    epochs: 100
  - name: EntitySynonymMapper

mitie.yml
language: "en"  # your two-letter language code

pipeline:
  - name: MitieNLP
    model: "data/total_word_feature_extractor.dat"
  - name: MitieTokenizer
  - name: MitieFeaturizer
  - name: RegexFeaturizer
  - name: LexicalSyntacticFeaturizer
  - name: CountVectorsFeaturizer
  - name: CountVectorsFeaturizer
    analyzer: "char_wb"
    min_ngram: 1
    max_ngram: 4
  - name: DIETClassifier
    epochs: 100
  - name: EntitySynonymMapper

Content of domain file (domain.yml) (if relevant): (not relevant here)

In addition, could you provide me with directions on where to find the documentation for how to interpret pipeline comparison results?  Many thanks in advance! :-)

Thanks for the issue, @koernerfelicia will get back to you about it soon!
You may find help in the docs and the forum, too 🤗

@ganeshviyer looks like this has already been routed to the right Inbox. Thanks for submitting the issue! :)

@koernerfelicia I think this should probably be handled by research as it's "owned" by them according to the code boundaries. (rasa.nlu.test). WDYT?

@joejuzl yeah, that seems fair!

@ganeshviyer This was fixed as part of https://github.com/RasaHQ/rasa/pull/8859 . Will be released in the next micro version.


Tickets/DM-46411: pairing vs stacking technote

Yes, this is an illustration of a pipeline.  A separate notebook would go to ts_aos_analysis, running on lsstComCam data from OR3.

View entire conversation on ReviewNB

Added description of the stacking process.

View entire conversation on ReviewNB

fixed

View entire conversation on ReviewNB

Added absolute differences

View entire conversation on ReviewNB


fix(blockstore): blocksync fixes ported from 0.38.8

Fixes ASA-2024-008 on the main branch.

PR checklist

[X] Tests written/updated
[X] Changelog entry added in .changelog (we use unclog to manage our changelog)
[ ] Updated relevant documentation (docs/ or spec/) and code comments
[X] Title follows the Conventional Commits spec

The decision made here to panic when no extended commit is found was not ported back to v0.38 and/or v0.37. Because of that, TestCheckExtendedCommitMissing fails on main and v1.x while it succeeds on v0.38.8 and v0.37.7 with the fixes in ASA-2024-008. (This PR.)
I'm planning to catch the panic in the test on main and v1.x and fail if the test did not panic. Currently, the issue is that the panic happens in a sub-thread that testify's require.Panics does not seem to catch. I'm open for ideas.


Update views.py - String quotation fix

Applied a fix for the SyntaxError: unexpected character after line continuation character error when trying to execute the python manage.py makemigrations background_task command during the installation process.

Python 3.11 dislikes using the same quotes in an f-string as used to define the f-string. However, Python 3.12 tolerates this. Accepted the merge request to support older versions of Python.
(see discussion on https://github.com/astral-sh/ruff/issues/10184 for similar problem)


Terminating child process makes whole cluster go away.

testing #71063:

start cluster debugging
terminate one child

Observe:
whole debug session tree disappears
This works fine in stable release

Aha, I understand what the problem is: selecting a child session from the dropdown menu doesn't update the selected session correctly: the red terminate icon doesn't become a "Plug". So the selected session is always the master process. And terminating it kills the whole tree.

/duplicate #71183


Should central server be run on node that contains outcome?

I don't know why I didn't see this before (did we discuss this @fvandaalen?)
The first order derivative is defined by:
$$   \frac{\partial L_{\overline{z}}}{\partial \overline{z}u} =
\sum \limits{t=1}^{t_u} \left[ d_t \frac{ K exp( Kz_{\overline{u}})}{\sum \limits_{j \in R_t} exp(K \overline{z}_j)} \right]
+
K \rho \left[ \overline{z}_u - \overline{\sigma}_u - \frac{\overline{\gamma}_u^{(p-1)}}{\rho}  \right]$$
We had already identified one privacy-infringing component (component 2) that will be taken care of by the n party scalar protocol:
$$  \frac{\partial L_{\overline{z}}}{\partial \overline{z}u} = \sum \limits{t=1}^{t_u} \left[ d_t \frac{ K exp( Kz_{\overline{u}})}{component2} \right]
+
K \rho \left[ \overline{z}_u - \overline{\sigma}_u - \frac{\overline{\gamma}_u^{(p-1)}}{\rho}  \right]
$$
However, there is still the issue that we have to sum from $t=1$ to $t_u$, where $t_u$ is the event time that belongs to the sample at index $u$.
This could in theory be solved by another call to the n-scalar-protocol, but this will affect performance even more than component 2 because the elements of component 2 can be reused.
This is leading me to wonder why we wouldn't run the central server on the node that contains the outcome data. Then we wouldn't have to use the n-scalar protocol on the central server and everything can stay efficient.
I will go ahead and experiment with this approach. It's possible I have forgotten some things that we discussed @fvandaalen . In this approach the n scalar protocol will still be used at the datanodes but not at the central aggregator.
One privacy concern would be that by running this on the outcome node, you would be able to link the data coming from the other nodes to a specific patient. If there are only a few variables used per location, you would be able to interpret this, and e.g. see which patients have high blood pressure. This could be solved though by adding a disclaimer or setting a limit on the minimum number of features at a node.

If the issue can be solved by running the central server on the outcome node, then verticox+ is already done (except for the encryption part) and we can close this issue


Update db-query-builder.md

Minor syntax changes.
I don't know Yii well enough, but I would suggest adding two additional examples: IS NOT NULL and LIKE 'foo%'

https://github.com/yiisoft/yii2/commit/cd1e9f4a99c7279a72b6a87dfcbb2f4334f9cc6b


Add Component Locking

This is a draft PR to get feedback on an early implementation of Component Locking.
This is not a complete implementation, but opening for early feedback and to align with changes proposed in https://github.com/ably-labs/spaces/pull/101.
[UPDATE: 2023-08-07]
I have now implemented all four proposed locking features: Query, Acquire, Release, and Subscribe.
Query
space.locks.get is used to query whether a lock identifier is currently locked and by whom. It returns a Lock type which has the following fields:
type Lock = {
  member: SpaceMember;
  request: LockRequest;
};

For example:
// check if the id is locked
const isLocked = space.locks.get(id) !== undefined;

// check which member has the lock
const { member } = space.locks.get(id);

// check the lock attributes assigned by the member holding the lock
const { request } = space.locks.get(id);
const value = request.attributes.get(key);

Acquire
space.locks.acquire initialises a lock request, adds it to the locks field of presence message extras, and calls presence.update.
It returns a Promise which resolves once presence.update resolves.
const req = await space.locks.acquire(id);

// or with some attributes
const attributes = new Map();
attributes.set('key', 'value');
const req = await space.locks.acquire(id, { attributes });

It throws an error if a lock request already exists for the given identifier with a status of PENDING or LOCKED.
Release
space.locks.release releases a previously requested lock by removing it from the locks field of presence message extras, and calls presence.update.
It returns a Promise which resolves once presence.update resolves.
await space.locks.release(id);

Subscribe
space.locks.subscribe subscribes to changes in lock status across all members.
The callback is called with a value of type Lock.
space.locks.subscribe('update', (lock) => {
  // lock.member is the requesting member
  // lock.request is the request made by the member
});

// or with destructuring:
space.locks.subscribe('update', ({ member, request }) => {
  // request.status is the status of the request, one of PENDING, LOCKED, or UNLOCKED
  // request.reason is an ErrorInfo if the status is UNLOCKED
});

Such changes occur when:

a PENDING request transitions to LOCKED (i.e. the requesting member now holds the lock)
a PENDING request transitions to UNLOCKED (i.e. the requesting member does not hold the lock since another member already does)
a LOCKED request transitions to UNLOCKED (i.e. the lock was either released or invalidated by a concurrent request which took precedence)
an UNLOCKED request transitions to LOCKED (i.e. the requesting member reacquired a lock)

I've also added an example in examples/locks.ts which can be run with npm run examples:locks:
$ npm run examples:locks

><EMAIL_ADDRESS>examples:locks
> ts-node ./examples/locks.ts

2023-08-07T20:26:16.220Z INFO initialising Ably client
2023-08-07T20:26:16.222Z INFO entering the "example" space
2023-08-07T20:26:16.670Z INFO setting location to {"slideId":"123","elementId":"456"}
2023-08-07T20:26:16.976Z INFO checking if location is locked
2023-08-07T20:26:16.978Z INFO location is not locked
2023-08-07T20:26:16.978Z INFO subscribing to lock events
2023-08-07T20:26:16.978Z INFO attempting to lock "/slides/123/element/456"
2023-08-07T20:26:17.012Z INFO waiting for lock event
2023-08-07T20:26:17.017Z INFO received lock update for "/slides/123/element/456", status=locked
2023-08-07T20:26:17.018Z INFO unsubscribing from lock events
2023-08-07T20:26:17.018Z INFO lock status is "locked"
2023-08-07T20:26:17.018Z INFO releasing the lock
2023-08-07T20:26:17.514Z INFO done

MMB-213 [MMB-214] [MMB-215]

I have now implemented the four proposed locking features: Query, Acquire, Release, and Subscribe (see the PR description).
I've also added an example of locking locations in examples/lock.ts (I wanted to add this to the demo, but I was finding that difficult).
This is not yet ready for merge since it requires an ably-js release including https://github.com/ably/ably-js/pull/1418, but it is otherwise ready for review.
cc @dpiatek @paddybyers @mattheworiordan

Thanks @lmars , a few questions/ comments:

In Query, const value = request.attributes.get(key); - does this mean we'll expose a get method for the attributes as well? How is that different from request.attributes.key or request.attributes[key]?
With the release() method, one can only ever release a lock they acquired because it's part of their presence data. Due to this there won't be any case of accidentally releasing someone else's lock right? I know that's not a use-case we are supporting right now but just wanted to confirm that's the case.
For Subscribe, given that we don't send a message when a lock enters the pending state from scratch (following someone requesting it), what's the thinking behind including this event: a PENDING request transitions to UNLOCKED (i.e. the requesting member does not hold the lock since another member already does)? Who is it meant for and what is the use-case?
re an UNLOCKED request transitions to LOCKED (i.e. the requesting member reacquired a lock) - if they require, will it not go into the pending state again triggering the first subscription event?

This encompasses the foundation of what we discussed in our work. I've left a couple of small comments, questions, but I think this is a great first iteration of the API.

@dpiatek I've addressed your comments, PTAL

@lmars two small comments otherwise LGTM

@mattheworiordan I had requested your review but I've gone ahead and merged this, please feel free to leave comments after the fact though and I can follow up in future PRs.

I had requested your review but I've gone ahead and merged this, please feel free to leave comments after the fact though and I can follow up in future PRs.

Yup, sorry, I've not had time.  I will add comments in a separate post or inline shortly.

Thanks for this @lmars, great start and simple which is nice.

This is not a complete implementation, but opening for early feedback

I am finding it hard what to comment on or not in terms of functionality because you're not saying which parts of the implementation you have omitted, vs decided not to implement. This makes it hard for me to comment on what's not here as a result. On the call, for example, I mentioned you had not included a way to iterate over the locks, and you said you'd add that. Had you decided not to include that method, or just not implemented it yet?  Given this, I feel it would be a waste of time to comment on what's not here, and would prefer to do that once you present an IDL of a complete API. Would it make sense to have that evolve as you do PRs, so we can comment on what you're removing/adding/changing to the complete API as well?

It returns a Lock type which has the following fields:
type Lock = { member: SpaceMember;   request: LockRequest; };

Referring to Paddy's proposal, he stated:

By allowing userdata to be provided as part of a lock entry, this enables the feature to be used as a convenient primitive for other higher-level services to be constructed client-side (eg leader election, state synchronisation). (These kinds of use-cases are described in the Chubby paper).

Do you envisage userData is just part of attributes for the lock, and immutable whilst the lock is held? I assume so, but just checking.

It returns a Lock type which has the following fields:
type Lock = { member: SpaceMember;   request: LockRequest; };

I am not mad about the attribute request for LockRequest. The lock has an ID, a status, attributes, not the request. I don't think the naming / modelling is valid from a user's perspective, and instead we're using the name request simply because a request is made to obtain the lock, but once locked, the naming request does not make much sense. Why aren't the LockRequest attributes set at the root level of the Lock object?
In this code example:
const isLocked = space.locks.get(lockId) !== undefined;
  if (isLocked) {
    info('location is already locked');
    process.exit();
  }

I have two comments/questions:

Calling if (space.locks.get(lockId)) { ..handle can't obtaining lock.. }}  if the user happens to have the lock already, will end up calling the block. I think we should be providing a convenience that provides filters for my locks, or all locks by my locks. This could be done with chaining such as space.locks.mine.get(lockId) or arguments, I'm unopinionated on that. However I believe the pattern will be so common it would be odd for us to not offer that. I see competing APIs do something similar.
I realise get is just a proxy to finding out if isLocked, but I feel like we could do better for readability with something like:

  if (space.locks.isLocked(lockId, exclude_self: true) {
    info('location is already locked');
    process.exit();
  }

is just nicer to read and more intuitive.
⚠ Please note I see I missed some of the replies to my comments on https://ably.atlassian.net/wiki/spaces/product/pages/2668298321/MMDR19+Component+Locking+API, I've since replied to them all. Note that some apply to this API.
❓ Should we close https://github.com/ably/ideas/issues/459 now that this work supersedes it?

@mattheworiordan

I am finding it hard what to comment on or not in terms of functionality because you're not saying which parts of the implementation you have omitted, vs decided not to implement

Apologies, that initial description was when this PR was a draft for early feedback, I subsequently implemented all functionality from our internal API design doc (i.e. Query, Acquire, Release, and Subscribe), which this now covers. If there is anything missing from the design doc, please point it out, but I realise I omitted getAll, which I'll follow up with.

Given this, I feel it would be a waste of time to comment on what's not here, and would prefer to do that once you present an IDL of a complete API

Can you clarify what you mean by IDL? Something beyond what is in the API design doc? I wasn't planning to write anything beyond that.

Do you envisage userData is just part of attributes for the lock, and immutable whilst the lock is held? I assume so, but just checking.

Yes, attributes are the equivalent of userData from that proposal. There is currently no API to modify the attributes, but it isn't enforced, there's nothing stopping a member updating the attributes directly in presence data, and that being broadcast to all members. Do you think there is a requirement that they should be immutable?

Why aren't the LockRequest attributes set at the root level of the Lock object?

Yes we could have Lock extend LockRequest, I'll consider that in a future PR.

I think we should be providing a convenience that provides filters for my locks, or all locks by my locks ... I feel like we could do better for readability with something like

Roger that, I'll write something up specific to enhancing the Query implementation rather than discussing here.

❓ Should we close https://github.com/ably/ideas/issues/459 now that this work supersedes it?

I'll do that once I've written the server-side implementation plan (https://ably.atlassian.net/browse/MMB-197).

There is currently no API to modify the attributes, but it isn't enforced, there's nothing stopping a member updating the attributes directly in presence data, and that being broadcast to all members. Do you think there is a requirement that they should be immutable?

Perhaps one for @paddybyers to comment on given he was the one who suggested it. My sense is we should either reject changes once a lock is obtained, or provide an API to actually make and subscribe to lock attribute changes.

I think the main aim of having the userdata is that participants can rely on the state set there being globally authoritative for the channel. If it can change, then that just detracts from that - to be useful, there would need then to be explicit versioning, events when it changes, etc. So I think, at least for now, we should say that it's immutable. We could consider it being mutable but that would require us thinking about what API a user would really need around that


Fix ForeignKeys in views, make it possible to use a database alias for synchronisation

Hey there!
This is a combination of https://github.com/mypebble/django-pgviews/pull/52/files and another fix that we use in production (ForeignKeys don't work properly with views unless deep copy is used on copied fields). Sorry for two things at once, but it'd just be nice to have it all upstreamed.
Paging @seroy who did both fixes.

Hi, I think code looks good, but I would appreciate some tests to test the behaviour before pushing it out

Implemented in https://github.com/mikicz/django-pgviews/pull/10


Add array_trim function.

@laithsakka I can't reproduce the build failure locally or in meta internal system.


Make POM release more like java-control-plane

Signed-off-by: Ryan Michela<EMAIL_ADDRESS>

Still need to set up Circle CI as described in https://github.com/envoyproxy/protoc-gen-validate/issues/168#issuecomment-483506361

Blocked by #183

@mattklein123 added required environment vars to CircleCI. After merge publishing to Maven Central should "just work".

Thanks Ryan!

https://circleci.com/gh/envoyproxy/protoc-gen-validate/95


Add Dependabot to the repository

@akaszynski Oh so Dependabot will only deal with pinned down dependencies. The thing is we do not necessarily want to pin down all of our dependencies. I understand this is maybe best practice, but IMO it might create more versioning conflicts when someone tries to install our package along with another one.
I discussed this with other DPF members and the feeling is that we might want to actually manage a "=<" version for each package. This way a released package gives a range of compatible dependencies.
Does that make sense?
TLDR; not sure how pinning down every dependency solves more problems than it creates.

@akaszynski Oh so Dependabot will only deal with pinned down dependencies. The thing is we do not necessarily want to pin down all of our dependencies. I understand this is maybe best practice, but IMO it might create more versioning conflicts when someone tries to install our package along with another one. I discussed this with other DPF members and the feeling is that we might want to actually manage a "=<" version for each package. This way a released package gives a range of compatible dependencies. Does that make sense? TLDR; not sure how pinning down every dependency solves more problems than it creates.

The thing is we do not necessarily want to pin down all of our dependencies.

Agreed 100%. PyAnsys packages should have loose requirements (or as loose as possible) to ensure multiple packages can work in the same environment.

IMO it might create more versioning conflicts when someone tries to install our package along with another one.

Absolutely. Anything in your setup.py or pyproject.toml that's actually required by your package should allow the oldest version possible to run your library.

I discussed this with other DPF members and the feeling is that we might want to actually manage a "=<" version for each package.

That's one way of ensuring that upstream dependencies don't break your current project. However, even though it's a potential risk I'd instead only a minimum version requirement. For example, vtk>=9.0.
So, at this point you're probably asking, why use dependabot at all since, as you stated, it only works with "pinned down" dependencies.

Oh so Dependabot will only deal with pinned down dependencies.

It works with fully pinned or those with "upper limits" like pyvista <= 0.34.0, which we've established isn't the best practice for run-time dependencies. However, and perhaps it's because I wasn't clear, the only requirements you should pin down should be your testing and documentation dependencies.
By pinning down requirements_test.txt to the latest dependencies, you can ensure that your package works with the latest version of a package as soon as it's released to PyPI. This ensures "the best of both worlds", you can allow for the "latest" package in your install_requires section of setup.py, but you always test with the latest package in your CI/CD. The only thing dependabot does is create PRs that bump your requirements_*.txt dependencies and the workflow will either run or not.
This gives you a chance to either patch the issue due to the upstream changes, or provide an upper bound for your install_requires for that package until you patch it. Either way, you'll have to patch it, and it's much better for a "bot" to tell you that your library can't support the latest version of <some-dependency> rather than your users, because users will always be trying to use the latest version of everything.
Ping me on teams if we need to discuss this further, or if our docs at https://dev.docs.pyansys.com/ need to include this discussion. (actually, there already is an issue at https://github.com/pyansys/dev-guide/issues/125).


fix: create the symbol table in the workspace path

Current dependencies on/for this PR:

main

PR #124

PR #125   👈

PR #126

This comment was auto-generated by Graphite.


Validate dpLGAR using Streamflow

We have to do the following to tune parameters

[x] Check the gradient chain
[x] Toy with max ponded depth
[ ] Get streamflow data
[ ] Test percolation
[x] See if params update
[ ] Train model

Testing my mass balance with a different threshold:
[2023-07-09 16:26:36,770][models.physics.MassBalance][INFO] - *********************************************************
[2023-07-09 16:26:36,770][models.physics.MassBalance][INFO] - -------------------- Simulation Summary -----------------
[2023-07-09 16:26:36,770][models.physics.MassBalance][INFO] - ------------------------ Mass balance -------------------
[2023-07-09 16:26:36,770][models.physics.MassBalance][INFO] - Initial water in soil    =      45.115850 cm
[2023-07-09 16:26:36,770][models.physics.MassBalance][INFO] - Total precipitation      =       1.270000 cm
[2023-07-09 16:26:36,770][models.physics.MassBalance][INFO] - Total infiltration       =       1.270000 cm
[2023-07-09 16:26:36,770][models.physics.MassBalance][INFO] - Final water in soil      =      44.850618 cm
[2023-07-09 16:26:36,770][models.physics.MassBalance][INFO] - Surface ponded water     =       0.000000 cm
[2023-07-09 16:26:36,770][models.physics.MassBalance][INFO] - Surface runoff           =       0.000000 cm
[2023-07-09 16:26:36,771][models.physics.MassBalance][INFO] - GIUH runoff              =       0.000000 cm
[2023-07-09 16:26:36,771][models.physics.MassBalance][INFO] - Total percolation        =       0.000000 cm
[2023-07-09 16:26:36,771][models.physics.MassBalance][INFO] - Total AET                =       1.535232 cm
[2023-07-09 16:26:36,771][models.physics.MassBalance][INFO] - Total PET                =       9.336486 cm
[2023-07-09 16:26:36,771][models.physics.MassBalance][INFO] - Total discharge (Q)      =       0.000000 cm
[2023-07-09 16:26:36,771][models.physics.MassBalance][INFO] - Global balance           =   1.255671e-10 cm
self.model.top_layer.print()
[2023-07-09 16:27:01,705][models.physics.layers.Layer][INFO] - [     Depth   Theta       Layer_num   dzdt     k_cm_hr    psi   ]
PyDev console: starting.

[2023-07-09 16:27:01,706][models.physics.layers.WettingFront][INFO] - [24.5311, 0.1816941172, 0, 0.149522, 0.000118, 1769.2941]
[2023-07-09 16:27:01,707][models.physics.layers.WettingFront][INFO] - [26.4522, 0.1788332030, 0, 0.139832, 0.000103, 1837.9781]
[2023-07-09 16:27:01,707][models.physics.layers.WettingFront][INFO] - [34.3777, 0.1765984955, 0, 0.076836, 0.000093, 1894.5824]
[2023-07-09 16:27:01,707][models.physics.layers.WettingFront][INFO] - [38.7353, 0.1696611346, 0, 0.103690, 0.000065, 2089.0180]
[2023-07-09 16:27:01,708][models.physics.layers.WettingFront][INFO] - [44.0000, 0.1684463074, 0, 0.000000, 0.000061, 2126.3183]
[2023-07-09 16:27:01,708][models.physics.layers.WettingFront][INFO] - [175.0000, 0.2491213663, 1, 0.000000, 0.000001, 2126.3183]
[2023-07-09 16:27:01,708][models.physics.layers.WettingFront][INFO] - [200.0000, 0.1755696091, 2, 0.000000, 0.000050, 2126.3183]

This is a threshold of 1e-12
[2023-07-09 16:34:08,588][models.physics.MassBalance][INFO] - *********************************************************
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - -------------------- Simulation Summary -----------------
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - ------------------------ Mass balance -------------------
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - Initial water in soil    =      45.115850 cm
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - Total precipitation      =       1.270000 cm
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - Total infiltration       =       1.270000 cm
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - Final water in soil      =      44.850691 cm
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - Surface ponded water     =       0.000000 cm
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - Surface runoff           =       0.000000 cm
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - GIUH runoff              =       0.000000 cm
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - Total percolation        =       0.000000 cm
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - Total AET                =       1.535160 cm
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - Total PET                =       9.336486 cm
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - Total discharge (Q)      =       0.000000 cm
[2023-07-09 16:34:08,589][models.physics.MassBalance][INFO] - Global balance           =   1.308251e-10 cm
self.model.top_layer.print()
[2023-07-09 16:34:24,602][models.physics.layers.Layer][INFO] - [     Depth   Theta       Layer_num   dzdt     k_cm_hr    psi   ]
PyDev console: starting.
[2023-07-09 16:34:24,602][models.physics.layers.WettingFront][INFO] - [24.5357, 0.1816947524, 0, 0.149497, 0.000118, 1769.2792]
[2023-07-09 16:34:24,602][models.physics.layers.WettingFront][INFO] - [26.4578, 0.1788333258, 0, 0.139822, 0.000103, 1837.9750]
[2023-07-09 16:34:24,603][models.physics.layers.WettingFront][INFO] - [34.3805, 0.1765990562, 0, 0.076830, 0.000093, 1894.5678]
[2023-07-09 16:34:24,603][models.physics.layers.WettingFront][INFO] - [38.7417, 0.1696609414, 0, 0.103690, 0.000065, 2089.0238]
[2023-07-09 16:34:24,603][models.physics.layers.WettingFront][INFO] - [44.0000, 0.1684463074, 0, 0.000000, 0.000061, 2126.3183]
[2023-07-09 16:34:24,603][models.physics.layers.WettingFront][INFO] - [175.0000, 0.2491213663, 1, 0.000000, 0.000001, 2126.3183]
[2023-07-09 16:34:24,603][models.physics.layers.WettingFront][INFO] - [200.0000, 0.1755696091, 2, 0.000000, 0.000050, 2126.3183]

It looks like the SE threshold is sensitive, this is something to note between versions of LGAR

For context, this is the mass balance I want to benchmark against with LGAR-C:
[ (24.535741,0.18169475352642,1,1,0, 1.494965e-01, 0.000118 1769.27922230005083)
(26.457777,0.17883332607650,1,2,0, 1.398221e-01, 0.000103 1837.97502802446888)
(34.380516,0.17659905720241,1,3,0, 7.683014e-02, 0.000093 1894.56779938911541)
(38.741668,0.16966094100581,1,4,0, 1.036899e-01, 0.000065 2089.02381680663939)
(44.000000,0.16844630744222,1,5,1, 0.000000e+00, 0.000061 2126.31830323560462)
(175.000000,0.24912136632205,2,6,1, 0.000000e+00, 0.000001 2126.31830323560462)
(200.000000,0.17556960906662,3,7,1, 0.000000e+00, 0.000050 2126.31830323560507) ]

*********************************************************
-------------------- Simulation Summary -----------------
------------------------ Mass balance -------------------
Initial water in soil    =  45.1158503564 cm
Total precipitation      =   1.2700000000 cm
Total infiltration       =   1.2700000000 cm
Final water in soil      =  44.8506907205 cm
Surface ponded water     =   0.0000000000 cm
Surface runoff           =   0.0000000000 cm
GIUH runoff              =   0.0000000000 cm
Total percolation        =   0.0000000000 cm
Total AET                =   1.5351596358 cm
Total PET                =   9.3364856806 cm
Total discharge (Q)      =   0.0000000000 cm
Global balance           =   1.292975e-10 cm
Time                     =   0.2508 sec

See PR #10 for why our mass balance is now different

Setting ponded_depth_cm to be a nn.Parameter() since it is sensitive and varies from site to site.
Also, I changed the runoff calculation to this:
            ponded_depth = ponded_depth_temp
            _runoff_ = ponded_depth - infiltration
            runoff = torch.clamp(_runoff_, min=0.0)

so there is always a runoff grad

Debugging the updating soil parameters:
Before update benchmark:
[2023-07-11 11:27:43,459][models.physics.layers.Layer][INFO] - [     Depth   Theta       Layer_num   dzdt     k_cm_hr    psi   ]
[2023-07-11 11:27:43,461][models.physics.layers.WettingFront][INFO] - [24.5357, 0.1816947524, 0, 0.149497, 0.000118, 1769.2792]
[2023-07-11 11:27:43,461][models.physics.layers.WettingFront][INFO] - [26.4578, 0.1788333258, 0, 0.139822, 0.000103, 1837.9750]
[2023-07-11 11:27:43,461][models.physics.layers.WettingFront][INFO] - [34.3805, 0.1765990562, 0, 0.076830, 0.000093, 1894.5678]
[2023-07-11 11:27:43,461][models.physics.layers.WettingFront][INFO] - [38.7417, 0.1696609414, 0, 0.103690, 0.000065, 2089.0238]
[2023-07-11 11:27:43,461][models.physics.layers.WettingFront][INFO] - [44.0000, 0.1684463074, 0, 0.000000, 0.000061, 2126.3183]
[2023-07-11 11:27:43,462][models.physics.layers.WettingFront][INFO] - [175.0000, 0.2491213663, 1, 0.000000, 0.000001, 2126.3183]
[2023-07-11 11:27:43,462][models.physics.layers.WettingFront][INFO] - [200.0000, 0.1755696091, 2, 0.000000, 0.000050, 2126.3183]

Parameter updates work:
[2023-07-12 10:18:52,848][models.dpLGAR][INFO] - Alpha for soil 1: 0.0021
[2023-07-12 10:18:52,849][models.dpLGAR][INFO] - Alpha for soil 2: 0.0073
[2023-07-12 10:18:52,849][models.dpLGAR][INFO] - Alpha for soil 3: 0.0037
[2023-07-12 10:18:52,849][models.dpLGAR][INFO] - n for soil 1: 1.6868
[2023-07-12 10:18:52,849][models.dpLGAR][INFO] - n for soil 2: 1.2980
[2023-07-12 10:18:52,849][models.dpLGAR][INFO] - n for soil 3: 1.6151
[2023-07-12 10:18:52,849][models.dpLGAR][INFO] - Ksat for soil 1: 0.4510
[2023-07-12 10:18:52,849][models.dpLGAR][INFO] - Ksat for soil 2: 0.0690
[2023-07-12 10:18:52,849][models.dpLGAR][INFO] - Ksat for soil 3: 0.4500
[2023-07-12 10:18:52,849][models.dpLGAR][INFO] - Max Ponded Depth: 0.0000

See PR #11 For more information on parameter updates and training

First run with the new data:
[2023-07-13 06:33:46,013][models.physics.MassBalance][INFO] - *********************************************************
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - -------------------- Simulation Summary -----------------
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - ------------------------ Mass balance -------------------
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - Initial water in soil    =      15.487900 cm
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - Total precipitation      =      16.406079 cm
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - Total infiltration       =      16.406079 cm
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - Final water in soil      =      26.638837 cm
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - Surface ponded water     =       0.000000 cm
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - Surface runoff           =       0.000000 cm
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - GIUH runoff              =       0.000000 cm
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - Total percolation        =       0.000000 cm
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - Total AET                =       5.255142 cm
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - Total PET                =       8.277200 cm
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - Total discharge (Q)      =       0.000000 cm
[2023-07-13 06:33:46,014][models.physics.MassBalance][INFO] - Global balance           =   1.748894e-10 cm```

There is a small discrepency when using hourly intervals vs 15 min intervals:
Epoch 1 Training:   4%|▍         | 745/17544 [03:58<95:44:02, 20.52s/it]>>> self.mass_balance.report_mass(self.model)
[2023-07-13 06:40:39,385][models.physics.MassBalance][INFO] - *********************************************************
[2023-07-13 06:40:39,386][models.physics.MassBalance][INFO] - -------------------- Simulation Summary -----------------
[2023-07-13 06:40:39,386][models.physics.MassBalance][INFO] - ------------------------ Mass balance -------------------
[2023-07-13 06:40:39,386][models.physics.MassBalance][INFO] - Initial water in soil    =      15.487900 cm
[2023-07-13 06:40:39,386][models.physics.MassBalance][INFO] - Total precipitation      =      16.406079 cm
[2023-07-13 06:40:39,386][models.physics.MassBalance][INFO] - Total infiltration       =      16.406079 cm
[2023-07-13 06:40:39,386][models.physics.MassBalance][INFO] - Final water in soil      =      26.446677 cm
[2023-07-13 06:40:39,386][models.physics.MassBalance][INFO] - Surface ponded water     =       0.000000 cm
[2023-07-13 06:40:39,386][models.physics.MassBalance][INFO] - Surface runoff           =       0.000000 cm
[2023-07-13 06:40:39,386][models.physics.MassBalance][INFO] - GIUH runoff              =       0.000000 cm
[2023-07-13 06:40:39,386][models.physics.MassBalance][INFO] - Total percolation        =       0.000000 cm
[2023-07-13 06:40:39,386][models.physics.MassBalance][INFO] - Total AET                =       5.447301 cm
[2023-07-13 06:40:39,387][models.physics.MassBalance][INFO] - Total PET                =       8.277200 cm
[2023-07-13 06:40:39,387][models.physics.MassBalance][INFO] - Total discharge (Q)      =       0.000000 cm
[2023-07-13 06:40:39,387][models.physics.MassBalance][INFO] - Global balance           =   6.616716e-10 cm

See #15

training the model using torchrun:
torchrun --nproc_per_node=1 --master_port=47800 dpLGAR/__main__.py ++nproc=1 ++save_name=debug_inplace

Starting to document on this issue again to make sure I don't forget things.
Currently, we're looking into getting an LSTM to work, and the normalization of inputs. Attributes look good (see below) , but forcings need work

Updates from work today:

Switching back to an MLP away from the LSTM (for now) so that I can get results in the near term for AGU

Going to be using all attributes (adding in bulk density too) in the MLP to predict alpha, n, ponded_water_max, theta_e, theta_r,  and ksat
This mimics what https://github.com/usda-ars-ussl/rosetta-soil/blob/main/rosetta/ANN_Module.py does

Data loading and normalization is complete
LGAR instantiation and MLP is good to go
Tomorrow i'll work on doing some training runs and testing.


IPv6Network, etc. not serializable

Greetings,
We're using your class as part of an InetRange class we use internally and were wondering if you would be able to to make the classes in this project serializable? When we're transmitting InetRanges for Spark jobs over the network they fail to serialize/deserialize.
Thanks!
Justin

Implemented, thanks for the suggestion.


fix(versions): update Activiti/example-runtime-bundle versions into master

UpdateBot pushed maven dependency: org.activiti.cloud.rb:activiti-cloud-runtime-bundle-dependencies to: 7.1.213

UpdateBot commands:
updatebot push-version --kind maven org.activiti.cloud.rb:activiti-cloud-runtime-bundle-dependencies 7.1.213


remove operation-type enum

Summary
Enums are useful for capturing controlled a vocabulary, things that do not change very often or are static in nature.
Since we are adding and removing operators at a fairly quick cadence this is not an appropriate data structure, especially if we want to model this in the DB which will likely mean a data-migration every time we add or remove operators.
In additional, while the keys are unique,  there is no guarantee on the value so there isn't really a safe guarantee.
Change this to a frozen-object, and the datatype of operator name to just a string.
Testing
I removed a few references to operators that are deprecated, otherwise no functional changes, app should work as before.

I do not understand what this resolves, but it looks good to me.

@YohannParis Having an enum here does not help anything, having an enum does mean a higher development cost - especially if we model this into Java/Hibernate combination, where hibernate will create a table-constraint based on the enum.  For example we have an enum representing Operation-status, if you look at the "simulation" table:
Check constraints:
    "simulation_engine_check" CHECK (engine::text = ANY (ARRAY['SCIML'::character varying, 'CIEMSS'::character varying]::text[]))
    "simulation_status_check" CHECK (status::text = ANY (ARRAY['CANCELLED'::character varying, 'COMPLETE'::character varying, 'ERROR'::character varying, 'FAILED'::character varying, 'QUEUED'::character varying, 'RETRIEVING'::character varying, 'RUNNING'::character varying]::text[]))

A similar check would be generated for operator-type, and we'd have to update the constraint as often as we update our operators.

That is what I thought, but because this enum was not saved on the back-end and just a front-end tool, I was confused.
Would you like we stop using ENUM for operators and things not 100% defined?

I do not understand what this resolves, but it looks good to me.

@YohannParis Having an enum here does not help anything, having an enum does mean a higher development cost - especially if we model this into Java/Hibernate combination, where hibernate will create a table-constraint based on the enum. For example we have an enum representing simulation-status, if you look at the "simulation" table:
Check constraints:
    "simulation_engine_check" CHECK (engine::text = ANY (ARRAY['SCIML'::character varying, 'CIEMSS'::character varying]::text[]))
    "simulation_status_check" CHECK (status::text = ANY (ARRAY['CANCELLED'::character varying, 'COMPLETE'::character varying, 'ERROR'::character varying, 'FAILED'::character varying, 'QUEUED'::character varying, 'RETRIEVING'::character varying, 'RUNNING'::character varying]::text[]))

A similar check would be generated for operator-type, and we'd have to update the constraint as often as we update our operators.

That is what I thought, but because this enum was not saved on the back-end and just a front-end tool, I was confused.
Would you like we stop using ENUM for operators and things not 100% defined?

For me, it isn't about how well objects are defined, but what we are using enums for. You wouldn't normally have an enum for team members for example if you are modelling an engineering project.
TeamMember.Shawn
TeamMember.Derek
...

Because things change, and you generally want enum to be fixed and stable(e.g. Status.fail, Status.in_progress, Status.success ).


Performance analysis range<1> vs range<3>

Hi,
I wanted to make a quick analysis on the usage of range<1> vs range<3> on some of my data.
In general I need a range<3> model since I do matrix operations on my data. Sometimes the operations on the matrix are very linear though, so I might choose to linearize my data and use range<1>.
On a small test I noticed that range<1> is about 3x faster than range<3> when analysing with nsight systems when running on a RTX 2080 Ti.
My sample used 960^3 sycl::float2 values, split in either sycl::range<3>{960, 960, 960} or sycl::range<1>{3375*64*64*64}.
Example code range<3>
#include <sycl/sycl.hpp>
#include <vector>
#include <iostream>
#include <chrono>
using namespace sycl;

int main()
{
    queue myQueue;
    size_t size = 3375;
    auto dims = sycl::range<3>{960, 960, 960};
    std::vector<sycl::float2> vecWithData(size * 64 * 64 * 64, sycl::float2{1.0f, 0.0f});
    std::cout << "vector size " << vecWithData.size() << std::endl;
    {
        sycl::buffer<sycl::float2, 3> complexIn = sycl::make_async_writeback_view(vecWithData.data(), dims, myQueue);

        myQueue.submit([&](sycl::handler &cgh)
                       {

                           cgh.parallel_for(dims, [=](sycl::id<3> index)
                                            { vec_acc[index] = vec_acc[index] * 3.0f; }); });
    }

    myQueue.wait();
}

Example code range<1>
#include <sycl/sycl.hpp>
#include <vector>
#include <iostream>
#include <chrono>
using namespace sycl;

int main()
{
    queue myQueue;
    size_t size = 3375;
    auto dims = sycl::range<1>{size * 64 * 64 * 64};
    std::vector<sycl::float2> vecWithData(size * 64 * 64 * 64, sycl::float2{1.0f, 0.0f});
    std::cout << "vector size " << vecWithData.size() << std::endl;
    {
        sycl::buffer<sycl::float2, 1> complexIn = sycl::make_async_writeback_view(vecWithData.data(), dims, myQueue);

        myQueue.submit([&](sycl::handler &cgh)
                       {

                           cgh.parallel_for(dims, [=](sycl::id<1> index)
                                            { vec_acc[index] = vec_acc[index] * 3.0f; }); });
    }

    myQueue.wait();
}

NSight System output (Top displays range<1>, bottom displays range<3>):

I also noticed some big fluctuation about the memcpy HtoD and DtoH for both cases, but that's not of importance right now.
Any help appreciated
Kind Regards

There seems to be some code missing in your snippets, e.g. where your accessors are coming from.
There is some additional indexing work in 3D to calculate the linear id, but I would not expect such a big difference. As per your other issue, please provide additional information about software and build process.


Can't install addon

Hi,
I just tried installing your addon but Home-Asisstant threw the following error:

Can't install ghcr.io/robocopklaus/aarch64-addon-blocky:0.2.0: 404 Client Error for http+docker://localhost/v1.44/images/ghcr.io/robocopklaus/aarch64-addon-blocky:0.2.0/json: Not Found ("No such image: ghcr.io/robocopklaus/aarch64-addon-blocky:0.2.0")

Could you please look into this, since I'm very interested in your addon?
I'm running:
Core 2024.5.5
Supervisor 2024.05.1
Operating System 12.3
Frontend 20240501.1
Thanks a lot.
BR Philipp

Hey Philipp, that's weird. I'll have a look ✌

Hey @phischmi, I just released a new version. It should work now 🤞😊


add range

@Zaid-Ajaj I think make test / pulumi up fails because the generated schema requires Limit, Start, and Step whereas we only want to require Limit. Adding omitempty to the RangeArgs struct tag doesn't seem to help - not quite sure how to do this

@aq17 I think you wanted optional instead of omitempty. The pulumi up also didn't work because YAML doesn't know how to accept multi-arguments in an array (i.e. input: [limit, start, step]) this syntax is not supported. You need an explicit object with properties:
fn::std:range:
  limit: 10
  start: 1
  step: 1

with start and step being optional above.
I've fixed up those issues and modified make test to show the actual error when pulumi up fails ✅

Thanks @Zaid-Ajaj !


Merge Hotfix 2.6.2

Codecov Report

Merging #1890 (fe398ab) into develop (6d24e01) will decrease coverage by 0.17%.
The diff coverage is 46.15%.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #1890      +/-   ##
===========================================
- Coverage    83.40%   83.23%   -0.17%
===========================================
  Files          479      479
  Lines        16935    16954      +19
  Branches      2527     2527
===========================================
- Hits         14124    14111      -13
- Misses        2128     2148      +20
- Partials       683      695      +12

Files
Coverage Δ

...ernal/vitals/JankStatsActivityLifecycleListener.kt
68.70% <46.15%> (-3.64%)
:arrow_down:

... and 20 files with indirect coverage changes


Add support for .dockerignore (or similar mechanism)

I started to look into adding support for .dockerignore files when creating the build context, but I'm afraid I have no idea what I am doing. Seems like you can pull most of what you need from the standard Buildkit builder
Right now copying the build context takes quite a while due to my repository with a multi-gigabyte Git history

Yes, we can just take this part from the Dockerfile frontend
https://github.com/moby/buildkit/blob/a1cfefeaeb66501a95a4d2f5858c939211f331ac/frontend/dockerfile/builder/build.go#L895-L910


Cannot toggle visibility on meshes ("...different scaling and rotation pivots, ignoring scaling pivot!")

I've been banging my head against the proverbial wall trying to get this to work, to no avail.
Basically, all I'm trying to do is hide/reveal a mesh during my animation using scale keys. I have it keyed at a scale of 1 on one frame, then 0.001 on the next frame. The animation is baked to the geometry itself, so there are no additional pivots at play, just the mesh's pivot.... yet I keep getting the following error:
// Error: line 1: Transform 'musician_gamePad' has different scaling and rotation pivots, this is not supported, ignoring scaling pivot! //
I tried setting the keys to Stepped and using the -dsa 2 flag but nothing works.
After exporting, the mesh in question is invisible initially, then becomes visible in the correct clip--working as intended, so far... however, once I play the clip where the mesh is visible, it STAYS visible in all the other clips permanently when it should become invisible when playing another clip.
Am I missing something? Why doesn't scaling work?

Do you have a scene you can share with me?

Hey, thanks for the quick response!
After doing some more testing, I don't think that the pivots have anything to do with it--after trying every combination of parenting, constraining, baking keys, and anything else you can think of, I got the pivot error to go away but still, the mesh turns on permanently after activating the one clip where it's supposed to be visible.
I can't give you the full scene I'm working on, but I attached a test scene using the relevant assets. Animations are all set up in the trax editor. The controller mesh should be visible during the musician_Gaming_A clip but not visible in any other clips.
test_scene.zip
Any insight you could give me would be greatly appreciated!

Thanks for the scene. I'm currently abroad and not able to look at it, but as soon as I return home, I'll give it a shot.

@GregariousImpala could it be that you are scaling your object to zero scale (or near zero scale) at any point in time?

@GregariousImpala could it be that you are scaling your object to zero scale (or near zero scale) at any point in time?

My object is never scaled to 0, however, it is scaled it to 0.001 in X, Y and Z in order for it to "disappear". This is standard practice and works as intended when using FBX and OBJ exporters, as well as the Khronos glTF exporter, so I don't see why it shouldn't work using the Maya2glTF exporter.
As far as I'm aware, there is no other way to bake "visibility" into an animation.


⚠ Bofrost.es has degraded performance

In fb62aa5, Bofrost.es (https://www.bofrost.es) experienced degraded performance:

HTTP code: 200
Response time: 1504 ms

Resolved: Bofrost.es performance has improved in ef0f69c after 6 minutes.


Inject rest.Interface in k8s kv

The rest client that's coming from the fake clientset is nil. We need to inject a fake rest client in tests.

How did you find it?

Followed the stack trace. I also found that the panic didn't occur on every run. Maybe dlv made it even less probable.

24d6802
Yeah, that seem much smoother


Entire Prefix Dissappears including the vanilla minecraft one when you have a rank and delete it.

I'm not a dummy, so I've checked these

[ ] I am not using any other permission management plugin
[ ] I use the latest available version of PowerRanks
[ ] I am a human

Describe the bug
Entire Prefix Dissappears including the vanilla minecraft one when you have a rank and delete it.
To Reproduce
Steps to reproduce the behavior:

Create a rank
Set it as your rank
Delete the rank
Type in chat
5.Entire Prefix is gone, including the vanilla prefix "<>"

Expected behavior
It should 1.not remove the vanilla prefix or 2.give you the default rank
Desktop (please complete the following information):

OS: [Linux 3.1 (idk)]
Server type [Paper]
Minecraft Version [1.16.2]
PowerRanks version [e.g. 1.8]

Thanks for reporting that issue.
This is indeed a problem that I overlooked and I will fix it for the next version.
https://github.com/svenar-nl/PowerRanks/blob/e20ca8986f0ce0be13e3546b0fcf6a5d3f56dbeb/Legacy/PowerRanks/src/nl/svenar/PowerRanks/Data/Users.java#L544


thoughts about WebCAT.utils

Ok, @TomasBeuzen so i looked over webcat_utils and had a think —
For this project, we might want a single method that does it all..
Inputs:

camera of interest
a data time (or date-time range)
number of seconds between frame captures (default is 1 frame every 10 seconds… 61 frames for a 10 minute video)
the option to save or delete the raw video (default is trash the video since they are so big)
option to write a .csv

Outputs:

frames saved in a folder called jpgs
optionally, the video
optionally, a .csv with some info

the method would do this:

from date-time/ date range — generate the URLS needed, using the WebCAT format
download each of files from the URLs (vr.download_url does this)
split the video into frames (at user specified interval) and save frames in a folder called jpegs (vr.save_frames does this)
optionally delete the video
optionally write to a csv that has:

the file names of each capture,
the date time of the video,
the frame number for the capture,
the Webcat URL (This CSV will be used to join with the labeled data)

All sounds good. At the moment that functionality is almost there and would kind of work like this:
url_list = ["url1", "url2", "url3"]
vr = VidViewer()
for url in url_list:
    vr.fin = url
    vr.download_url()  # download if desired (but this is not actually necessary to split into frames)
    vr.save_frames(delta_t=600)  # save
    vr.save_metadata()  # this method doesn't exist yet but would save all the metadata of the video in a .csv file

I'll specifically note that this class never actually requires you to download the video locally to split it into frames or extract metadata. It reads the video directly from the url into a local session object which is never saved, unless you want to save it by running vr.download_url().
I did originally off-load the url generation into a class attribute, where you could pass in the station/date/time range and the url was generated automatically but it was difficult to catch all the potential errors due to bad inputs and I decided it was better to have the user explicitly pass valid urls to the class instance to download. We can talk more about this tomorrow.

I just wrote a URL generator... I'll upload in just a moment..

i think this can be closed


netpbm: 11.7.1 -> 11.7.2

Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 345261 run on x86_64-linux 1

3 packages marked as broken and skipped:

ezminc
spring
springLobby

2 packages failed to build:

cqrlog
pynac

78 packages built:

asciidoc-full
asciidoc-full-with-plugins
asciidoc-full-with-plugins.dist
asciidoc-full.dist
auto-multiple-choice
bicgl
bicpl
cantor
clevis
clevis.man
conglomerate
denemo
diffoscope
diffoscope.dist
diffoscope.man
diffoscopeMinimal
diffoscopeMinimal.dist
diffoscopeMinimal.man
disorderfs
dosbox-x
fbcat
foomatic-db-ppds
foomatic-db-ppds-withNonfreeDb
fped
frescobaldi
frescobaldi.dist
geeqie
img2pdf (python312Packages.img2pdf)
img2pdf.dist (python312Packages.img2pdf.dist)
kakounePlugins.rep
kstars
labplot
latex2html
lilypond
lilypond-unstable
lilypond-unstable-with-fonts
lilypond-with-fonts
luksmeta
manga-cli
mpvScripts.convert
mup
netpbm
netpbm.bin
netpbm.dev
nhentai
nhentai.dist
ocrmypdf (python312Packages.ocrmypdf)
ocrmypdf.dist (python312Packages.ocrmypdf.dist)
oobicpl
paperless-ngx
pcb
pdfarranger
pdfarranger.dist
pnglatex (python312Packages.pnglatex)
pnglatex.dist (python312Packages.pnglatex.dist)
polymake
protontricks
protontricks.dist
python311Packages.abjad
python311Packages.abjad.dist
python311Packages.img2pdf
python311Packages.img2pdf.dist
python311Packages.ocrmypdf
python311Packages.ocrmypdf.dist
python311Packages.pnglatex
python311Packages.pnglatex.dist
sage
sageWithDoc
singular
sng
sng.man
solfege
steamtinkerlaunch
tang
tang.man
tuxpaint
xplanet
yad

Result of nixpkgs-review pr 345261 run on x86_64-linux 1

3 packages marked as broken and skipped:

ezminc
spring
springLobby

6 packages failed to build:

cantor
cqrlog
labplot
pynac
sage
sageWithDoc

74 packages built:

asciidoc-full
asciidoc-full-with-plugins
asciidoc-full-with-plugins.dist
asciidoc-full.dist
auto-multiple-choice
bicgl
bicpl
clevis
clevis.man
conglomerate
denemo
diffoscope
diffoscope.dist
diffoscope.man
diffoscopeMinimal
diffoscopeMinimal.dist
diffoscopeMinimal.man
disorderfs
dosbox-x
fbcat
foomatic-db-ppds
foomatic-db-ppds-withNonfreeDb
fped
frescobaldi
frescobaldi.dist
geeqie
img2pdf (python312Packages.img2pdf)
img2pdf.dist (python312Packages.img2pdf.dist)
kakounePlugins.rep
kstars
latex2html
lilypond
lilypond-unstable
lilypond-unstable-with-fonts
lilypond-with-fonts
luksmeta
manga-cli
mpvScripts.convert
mup
netpbm
netpbm.bin
netpbm.dev
nhentai
nhentai.dist
ocrmypdf (python312Packages.ocrmypdf)
ocrmypdf.dist (python312Packages.ocrmypdf.dist)
oobicpl
paperless-ngx
pcb
pdfarranger
pdfarranger.dist
pnglatex (python312Packages.pnglatex)
pnglatex.dist (python312Packages.pnglatex.dist)
polymake
protontricks
protontricks.dist
python311Packages.abjad
python311Packages.abjad.dist
python311Packages.img2pdf
python311Packages.img2pdf.dist
python311Packages.ocrmypdf
python311Packages.ocrmypdf.dist
python311Packages.pnglatex
python311Packages.pnglatex.dist
singular
sng
sng.man
solfege
steamtinkerlaunch
tang
tang.man
tuxpaint
xplanet
yad

nixpkgs-review result
Generated using nixpkgs-review.
Command: nixpkgs-review pr 345261

x86_64-linux

:fast_forward: 2 packages marked as broken and skipped:

spring
springLobby

:x: 2 packages failed to build:

cqrlog
pynac

:white_check_mark: 80 packages built:

asciidoc-full
asciidoc-full-with-plugins
asciidoc-full-with-plugins.dist
asciidoc-full.dist
auto-multiple-choice
bicgl
bicpl
cantor
clevis
clevis.man
conglomerate
denemo
diffoscope
diffoscope.dist
diffoscope.man
diffoscopeMinimal
diffoscopeMinimal.dist
diffoscopeMinimal.man
disorderfs
dosbox-x
ezminc
fbcat
foomatic-db-ppds
foomatic-db-ppds-withNonfreeDb
fped
frescobaldi
frescobaldi.dist
geeqie
img2pdf (python312Packages.img2pdf)
img2pdf.dist (python312Packages.img2pdf.dist)
kakounePlugins.rep
kstars
labplot
latex2html
lilypond
lilypond-unstable
lilypond-unstable-with-fonts
lilypond-with-fonts
luksmeta
manga-cli
mpvScripts.convert
mup
netpbm
netpbm.bin
netpbm.dev
nhentai
nhentai.dist
ocrmypdf (python312Packages.ocrmypdf)
ocrmypdf.dist (python312Packages.ocrmypdf.dist)
oobicpl
paperless-ngx
pcb
pdfarranger
pdfarranger.dist
pnglatex (python312Packages.pnglatex)
pnglatex.dist (python312Packages.pnglatex.dist)
polymake
protontricks
protontricks.dist
python311Packages.abjad
python311Packages.abjad.dist
python311Packages.img2pdf
python311Packages.img2pdf.dist
python311Packages.ocrmypdf
python311Packages.ocrmypdf.dist
python311Packages.pnglatex
python311Packages.pnglatex.dist
sage
sageWithDoc
singular
sng
sng.man
solfege
steamtinkerlaunch
steamtinkerlaunch.steamcompattool
tang
tang.man
tuxpaint
xplanet
yad

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-ready-for-review/3032/4766


display 'Using environment' within 'bosh login' [fixes #347]

Shows what environment is being targetted:
$ out/bosh login
Using environment 'https://<IP_ADDRESS>:25555'

Email ():

I think it makes sense to show environment header at login.
Reminder to submit PRs here to develop branch instead of master.

@dpb587-pivotal thanks for changing base; sorry about that

story: https://www.pivotaltracker.com/story/show/152735916


Needs update for Steam download changes

Describe the bug/error
After Steam download changes the game no longer launches, instead it says it was launched incorrectly:
EliteDangerous64.exe  - Application Error
The application was unable to start correctly 0xc000007b.
Click OK to close the application.

Using Steam command line: cmd /c "MinEdLauncher.exe %command% /autorun /autoquit /edo"
Debug log

Expand to view
2023-06-19 17:51:39.695 +02:00 [INF] Elite Dangerous: Minimal Launcher - v0.8.2+4646842b
2023-06-19 17:51:39.812 +02:00 [DBG]
    Args: [|"D:\Games\Steam\steamapps\common\Elite Dangerous\EDLaunch.exe"; "/Steam";
  "/novr"; "/autorun"; "/autoquit"; "/edo"|]
    OS: Win64
    Env:

2023-06-19 17:51:39.816 +02:00 [DBG] Reading settings from 'C:\Users\Jacklul\AppData\Local\min-ed-launcher\settings.json'
2023-06-19 17:51:40.072 +02:00 [DBG] Settings: { Platform = Steam
  DisplayMode = Pancake
  AutoRun = true
  AutoQuit = true
  WatchForCrashes = false
  ProductWhitelist = seq ["edo"]
  ForceLocal = true
  CompatTool = None
  CbLauncherDir = "D:\Games\Steam\steamapps\common\Elite Dangerous"
  PreferredLanguage = None
  ApiUri = https://api.zaonce.net/
  Restart = None
  AutoUpdate = true
  CheckForLauncherUpdates = true
  MaxConcurrentDownloads = 4
  ForceUpdate = set ["FORC-FDEV-DO-1000"; "FORC-FDEV-DO-38-IN-40"]
  Processes = []
  ShutdownProcesses = []
  FilterOverrides =
   seq [[FORC-FDEV-DO-1000, edo]; [FORC-FDEV-DO-38-IN-40, edh4]]
  AdditionalProducts = []
  DryRun = false }
2023-06-19 17:51:40.087 +02:00 [INF] Elite Runtime
    Platform: Steam
    CobraBay Version: 0.4.6779.0
    Products Dir: D:\Games\Steam\steamapps\common\Elite Dangerous\Products
2023-06-19 17:51:40.105 +02:00 [DBG] Getting machine id
2023-06-19 17:51:40.124 +02:00 [INF] Logging in
2023-06-19 17:51:40.126 +02:00 [DBG] Getting remote time
2023-06-19 17:51:40.642 +02:00 [DBG] Got steam client
2023-06-19 17:51:40.642 +02:00 [DBG] Got steam pipe
2023-06-19 17:51:40.642 +02:00 [DBG] Got steam global user
2023-06-19 17:51:40.644 +02:00 [DBG] Got steam user
2023-06-19 17:51:40.644 +02:00 [DBG] Requesting steam auth ticket
2023-06-19 17:51:40.645 +02:00 [DBG] Got steam auth ticket
2023-06-19 17:51:40.649 +02:00 [DBG] Authenticating via Steam
2023-06-19 17:51:41.207 +02:00 [DBG] Successfully authenticated
2023-06-19 17:51:41.207 +02:00 [DBG] Disposing Steam resources
2023-06-19 17:51:41.207 +02:00 [DBG] Cancelling auth ticket
2023-06-19 17:51:41.208 +02:00 [DBG] closing steam
2023-06-19 17:51:41.211 +02:00 [INF] Logged in via Steam as: ***********
2023-06-19 17:51:41.212 +02:00 [DBG] Getting authorized products
2023-06-19 17:51:42.805 +02:00 [DBG] Purchases Response:
{"purchases":[{"colour":"#f07b05","filter":"ed","directory":"elite-dangerous-64","serverargs":"","gameargs":"SeasonOne","sortkey":"06","product_name":"Legacy Elite Dangerous","imageset":{"box":"http://hosting.zaonce.net/launcher_new/images/box_ed.jpg","hero":"http://hosting.zaonce.net/launcher_new/images/Elite_Dangerous_LauncherHero.png","logo":"http://hosting.zaonce.net/launcher_new/logos/ED_Refresh_Logo_Full_White.png"},"ratings":{"esrb":"teen","pegi":"7"},"game_api":"https://api3x.orerve.net","game_branch":38,"patch_notes_game_code":13,"product_sku":"FORC-FDEV-D-1010","template":"http://hosting.zaonce.net/launcher-steam/elite/en.html"},{"colour":"#f07b05","filter":"eda","directory":"elite-dangerous-64","serverargs":"","gameargs":"CQCArena","sortkey":"20","product_name":"Elite Dangerous: Arena","imageset":{"box":"http://hosting.zaonce.net/launcher_new/images/box_arena.jpg","hero":"http://hosting.zaonce.net/launcher_new/images/Elite_Dangerous_Arena_LauncherHero.png","logo":"http://hosting.zaonce.net/launcher_new/logos/ED_Refresh_Logo_Full_White.png"},"ratings":{"esrb":"teen","pegi":"7"},"game_api":"https://api3x.orerve.net","game_branch":38,"patch_notes_game_code":13,"product_sku":"FORC-FDEV-D-1012","template":"http://hosting.zaonce.net/launcher-steam/elite-dangerous-arena/en.html"},{"colour":"#f07b05","filter":"ed","directory":"COMBAT_TUTORIAL_DEMO","serverargs":"","gameargs":"","sortkey":"30","product_name":"Single Player Combat Training","imageset":{"box":"http://hosting.zaonce.net/launcher_new/images/box_ed.jpg","hero":"http://hosting.zaonce.net/launcher_new/images/Elite_Dangerous_LauncherHero.png","logo":"http://hosting.zaonce.net/launcher_new/logos/ED_Refresh_Logo_Full_White.png"},"ratings":{"esrb":"teen","pegi":"7"},"game_api":"https://api3x.orerve.net","game_branch":38,"patch_notes_game_code":13,"product_sku":"COMBAT_TUTORIAL_DEMO","template":"http://hosting.zaonce.net/launcher-steam/elite/en.html"},{"colour":"#0a8bd6","filter":"edh","directory":"elite-dangerous-64","serverargs":"","gameargs":"","sortkey":"05","product_name":"Legacy Horizons","imageset":{"box":"http://hosting.zaonce.net/launcher_new/images/box_edh.png","hero":"http://hosting.zaonce.net/launcher_new/images/Elite_Dangerous_Horizons_LauncherHero.png","logo":"http://hosting.zaonce.net/launcher_new/logos/ED_Refresh_Logo_Full_White.png"},"ratings":{"esrb":"teen","pegi":"7"},"game_api":"https://api3x.orerve.net","game_branch":38,"patch_notes_game_code":13,"product_sku":"FORC-FDEV-D-1013","template":"http://hosting.zaonce.net/launcher-steam/elite-dangerous-horizons-38/en.html"},{"colour":"#f07b05","filter":"ed","directory":"elite-dangerous-odyssey-64","serverargs":"","gameargs":"SeasonTwo","sortkey":"04","product_name":"Elite Dangerous: Horizons","imageset":{"box":"http://hosting.zaonce.net/launcher_new/images/box_edh.png","hero":"http://hosting.zaonce.net/launcher_new/images/Elite_Dangerous_Horizons_LauncherHero.png","logo":"http://hosting.zaonce.net/launcher_new/logos/ED_Refresh_Logo_Full_White.png"},"ratings":{"esrb":"teen","pegi":"7"},"game_api":"https://api.orerve.net","game_branch":40,"patch_notes_game_code":943,"product_sku":"FORC-FDEV-DO-38-IN-40","template":"http://hosting.zaonce.net/launcher-steam/elite-dangerous-horizons/en.html"},{"colour":"#0a8bd6","filter":"edh","directory":"elite-dangerous-odyssey-64","serverargs":"","gameargs":"","sortkey":"00","product_name":"Elite Dangerous: Odyssey","imageset":{"box":"http://hosting.zaonce.net/launcher_new/images/box_edo.jpg","hero":"http://hosting.zaonce.net/launcher_new/images/Elite_Dangerous_Odyssey_LauncherHero.png","logo":"http://hosting.zaonce.net/launcher_new/logos/ED_Watson_Logo_Full_White.png"},"ratings":{"esrb":"teen","pegi":"16"},"game_api":"https://api.orerve.net","game_branch":40,"patch_notes_game_code":943,"product_sku":"FORC-FDEV-DO-1000","template":"http://hosting.zaonce.net/launcher-steam/odyssey/en.html"}]}
2023-06-19 17:51:42.823 +02:00 [DBG] Authorized Products: Elite Dangerous: Odyssey,Elite Dangerous: Horizons,Legacy Horizons,Legacy Elite Dangerous,Elite Dangerous: Arena,Single Player Combat Training
2023-06-19 17:51:42.823 +02:00 [INF] Checking for updates
2023-06-19 17:51:43.544 +02:00 [DBG] Launcher is latest release 0.8.2
2023-06-19 17:51:43.552 +02:00 [DBG] Disabling 'Legacy Horizons'. Unable to find product at 'D:\Games\Steam\steamapps\common\Elite Dangerous\Products\elite-dangerous-64\VersionInfo.txt'
2023-06-19 17:51:43.552 +02:00 [DBG] Disabling 'Legacy Elite Dangerous'. Unable to find product at 'D:\Games\Steam\steamapps\common\Elite Dangerous\Products\elite-dangerous-64\VersionInfo.txt'
2023-06-19 17:51:43.553 +02:00 [DBG] Disabling 'Elite Dangerous: Arena'. Unable to find product at 'D:\Games\Steam\steamapps\common\Elite Dangerous\Products\elite-dangerous-64\VersionInfo.txt'
2023-06-19 17:51:43.553 +02:00 [DBG] Disabling 'Single Player Combat Training'. Unable to find product at 'D:\Games\Steam\steamapps\common\Elite Dangerous\Products\COMBAT_TUTORIAL_DEMO\VersionInfo.txt'
2023-06-19 17:51:45.213 +02:00 [INF] Available Products:
	Elite Dangerous: Odyssey  FORC-FDEV-DO-1000     Up to Date
	Elite Dangerous: Horizons FORC-FDEV-DO-38-IN-40 Up to Date
2023-06-19 17:51:45.487 +02:00 [INF] Launching Elite Dangerous: Odyssey
2023-06-19 17:51:45.496 +02:00 [DBG] Process: D:\Games\Steam\steamapps\common\Elite Dangerous\Products\elite-dangerous-odyssey-64\EliteDangerous64.exe "ServerToken xxx...xxx[110] xxx...xxx[1844] " /language English\\UK /steam /novr
2023-06-19 17:53:46.748 +02:00 [INF] Shutdown Elite Dangerous: Odyssey

I've disabled Steam overlay and it started working, weird.


Add type declarations to package?

Hi,
at the moment, the type declarations available for numeral seem outdated (see https://www.npmjs.com/package/@types/numeral or directly here: https://github.com/DefinitelyTyped/DefinitelyTyped/blob/types-2.0/numeral/index.d.ts). The recommended way instead of keeping separate declarations in the @types/DefinitelyTyped repo is to include them in your NPM package, anyway:
https://www.typescriptlang.org/docs/handbook/declaration-files/publishing.html
Do you see any chance for this?
Thank you!

I do not use type script so if someone would like to create a pull request with this i will add it.

Thanks for the quick reply.
Creating an initial type declaration and adding numeral to the exclusion list of definitely typed wouldn't be that hard. The problem however is that it would require monitoring this repo and apply updates continuously, whenever you change your API surface. Even worse, you wouldn't want to keep an outdated version of the declarations in your package, as this would make it really hard for people to integrate proper ones from somewhere else. So, it's not done with a single pull request, and that's some comittment I'm probably not able to keep up. Do you have any regular contributors with a TypeScript background that could take on the task?

There are typings already available:
https://github.com/DefinitelyTyped/DefinitelyTyped/blob/master/types/numeral/index.d.ts
"@types/numeral": "^2.0.4",


Make high level plan for hiring 2 developer advocates

Make high level plan for hiring 2 developer advocates

I am interested

I am bsed in India and searching for remote opportunities

@kumaradityaraj: unfortunately, the first hires will be in the US and Europe, sorry. Feel free to add me on LinkedIn, and I can share your profile in my network.
https://www.linkedin.com/in/fredericharper