Embedded Python Prototyping

Some experimental code to see if we can make Python RADAE code C-callable.

[x] try embedded example (embed1.c & mult.py)
[x] script runs some numpy and PyTorch
[x] pass in a numpy array
[x] run some init-time code (e.g. create model)
[x] return a numpy array
[x] try to Embed a RADAE example, OK to use Python for I/O  (embed_dec.py & embed_dec.c)
[x] (tmiw) try RADAE embed_dec on Windows
[x] RADAE example that uses C for I/O
[x] Refactor radae_tx to be used in Embedded framework
[x] Top level C program for radae_tx.c that works with stdin/stdout
[x] Refactor radae_rx to be used in Embedded framework, matching radae_rx.c, first pass
[x] Refactor radae_rx to be used in Embedded framework, matching radae_rx.c, second pass that allows init-time setup for ctests used on radae/radae_rx
[ ] Reconcile radae_tx and embed/radae_tx, we only need one
[ ] Reconcile radae_rx and embed/radae_rx, we only need one, and it will mean complete ctest coverage
[x] Refactor to libradae library, with top level main() test C programs
[x] ctests for Ubuntu 22 versions of embedded Python/C/libradae radae_tx & radae_rx
[ ] ctest for memory leaks
[ ] plan cleanup/refactor, either do here or log an Issue (a lot of this code is being rushed through to expedite a V1.0 Windows release)
[ ] (tmiw) Build library and top level test C program for Windows
[ ] (tmiw) Attempt to link library with freedv-gui, (Note - do we also need to link libopus?)

@tmiw - I've spent a few hours building up the features we need with the Embedded Python approach.  Returning numpy vectors is a bit messy, but works.  It's not choking when I import numpy and Torch.
I took a look at Cython, however the dosc I have read suggest it needs a top level Python module.  The embedding approach has C as the top level.  I'm adding notes to the README as I go on various approaches.

Would we still need to have the prerequisite Python packages (i.e. pytorch) installed with the embedding approach? I'm thinking based on how to get this working on Windows (which is most of the current user base).

Yes I think the Embedding approach is just like running the Python script natively in terms of packages required.  However I think it's possible to install/import Python stuff locally, so we might not need to install PyTorch/numpy etc system wide, and we could provide all the files reqd in our installer.
In the medium term, I see more and more of this code getting converted to C.

@tmiw - I have some nom-trivial RADAE code (the decoder) working with the Embedded approach on Ubuntu 22.  Could you please try the embed_dec example on Windows? Command lines in the README.md

@tmiw - I think embed/radae_tx is now working as a C program.  This is very close to being linkable with freedv-gui, we just need to convert it to a library.  I might need some of your help with doing the C building steps/cmake/portability. Might be useful to test for memory leaks too.

I had to do this from my Windows VM since the .h files from the python3.12-devel package wouldn't work (and they weren't in the "Windows embeddable package" Python provides). Even then, it couldn't seem to find the .h files for some reason:
mooneer@WinDev2407Eval:~$ wget https://github.com/mstorsjo/llvm-mingw/releases/download/20240917/llvm-mingw-20240917-ucrt-x86_64.zip
[...]
mooneer@WinDev2407Eval:~$ unzip https://github.com/mstorsjo/llvm-mingw/releases/download/20240917/llvm-mingw-20240917-ucrt-x86_64.zip
[...]
mooneer@WinDev2407Eval:~$ git clone https://github.com/drowe67/radae.git
Cloning into 'radae'...
remote: Enumerating objects: 1726, done.
remote: Counting objects: 100% (552/552), done.
remote: Compressing objects: 100% (234/234), done.
remote: Total 1726 (delta 354), reused 448 (delta 309), pack-reused 1174 (from 1)
Receiving objects: 100% (1726/1726), 82.67 MiB | 21.83 MiB/s, done.
Resolving deltas: 100% (1180/1180), done.
mooneer@WinDev2407Eval:~$ cd radae/
mooneer@WinDev2407Eval:~/radae$ git checkout dr-embed
Branch 'dr-embed' set up to track remote branch 'dr-embed' from 'origin'.
Switched to a new branch 'dr-embed'
mooneer@WinDev2407Eval:~/radae$ cd embed/
mooneer@WinDev2407Eval:~/radae/embed$ ~/llvm-mingw-20240917-ucrt-x86_64/bin/x86_64-w64-mingw32-gcc.exe -o embed1.exe embed1.c -I/mnt/c/Users/User/AppData/Local/Programs/Python/Python311/include -L/mnt/c/Users/User/AppData/Local/Programs/Python/Python311/lib -lpython3
embed1.c:2:10: fatal error: 'Python.h' file not found
    2 | #include <Python.h>
      |          ^~~~~~~~~~
1 error generated.
mooneer@WinDev2407Eval:~/radae/embed$ ls /mnt/c/Users/User/AppData/Local/Programs/Python/Python311/include/Python.h
mooneer@WinDev2407Eval:~/radae/embed$

(This is Windows Subsystem for Linux BTW.)
Actually, LLVM MinGW seems to want Windows-style paths:
mooneer@WinDev2407Eval:~/radae/embed$ ~/llvm-mingw-20240917-ucrt-x86_64/bin/x86_64-w64-mingw32-gcc.exe -o embed1.exe embed1.c -IC:/Users/User/AppData/Local/Programs/Python/Python311/include -L/mnt/c/Users/User/AppData/Local/Programs/Python/
Python311/lib -lpython3
embed1.c:4:10: fatal error: 'numpy/arrayobject.h' file not found
    4 | #include "numpy/arrayobject.h"
      |          ^~~~~~~~~~~~~~~~~~~~~
1 error generated.

After figuring out where everything was:
mooneer@WinDev2407Eval:~/radae/embed$ ~/llvm-mingw-20240917-ucrt-x86_64/bin/x86_64-w64-mingw32-gcc.exe -o embed1.exe embed1.c -IC:/Users/User/miniconda3/envs/radae/include -IC:/Users/User/miniconda3/envs/radae/Lib/site-packages/numpy/core/include -LC:/Users/User/miniconda3/envs/radae/libs -lpython3
embed1.c:47:51: warning: incompatible pointer types passing 'long *' to parameter of type 'const npy_intp *'
      (aka 'const long long *') [-Wincompatible-pointer-types]
   47 |             pValue = PyArray_SimpleNewFromData(1, &dims, NPY_FLOAT, arr_in);
      |                                                   ^~~~~
C:/Users/User/miniconda3/envs/radae/Lib/site-packages/numpy/core/include/numpy/ndarrayobject.h:121:40: note: expanded
      from macro 'PyArray_SimpleNewFromData'
  121 |         PyArray_New(&PyArray_Type, nd, dims, typenum, NULL, \
      |                                        ^~~~
embed1.c:56:51: warning: incompatible pointer types passing 'long *' to parameter of type 'const npy_intp *'
      (aka 'const long long *') [-Wincompatible-pointer-types]
   56 |             pValue = PyArray_SimpleNewFromData(1, &dims, NPY_FLOAT, arr_out);
      |                                                   ^~~~~
C:/Users/User/miniconda3/envs/radae/Lib/site-packages/numpy/core/include/numpy/ndarrayobject.h:121:40: note: expanded
      from macro 'PyArray_SimpleNewFromData'
  121 |         PyArray_New(&PyArray_Type, nd, dims, typenum, NULL, \
      |                                        ^~~~
2 warnings generated.
mooneer@WinDev2407Eval:~/radae/embed$

I had to try a few things before I could get embed1.exe to run, though:
mooneer@WinDev2407Eval:~/radae/embed$ chmod ugo+x embed1.exe
mooneer@WinDev2407Eval:~/radae/embed$ ./embed1.exe
Usage: embed1.exe pythonfile funcname mult_arg1 mult_arg2
mooneer@WinDev2407Eval:~/radae/embed$ ./embed1.exe mult multiply 2 2
import_array returned: -1
Traceback (most recent call last):
  File "\\wsl.localhost\Ubuntu\home\mooneer\radae\embed\mult.py", line 1, in <module>
    import torch
ModuleNotFoundError: No module named 'torch'
Failed to load "mult"
mooneer@WinDev2407Eval:~/radae/embed$ PYTHONPATH=C:/Users/User/miniconda3/envs/radae ./embed1.exe mult multiply 2 2
import_array returned: -1
Traceback (most recent call last):
  File "\\wsl.localhost\Ubuntu\home\mooneer\radae\embed\mult.py", line 1, in <module>
    import torch
ModuleNotFoundError: No module named 'torch'
Failed to load "mult"
mooneer@WinDev2407Eval:~/radae/embed$ PYTHONPATH=C:\\Users\\User\\miniconda3\\envs\\radae ./embed1.exe mult multiply 2 2
import_array returned: -1
Traceback (most recent call last):
  File "\\wsl.localhost\Ubuntu\home\mooneer\radae\embed\mult.py", line 1, in <module>
    import torch
ModuleNotFoundError: No module named 'torch'
Failed to load "mult"
mooneer@WinDev2407Eval:~/radae/embed$ PYTHONPATH=/mnt/c/Users/User/miniconda3/envs/radae ./embed1.exe mult multiply 2 2
import_array returned: -1
Traceback (most recent call last):
  File "\\wsl.localhost\Ubuntu\home\mooneer\radae\embed\mult.py", line 1, in <module>
    import torch
ModuleNotFoundError: No module named 'torch'
Failed to load "mult"
mooneer@WinDev2407Eval:~/radae/embed$ cd /mnt/c/Users/User/miniconda3/envs/radae
mooneer@WinDev2407Eval:/mnt/c/Users/User/miniconda3/envs/radae$ PYTHONPATH="." ~/radae/embed/embed1.exe mult multiply 2 2
import_array returned: 0
some init-time numpy
numpy vector from init: [1. 1. 1.] float64
Will compute 2 times 2
torch tensor: tensor([[1., 1., 1.]])
arr_in: [1. 2. 3.] float32
arr_out: [0. 0. 0.] float32
Result of call: 4
returned array: 1.000000 1.000000 1.000000
mooneer@WinDev2407Eval:/mnt/c/Users/User/miniconda3/envs/radae$

Anyway, this currently seems like a fair number of hoops to jump through. I wonder if it's possible to zip up the RADAE venv that conda created, unzip it onto my normal freedv-gui build environment (Ubuntu) and use LLVM MinGW from there? I'm still not fully sure how to distribute such a binary yet, however.
Going to try embed_dec now...

Hmm...
mooneer@WinDev2407Eval:~/radae/embed$ ~/llvm-mingw-20240917-ucrt-x86_64/bin/x86_64-w64-mingw32-gcc.exe -o embed_dec.exe
embed_dec.c -IC:/Users/User/miniconda3/envs/radae/include -IC:/Users/User/miniconda3/envs/radae/Lib/site-packages/numpy/
core/include -LC:/Users/User/miniconda3/envs/radae/libs -lpython3
mooneer@WinDev2407Eval:~/radae/embed$ chmod ugo+x embed_dec.exe
mooneer@WinDev2407Eval:~/radae/embed$ cd /mnt/c/Users/User/miniconda3/envs/radae
mooneer@WinDev2407Eval:/mnt/c/Users/User/miniconda3/envs/radae$ PYTHONPATH="." ~/radae/embed/embed_dec.exe embed_dec my_decode
import_array returned: 0
Traceback (most recent call last):
  File "\\wsl.localhost\Ubuntu\home\mooneer\radae\embed\embed_dec.py", line 35, in <module>
    from radae import RADAE, distortion_loss
ImportError: cannot import name 'RADAE' from 'radae' (unknown location)
Failed to load "embed_dec"
mooneer@WinDev2407Eval:/mnt/c/Users/User/miniconda3/envs/radae$

I'll have to come back to this later. It looks like we might have to copy the RADAE files into the venv conda created for this to work, but maybe there's another route.

@tmiw - good to see you got the embed_dec example working.  I saw many of the same errors when I was developing.
I find Python import paths really confusing, I resolved it on the Ubuntu 22 version with sys.append.  I run from the radae/embed dir.
Yep, a few hoops to jump through, and some careful thought required around configuration, but the benefits are huge:

Getting this into mainstream use in weeks rather than (6?) months of grant time.
Less bugs
Ease of maintenance and rapid development moving fwd
We can focus on improving the waveform, not porting already working code and chasing C<->Python bugs

Re distribution, a number of the Python scripts would have to be included (e.g. radae_tx.py, radae_rx.py, radae.py. dsp.py), plus PyTorch, numpy, etc, which are already available for Windows, although especially for Torch I imagine we might want to make sure a preferred version is installed.

Hi @tmiw - how are you going with radae_tx on win32?  I'm keen to keep this moving along, so please let me know if you get stuck.

Sorry, haven't had time the last few days. I did find out that it was getting confused between the Python 3.12 in my conda environment vs. Python 3.11 that I downloaded from python.org, so after removing the latter and trying again through a normal command prompt window:
Y:\home\mooneer\radae\embed>y:\home\mooneer\llvm-mingw-20240917-ucrt-x86_64\bin\x86_64-w64-mingw32-gcc.exe -o radae_tx.exe radae_tx.c -IC:/Users/User/miniconda3/envs/radae/include -IC:/Users/User/miniconda3/envs/radae/lib/site-packages/numpy/core/include -Lc:/Users/User/miniconda3/envs/radae/libs -lpython3
radae_tx.c:74:51: warning: incompatible pointer types passing 'long *' to parameter of type 'const npy_intp *'
      (aka 'const long long *') [-Wincompatible-pointer-types]
   74 |             pValue = PyArray_SimpleNewFromData(1, &nb_floats, NPY_FLOAT, buffer_f32);
      |                                                   ^~~~~~~~~~
C:/Users/User/miniconda3/envs/radae/lib/site-packages/numpy/core/include/numpy/ndarrayobject.h:121:40: note: expanded
      from macro 'PyArray_SimpleNewFromData'
  121 |         PyArray_New(&PyArray_Type, nd, dims, typenum, NULL, \
      |                                        ^~~~
radae_tx.c:83:51: warning: incompatible pointer types passing 'long *' to parameter of type 'const npy_intp *'
      (aka 'const long long *') [-Wincompatible-pointer-types]
   83 |             pValue = PyArray_SimpleNewFromData(1, &Nmf, NPY_CFLOAT, tx_out);
      |                                                   ^~~~
C:/Users/User/miniconda3/envs/radae/lib/site-packages/numpy/core/include/numpy/ndarrayobject.h:121:40: note: expanded
      from macro 'PyArray_SimpleNewFromData'
  121 |         PyArray_New(&PyArray_Type, nd, dims, typenum, NULL, \
      |                                        ^~~~
radae_tx.c:116:51: warning: incompatible pointer types passing 'long *' to parameter of type 'const npy_intp *'
      (aka 'const long long *') [-Wincompatible-pointer-types]
  116 |             pValue = PyArray_SimpleNewFromData(1, &Neoo, NPY_CFLOAT, eoo_out);
      |                                                   ^~~~~
C:/Users/User/miniconda3/envs/radae/lib/site-packages/numpy/core/include/numpy/ndarrayobject.h:121:40: note: expanded
      from macro 'PyArray_SimpleNewFromData'
  121 |         PyArray_New(&PyArray_Type, nd, dims, typenum, NULL, \
      |                                        ^~~~
3 warnings generated.

Y:\home\mooneer\radae\embed>set PYTHONPATH=C:\Users\User\miniconda3\envs\radae\Lib\site-packages

Y:\home\mooneer\radae\embed>radae_tx radae_tx < features.f32 >tx.f32

I get the following:

Trying again from the conda environment:
(base) C:\Users\User>conda activate radae

(radae) C:\Users\User>y:

(radae) Y:\>cd home\mooneer\radae\embed

(radae) Y:\home\mooneer\radae\embed>radae_tx radae_tx < features.f32 >tx.f32
import_array returned: 0
encoder: 937200 weights
decoder: 907764 weights
encoder: 937200 weights
decoder: 907764 weights
Rs: 33.33 Rs': 50.00 Ts': 0.020 Nsmf: 120 Ns:   4 Nc:  30 M: 160 Ncp: 32
Y:\home\mooneer\radae\embed\radae_tx.py:62: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  checkpoint = torch.load(model_name, map_location='cpu')
nb_floats: 432 Nmf: 960 Neoo: 1152
ValueError: negative dimensions are not allowed
Error setting up numpy array for buffer_f32

(radae) Y:\home\mooneer\radae\embed>

features.f32 was generated on my Mac via lpcnet_demo -features mooneer.wav features.f32 as that's the command I remembered offhand, so that might not be quite correct. Also, radae_tx took quite a long time to error out for some reason. In any case, it does seem to run once it's able to find the Python DLL file.
BTW I just realized that lpcnet_demo needs to run on raw samples, not .wav files. Still got the same error after doing that, though.

Thanks for progressing this @tmiw.  OK so this is in the code that sets up buffer_f32, so it's not the contents of features.f32, rather the C side of setting up the numpy array.
Hmm, perhaps it's interpreting nb_floats and a negative number, and yet it's printed out just fine above, and it works under Linux :thinking: Maybe a Windows/Linux type thing?  Ahh, check out that warning above:
Y:\home\mooneer\radae\embed>y:\home\mooneer\llvm-mingw-20240917-ucrt-x86_64\bin\x86_64-w64-mingw32-gcc.exe -o radae_tx.exe radae_tx.c -IC:/Users/User/miniconda3/envs/radae/include -IC:/Users/User/miniconda3/envs/radae/lib/site-packages/numpy/core/include -Lc:/Users/User/miniconda3/envs/radae/libs -lpython3
radae_tx.c:74:51: warning: incompatible pointer types passing 'long *' to parameter of type 'const npy_intp *'
      (aka 'const long long *') [-Wincompatible-pointer-types]
   74 |             pValue = PyArray_SimpleNewFromData(1, &nb_floats, NPY_FLOAT, buffer_f32);

Looks like the type is different for Windows & Linux.  Hmm, I wonder how we should handle that.

@tmiw I've sent you an invite as a collaborator - feel free to check in code to this PR to mess with things.

Y:\home\mooneer\radae\embed>y:\home\mooneer\llvm-mingw-20240917-ucrt-x86_64\bin\x86_64-w64-mingw32-gcc.exe -o radae_tx.exe radae_tx.c -IC:/Users/User/miniconda3/envs/radae/include -IC:/Users/User/miniconda3/envs/radae/lib/site-packages/numpy/core/include -Lc:/Users/User/miniconda3/envs/radae/libs -lpython3
radae_tx.c:74:51: warning: incompatible pointer types passing 'long *' to parameter of type 'const npy_intp *'
      (aka 'const long long *') [-Wincompatible-pointer-types]
   74 |             pValue = PyArray_SimpleNewFromData(1, &nb_floats, NPY_FLOAT, buffer_f32);

Looks like the type is different for Windows & Linux. Hmm, I wonder how we should handle that. Is a long long on Win32 the same length as long on Linux?

I'm guessing the calling conventions are just different enough where we can't just make the compiler force function calls on Windows like we can elsewhere. I changed nb_floats et al to npy_intp (along with bringing in the stdin/stdout stuff from #24) and I have a tx.f32 with data now.

@tmiw I've sent you an invite as a collaborator - feel free to check in code to this PR to mess with things.

Thanks!

That's great @tmiw ... we're well on the way....
How about:

I'll work on getting radae_rx to the same stage.
I'll then create a libradae that calls the core functions of tx/rx plus radae_tx, radae_rx applications that call libradae.
Would you then be able to handle getting the library building and test apps running under Windows, and maybe the appropriate cmake code to automate the build for Windows?
If that works you're ready to link with freedv-gui.

No rush on this, I appreciate you have a lot going on, just wanted to coordinate on the general plan.

That works. We should probably review what the API will look like before moving to 3 and 4 (making sure it's well documented if nothing else).

@tmiw .. thinking about this Python path problem.  Would using the PYTHONPATH env var, e.g,:
cd radae/embed
cat ../rx.f32 | PYTHONPATH="../" python3 radae_rx.py > ../features_out.f32

be better that setting the relative path in radae_rx.py code, e.g.
sys.path.append("../")
from radae import RADAE,complex_bpf,acquisition,receiver_one

So when we come to a libradae library, you would just have to set PYTHONPATH to wherever you decide to put the distributed files.  I presume we can cat multiple paths, like $PATH

@tmiw .. thinking about this Python path problem. Would using the PYTHONPATH env var, e.g,:
cd radae/embed
cat ../rx.f32 | PYTHONPATH="../" python3 radae_rx.py > ../features_out.f32

be better that setting the relative path in radae_rx.py code, e.g.
sys.path.append("../")
from radae import RADAE,complex_bpf,acquisition,receiver_one

So when we come to a libradae library, you would just have to set PYTHONPATH to wherever you decide to put the distributed files. I presume we can cat multiple paths, like $PATH

Actually, I think the usage of Python should be as hidden as possible from the user of the library (and definitely from any users of applications that use it). Maybe some sort of API function to effectively do sys.path.append() that has to be called before doing anything else with the library?

Actually, I think the usage of Python should be as hidden as possible from the user of the library (and definitely from any users of applications that use it).

Yes those are good requirements.  There are many Python application that are "installed" for end users, so I imagine this problem has been solved, e.g. all Python files installed into the same folder, or with a fixed dir structure relationship.  That way no thought is rqd by the API user or end user.

So when we come to a libradae library, you would just have to set PYTHONPATH to wherever you decide to put the distributed files. I presume we can cat multiple paths, like $PATH

Actually, I think the usage of Python should be as hidden as possible from the user of the library (and definitely from any users of applications that use it). Maybe some sort of API function to effectively do sys.path.append() that has to be called before doing anything else with the library?

@tmiw - I've temporarily gone with the PYTHONPATH approach, but hope to simply this with some refactoring that is in progress.

@tmiw - I've pushed a draft of the C API, please tell me what you think.  Some context/suggestions for scope:

The aim to have just enough for you to link with freedv-gui and get the average Windows freedv-gui use on the air with the Radio Autoencoder.
Not aimed at external API users just yet.
I've started moving the "working acronym" from RADAE->RADE as a more pronounceable name.
The idea is to push any features that don't directly address (1) into a catch all "further work" list.  We can then iterate that list for V2.
The libopus feature extraction/vocoder is external to the API, at least for this version.

@tmiw - we now have a first pass C API, that passes a few tests.  It's basically just rade_api.c (so a single file library).  Please feel free to use this for a freedv-gui integration.
Suggest you build radae_tx and radae_rx first, and make sure they work on Win32 with some sort of test (like diffing the output compared to the Ubuntu 22 versions of the same programs, see ctests).
I now have some refactoring of the internals to do, as per task list at top of PR, so will continue to work on that in this PR.  In particular I want to make sure the "embedded" version of radae_rx.py passes all the ctests I had previously set up.  We don't want any :bug: in the C API version....

I did a first pass review and left some comments. I have a bit more time so I'll see if I can get it to build on Windows.

Should the API be coded in C or C++ (e.g. native complex types)?  The Python embedding stuff is C.  I'm currently coding rade_api.c in C.

I think C is fine. If we need to, we can always create a C++ wrapper later.

Hmm, I'm trying to cross-compile for Windows (like what's done for freedv-gui) and I'm running into problems getting FindPython to use the correct libraries. One suggestion seems to be to just have special handling for that case. That's probably the easiest way to go, I think.
More concerning is that the conda environment I'm using in Windows is almost 3GB in size (693MB compressed):
mooneer@ubuntu-vm:~/radae/build_windows$ du -sh ../windows/radae/
2.7G	../windows/radae/
mooneer@ubuntu-vm:~/radae/build_windows$

PyTorch itself is ~1 GB:
mooneer@ubuntu-vm:~/radae/build_windows$ du -sh ../windows/radae/Lib/site-packages/torch
1.1G	../windows/radae/Lib/site-packages/torch
mooneer@ubuntu-vm:~/radae/build_windows$

That seems like a lot to make people download, to be honest, even if we defer the download and installation of Python etc. until the freedv-gui installer runs. Thoughts on that?

That seems like a lot to make people download, to be honest, even if we defer the download and installation of Python etc. until the freedv-gui installer runs. Thoughts on that?

So 693Mbyte is what people have to download? Yes it does seem like a lot, but perhaps that's the world we're in now (like needing more CPU).  I have to install 300 GB packages at the day job for FPGA work these days .... RIP my SSD. My OS regularly downloads ~300MB (like weekly) updates. In comparison 0.7GB is an hour of streaming TV at 480p.
How about you ask the PLT or test campaign google group how they feel?  Would the early adopter Hams we are targeting have a problem downloading say 1Gbyte to get RADAE?

@tmiw - just a small thing but every time I hit make it rebuilds the Opus stuff, even if nothing has changed.  I'm happy to raise an issue/add to V2 list, I can just e.g.  make radae_tx to build what I need to.

@tmiw - just a small thing but every time I hit make it rebuilds the Opus stuff, even if nothing has changed. I'm happy to raise an issue/add to V2 list, I can just e.g. make radae_tx to build what I need to.

Updated to avoid this. I remember running into this in freedv-gui as well and it had something to do with the "git" download method (vs. downloading from a URL).

How about you ask the PLT or test campaign google group how they feel? Would the early adopter Hams we are targeting have a problem downloading say 1Gbyte to get RADAE?

I just sent messages to both so we'll see what happens.

Question: should the Python portions of RADAE be made into a wheel? I can see that making things a bit simpler on my end (for example, by spinning up a command window during the installer's execution or first startup that basically runs "pip install radae" and brings down all its dependencies at the same time).
I'm also considering only releasing Windows binaries, at least initially. That probably means that the FreeDV builds I'll be releasing will be i.e. "2.0.0-20240928" like I used to do with test builds; I'm not sure it's a good idea to release 2.0.0 officially until all our existing platforms are supported. (I've done a bit of research on the macOS side and may have a way forward, but limited time means I can't try it right now.)

i @tmiw,

Question: should the Python portions of RADAE be made into a wheel? I can see that making things a bit simpler on my end (for example, by spinning up a command window during the installer's execution or first startup that basically runs "pip install radae" and brings down all its dependencies at the same time).

I'm happy to take you lead the most suitable packaging approach.  I'm a Python noob, and get confused with wheels, venv, and even paths tp find radae .... :shrug:

I'm also considering only releasing Windows binaries, at least initially. That probably means that the FreeDV builds I'll be releasing will be i.e. "2.0.0-20240928" like I used to do with test builds; I'm not sure it's a good idea to release 2.0.0 officially until all our existing platforms are supported. (I've done a bit of research on the macOS side and may have a way forward, but limited time means I can't try it right now.)

Yep that's what I had in mind - Windows first, other OSes later.  That'll get us 90% of our target audience.  Also non-Windows Hams will have the skills to get it running on their OSes without an installer.

PR: https://github.com/drowe67/freedv-gui/pull/748
So far I got the installer to install the files for a FreeDV copy of Python 3.12 and have a batch file (not integrated with the installer yet) that will install pip and the needed dependencies. Next step is to do the actual integration, I think, then have the C API be built during the normal FreeDV build process.

PR: drowe67/freedv-gui#748
So far I got the installer to install the files for a FreeDV copy of Python 3.12 and have a batch file (not integrated with the installer yet) that will install pip and the needed dependencies. Next step is to do the actual integration, I think, then have the C API be built during the normal FreeDV build process.

That's fine progress @tmiw.  Towards the end of this week, I will work on the "reconcile" tasks above, I want to make sure the embedded versions pass all of the tests the original versions of radae_tx/radae_tx did.  Need to move all the states into the radae_rx class to make that happen.

@tmiw - is there a way you can run some sort of basic functional test (or ctest) on Windows to make sure nothing is broken?  For example make radae_tx/radae_tx do a basic encode/decode.  The loss.py tool is Python so you might even be able to make sure the encode/decode is doing something sensible.
Just looking for some basic coverage to make sure Linux == Windows.

@tmiw - is there a way you can run some sort of basic functional test (or ctest) on Windows to make sure nothing is broken? For example make radae_tx/radae_tx do a basic encode/decode. The loss.py tool is Python so you might even be able to make sure the encode/decode is doing something sensible.
Just looking for some basic coverage to make sure Linux == Windows.

Actually, I think it's possible to have GH actions run on Windows instances. Is it worth it if I spent a couple of hours looking into creating a separate GH action for the Windows side? In theory since it's all console-based I would think it'd be possible to build and run the existing ctests just like on Ubuntu.
Failing that, maybe we can just do some basic testing using WINE and something like https://wenv.readthedocs.io/en/latest/introduction.html.

Interesting, I shouldn't be running NumPy 1.x:
C:\Program Files\FreeDV 2.0.0-devel-2024-10-02-59697a70\bin>radae_tx
model file: dummy
A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.1.1 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.

Traceback (most recent call last):import_array returned: -1

From pip.exe:
C:\Program Files\FreeDV 2.0.0-devel-2024-10-02-59697a70\bin>Scripts\pip.exe show numpy
Name: numpy
Version: 2.1.1
Summary: Fundamental package for array computing in Python
Home-page: https://numpy.org
Author: Travis E. Oliphant et al.
Author-email:
License: Copyright (c) 2005-2024, NumPy Developers.
All rights reserved.

I'll have to look into this further before I can say whether the Windows version works like the Linux one.

Even more interesting--running the .py file directly works:
C:\Program Files\FreeDV 2.0.0-devel-2024-10-03-94277141\bin>.\python.exe radae_tx.py
encoder: 937200 weights
decoder: 907764 weights
encoder: 937200 weights
decoder: 907764 weights
Rs: 33.33 Rs': 50.00 Ts': 0.020 Nsmf: 120 Ns:   4 Nc:  30 M: 160 Ncp: 32
C:\Program Files\FreeDV 2.0.0-devel-2024-10-03-94277141\bin\radae_tx.py:61: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  checkpoint = torch.load(model_name, map_location='cpu')
^Z

Oh. Somehow the conda environment I created (which I'm using to provide python3.dll/.lib) ended up with numpy 1.x. Going to build a new environment with just numpy 2.x and see how that goes.

Okay, that environment worked. On Windows:
C:\Program Files\FreeDV 2.0.0-devel-2024-10-03-94277141\bin>radae_tx.exe < z:\mooneer\Downloads\features_in.f32 > z:\mooneer\Downloads\rx.f32
model file: dummyimport_array returned: 0
encoder: 937200 weights
decoder: 907764 weights
encoder: 937200 weights
decoder: 907764 weights
Rs: 33.33 Rs': 50.00 Ts': 0.020 Nsmf: 120 Ns:   4 Nc:  30 M: 160 Ncp: 32
C:\Program Files\FreeDV 2.0.0-devel-2024-10-03-94277141\bin\radae_tx.py:61: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  checkpoint = torch.load(model_name, map_location='cpu')
n_features_in: 432 Nmf: 960 Neoo: 1152
encoder: 937200 weights
decoder: 907764 weights
encoder: 937200 weights
decoder: 907764 weights
Rs: 33.33 Rs': 50.00 Ts': 0.020 Nsmf: 120 Ns:   4 Nc:  30 M: 160 Ncp: 32
Input BPF bandwidth: 1740.000244 centre: 1475.000000
n_features_out: 432 nin_max: 1120 nin: 960

C:\Program Files\FreeDV 2.0.0-devel-2024-10-03-94277141\bin>

and subsequently on Linux:
(radae-venv) mooneer@ubuntu-vm:~/radae$ cat /mnt/hgfs/mooneer/Downloads/rx.f32 | python3 radae_rx.py model19_check3/checkpoints/checkpoint_epoch_100.pth -v 1 --auxdata > features_txs_out.f32; \
                       python3 loss.py features_in.f32 features_txs_out.f32 --loss_test 0.15 --acq_time_test 0.5
encoder: 937200 weights
decoder: 907764 weights
encoder: 937200 weights
decoder: 907764 weights
Rs: 33.33 Rs': 50.00 Ts': 0.020 Nsmf: 120 Ns:   4 Nc:  30 M: 160 Ncp: 32
/home/mooneer/radae/radae_rx.py:104: DeprecationWarning: __array__ implementation doesn't accept a copy keyword, so passing copy=False failed. __array__ must implement 'dtype' and 'copy' keyword arguments.
  p = np.array(model.p)
/home/mooneer/radae/radae_rx.py:107: DeprecationWarning: __array__ implementation doesn't accept a copy keyword, so passing copy=False failed. __array__ must implement 'dtype' and 'copy' keyword arguments.
  w = np.array(model.w)
Input BPF bandwidth: 1740.000244 centre: 1475.000000
  1 state: search     valid: 1 0  0 Dthresh:     2.44 Dtmax12:     5.32     0.00 tmax:  274 fmax:   0.00
  2 state: candidate  valid: 1 0  1 Dthresh:     5.44 Dtmax12:    10.10     0.00 tmax:  275 fmax:   0.00
  3 state: candidate  valid: 1 0  2 Dthresh:     6.10 Dtmax12:    10.56     0.00 tmax:  276 fmax:   0.00
  4 state: candidate  valid: 1 0  3 Dthresh:     6.12 Dtmax12:    10.55     0.00 tmax:  276 fmax:   0.00
  5 state: sync       valid: 1 0 25 Dthresh:     5.55 Dtmax12:    10.55     1.86 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
torch.Size([1, 982, 20]) torch.Size([1, 936, 20])
Loss between features_in.f32 and features_txs_out.f32
  loss: 0.128 start: 36 acq_time:  0.36 s
36 900
PASS
(radae-venv) mooneer@ubuntu-vm:~/radae$

Will try radae_rx.exe now.

radae_rx on Windows:
C:\Program Files\FreeDV 2.0.0-devel-2024-10-03-94277141\bin>radae_rx.exe < z:\mooneer\Downloads\rx.f32 > z:\mooneer\Down
loads\features_out.f32
model file: dummyimport_array returned: 0
encoder: 937200 weights
decoder: 907764 weights
encoder: 937200 weights
decoder: 907764 weights
Rs: 33.33 Rs': 50.00 Ts': 0.020 Nsmf: 120 Ns:   4 Nc:  30 M: 160 Ncp: 32
C:\Program Files\FreeDV 2.0.0-devel-2024-10-03-94277141\bin\radae_tx.py:61: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  checkpoint = torch.load(model_name, map_location='cpu')
n_features_in: 432 Nmf: 960 Neoo: 1152
encoder: 937200 weights
decoder: 907764 weights
encoder: 937200 weights
decoder: 907764 weights
Rs: 33.33 Rs': 50.00 Ts': 0.020 Nsmf: 120 Ns:   4 Nc:  30 M: 160 Ncp: 32
Input BPF bandwidth: 1740.000244 centre: 1475.000000
n_features_out: 432 nin_max: 1120 nin: 960
  1 state: search     valid: 1 0  0 Dthresh:     2.44 Dtmax12:     5.32     0.00 tmax:  274 fmax:   0.00
  2 state: candidate  valid: 1 0  1 Dthresh:     5.44 Dtmax12:    10.10     0.00 tmax:  275 fmax:   0.00
  3 state: candidate  valid: 1 0  2 Dthresh:     6.10 Dtmax12:    10.56     0.00 tmax:  276 fmax:   0.00
  4 state: candidate  valid: 1 0  3 Dthresh:     6.11 Dtmax12:    10.55     0.00 tmax:  276 fmax:   0.00
  5 state: sync       valid: 1 0 25 Dthresh:     5.54 Dtmax12:    10.55     1.84 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
  6 state: sync       valid: 1 0 25 Dthresh:     5.54 Dtmax12:    10.49     1.94 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
  7 state: sync       valid: 1 0 25 Dthresh:     5.53 Dtmax12:    10.49     1.19 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
  8 state: sync       valid: 1 0 25 Dthresh:     5.53 Dtmax12:    10.52     1.46 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
  9 state: sync       valid: 1 0 25 Dthresh:     5.54 Dtmax12:    10.53     1.66 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 10 state: sync       valid: 1 0 25 Dthresh:     5.53 Dtmax12:    10.63     1.02 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 11 state: sync       valid: 1 0 25 Dthresh:     5.52 Dtmax12:    10.70     1.48 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 12 state: sync       valid: 1 0 25 Dthresh:     5.53 Dtmax12:    10.65     0.85 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 13 state: sync       valid: 1 0 25 Dthresh:     5.51 Dtmax12:    10.58     2.28 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 14 state: sync       valid: 1 0 25 Dthresh:     5.50 Dtmax12:    10.52     1.19 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 15 state: sync       valid: 1 0 25 Dthresh:     5.49 Dtmax12:    10.46     1.59 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 16 state: sync       valid: 1 0 25 Dthresh:     5.50 Dtmax12:    10.48     1.52 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 17 state: sync       valid: 1 0 25 Dthresh:     5.50 Dtmax12:    10.53     1.83 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 18 state: sync       valid: 1 0 25 Dthresh:     5.49 Dtmax12:    10.57     2.39 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 19 state: sync       valid: 1 0 25 Dthresh:     5.46 Dtmax12:    10.58     2.20 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 20 state: sync       valid: 1 0 25 Dthresh:     5.47 Dtmax12:    10.63     1.43 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 21 state: sync       valid: 1 0 25 Dthresh:     5.46 Dtmax12:    10.65     1.03 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 22 state: sync       valid: 1 0 25 Dthresh:     5.47 Dtmax12:    10.56     1.06 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 23 state: sync       valid: 1 0 25 Dthresh:     5.46 Dtmax12:    10.57     0.89 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 24 state: sync       valid: 1 0 25 Dthresh:     5.44 Dtmax12:    10.62     0.81 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 25 state: sync       valid: 1 0 25 Dthresh:     5.44 Dtmax12:    10.58     1.11 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 26 state: sync       valid: 1 0 25 Dthresh:     5.44 Dtmax12:    10.66     0.95 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 27 state: sync       valid: 1 0 25 Dthresh:     5.45 Dtmax12:    10.58     0.75 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 28 state: sync       valid: 1 0 25 Dthresh:     5.45 Dtmax12:    10.51     1.67 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 29 state: sync       valid: 1 0 25 Dthresh:     5.44 Dtmax12:    10.51     1.52 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 30 state: sync       valid: 1 0 25 Dthresh:     5.43 Dtmax12:    10.51     1.95 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 31 state: sync       valid: 1 0 25 Dthresh:     5.43 Dtmax12:    10.60     1.74 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 32 state: sync       valid: 1 0 25 Dthresh:     5.44 Dtmax12:    10.61     1.68 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 33 state: sync       valid: 1 0 25 Dthresh:     5.43 Dtmax12:    10.60     2.39 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 34 state: sync       valid: 1 0 25 Dthresh:     5.44 Dtmax12:    10.60     0.34 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 35 state: sync       valid: 1 0 25 Dthresh:     5.43 Dtmax12:    10.47     0.14 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 36 state: sync       valid: 1 0 25 Dthresh:     5.43 Dtmax12:    10.46     1.84 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 37 state: sync       valid: 1 0 25 Dthresh:     5.40 Dtmax12:    10.45     1.69 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 38 state: sync       valid: 1 0 25 Dthresh:     5.39 Dtmax12:    10.46     2.52 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 39 state: sync       valid: 1 0 25 Dthresh:     5.42 Dtmax12:    10.47     1.41 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 40 state: sync       valid: 1 0 25 Dthresh:     5.42 Dtmax12:    10.43     1.20 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 41 state: sync       valid: 1 0 25 Dthresh:     5.42 Dtmax12:    10.60     1.64 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 42 state: sync       valid: 1 0 25 Dthresh:     5.40 Dtmax12:    10.66     1.70 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 43 state: sync       valid: 1 0 25 Dthresh:     5.39 Dtmax12:    10.62     0.40 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 44 state: sync       valid: 1 0 25 Dthresh:     5.40 Dtmax12:    10.62     1.38 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 45 state: sync       valid: 1 0 25 Dthresh:     5.40 Dtmax12:    10.62     2.02 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 46 state: sync       valid: 1 0 25 Dthresh:     5.38 Dtmax12:    10.51     2.08 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 47 state: sync       valid: 1 0 25 Dthresh:     5.38 Dtmax12:    10.52     1.83 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 48 state: sync       valid: 1 0 25 Dthresh:     5.38 Dtmax12:    10.58     1.18 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 49 state: sync       valid: 1 0 25 Dthresh:     5.37 Dtmax12:    10.61     1.54 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 50 state: sync       valid: 1 0 25 Dthresh:     5.37 Dtmax12:    10.57     0.67 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 51 state: sync       valid: 1 0 25 Dthresh:     5.40 Dtmax12:    10.47     2.21 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 52 state: sync       valid: 1 0 25 Dthresh:     5.40 Dtmax12:    10.53     1.16 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 53 state: sync       valid: 1 0 25 Dthresh:     5.42 Dtmax12:    10.55     1.81 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 54 state: sync       valid: 1 0 25 Dthresh:     5.44 Dtmax12:    10.59     0.39 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 55 state: sync       valid: 1 0 25 Dthresh:     5.43 Dtmax12:    10.54     0.82 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 56 state: sync       valid: 1 0 25 Dthresh:     5.43 Dtmax12:    10.48     1.74 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 57 state: sync       valid: 1 0 25 Dthresh:     5.43 Dtmax12:    10.59     0.96 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 58 state: sync       valid: 1 0 25 Dthresh:     5.44 Dtmax12:    10.60     1.18 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 59 state: sync       valid: 1 0 25 Dthresh:     5.44 Dtmax12:    10.57     1.28 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 60 state: sync       valid: 1 0 25 Dthresh:     5.44 Dtmax12:    10.51     0.83 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 61 state: sync       valid: 1 0 25 Dthresh:     5.44 Dtmax12:    10.58     1.52 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 62 state: sync       valid: 1 0 25 Dthresh:     5.45 Dtmax12:    10.64     1.38 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 63 state: sync       valid: 1 0 25 Dthresh:     5.44 Dtmax12:    10.55     2.59 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 64 state: sync       valid: 1 0 25 Dthresh:     5.44 Dtmax12:    10.42     1.31 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 65 state: sync       valid: 1 0 25 Dthresh:     5.44 Dtmax12:    10.48     0.34 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 66 state: sync       valid: 1 0 25 Dthresh:     5.44 Dtmax12:    10.62     1.76 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 67 state: sync       valid: 1 0 25 Dthresh:     5.44 Dtmax12:    10.55     2.14 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 68 state: sync       valid: 1 0 25 Dthresh:     5.42 Dtmax12:    10.52     1.21 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 69 state: sync       valid: 1 0 25 Dthresh:     5.41 Dtmax12:    10.64     0.13 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 70 state: sync       valid: 1 0 25 Dthresh:     5.39 Dtmax12:    10.66     0.46 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 71 state: sync       valid: 1 0 25 Dthresh:     5.39 Dtmax12:    10.52     0.97 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 72 state: sync       valid: 1 0 25 Dthresh:     5.38 Dtmax12:    10.36     0.28 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 73 state: sync       valid: 1 0 25 Dthresh:     5.39 Dtmax12:    10.36     0.99 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 74 state: sync       valid: 1 0 25 Dthresh:     5.39 Dtmax12:    10.50     1.38 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 75 state: sync       valid: 1 0 25 Dthresh:     5.39 Dtmax12:    10.52     1.60 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 76 state: sync       valid: 1 0 25 Dthresh:     5.38 Dtmax12:    10.57     1.55 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 77 state: sync       valid: 1 0 25 Dthresh:     5.38 Dtmax12:    10.53     0.82 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 78 state: sync       valid: 1 0 25 Dthresh:     5.36 Dtmax12:    10.38     0.61 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 79 state: sync       valid: 1 0 25 Dthresh:     5.35 Dtmax12:    10.51     2.02 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 80 state: sync       valid: 1 0 25 Dthresh:     5.34 Dtmax12:    10.68     0.55 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0
 81 state: sync       valid: 1 0 25 Dthresh:     5.33 Dtmax12:    10.63     0.63 tmax:  276 fmax:  -0.00 aux: [0 0 0] uw_err: 0

C:\Program Files\FreeDV 2.0.0-devel-2024-10-03-94277141\bin>

and on Linux:
(radae-venv) mooneer@ubuntu-vm:~/radae$ python3 loss.py features_in.f32 /mnt/hgfs/mooneer/Downloads/features_out.f32 --loss_test 0.15 --acq_time_test 0.5
torch.Size([1, 982, 20]) torch.Size([1, 924, 20])
Loss between features_in.f32 and /mnt/hgfs/mooneer/Downloads/features_out.f32
  loss: 0.128 start: 36 acq_time:  0.36 s
36 888
PASS
(radae-venv) mooneer@ubuntu-vm:~/radae$

So yeah, the Windows version looks like it works the same as the Linux one now.

Bonus screenshot showing freedv-gui code calling an API function:

Great work @tmiw, yes the 0.128 is the mean distance in feature vectors I was seeing for that sample
Loss between features_in.f32 and /mnt/hgfs/mooneer/Downloads/features_out.f32
  loss: 0.128 start: 36 acq_time:  0.36 s

We're quite lucky to have a meaningful objective measure for "speech quality" a speech system - this has traditionally been a problem, unlike say modems where you have BER.

Hi @tmiw - I just tried cmake .. from https://github.com/drowe67/radae/pull/27/commits/300f2e302298577efac78748c941a3d5433a931b:
david@deep:~/radae/build dr-embed $ cmake ..
-- The C compiler identification is GNU 11.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Will build opus with FARGAN
-- ./autogen.sh;&&;./configure;--enable-dred;--disable-shared
CMake Error at /usr/share/cmake-3.22/Modules/ExternalProject.cmake:2806 (message):
  At least one entry of URL is a path (invalid in a list)
Call Stack (most recent call first):
  /usr/share/cmake-3.22/Modules/ExternalProject.cmake:3716 (_ep_add_download_command)
  cmake/BuildOpus.cmake:12 (ExternalProject_Add)
  CMakeLists.txt:18 (include)

-- Configuring incomplete, errors occurred!
See also "/home/david/radae/build/CMakeFiles/CMakeOutput.log".

I tried rm-ing the build dir but same.  Any ideas?

Hi @tmiw - I just tried cmake .. from 300f2e3:
david@deep:~/radae/build dr-embed $ cmake ..
-- The C compiler identification is GNU 11.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Will build opus with FARGAN
-- ./autogen.sh;&&;./configure;--enable-dred;--disable-shared
CMake Error at /usr/share/cmake-3.22/Modules/ExternalProject.cmake:2806 (message):
  At least one entry of URL is a path (invalid in a list)
Call Stack (most recent call first):
  /usr/share/cmake-3.22/Modules/ExternalProject.cmake:3716 (_ep_add_download_command)
  cmake/BuildOpus.cmake:12 (ExternalProject_Add)
  CMakeLists.txt:18 (include)

-- Configuring incomplete, errors occurred!
See also "/home/david/radae/build/CMakeFiles/CMakeOutput.log".

I tried rm-ing the build dir but same. Any ideas?

I'm not sure since my most recent changes shouldn't have impacted that. My Ubuntu VM's running a newer version of CMake, though:
mooneer@ubuntu-vm:~/radae/src$ cmake --version
cmake version 3.28.3

CMake suite maintained and supported by Kitware (kitware.com/cmake).
mooneer@ubuntu-vm:~/radae/src$

BTW I have RX working in freedv-gui now with the RADE API. Hopefully I'll get some time over this coming week to implement TX.

@tmiw - I've been doing quite a bit of refactoring to make sure the embedded versions of radae_tx.py and radae_rx.py are properly ctested in commits up to  7299be. I hope it doesn't mess any of your work up.  The ctests did trap a few small bugs, which I have fixed.

@tmiw - I've been doing quite a bit of refactoring to make sure the embedded versions of radae_tx.py and radae_rx.py are properly ctested in commits up to 7299be. I hope it doesn't mess any of your work up. The ctests did trap a few small bugs, which I have fixed.

No worries, just needed to change some paths for the Windows installer on my end. TX is now working too, so I'm having Walter test out the initial Windows installer before I distribute more widely.

No worries, just needed to change some paths for the Windows installer on my end. TX is now working too, so I'm having Walter test out the initial Windows installer before I distribute more widely.

That's fine progress @tmiw :+1:  I'm pleased it was relatively painless.  I have a few more tweaks to make like file renaming.

No worries, just needed to change some paths for the Windows installer on my end. TX is now working too, so I'm having Walter test out the initial Windows installer before I distribute more widely.

That's fine progress @tmiw 👍 I'm pleased it was relatively painless. I have a few more tweaks to make like file renaming.

One note, though--I did have to pass -DLPCNET_DISABLE=1 to the freedv-gui cmake to avoid conflicting symbols during linking. Doing so disables 2020/2020B but we were planning on removing those eventually anyway, right?
(Weirdly, the conflicting symbols are only an issue when building for Windows. The normal Linux build is somehow able to handle them witihout any problems AFAIK.)

One note, though--I did have to pass -DLPCNET_DISABLE=1 to the freedv-gui cmake to avoid conflicting symbols during linking. Doing so disables 2020/2020B but we were planning on removing those eventually anyway, right?

Sure, I think it's fine to not have the legacy 2020x modes for the the RADAE V1 release of freedv-gui.  If anyone asks for them we can consider a way to keep them running.  I feel that on a practical note there is no need for 2020x any more.

One note, though--I did have to pass -DLPCNET_DISABLE=1 to the freedv-gui cmake to avoid conflicting symbols during linking. Doing so disables 2020/2020B but we were planning on removing those eventually anyway, right?

Sure, I think it's fine to not have the legacy 2020x modes for the the RADAE V1 release of freedv-gui.  If anyone asks for them we can consider a way to keep them running.  I feel that on a practical note there is no need for 2020x any more.

@tmiw - Starting from Take's comments on the test google group I've been exploring the idea of filtering the Tx signal in #30.  It's turning into a rabbit hole so I've merged what I have into this PR, and bumped the final resolution to V2.  radae_tx.py now has the --txbpf option however it has some side effects so I recommend against providing any sort of end user access to this feature (even via an option). I think the whole issue of Tx spectra can be fully resolved in V2 with a few weeks work, and when resolved won't require any options of end user control, and will work without significant degradation in PAPR.

@tmiw Ok I've finished up my cleaning up and refactoring.  When you're ready, I'm happy to merge as it's getting quite long.  We could always do any bug fixes in a new PR.  However if you are still WIP then no problems leaving this open.

@tmiw Ok I've finished up my cleaning up and refactoring. When you're ready, I'm happy to merge as it's getting quite long. We could always do any bug fixes in a new PR. However if you are still WIP then no problems leaving this open.

I think this is OK to merge. I'm still working on macOS support but the problems are on the freedv-gui side right now.


Gta 4 not launching with new dxvk versions

Hey, so I tried to use dxvk 2.0, 2.1, 2.2 but none of them works with gta 4 when i launch the game nothing happens. I can see the game running in task manager but the game window just doesnt pop up on my screen, only dxvk 1.10.3  and older works for me but I would really like to use the new versions with the game. If anyone could help me it would mean alot thanks.

Post dxvk logs.
Likely means outdated driver or old GPU that doesn't get new features anymore in the drivers.

Post dxvk logs. Likely means outdated driver or old GPU that doesn't get new features anymore in the drivers.

GTAIV_dxgi.log
is it this one dxgi.log?

Post dxvk logs. Likely means outdated driver or old GPU that doesn't get new features anymore in the drivers.

GTAIV_d3d9.log
here is also d3d9.log

idk if its my driver tho, since the new dxvk versions work fine in mafia 2.

Log just ends early.
What is your GPU and driver version

Log just ends early.
What is your GPU and driver version

Gtx 1060 3gb, driver version 535.98 (latest)

The strange thing is, the new versions did work in gta 4 before but suddenly stopped working which idk why

I would either try to downgrade to the driver version before or delete cache.
If you have a .cache file next to exe try to remove that and also look up how to clear Nvidia driver cache. Just a shot in the dark.

I would either try to downgrade to the driver version before or delete cache.
If you have a .cache file next to exe try to remove that and also look up how to clear Nvidia driver cache. Just a shot in the dark.

Thanks, will try clearing the nvidia driver cache.

I would either try to downgrade to the driver version before or delete cache.
If you have a .cache file next to exe try to remove that and also look up how to clear Nvidia driver cache. Just a shot in the dark.

Did not work unfortunately, guess im gonna have to use the older dxvk versions.

Sorry not sure what the issue could be, there was quite alot of changes for dxvk 2.0 and the game in general works with dxvk, moreso with 2.2.
Since you say it used to work i'm doubting it's dxvk itself.

Sorry not sure what the issue could be, there was quite alot of changes for dxvk 2.0 and the game in general works with dxvk, moreso with 2.2.
Since you say it used to work i'm doubting it's dxvk itself.
If you haven't already i'd suggest taking a skim through the Windows section in the wiki https://github.com/doitsujin/dxvk/wiki/Windows

Thanks for the help anyway :)

Works for me. No idea why it doesn't work for you but it's almost certainly something out of our control anyway.

Works for me. No idea why it doesn't work for you but it's almost certainly something out of our control anyway.

damn yeah i honestly dont know why it doesnt work for me, its super strange.

delete dxvk's dxgi.dll, it may cause problems in newer versions. dxvk works without it.


Fails to extract Putty 0.68 installer

Site: http://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html
Direct link: https://the.earth.li/~sgtatham/putty/latest/w32/putty-0.68-installer.msi
Error message
The following error occured extracting the MSI:

System.Runtime.InteropServices.ExternalException (0x0000064F):

Failed to create view with query: SELECT * FROM `_Streams` WHERE `Name` = 'putty.cab'

   at Microsoft.Tools.WindowsInstallerXml.Msi.View..ctor(Database db, String query)
   at Microsoft.Tools.WindowsInstallerXml.Msi.Database.OpenExecuteView(String query)
   at LessMsi.Msi.Wixtracts.ExtractCabFromPackage(Path filePath, String cabName, Database inputDatabase)
   at LessMsi.Msi.Wixtracts.CabsFromMsiToDisk(Path msi, Database msidb, String outputDir)
   at LessMsi.Msi.Wixtracts.ExtractFiles(Path msi, String outputDir, MsiFile[] filesToExtract, AsyncCallback progressCallback)
   at LessMsi.Gui.MainForm.btnExtract_Click(Object sender, EventArgs e)

Exception
See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.

************** Exception Text **************
System.NullReferenceException: Object reference not set to an instance of an object.
   at LessMsi.Gui.MainForm.fileGrid_KeyDown(Object sender, KeyEventArgs e)
   at System.Windows.Forms.Control.OnKeyDown(KeyEventArgs e)
   at System.Windows.Forms.DataGridView.OnKeyDown(KeyEventArgs e)
   at System.Windows.Forms.Control.ProcessKeyEventArgs(Message& m)
   at System.Windows.Forms.DataGridView.ProcessKeyEventArgs(Message& m)
   at System.Windows.Forms.Control.ProcessKeyMessage(Message& m)
   at System.Windows.Forms.Control.WmKeyChar(Message& m)
   at System.Windows.Forms.Control.WndProc(Message& m)
   at System.Windows.Forms.DataGridView.WndProc(Message& m)
   at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
   at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
   at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

************** Loaded Assemblies **************
mscorlib
    Assembly Version: <IP_ADDRESS>
    Win32 Version: 4.6.1087.0 built by: NETFXREL4STAGE
    CodeBase: file:///C:/Windows/Microsoft.NET/Framework/v4.0.30319/mscorlib.dll
----------------------------------------
lessmsi-gui
    Assembly Version: <IP_ADDRESS>
    Win32 Version: 1.5.1
    CodeBase: file:///C:/Soft/LessMSI/lessmsi-gui.exe
----------------------------------------
System.Windows.Forms
    Assembly Version: <IP_ADDRESS>
    Win32 Version: 4.6.1087.0 built by: NETFXREL4STAGE
    CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Windows.Forms/v4.0_<IP_ADDRESS>__b77a5c561934e089/System.Windows.Forms.dll
----------------------------------------
System
    Assembly Version: <IP_ADDRESS>
    Win32 Version: 4.6.1087.0 built by: NETFXREL4STAGE
    CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System/v4.0_<IP_ADDRESS>__b77a5c561934e089/System.dll
----------------------------------------
System.Drawing
    Assembly Version: <IP_ADDRESS>
    Win32 Version: 4.6.1087.0 built by: NETFXREL4STAGE
    CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Drawing/v4.0_<IP_ADDRESS>__b03f5f7f11d50a3a/System.Drawing.dll
----------------------------------------
System.Core
    Assembly Version: <IP_ADDRESS>
    Win32 Version: 4.6.1087.0 built by: NETFXREL4STAGE
    CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Core/v4.0_<IP_ADDRESS>__b77a5c561934e089/System.Core.dll
----------------------------------------
System.Configuration
    Assembly Version: <IP_ADDRESS>
    Win32 Version: 4.6.1087.0 built by: NETFXREL4STAGE
    CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Configuration/v4.0_<IP_ADDRESS>__b03f5f7f11d50a3a/System.Configuration.dll
----------------------------------------
System.Xml
    Assembly Version: <IP_ADDRESS>
    Win32 Version: 4.6.1087.0 built by: NETFXREL4STAGE
    CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Xml/v4.0_<IP_ADDRESS>__b77a5c561934e089/System.Xml.dll
----------------------------------------
Microsoft.GeneratedCode
    Assembly Version: <IP_ADDRESS>
    Win32 Version: 4.6.1087.0 built by: NETFXREL4STAGE
    CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Xml/v4.0_<IP_ADDRESS>__b77a5c561934e089/System.Xml.dll
----------------------------------------
wix
    Assembly Version: 2.0.2110.0
    Win32 Version: 2.0.2110.0
    CodeBase: file:///C:/Soft/LessMSI/wix.DLL
----------------------------------------
lessmsi.core
    Assembly Version: <IP_ADDRESS>
    Win32 Version: 1.5.1
    CodeBase: file:///C:/Soft/LessMSI/lessmsi.core.DLL
----------------------------------------
LessIO
    Assembly Version: <IP_ADDRESS>
    Win32 Version: 0.5.0
    CodeBase: file:///C:/Soft/LessMSI/LessIO.DLL
----------------------------------------
libmspackn
    Assembly Version: <IP_ADDRESS>
    Win32 Version: 0.8
    CodeBase: file:///C:/Soft/LessMSI/libmspackn.DLL
----------------------------------------

************** JIT Debugging **************
To enable just-in-time (JIT) debugging, the .config file for this
application or computer (machine.config) must have the
jitDebugging value set in the system.windows.forms section.
The application must also be compiled with debugging
enabled.

For example:

<configuration>
    <system.windows.forms jitDebugging="true" />
</configuration>

When JIT debugging is enabled, any unhandled exception
will be sent to the JIT debugger registered on the computer
rather than be handled by this dialog box.

Weird. No _Streams table present. Will have to investigate this...

Seems wix does something funny here. Others have noticed the missing _Streams table as noted in the following thread: http://windows-installer-xml-wix-toolset.687559.n2.nabble.com/CAB-not-being-written-to-Streams-table-td699090.html#none
Since the installer works, obviously there is some way to make an assumption and find the CAB's stream. So I'm trying to chase that down next...

Interestingly, Dark.exe which is a wix's own decompiler, also isn't able to decompile this msi and fails in exactly the same way:
C:\Users\scott\Downloads\putty-extract-wix>c:\tools\wix\dark.exe -x .\ ./putty-0.68-installer.msi
Windows Installer XML Toolset Decompiler version 3.8.1128.0
Copyright (c) Outercurve Foundation. All rights reserved.

putty-0.68-installer.msi
dark.exe : error DARK0001 : Internal MSI failure. Win32 error: 1615, MSI error: 2228, detail: C:\Users\scott\Downloads\putty-extract-wix\putty-0.68-in
staller.msi, _Streams, SELECT `Data` FROM `_Streams` WHERE `Name` = ?

Exception Type: Microsoft.Tools.WindowsInstallerXml.Msi.MsiException

Stack Trace:
  at Microsoft.Tools.WindowsInstallerXml.Msi.View..ctor(Database db, String query)
  at Microsoft.Tools.WindowsInstallerXml.Unbinder.ExtractCabinets(Output output, Database database, String databaseFile, String exportBasePath)
  at Microsoft.Tools.WindowsInstallerXml.Unbinder.UnbindDatabase(String databaseFile, OutputType outputType, String exportBasePath)
  at Microsoft.Tools.WindowsInstallerXml.Unbinder.Unbind(String file, OutputType outputType, String exportBasePath)
  at Microsoft.Tools.WindowsInstallerXml.Tools.Dark.Run(String[] args)

C:\Users\scott\Downloads\putty-extract-wix>c:\tools\wix\dark.exe -x .\ ./putty-0.68-installer.msi

Everything I read suggests that the _Streams table should be available (and maybe auto-generated upon request). However, I cannot figure out how to get it to work with this particular installer.
msi2xml also cannot read this file, but it reads others. Something goofy about this one that I cannot put my finger on. I'm putting some hacked code in a branch 78_no_streams_table. See OleStorageHelper.cs for how to get the raw OLE streams, but still that _Streams table should be accessible somehow.

@beatcracker Thank you for the feedback. It will be great if you can check out the new release and do a bit of testing. Note I've implemented a fallback that does a trick making a pretty bold assumption but it should work for most (but not all) MSIs. Fortunately it is a fallback that won't be resorted to in most cases.
If you have a moment to tinker with the other items in this release that would be much appreciated!

Thanks and sorry for the late reply. Extracting works perfectly.
> If you have a moment to tinker with the other items in this release that would be much appreciated!
It looks like the Streams tab makes some invalid assumptions about character encoding:


Search addition

Implements the SEARCH command.

I did add two tests for search request builder. I have to admit, my understanding of webdav is limited, especially to build more detailed tests like those already existent.

Thanks a lot for the contribution, I'll merge it and release a new version.

A new version is in nuget.
@usselite I also did some refactoring & minor code changes.


MINOR: Use new Admin interface instead of KafkaAdminClient class

More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.
Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Call for review @guozhangwang @bbejeck @vvcephei @ableegoldman @cadonna @abbccdda

Java 11/2.12 failed with kafka.api.SaslPlainSslEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl
Java 8 passed
Retest this please

Merged #7232 into trunk.


How to implement SchemaDirectiveVisitor like in Apollo?

I'm trying to implement directives similarly to how Apollo does it with the SchemaDirectiveVisitor class. Is this possible?
I'm trying a Type Config Decorator function to implement directives, because my schema is written in the GraphQL Type Language in a .graphql file. One issue I'm having is that I can't access schema definitions from this function (GraphQL\Type\Definition\Directive instances). I can only access directives as GraphQL\Language\AST\DirectiveNode instances - specific directive assignments to specific types. I need the actual Directive definition to be able to get the default value, for example.
It looks like if I'd build the entire schema using classes not with the Type Language I could access what I need, but that isn't an option, defining the schema in a .graphql file is a lot easier and less verbose.
Any suggestions on how to deal with this? Maybe you have suggestions on how to implement Directives similarly to how Apollo does it?
Secondary question:
I'm stitching together multiple .graphql files like this (this was suggested in another ticket):
    public function buildTypeDefsString(GraphQLApiConfig $config): string
    {
        $typeDefs = $config->getTypeDefFiles();

        /** @var ?Schema $schema */
        $schema = null;
        foreach ($typeDefs as $typeDef) {
            $typeDefContent = file_get_contents($typeDef);
            $documentNode = Parser::parse($typeDefContent);

            // Initialize schema or extend it
            $schema = $schema
                ? SchemaExtender::extend($schema, $documentNode)
                : BuildSchema::build($documentNode);
        }

        return BetterSchemaPrinter::doPrint($schema);
    }

So I do this first just to get the total combined schema, then I print it to a string and pass it into BuildSchema::build() to build it again, just so that I can attach a type config decorator. Seems inefficient - building the schema, serializing it to string and then building the schema from the string again.
Is there any way to adjust an existing schema with a type config decorator without this inefficiency?

Also - why is the fields field of the $typeConfig passed into a type config decorator a callable? If it would have the same format as described in the docs (an array of field configs), then this decorator could be used to also change field description, customize field resolver etc. But currently it's a callable and calling it manually throws all sorts of errors. Any way to get around this?

I managed to do this. Basically the SchemaDirectiveVisitor replaces the 'fields' callable with a new callable, that besides executing the original one, also does some extra things like changes field config.
Also I got around the schema combination issue by just manually extending the AST, it wasn't that difficult.


Support CLI search with empty filter

Description
Support CLI search with empty filters
Issues Resolved
Fixes #346
Check List

[x] New functionality includes testing.

[x] All tests pass ($ rake test).

[x] New functionality has been documented in the README if applicable.

[x] New functionality has been thoroughly documented in the examples (please include helpful comments).

[x] New endpoints supported are updated in the endpoints-support.md file.
[x] Changes are documented in the CHANGELOG.

@jsmartt - Thanks Jared for the contribution. I think we should update the README.md file as well to include this example which works without filter in the CLI section.

Done :)

Hi @sijeesh, yeah, the list command would be another fine place to put this functionality. It doesn't currently support showing specific attributes, so I put it in the search command because it was easier, but if you'd prefer me to add the option to show specific attributes in the list command, I can do that also.

I've created PR https://github.com/HewlettPackard/oneview-sdk-ruby/pull/348 with a better option to support the functionality I'm looking for. Closing this for now


Submission willekld lacerdar

https://github.com/SSE-Exercise-2/git-exercise

Looks good :+1:

Don't add temporary files, such as __pycache__ to version control. A ''.gitignore'' file can help preventing this.
Commit messages look good except "Solve ex. 2 task A"


the 3d model not stable（ flash of move always）

Dear juangallostra:
I can show the AR model by the code ,however, the 3d model not stable（ flash of move）.
And a problem could I get the postion and rotate of the camera?
Thanks.

Has the model drift problem been solved? Can you tell me how to do it


Add Russian translations for documentation

This pull request introduces comprehensive documentation translations and structural enhancements to support Russian-speaking users.

Translates documentation: All specified README.md files across various directories (apps, examples) are now available in Russian, providing accessibility for Russian-speaking users.
Adds new directories for Russian translations: For each translated document, a new directory named ru is created within its respective parent directory to house the Russian version of the README.md.
Improves user accessibility: By including cultural and regional adjustments in the translations, the documentation is made more relatable and understandable for the target audience.

For more details, open the Copilot Workspace session.

Moved Russian README files into new 'ru' subdirectories for better organization.

Renamed and moved README.md files for apps/chatbot-kickstarter, apps/embeddings-playground, and apps/enterprise-knowledge-retrieval into newly created ru subdirectories within their respective directories.
This change organizes the Russian translations separately, making the repository structure cleaner and more navigable for users looking for localized documentation.

For more details, open the Copilot Workspace session.

Restore original English content in README.md and add new README.md files for Russian translations in specific directories.

Restore the original English content in the root README.md file, removing Russian translations.
Add new README.md files with Russian translations in their respective directories for apps/chatbot-kickstarter and apps/embeddings-playground.
Restore original English content in examples/vector_databases/kusto/README.md and ensure Russian translations are correctly placed in their respective directories.
Ensure that no files are deleted in the process, maintaining the integrity of the repository.

For more details, open the Copilot Workspace session.


Return timestamp of latest unit for file in seconds not nanoseconds

The doc comment said that the timestamp was in seconds, but we were returning a timestamp in nanoseconds.

@swift-ci Please test


Add high-level types for musig2 primitives provided by secp256k1-kmp

This PR is based on https://github.com/ACINQ/bitcoin-kmp/pull/108 because of https://github.com/ACINQ/secp256k1-kmp/pull/93

Why did you keep the prototype musig2 implementation (in the test folders)? Shouldn't we get rid of it entirely, we want to test the "real" implementation and have no use for the prototype one anymore?

I kept it for reference but it's not supposed to be used in Musig2TestsCommon.kt I'll fix that.
I don't think it's too soon to start reviewing this PR as the API in https://github.com/ACINQ/secp256k1-kmp/pull/93 should not change much (though we still have a few weeks before it gets merged).


fix: client side SingletonRouter

What kind of change does this PR introduce?
Enables client side SingletonRouter  routing exported from next/router.
Fixes https://github.com/toomuchdesign/next-page-tester/issues/160
What is the current behaviour?
It is not possible to use client side SingletonRouter  routing exported from next/router.
Does this PR introduce a breaking change?
No
Other information:
Please check if the PR fulfills these requirements:

[x] Tests for the changes have been added
[ ] Docs have been added / updated

This is a super draft implementation that demonstrated one possible way to do it. As we have to "re-wire" a locally scoped variable in a module I don't see a better way than to somehow "hack" ourselves into the initialisation.
Open question:

Where should we call this logic? Is the current place okay?
How to handle non-jest environments? Do we even have to worry about them? I'm a bit sceptical its possible to run the library in other test runners anyway.

Coverage remained the same at 100.0% when pulling<PHONE_NUMBER>fb8180be725da58492078ab24f4e68 on fix-client-singleton-router into c053ccec56a3e7bd9918ba233a8a2ef3907e0d76 on master.

I know it might sound a bit punk but... how about not providing support SingletonRouter?  Is there anything SingletonRouter can do which useRouter can't?

I know it might sound a bit punk but... how about not providing support SingletonRouter? Is there anything SingletonRouter can do which useRouter can't?

No, but it means that projects that are using it have no change to use next-page-tester. I'm not saying its good usage (I'm against it personally) and it doesn't feel React'ish.
In a project I'm currently trying to integrate next-page-tester we use it all over the place and it would be impossible to get rid of it ATM 😢

Hi @Meemaw,
sorry for keeping you waiting, I'haven't had much time to spend in extra screen activity lately.
If I get it right, here we're patching the default next/router entry point with an object created in src/router/makeRouterMock.ts by makeRouterMock function.
Since makeRouterMock replaces the mock instance (SingletonRouter) at every run, is the newly created mock always available through next/router default entry point? Or the mock is then one available at jest.mock time?
Alternative 1
This PR seems quite all about Jest configuration. Shall we consider providing the minimum necessary API to let users mock next/router on their side?
Alternative 2
Would it be much of a hassle wrapping the pages consuming SingletonRouter with a HOC providing the hook version of Next.js router?

Alternative 1
This PR seems quite all about Jest configuration. Shall we consider providing the minimum necessary API to let users mock next/router on their side?

I was thinking about that -- and tried it but... I couldn't get it to wrong. Actually any change to the current approach breaks it 😢

Alternative 2
Would it be much of a hassle wrapping the pages consuming SingletonRouter with a HOC providing the hook version of Next.js router?

Yes, not really possible as the SingletonRouter is used across router helper utility functions that are spread all over the place ...
I mentioned this in one of the PRs some time ago: we shouldn't require users to change their code in order to adopt/use to library. They will more likely just give up and not use it.

Another punk idea. What if we tried to misuse router option to notify jest (or any other test runner) that next/router mock should be updated? After all we call it every time SingletonRouter should be updated.

Another punk idea. What if we tried to misuse router option to notify jest (or any other test runner) that next/router mock should be updated? After all we call it every time SingletonRouter should be updated.

Not sure what you mean by this 🤔

I'll try to put down something :)

Howdy @Meemaw?
I set up a demo branch to mock singletonRouter in user land by misusing router option:
https://github.com/toomuchdesign/next-page-tester/tree/fix-client-singleton-router-II
I'm sure it's improvable and I'd be happy if you validate it and take care of it. If you found it decent enough, we might merge new tests and provide this setup as docs in our /example folder.
PS. I'd like to keep commit 638f6bc, since it helped me a lot understanding what tests were doing :)
Let me know! 👊

Hey @toomuchdesign
That looks promising 🎉
Do you need to override each of the properties on SingletonRouter manually? If SingletonRouter = router doesn't work I think we can do this:
Object.assign(SingletonRouter, router);

I'd like to be wrong, but I'd say yes, you need to mutate SingletonRouter object.

I remember Object.assign(SingletonRouter, router) causing issues with the way SingletonRouter is generated (some getter fun involved).

I remember Object.assign(SingletonRouter, router) causing issues with the way SingletonRouter is generated (some getter fun involved).

Managed to get it to work in https://github.com/toomuchdesign/next-page-tester/pull/173


Filter files option

Would like to see an option to pass a filter function to be applied on each file, such as filtering on the name of file or the Stats object etc.

Sure, but why? Isn't it better to do a filter after you know what's in the directory?


Fix link to Boost.Hana in README

The link is currently broken because it points to a location within the repository.

Thanks!


Add a token key to the AP721DatabaseV1 getAllData returns

right now calling getAllData or returns all of the active tokenData for a given database. the problem is it doesnt tell you what tokenId this is associated with, so in cases where tokens have been burned this may be confusing. would just need to adjust the return value to add in the tokenId whose data is being returned. would make those functions much nicer for front ends

would also need to add back in the "does token exist check" and if not then continue into the for loop that used to be there

Closing due to protocol changes.


Add loadReadsFromSamString function into ADAMContext

add loadReadsFromSamString function into ADAMContext,
add some annotation for loadReadsFromSamString
link: https://github.com/bigdatagenomics/adam/issues/1425
I have finished it in adam-0.19.0 and update for Adam-0.21.1-SNAPSHOT today.
@fnothaft @heuermh
The first and second commits is not related about link: https://github.com/bigdatagenomics/adam/issues/1425. But I do not known how to remove it, sorry.
The two commits has been PR: https://github.com/bigdatagenomics/adam/pull/1433

Can one of the admins verify this patch?

Coverage decreased (-0.07%) to 76.325% when pulling 63b0a6e71b803c907cdd725559cc95763b35b754 on xubo245:ADAM_SNAPSHOT into 07c1982fb6fafb959b96d4c3cb83fa7edbe25721 on bigdatagenomics:master.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1853/Build result: FAILURE[...truncated 16 lines...] > /home/jenkins/git2/bin/git rev-parse origin/pr/1434/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 86777abd59cd3f38efc86bb19fd76eb11707852c # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1434/merge^{commit} # timeout=10Checking out Revision 86777abd59cd3f38efc86bb19fd76eb11707852c (origin/pr/1434/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 86777abd59cd3f38efc86bb19fd76eb11707852cFirst time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result FAILURENotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

I can not understand the FAILed information...It is different from mine.
Where error in my code?

Hi @xubo245! Similar to my comment on your other PR, this is just our source formatting check failing; you can fix this by running ./scripts/format-source. Let me know if you have any problems running that. Thank you again for the contribution, and I hope to make a review pass tomorrow!

I just run ./scripts/format-source and PR in this issue.
Is it OK?

Jenkins, test this please.

Coverage increased (+0.4%) to 76.776% when pulling 3c6a2a4311eea8a3d38e8c31d95c9d70f5fb89b2 on xubo245:ADAM_SNAPSHOT into 07c1982fb6fafb959b96d4c3cb83fa7edbe25721 on bigdatagenomics:master.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1865/Build result: FAILURE[...truncated 16 lines...] > /home/jenkins/git2/bin/git rev-parse origin/pr/1434/merge^{commit} # timeout=10 > /home/jenkins/git2/bin/git branch -a --contains 57029b136cfac82a7c1ad5f457a9f33b06e88c09 # timeout=10 > /home/jenkins/git2/bin/git rev-parse remotes/origin/pr/1434/merge^{commit} # timeout=10Checking out Revision 57029b136cfac82a7c1ad5f457a9f33b06e88c09 (origin/pr/1434/merge) > /home/jenkins/git2/bin/git config core.sparsecheckout # timeout=10 > /home/jenkins/git2/bin/git checkout -f 57029b136cfac82a7c1ad5f457a9f33b06e88c09First time build. Skipping changelog.Triggering ADAM-prb ? 2.3.0,2.11,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.3.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.11,2.0.0,centosTriggering ADAM-prb ? 2.6.0,2.10,2.0.0,centosTriggering ADAM-prb ? 2.3.0,2.10,1.6.1,centosTriggering ADAM-prb ? 2.6.0,2.11,1.6.1,centosADAM-prb ? 2.3.0,2.11,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.11,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,2.0.0,centos completed with result FAILUREADAM-prb ? 2.6.0,2.11,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.10,2.0.0,centos completed with result SUCCESSADAM-prb ? 2.3.0,2.10,1.6.1,centos completed with result SUCCESSADAM-prb ? 2.6.0,2.11,1.6.1,centos completed with result SUCCESSNotifying endpoint 'HTTP:https://webhooks.gitter.im/e/ac8bb6e9f53357bc8aa8'
Test FAILed.

Jenkins, test this please.

Coverage increased (+0.3%) to 76.672% when pulling 3c6a2a4311eea8a3d38e8c31d95c9d70f5fb89b2 on xubo245:ADAM_SNAPSHOT into 07c1982fb6fafb959b96d4c3cb83fa7edbe25721 on bigdatagenomics:master.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/1868/
Test PASSed.

The conflicts need be resolved by me？

Is it correct that the conflicts  be resolved?

Cloud it be merge?
It is second time that show conflicts  with ADAMContext.scala

@xubo245 sorry about the conflicts, we'll review and help rebase/squash commits after the 0.22.0 release is cut tomorrow.

@heuermh  Clould you help me to deal with this commits ?

This PR has more than a month...

Cloud you help me to deal with this commits ? @fnothaft @heuermh

@xubo245 I took a closer look at this last week.  Even with the commits straightened out, I feel like I'm missing the use case.  What would this provide that say the Pipe API does not?

the use case: When we invoke read mapping tools in spark, read mapping return RDD(sam format), It need to transform Adam. If write to local Filesystem, it waste time...
It has been described in https://github.com/bigdatagenomics/adam/issues/1425