TST: residue/residuez/invres/invresz don't have any tests

Unless I'm mistaken, there are no tests for these functions.  Here are some I tried, which all pass mathematically, but may produce different arrays:
From http://www.mathworks.com/help/matlab/ref/residue.html:
This is correct mathematically, but terms are in a different order:
b = [ 5, 3, -2, 7]
a = [-4, 0, 8, 3]
r, p, k = residue(b, a)

r == [-1.4167, -0.6653,  1.3320]
p == [ 1.5737, -1.1644, -0.4093]
k == -1.2500

Matches:
b, a = invres(r, p, k)
b == [-1.25, -0.75,  0.50, -1.75]
a == [ 1.00, -0.00, -2.00, -0.75]

From http://www.mathworks.com/help/matlab/math/partial-fraction-expansions.html:
SciPy gives k = [0], which is mathematically the same, but not the same:
b = [-4, 8]
a = [1, 6, 8]
r, p, k = residue(b, a)

r == [-12,  8]
p == [ -4, -2]
k == []

Matches SciPy:
b2, a2 = invres(r, p, k)
b2 == [-4, 8]
a2 == [1, 6, 8]

From http://eeweb.poly.edu/iselesni/EL6113/notes/residues.pdf:
Equivalent, but different order and [] vs [0]:
num = [4, 1]
den = [1, -1, -2]
r, p, k = residue(num, den)
r == [3, 1]
p == [2, -1]
k == []

Complex poles
Same, but different order and [0] vs []:
num = [4 ,3]
den = [2, -3.4, 1.98, -0.406]
r, p, k = residue(num, den)
r == [36.25, -18.125 +13.125j, -18.125 -13.125j]
p == [0.7, 0.5 + 0.2j, 0.5 - 0.2j]
k == []

Repeated pole
Same except [0] vs []:
num = [2, 1]
den = [1, 5, 8, 4]
r, p, k = residue(num, den)
r == [1, 3, -1]
p == [-2, -2, -1]
k == []

Matches:
r = [5, 6, -7]
p = [3, -4, -1/5]
num, den = invres(r, p, [])
num == [4, -2.8, 84.4]
den == [1, 1.2, -11.8, -2.4]

Degree of numerator greater than denominator
Equivalent, but different order:
num = [3, -1.1, 0.88, -2.396, 1.348]
den = [1, -0.7, -0.14, 0.048]
r, p, k = residue(num, den)
r == [1, 4, -3]
p == [0.8, -0.3, 0.2]
k == [3, 1]

From http://en.wikipedia.org/wiki/Partial_fraction_decomposition:
1 / (x^2 + 2x - 3) =
-1/4 / (x + 3) + 1/4 / (x - 1)
b = [1]
a = [1, 2, -3]
r, p, k = residue(b, a)
assert_allclose(r, [-1/4, 1/4])
assert_allclose(p, [-3, 1])
assert_allclose(k, [0])

(z^2 - 5) / (z^2 - 1)*(z^2 + 1) =
1/(z+1) - 1/(z-1) + 3j/2/(z+1j) - 3j/2/(z-1j)
b = [1, 0, -5]
a = polymul([1, 0, -1], [1, 0, 1])
r, p, k = residue(b, a)
assert_allclose(r, [-3j/2, 3j/2, -1, 1])
assert_allclose(p, [1j, -1j, 1, -1])
assert_allclose(k, [0])

From http://faculty.uml.edu/ahmed_abuhajar/documents/ResidueOfRepeatedRoots.pdf
F(s) = (3s^2 + 8s + 6) / (s^3 + 3s^2 + 3s + 1)
b = [3, 8, 6]
a = [1, 3, 3, 1]
r, p, k = residue(b, a)
assert_allclose(r, [3, 2, 1])
assert_allclose(p, [-1, -1, -1])
assert_allclose(k, [0])

http://www.ws.binghamton.edu/fowler/fowler personal page/EE301_files/EECE 301 Note Set 33a PFE for Z Transform.pdf
Positive powers of z digital filters:
No Direct Terms, No Repeated Roots, No Complex Roots
Different order, [] vs [0]:
r, p, k = residue([3, -1], [1, -3, 2])
r == [5, -2]
p == [2, 1]
k == []

One Direct Term, No Repeated Roots, No Complex Roots
Different order:
r, p, k = residue([2, 3, -1], [1, -3, 2])
r == [13, -4]
p == [2, 1]
k == 2

Two Direct Terms, No Repeated Roots, No Complex Roots
Different order:
r, p, k = residue([7, 2, 3, -1], [1, -3, 2])
r == [69, -11]
p == [2, 1]
k == [7, 23]

No Direct Terms, A Double Root, No Complex Roots
Repeated poles are in the correct order, but others are not:
r, p, k = residue([2, 3, -1], [1, -5, 8, -4])
r == [-2, 13, 4]
p == [2, 2, 1]
k == []

No Direct Terms, No Repeated Roots, Two Complex Roots
Different order:
r, p, k = residue([2, 3, -1], [1, -3, 4, -2])
r == [-1 - 3.5j, -1 + 3.5j, 4]
p == [1 + 1j, 1 - 1j, 1]
k == []

I see a few tests for invres in test_signaltools.py, but the rest is indeed untested.

  These are mathematically equivalent so it doesn't matter?  For repeated roots the order does match, though, which is necessary.
Octave, Matlab and DADiSP all return k=[] while SciPy returns k=[0].  Mathematically equivalent, but probably should be changed.
Octave and Matlab accept b=1 but SciPy fails unless it's written as b=[1]

Do you want to put the above tests in a PR?

I'm not going to right now, so someone else can if they want.
Another bug I noticed:
residue([6, 0], [1, 5, 8, 4])
Out[33]: (array([  6.,  12.,  -6.]), array([-2., -2., -1.]), array([ 0.]))

residue([0, 0, 6, 0], [1, 5, 8, 4])
Out[34]: (array([  6.,  12.,  -6.]), array([-2., -2., -1.]), array([ 0.]))

residue([0, 0, 6, 0], [0, 0, 0, 1, 5, 8, 4])
Out[36]: (array([ inf,  inf, -inf]), array([-2., -2., -1.]), array([ 0.]))

These should all be identical.  Same for these:
residuez([6], [1, 5, 8, 4])
Out[37]: (array([-12.,  12.,   6.]), array([-2., -2., -1.]), array([ 0.]))

residuez([6, 0, 0], [1, 5, 8, 4])
Out[38]: (array([-12.,  12.,   6.]), array([-2., -2., -1.]), array([ 0.]))

residuez([6, 0, 0], [1, 5, 8, 4, 0, 0])
Out[39]:
(array([-12.,  12.,   6.,  nan,  nan]),
 array([-2., -2., -1.,  0.,  0.]),
 array([ 0.]))

invresz(array([-9.,  3.,  6.]), array([-2., -2., -1.]), [])
Out[55]: (array([ 0.,  0.,  6.]), array([ 1.,  5.,  8.,  4.]))

invresz(array([-9.,  3.,  6.]), array([-2., -2., -1.]), [0])
Out[56]: (array([ 0.,  0.,  6.,  0.]), array([ 1.,  5.,  8.,  4.]))

(for negative powers tf like in residuez and invresz, trailing zeros don't matter, so shouldn't be returned.  for positive powers tf like in residue, invres, and tf2__, leading zeros don't matter.)
invres(array([  6.,  12.,  -6.]), array([-2., -2., -1.]), array([ 0.]))
Out[71]: (array([ 6.,  0.]), array([ 1.,  5.,  8.,  4.]))

invres(array([  6.,  12.,  -6.]), array([-2., -2., -1.]), array([ ]))
Out[72]: (array([ 6.,  0.]), array([ 1.,  5.,  8.,  4.]))

invres(array([  6.,  12.,  -6.]), array([-2., -2., -1.]), array([ 0,0]))
Out[73]: (array([ 6.,  0.]), array([ 1.,  5.,  8.,  4.]))

SciPy ordering of terms matches Octave, but not Matlab. These are mathematically equivalent so it doesn't matter? For repeated roots the order does match, though, which is necessary.

Agree, doesn't matter so leave as is.
Octave, Matlab and DADiSP all return k=[] while SciPy returns k=[0]. Mathematically equivalent, but probably should be changed.

Sounds fine to change
Octave and Matlab accept b=1 but SciPy fails unless it's written as b=[1]

Looks like this can easily be fixed by using atleast_1d.

Might indeed be good to keep this issue about test coverage, and open a new one (or multiple) for the bugs. One issue for multiple functions seems OK, would be good to group the same bugs that affect multiple functions.

Adding the label easy-fix for improving the test coverage.

Nice detective work @endolith!


Fix Incorrect password hash on documentation

Changes proposed in this pull request:

Changed the jsonrpc client readme to correct the function for hashing passwords.

Changed from: SHA0( UpperCase(username_ascii_string) + password_ascii_string)
To: SHA0( password_ascii_string + UpperCase(username_ascii_string))

Thank you.


Bug Fix: Fix ContextMenu's paste option retaining style

Description
The contextmenu's paste does not retain style such as font size, highlighting, etc due to the clipboard API not supporting application/x-lexical-format. Even if you tried to register the custom web format in the clipboard, you will have to manually get the data string for the lexical form of the text. The context menu's copy option never had any problem saving the lexical format. The problem lies with the paste option only. My solution to this was to use the html/text of what's about to be pasted and convert it into LexicalJSON with the style saved to manually save in the data during insertion. This will only be done for paste that are used through ClipboardEvent. This approach serves as a band aid to the clipboard problem and I added a TODO note. Hopefully in the future, clipboard API will give us an easier way to fix this issue.
Closes #6720
Test plan
added an e2e test case that tests the context menu successfully pasting an altered rich text
Before
https://github.com/user-attachments/assets/b82558f0-f485-4c14-aec0-36bc61feb716
After
https://github.com/user-attachments/assets/58c8a97f-5fd9-4290-a558-ade5b64e20c7

The reason why this works with JSON but not with HTML is just because the JSON is trusted, it's known to be generated with that specific lexical editor configuration (based on the namespace in the json), so the styles are taken as-is by the importJSON implementations of the respective nodes. The HTML content could be from anywhere and might not make sense in the document so it is trusted less and must be parsed much more conservatively.

Unfortunately this isn't the best approach to solving the issue, the way it should work is to have importDOM support all of the allowed styles. By default a very limited set of styles are supported for import (see applyTextFormatFromStyle in LexicalTextNode.ts) and font-size is not included.
There isn't really a very simple way to do it other than to add font-size support globally by patching applyTextFormatFromStyle which would not be backwards compatible (and not desired in many use cases, it would be unexpected to paste text from an email or spreadsheet and have it show up as a different size in many cases) or to configure the playground with a custom html import configuration that will preserve font-size (re #6770) in these cases.

ok i will try to implement this approach!


Impossible to start Netflix Conductor with dependencies using docker-compose

Hi everyone. I would appreciate some help.
I am using the current master branch. I can use docker to start conductor with db and elasticsearch in memory both using the conductor:serverAndUI container, which can be built in docker/serverAndUI, and using the docker-compose after changing both db and workflow.elasticsearch.instanceType values to memory on the docker/server/config/config.properties file.
When I use docker-compose build && docker compose up with the dynomite db and workflow.elasticsearch.instanceType=external, which is the default configuration, conductor-server exits with code 3 and throws some exceptions. I can only use the docker-compose successfully after changing the docker-compose.yaml file to have the following configuration for the elasticsearch container:
elasticsearch:
image: elasticsearch:5
command: "-Etransport.host=<IP_ADDRESS>"
I also need to insert a "sleep 30" before the command that starts conductor-server in the docker/server/bin/startup.sh file. This allows elasticsearch to start prior to conductor-server.
Finally, I also need to increase the max virtual memory chunks that can be used using the command: sudo sysctl -w vm.max_map_count=262144
However with this configuration, that is conductor-ui, conductor-server, elasticsearch and dynomite all in different docker containers in the same machine, makes conductor really slow in comparison to running it with the memory config or using the conductor:serverAndUI container. For example, with the in memory solution I can poll a task normally every second, but using dynomite, elasticsearch and conductor server and ui in different containers it takes about 10 seconds to poll a task... Is this normal or is it a consequence of my configurations?
Can you check what is the problem in building and running conductor with dynomite and elasticsearch using docker-compose? Do I need to run these systems in different machines? I don't need replication/HA for now... I just need to test the system using the production config (conductor + dynomite + elasticsearch).
Thank you for your help...

I think this issue is somehow solved on the issue #838.
Closing this issue...


Make a ^money mine (or just ^mine)

mining can get you money
you can only use the mining command once a 1 hour
i donut have a brain

uhhh the pickaxe should cost like 80 monies or so.
^money shop too (i know thats coming after like 5 days or so idfk)
^money buy ⛏
(copy of another bot loloololol)

What would mining do tho

And how would a pickaxe help that


Svelte 3 support?

Do we already support Svelte 3 preprocess option with an array of preprocessors?

Huh. It looks like this is not yet supported. I'm surprised this was missed. It needs to be added.


Children are classes not nodes

I believe that the children should point to e-classes, instead of particular node IDs.

Closing due to this being an intentional choice to represent extra information in egglog for proofs.


Mayhem Heroes Integration

Initial PR

@xansec Opened

@xansec I think this is resolved? Says its only 2 commits now


[stable/prometheus-mysql-exporter] updated gce-proxy to 1.14 & added myself to maintainers & owners

Signed-off-by: André Bauer<EMAIL_ADDRESS>
What this PR does / why we need it:

updated gce-proxy to 1.14
added myself to maintainers
added owners file

Which issue this PR fixes
(optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged)

fixes #

Special notes for your reviewer:
https://github.com/helm/charts/issues/13869
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[x] DCO signed
[x] Chart Version bumped
[x] Variables are documented in the README.md
[x] Title of the PR starts with chart name (e.g. [stable/chart])

/assign @juanchimienti
/assign @viglesiasce

/assign @koolamusic

Thanks for getting these changes in @monotek – I'm happy to be an OWNER too if that is useful

/ok-to-test
/lgtm
thanks a good luck!


[Event Request] Table 5050 "Contact" – New Event “OnCreateCustomerOnBeforeGetCustTemplate”

Could we please have a new  publisher “OnCreateCustomerOnBeforeGetCustTemplate” in Function “CreateCustomer”, Table 5050 "Contact”, including a Handle pattern?

    procedure CreateCustomer(CustomerTemplate: Code[10]) CustNo: Code[20]
    var
        Cust: Record Customer;
        CustTemplate: Record "Customer Template";
        DefaultDim: Record "Default Dimension";
        DefaultDim2: Record "Default Dimension";
        ContBusRel: Record "Contact Business Relation";
…
        RMSetup.Get();
        RMSetup.TestField("Bus. Rel. Code for Customers");

*** start new code ***
        IsHandled := false;
        OnCreateCustomerOnBeforeGetCustTemplate(CustTemplate, IsHandled);
        if not (IsHandled) then begin
*** stop new code ***
            if CustomerTemplate <> '' then
                if CustTemplate.Get(CustomerTemplate) then;
*** start new code ***
        end;
*** stop new code ***

        Clear(Cust);
        Cust.SetInsertFromContact(true);
…
*** start new code ***
    [IntegrationEvent(false, false)]
    local procedure OnCreateCustomerOnBeforeGetCustTemplate(var CustTemplate: record "Customer Template"; var IsHandled: Boolean)
    begin
    end;
*** stop new code ***

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.


Register names contain redundancy

From a quick glance, it appears that all registers are prefixed with the peripheral name. This is redundant if structs are generated from these registers. For example, one might generate the following struct for the SPI register (if the clusters are ignored, see https://github.com/espressif/svd/issues/2#issuecomment-886033289):
typedef struct {
    volatile uint32_t SPI_MEM_CMD_REG;
    volatile uint32_t SPI_MEM_ADDR_REG;
    volatile uint32_t SPI_MEM_CTRL_REG;
    volatile uint32_t SPI_MEM_CTRL1_REG;
    volatile uint32_t SPI_MEM_CTRL2_REG;
    volatile uint32_t SPI_MEM_CLOCK_REG;
    volatile uint32_t SPI_MEM_USER_REG;
    volatile uint32_t SPI_MEM_USER1_REG;
    volatile uint32_t SPI_MEM_USER2_REG;
} SPI_Type;

#define SPI ((*SPI_Type)(0x60002000))

// using it
SPI->SPI_MEM_CMD_REG = ...

Here, you can see there is a lot of redundancy in the field names. The "SPI" part is repeated, and the "_REG" part is unnecessary. It would be much more useful to have something like this:
SPI->MEM_CMD = ...

or even:
SPI->MEM.CMD = ...

In other words, a struct like this:
typedef struct {
    struct {
        volatile uint32_t CMD;
        volatile uint32_t ADDR;
        volatile uint32_t CTRL;
        volatile uint32_t CTRL1;
        volatile uint32_t CTRL2;
        volatile uint32_t CLOCK;
        volatile uint32_t USER;
        volatile uint32_t USER1;
        volatile uint32_t USER2;
    } MEM;
} SPI_Type;

In fact, it looks like this data is already present: the ESP-IDF contains two versions of each register, a "reg" and a "struct" version (see spi_mem_reg.h and spi_mem_struct.h). I would much prefer if the SVD file was more like the "struct" version rather than the "reg" version.
In practice, I'm not using C but some other language. But it has the same issue, I am using C here for ease of understanding.

Thank you for opening this issue! I agree that the register naming is quite verbose and redundant. Fortunately it should be rather easy to strip the prefixes/suffixes during the generation process, so I will look into this.

Yes, stripping prefixes/suffixes would make the SVD file a little bit easier to use.
However, what I'm asking is whether the same source as the *_struct.h header files can be used. For example, the SVD file contains entries like this:
        <register>
          <name>GPIO_FUNC1_IN_SEL_CFG_REG</name>
          <description>GPIO input function configuration register</description>
          <addressOffset>0x158</addressOffset>
          <size>32</size>
          <fields>
            ...
          </fields>
        </register>
        <register>
          <name>GPIO_FUNC2_IN_SEL_CFG_REG</name>
          <description>GPIO input function configuration register</description>
          <addressOffset>0x15c</addressOffset>
          <size>32</size>
          <fields>
            ...
          </fields>
        </register>

This makes all registers available to programs, but makes it hard to pick a GPIO number using an index.
Compare this with the following sample from gpio_struct.h:
    union {
        struct {
            uint32_t func_sel:         5;
            uint32_t sig_in_inv:       1;
            uint32_t sig_in_sel:       1;
            uint32_t reserved7:       25;
        };
        uint32_t val;
    } func_in_sel_cfg[128];

This is much easier to work with, because programs can refer to a register by an index number instead of having to use the register number directly. In the SVD file, this can be implemented as follows:
        <register>
          <dim>128</dim>
          <dimIncrement>0x04</dimIncrement>
          <name>FUNC_IN_SEL_CFG[%s]</name>
          <description>GPIO input function configuration register</description>
          <addressOffset>0x158</addressOffset>
          <fields>
            ...
          </fields>
        </register>

Okay thank you for the explanation. I understand now, sorry about that.
This seems to stem from our internal representation from which we are generating the SVD. I am unable to link to specific lines due to the size of the SVD, but you will find that there are already some registers using <dim>/<dimIndex> along with the %s placeholder. You can find these under the GPIOSD and IO_MUX peripherals. I will look into why the required information is not present for the remaining peripherals and hopefully come up with a solution.
I have also updated the generator to strip the peripheral name prefix from register/field names, and the _REG suffix from register names as mentioned above. I am currently blocked by a few internal issues but will update the public SVD once they are resolved.

I am unable to link to specific lines due to the size of the SVD, but you will find that there are already some registers using <dim>/<dimIncrement> along with the %s placeholder, so this feature is implemented in the generator.

Interesting! I found two instances: GPIOSD_SIGMADELTA%s_REG and IO_MUX_GPIO%s_REG. Somehow my parser still converts that to a list of registers, but that's definitely a bug on my side (the SVD file looks correct in that regard!).

I will look into why the required information is not present for the remaining peripherals and hopefully come up with a solution, I may be able to modify things to make it work regardless.

That would be great! SVD file quality varies a lot by vendors (there is a huge difference between Nordic and STMicroelectronics for example) but very few even provide a channel to suggest improvements. I'm really happy you are open to feedback and try to make a good SVD file - that makes supporting the ESP32C3 a lot easier!
(Off-topic sidenote: I think the ESP32-C3 will be a lot easier to support in TinyGo because it uses the RISC-V architecture, maybe I can even get WiFi/BT to work!).

I believe most of the issues raised here have been addressed in 11b28a1, thank you again for bringing these to my attention. There is still a bit of redundancy in some field names, but we're getting there.
I am closing this issue for now, if you feel it has not been resolved then it can be re-opened or additional issues can be opened.

Thanks a lot! This definitely makes the SVD file easier to use.

I found one thing in the new SVD file that's a bit odd: the TIMG peripheral still has _REG suffixes. There are some other miscellaneous _REG suffixes, but this is the only peripheral that has the suffix on all registers.

Thanks for pointing that out, I'll make a note to look into it when I'm working on this project next.

Turns out some of the register names have a _REG_REG suffix, so only one is being stripped. Easy enough, will be fixed in the next release.


Update flowchart.md

Added example to create arrow head link with text.
I had some troubles figuring out the right syntax.
Maybe for someone else useful, too?
Thankfully,
Robert

Many thanks!
I am reworking the docs as we speak so I'll add this there as well.

Thanks for your quick response,
Robert


Full Reset function in case of problems

Implementation of full reset function in case of problems -

full uninstall funcs are already implemented


Xcode 6.4 installation bug

Hi,
I trying install XCActionBar over script
curl -fsSL https://raw.githubusercontent.com/pdcgomes/XCActionBar/master/Scripts/install.sh | sh
after restart Xcode nothing happens, in Window menu doesn't appear Action bar item and key combination doesn't work.

Hi, I hadn't updated it to support Xcode 6.4's UUID yet. Should be fixed now, can you please try?

@pdcgomes, now all ok, thanks for very quick help!

:beers:


feat: add withStyles HOC

Please fill in the template below before submitting your pull request. This helps keep a consistent structure for project history.
Metadata

[x] Feature: a new feature.
[ ] Bug: a bug fix.
[ ] Refactor: a code change that neither fixes a bug nor adds a feature.
[ ] Style: changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc.).
[ ] Test: adding missing tests or correcting existing tests.
[ ] Performance: a code change that improves performance.
[ ] Documentation: changes to documentation.
[ ] Build: changes that affect the build system or external dependencies.
[ ] CI: changes to our CI configuration files and scripts.
[ ] Chore: housekeeping changes that do not affect production.

Changes
This HOC allows styles to be injected without ThemeProvider
Checklist

[x] This PR contains only a single concern and is small enough to be reviewed easily.
[x] Documentation for this issue has been added or updated with this PR.
[x] Unit tests have been added, if applicable.

Screenshots
Please include any relevant screenshots if this PR pertains to a visible UI change.
Additional Information
Please include any additional notes you feel are necessary here.

:tada: This PR is included in version 1.1.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


{Packaging} Revert BuildRpmPackageCentOS7.ArtifactName to yum

Description
#20918 changed the RPM artifact name from yum to rpm-centos7 to better reflect its content (an RPM built by CentOS 7), but this requires lots of changes in the release pipeline, as well as the RPM archive path, such as https://azurecliprod.blob.core.windows.net/archive/20220301.49/yum/azure-cli-2.34.1-1.el7.x86_64.rpm
This PR reverts the change and keep the name yum for backward compatibility, even though it is a not good name.

Packaging

Packaging


Let users add their own “model”

Support a node where users can write their own JSON API request for a model endpoint,
name their own model, and then access the model settings in a form generated from the API parameters.

This is still in the works, related to new Issues https://github.com/ianarawjo/ChainForge/issues/118 and https://github.com/ianarawjo/ChainForge/issues/116 and a replacement of the now-defunc Dalai.
I'd do it as a way to add Model Providers to the drop-down list on prompt nodes. I think the most reasonable thing to do, given CORS issues and that this is a local install feature, is to have people register a Completion Function in Python.
Then, optionally they can also register JSON for their Model Settings spec in the same place.

Hey wday-cs, one option that’s currently supported is to host your model on HuggingFace endpoints’ inference API. CF supports custom endpoints for that use case. May I ask how you are currently querying your custom model and whether there’s a particular API CF could call or use?

Thanks for pointing out the workaround. We can only use that in limited cases due to InfoSec requirements.
The schema of the important endpoint to generate predictions looks as follow:
POST /v1/predictions

{
  "target": {
    "provider": "custom_model",
    "model": "lis-text-generation-v2"
  },
  "task": {
    "type": "lis-text-generation-v2",
    "input": {
      "parameters": {
        "max_new_tokens": 32
      },
      "inputs": "Hey Dolly, could you explain the meaning of life?"
    }
  }
}

We have a standard models endpoint definitions. This is a minimal example. To support all variations it will likely need a base abstract class and require the developers to define the remaining schema for custom model integration. The inputs will be mapped to Prompts in CF.
Let me know if something is not clear.

Thanks, this is pretty clear. Having stewed on the custom models issue over the past weeks, I think the best case that would satisfy the broadest use cases would be to add a way for devs to register custom Python completion functions for their custom endpoints. I’m not a big fan of config files etc, but I don’t see another way of supporting the broadest use cases without having to add a new provider every month for some new API that came out yesterday.
So, I’d probably add a command line flag to CF, where you can specify a file with registered completion function(s). Optionally, you would also be able to define the ‘settings’ popup as JSON, if you wanted to. Then I’d add a tutorial on how to set this up to the docs.
Just for clarification, you might be interested in a ‘settings’ screen as well, if you had the ability to add your custom model endpoint?

I agree that using config files can be tricky, so we should try to avoid them if possible.
The fastest way to add a custom model would be by using a custom Python function. This way, the developer can create the logic for converting Prompts into a format the model understands, and CF will take care of the request. There would also need to be a function to change the model output into something CF can use. We can check the output if it's something more complicated.
Even though I don't like using config files, they might be needed if we want to add or change functions without restarting CF. This is good because users won't need to touch where CF is set up. The 'settings' part of the CF editor could make this easier. But we should remember that if we need to change the Python files for the custom model, we'll still need to restart CF with this setup.
Let's make sure we keep things simple and don't make this feature more complicated than it needs to be.

Update: I have this working now, with an example custom provider that calls Cohere. But there’s still plenty of UX wrinkles to iron out before release, like saving the scripts across runs. It should be coming in a few days’ time.

This issue is now solved by https://github.com/ianarawjo/ChainForge/pull/122
Let me know if you encounter any problems and we can start a new Issue.

Hi Ian,
Thanks for the update; it's really helpful 🙏. My POC for integrating our models wasn't as polished! Looking forward to seeing if the ChainForge POC with our custom models can be adopted organization-wide.


integration_tests/config_store: Fix build errors

This fixes the "no new variables on left side of :=" while running the config_store integration tests.

Likely the commands are exiting 0 even if building fails. I have a branch to investigate that I haven't pushed up yet.

The code was introduced in #112, and the integration tests weren't run as part of CI.  I suspect this is because it was a fork of the repo.

Integration tests and build examples have this at the top of their GHA yaml:
on: [push]

whereas validate (which does run) has:
on:
  push:
    branches:
      - main
  pull_request:

So we probably should add pull_request to the other two, but I haven't done any more digging than this.

Yep, I got that far and discovered the same (discussed in Slack). Thanks for confirming.
I'm doing more cleanup/efficiency work on the actions, but if smaller PRs are preferred I can do incremental changes.


60633 content - Update querySelector for article section

Please always provide the GitHub issue(s) your PR is for, as well as test URLs where your change can be observed (before and after):
Fix https://github.com/hlxsites/accenture-newsroom/issues/69
Test URLs:

Before: https://main--accenture-newsroom--hlxsites.hlx.live/news/2023/accenture-completes-acquisition-of-anser-advisory
https://main--accenture-newsroom--hlxsites.hlx.page/news/2023/accenture-song-wins-dan-wieden-titanium-grand-prix-for-tuvalu-the-first-digital-nation
After: https://60633-content--accenture-newsroom--hlxsites.hlx.live/news/2023/accenture-completes-acquisition-of-anser-advisory
https://60633-content--accenture-newsroom--hlxsites.hlx.page/news/2023/accenture-song-wins-dan-wieden-titanium-grand-prix-for-tuvalu-the-first-digital-nation

@jjjaspher this is a bit fragile selector. I think you're trying to find the section where this # # # present. May be you can just iterate over all the sections of main and choose the one before the aside-container, that's the section that ends the article, so your # # # should be there. Please update the code.

Hi @ravkiran . I updated the code and target the last section and last default-content-wrapper since there are cases of multiple default-content-wrapper. As per review points my basis is the aside-container. Please review once, thank you


docs: improve Gradle instructions

See https://blog.gradle.org/groovy-kotlin-dsl-samples
Add a better syntax for the Kotlin-DSL that also works for Android projects

I've fixed the versions and will merge.


[GRIFFIN-203] "Plaintext mode" for measure creation

Created json/yaml measure creation flow:

Added ability to view measure as yaml:

I did not add any input validation, for now only server side validation is working. I'm not quite sure if copying validation from service side is a good choice.

LGTM

Awesome!

Looks like there are still some rendering issues:

Awesome feature!
There seems an issue when zoom up the page, the textare is outside and the submit button can't be clicked/triggered.

Thank you everyone for feedback, I'm not UI guy so structure was messed little bit :) I changed it to be consistent with other pages and now it looks much better.


bug: Pulsar source failed when parsing avro message.

FYI #12871 unless using confluent schema registry, which is unlikely the case for pulsar, because pulsar has its own schema registry.

Duplicated #12871

FYI #12871 unless using confluent schema registry, which is unlikely the case for pulsar, because pulsar has its own schema registry.

Yes, according to our debug with pulsar team, this is caused by wrong api usage.

Which usage do we intend to support then? (1) No header; (2) Single-object encoding header; (3) pulsar schema registry?

FYI #12871 unless using confluent schema registry, which is unlikely the case for pulsar, because pulsar has its own schema registry.

Yes, according to our debug with pulsar team, this is caused by wrong api usage.

Which usage do we intend to support then? (1) No header; (2) Single-object encoding header; (3) pulsar schema registry?

If no schema registry, we should use single-object encoding.
I've not tested what's the behavior of pulsar java api when there is schema registry, we should test it first.


.gif images can't be processed.

In my index.html, I added a gif file with src as /images/loader.gif. Webpack runs just fine, but the gif image will not show like /images/logo.jpg. Can't figure out why.

I also tested with GIF animated pictures. Works fine for me.
Please, make sure that image path is correct. If you placed your loader.gif file in assets/images/ folder - in assets/html/index.html file you should use image tag like this: <img src="./../images/loader.gif">


Added support for background webserver for containerized applications

@Strech Opened new PR following our conversation in https://github.com/Strech/sidekiq-prometheus-exporter/pull/3.

@tameem92 Hey, thanks for the PR, I will throw some questions soon 😉

@tameem92 Can I ask you a bit of more examples to understand the problem you try to solve?
Because if we will do that it will narrow down the solution to a specific problem. Instead, I think you should do an ENTRYPOINT
Dockerfile
COPY entrypoint.sh /entrypoint.sh
ENTRYPOINT /entrypoint.sh

entrypoint.sh
#!/bin/sh

bundle exec rackup -d -p9292 -o<IP_ADDRESS> metrics.ru

metrics.ru
require 'sidekiq'
require 'sidekiq/prometheus/exporter'

Sidekiq.configure_client do |config|
  config.redis = {url: 'redis://<your-redis-host>:6379/0'}
end

run Sidekiq::Prometheus::Exporter.to_app

This will allow you to have anything you want in the background without a need to create a specific solution in the gem.
WDYT?

@Strech Let's say I have a Ruby on Rails app that I am deploying in a Docker container to AWS Elastic Beanstalk. In the Dockerfile we use one of the web servers like Puma or Unicorn. We ideally want one docker container with Rail's application running. Now in order to deploy Sidekiq in Elastic Beanstalk, I need to deploy a new container with a similar code base which only runs Sidekiq which has no web server.
Now the Sidekiq container entry point is:
# run.sh
bundle install
bundle exec sidekiq

In this case, our entry point is the Sidekiq worker but I also want a web server running in order to expose the Sidekiq metrics. This applies to environments like Elastic Beanstalk or Kuberentes. If you look at the source code of Gitlab, they had to implement something similar and add a web server just for exposing sidekiq metrics.
One way to accomplish this is to have supervisord running in the container and run two processes:

bundle exec rackup -d -p9292 -o<IP_ADDRESS> metrics.ru
bundle exec sidekiq

But I preferred the approach of having a simple function in the gem that runs the background server. With this approach, all I have to do is add that to my sidekiq config. And just run the command "bundle exec sidekiq"
Let me know what you think? Thanks.

@tameem92 in my current job we are also using Docker containers. And if I get what you want – you want to have a Sidekiq metrics within your Sidekiq container, but you ENTRYPOINT is already a Sidekiq process?
And you don't want to spawn a process from the same ENTRYPOINT file (a new web server with -d option), but instead, you want to have a co-running code in a thread within your Sidekiq? Or Rails?
Because if you want, you also can register an exporter and then if you have sidekiq/web enabled it will be there.

I understand that you don't want to run more than 1 process in a container, but what is preventing you from having code you add in you initializers/lib/etc of your Rails application if you want to run exporter there?
Can you give me an example you mention from the GitLab?
Can you clarify from 2 containers you have 1st – Rails, 2nd – Sidekiq, in which you want to add Exporter?

@Strech That's correct. I want the metrics to be available in the Sidekiq Docker Container and instead of spawning a second process in my ENTRYPOINT file. Spawning a second process in the file using "-d" is totally a valid solution, but I thought it would be good if the gem just had a function to do that. This gets rid of having an ENTRYPOINT file and you can just have a CMD at the end too.
I definitely noticed the solution of having it available on Sidekiq/web but with AWS infrastructure available with in a VPC, I don't want to have the dashboard sidekiq/web available publicly and its firewalled to whitelist a few IP addresses.
This is the issue from Gitlab: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/13082
They added options for a webrick server for exporting sidekiq metrics so its part of Gitlab and people won't have to manually add it in the ENTRYPOINT file.
To sum it up, having the function I added or adding a web server running in daemon mode in the ENTRYPOINT file are both functionally equivalent. A few advantages with having it available in the Gem and in the README.

A person using the gem won't have to do extra tweaks in their Docker configs for Sidekiq.
From the README it's clear if they have a container that doesn't run a web server they can use this solution.

If you think adding this to the ENTRYPOINT file is a better solution. I would just go ahead and update the README with that information regarding this case.

@tameem92 Thanks for your feedback! I would like to ask you a bit more.
Also it's quite confusing why I should use run from Rack even if it's Webrick.run in a Thread. And you include some middlewared I think overwhelming, like logging and so on.
Hence I think we should include it into README but not into a gem itself. It's too specific and opens many question, like: why this set of middleware, why can't I use block as Webrick allows me to do and so on.
But I'm happy to think about it one more time if we will see someone else need ❤

@Strech That makes sense. Will just add it to README and close off the PR. Thanks.

@tameem92 I will appreciate if you can enhance the readme it might help others 👏


Problem scrolling serie with multiple images using maxConcurrentMetadataRequests

Version of the viewer used : 4.2.13 (2020-07-23)
I decided to use the property maxConcurrentMetadataRequests to improve the first diplay time of the thumbnail images on the left (which really improved the first display time, thanks guys ! ) as I am using a proxy server to get the data, but I encountered a problem :
If I try to scroll down on a serie to see all its images while the metadatas are still loading, then the serie starts to act like crazy and just scroll on its own up and down.
If I try to scroll or to move the scrolling bar it doesn't change anything, and If i click on an other serie, then it goes automatically back to the faulty serie and it keeps scrolling up and down.
Once all the metadata requests are done, then I can move the scrollbar and it works normally.
I would expect the image to display a "loading wheel" if trying to access images which metadata has not been loaded yet, or at least not to act like crazy scrolling up and down without stopping.
Configuration used :
window.OHIFViewer.installViewer(
{
                routerBasename: '/Home/OhifViewerAllStudies',
                extensions: [],
                showStudyList: true,
                filterQueryParam: false,
                maxConcurrentMetadataRequests: 1,
                servers: {
                    dicomWeb: [
                      {
                        name: 'DCM4CHEE',
                        wadoUriRoot: '/api/WadoQidoAPI',
                        qidoRoot: '/api/WadoQidoAPI',
                        wadoRoot: '/api/WadoQidoAPI',
                        qidoSupportsIncludeField: true,
                        imageRendering: 'wadors',
                        thumbnailRendering: 'wadors',
                        enableStudyLazyLoad: true,
                      },
                    ],
                }
},
containerId,
componentRenderedOrUpdatedCallback);

Thanks for your help !

I think 'maxConcurrentMetadataRequests' not even work. same issue here, if the study is huge enough (many series + many instances each), make the viewer useless (until every file is recalled). it's a enormous waste of money (if your using goggle healtcare)

Please review the latest code in the master branch. This issue might have been resolved. If it persists, kindly reopen the issue with updated details.
Try viewer-dev.ohif.org instead of viewer.ohif.org
Our viewer.ohif.org is deployed from release branch while viewer-dev.ohif.org is our master  branch
Read more about branch explanations here https://docs.ohif.org/development/getting-started#developing


General fixes and improvements

Changes:

fixed methods visibility
removed redundant dependencies
renamed exception variable ($e -> $ex)
methods returns null explicitly
fixed error messages in DisableTest
added missing scenarios in tests - application configuration not found
reduced isset/empty checks complexity
added missing properties in test cases
FIX do not enable/disable development mode when exception on removing cached config

@weierophinney please have a look

Thanks, @webimpress


Heatwave alerts

Hello,
Would it be possible to also integrate alerts for heatwave and maybe fire hazards (where these are available)?

Hi, I can't find public resources for fire reports... Presidenza del Consiglio dei Ministri - Dipartimento della Protezione Civile.
For heatwave, you can refer to the phenomena. (I don't know with which criteria they are added on the map, I preferred to intercept the phenomena based on the radius from home).

Anyway, Meteoalarm seems to have access to regional data and has this kind of alert.


Fix the Android build

@swift-ci please smoke test

What is this about, are you getting compilation issues all of a sudden? Because my Android CI just cross-compiled the latest Nov. 14 trunk snapshot of SwiftPM and TSC yesterday without a problem, after applying these basic patches that were mostly just upstreamed with your approval.
I didn't know what to make of that linked Android overlay patch for TSC, as I didn't need it when building natively on Android last month or when cross-compiling the latest Termux 6.1 toolchain.

What is this about, are you getting compilation issues all of a sudden? Because my Android CI just cross-compiled the latest Nov. 14 trunk snapshot of SwiftPM and TSC yesterday without a problem, after applying these basic patches that were mostly just upstreamed with your approval.
I didn't know what to make of that linked Android overlay patch for TSC, as I didn't need it when building natively on Android last month or when cross-compiling the latest Termux 6.1 toolchain.

Yeah, Android builds are using the GNU-specific definition of strerror_r so I got a compilation failure here.
This was with the Swift 6.0.2 release and your Android SDK from https://github.com/finagolfin/swift-android-sdk/releases/tag/6.0.2, with top-of-tree main branch of SwiftTSC.
Using:
/Library/Developer/Toolchains/swift-6.0.2-RELEASE.xctoolchain/usr/bin/swift build --swift-sdk aarch64-unknown-linux-android24

Android builds are using the GNU-specific definition of strerror_r so I got a compilation failure here.

OK, this is a well-known issue with the Android build of this repo, which is remedied by passing in -Xswiftc -Xcc -Xswiftc -U_GNU_SOURCE to SwiftPM, as you can see in the CMake config here too.
Unfortunately, while this SwiftPM package manifest can add such defines easily, it could not undefine them without resorting to unsafe flags: maybe that should be fixed in SwiftPM.
After adding those flags and my linked Android overlay patch for this repo, I was able to cross-compile this repo for Android, then adding --build-tests -Xlinker -landroid-spawn allowed me to cross-compile its tests too (Android API 24 did not support the posix_spawn APIs yet, so I ship a compatibility library in my SDK bundle for that).
If you have any ideas on how to better handle these matters, I'm all ears.

Do you know where that _GNU_SOURCE definition is originating from?

I reverted the misc.swift changes for now, but I think we do still need the added import in Tracing.swift as I get a failure to find getpid otherwise.

@swift-ci please smoke test

Do you know where that _GNU_SOURCE definition is originating from?

The Swift compiler started defining it for Android for some extra math function declarations six years ago. We briefly discussed removing it years ago, but figured there were likely already Swift codebases relying on it, so left it in.

@swift-ci please smoke test

Looks reasonable, especially now that it's just adding the missing import

@swift-ci please test Linux


feat: helmfile build to produce rendered and flattened state file for GitOps

Problem
Helmfile's advanced templating features consisting of "release templates", "sub-helmfiles", "state values", "environments" are all GREAT for making your desired states DRY by avoiding repeated configs across environments.
But advanced templating has downside - less predictable results. This makes it very hard to debug and audit deployments made by Helmfile.
Helmfile Diff
helmfile diff helps by previewing changes to be made to the cluster before deploying. But you need real cluster connectivity from your CI to run diff, which is a security concern. And sometimes, you still need another level of visibility to "intermediate" result produced by Helmfile - rendered helmfile.yaml template.
Debug logs, Dry-Run, Captured Releases, ...
People usually use --helmfile --log-level=debug for that purpose. helmfile --dry-run #118 is being discussed for additional visibility and better U/X.
@bitsofinfo has proposed ability to write rendered releases to the local disk in #752. This sounds like a good idea for visibility and auditability. But one gotcha is that it may lack correctness due to how Helmfile is implemented today. Helmfile as of today applies rendered releases in-place. So if Helmfile is enhanced to write a slightly different representation of releases to disk, it's theoretically possible that what Helmfile wrote and what Helmfile applied differs, which makes the whole purpose of the feature meaningless.
GitOps
GitOps is recently emerging for better audited and reproducible deployments. But it doesn't really help Helmfile.
Highly templatized helmfile.yaml commited into Git for GitOps doesn't automatically provide useful git-diff - because the diff isn't based on rendered helmfile.yaml. You need either CI or your brain to render the helmfile.yaml template and take useful diff to give confidence that a new helmfile deployment doesn't break your system, which is hard.
Solution
I'm proposing the combination of the followings:

Embrace GitOps
GitOps-friendly Helmfile "ReleaseSet" API
helmfile build that writes a ReleaseSet
Alternative and internal RelesetSet processor(In-place processor we have today remains for backward-compatibility

A ReleaseSet is produced from the ordinary helmfile.yaml template you all have today and looks like:
apiVersion: helmfile.example.com/v1alpha1
kind: ReleaseSet
metadata:
  name: myapp
  annotations:
    approved: "false"
spec:
  releases:
  # or ns1/baz meaning both namespace and tillerNamespace set to ns1
  - name: bar
    chart: mychart
    namespace: ns1
    tillerNamespace: ns1
    dependencies:
    - somerandomlygeneratedgroupname:baz
    values:
    - inline:
        foo: bar
    secrets:
    - inline:
        # embedded secrets.yaml contents (=values encrypted by sops
    - # References to values in SSM
    - # References to values in Vault
  # or ns2/somerandomlygeneratedgroupname:baz meaning both namespace and tillerNamespace set to ns2
  - name: somerandomlygeneratedgroupname:baz
    chart: mychart2
    namespace: ns2
    tillerNamespace: ns2

All the releases, values and encrypted secrets and the selected environment included in your helmfileand its sub-helmfiles and will be rendered and flattened.
The environment will be erased hence there's no occurrence of "environment" in the build output.
We need a DAG of "dependencies"(#715 and #723) to flatten sub-helmfiles.
The typical usage of helmfile build would look as follows.
On your laptop(for testing) or CI(for production), helmfile build is run to produce a flattened ReleaseSet:
$ cd helmfile-template-repo/
$ helmfile -e prod -f helmfile.yaml.gotmpl build > ../gitops-repo/helmfile.prod.yaml

$ cd ../gitops-repo
$ git commit -m 'New deployment'
$ git push origin master

On your laptop(for testing) or CD(for production), the CD pipeline recognizes approved: "false" in your releaset set and runs helmfile diff so that you can review changes at K8s manifests level for approval:
$ helmfile -f helmfile.prod.yaml diff | some-cmd-to-send-github-comment

You approve it by commenting on the PR to the "helmfile-template-repo" with /approve.
On your laptop(for testing), or CI triggered via a GitHub pull request comment(for production), some command to modify the ReleasetSet to have a approved: "true" annotation:
# `helmfile approve` is just a nice-to-have here. You can use yq or sed instead
$ helmfile -f helmfile.prod.yaml approve > helmfile.prod.approved.yaml
$ cp helmfile.prod.approved.yaml helmfile.prod.yaml
$ git commit -m 'Automated approval' && git push

On your laptop or CD, the CD pipeline recognizes approved: "true" and therefore runs helmfile apply:
$ git clone git@github.com:yourorg/yourgitopsrepo.git -b your-pr
$ helmfile -f helmfile.prod.yaml apply

This way, you have complete audit logs of what's being rendered and deployed by Helmfile in Git.

cc/ @sstarcher @osterman @bitsofinfo @Stono I'm considering to enhance Helmfile this way for better auditing of Helmfile deployments with GitOps, and better security by optionally limiting CI to have no access to production clusters and exposing no github webhook endpoint.
WDYT?

I dig it!!! Solid idea.
Some questions:

The ReleaseSet example above; I assume this new format AND the existing format for a helmfile could both still be consumable by helmfile -f <releaseset format | non-releaseset format> Would absolutely have to have this compatibility or flexibility to use either. Just double checking your 4th bullet above :)

What I like about this is that it does not impose/force any particular flow. In my use case I could still just use build to give me this auditable record, what I or anyone else does with it is up to the operator of helmfile. This is important.

a ReleaseSet's annotations should be customizable. I.E. the name of the annotation keys and value(s). I don't see a need for complexity here, just some simple k=v,k=v kind of capability would be good here. Also perhaps consider adding labels, but this might not be needed as annotations are more appropriate, as my understanding of the k8s spec is labels is reserved maybe for actual objects created within k8s by a manifest, which we clearly are not doing here.

helmfile -e prod -f helmfile.yaml.gotmpl --set-annotations a1=y,a2=z build

Another thought, it would be nice to be able to add into an annotations a auto-generated string representation of the arguments passed to helmfile build that produced this result. This would be very helpful for my use case where the "generated" releases are very dynamic.

...
annotations:
  helmfilecmdargs: "-f helmfile.yaml -e env --state-values-set x=y ... etc build"

Just a note, currently I have some hacky tooling that is manually parsing out the debug produced releases: block for my auditing purposes. This would most definitely replace that.  Secondly I am also heavily using the --quiet & template option to get my the k8s yaml to store as well. I would still have that need, but its pretty un-related to this.

Overall, these are just my initial thoughts, but I'm excited about this, great idea!

I fully agree that if we can remove or make all templating of helmfiles first-class citizens it would be much better and easier to handle and debug.  Templating of a helmfile itself is essentially a crutch and a dirty work around.
Embedded secrets.yaml could result in a massive release set.  It would likely be better to create it as a separate object and to reference them as refs.
Everything in ReleaseSet should be cleanly diff-able.

The ReleaseSet example above; I assume this new format AND the existing format for a helmfile could both still be consumable by helmfile -f <releaseset format | non-releaseset format> Would absolutely have to have this compatibility or flexibility to use either. Just double checking your 4th bullet above :)

Your assumption is correct :) Btw, implementation-wise it should be a matter of Helmfile checking existence of apiVersion and Kind, which should be easy enough.

What I like about this is that it does not impose/force any particular flow. In my use case I could still just use build to give me this auditable record, what I or anyone else does with it is up to the operator of helmfile. This is important.

Exactly. I believe that's important.

Another thought, it would be nice to be able to add into an annotations a auto-generated string representation of the arguments passed to helmfile build that produced this result

Seems like a valid feature request! I like it as much as I like run annotation added to pods created by kubectl run :)

Just a note, currently I have some hacky tooling that is manually parsing out the debug produced releases: block for my auditing purposes. This would most definitely replace that.

Yep, I believe so too.

Secondly I am also heavily using the --quiet & template option to get my the k8s yaml to store as well. I would still have that need, but its pretty un-related to this.

Yep. You'll still need it.
FWIW, I'm currently forwarding helmfile diff outputs to the pull request triggered the helmfile deployment for the same purpose as you(I think)

I fully agree that if we can remove or make all templating of helmfiles first-class citizens it would be much better and easier to handle and debug.  Templating of a helmfile itself is essentially a crutch and a dirty work around.

I do envision that, after this enhancement, helmfile can be used in combination with other templating tool based on jsonnet, Lua, ECMAScript or anything else.
The minimum feature of Helmfile will be a solid ReleaseSet reconciler that supports DAG, hooks, and many pluggable values/secrets backends.

Embedded secrets.yaml could result in a massive release set.  It would likely be better to create it as a separate object and to reference them as refs.
Everything in ReleaseSet should be cleanly diff-able.

I agree, but I'm not yet sure how it should fully work.
Anything other than secrets.yaml collocated in the same repo as helmfile.yaml will remain references to actual secrets. Based on https://github.com/roboll/helmfile/issues/745#issuecomment-510889406 it may look like:
spec:
  releases:
  # or ns1/baz meaning both namespace and tillerNamespace set to ns1
  - name: bar
    chart: mychart
    set:
    - values:
        foo: bar
    - valuesFrom:
        sops:
          inline: |
            #embedded secrets.yaml #1
    - valuesFrom:
        sops:
          inline: |
            #embedded secrets.yaml #2
    - valuesFrom:
        sops:
          url: github.com/yourorg/yourrepo//yourenv@secrets.yaml?ref=<commit id>
    - valuesFrom:
         ssm: #ssm path and anything needed for importing kvs from ssm
    - valuesFrom:
         vault: #vault path and anything needed for importing kvs from vault

Perhaps small secrets.yamls can be embedded with set[].valuesFrom.sops.inline as shown in the above, and bigger secrets.yaml should be referenced by urls.
Maybe it's fine to omit the ability to embed small secrets.yaml and make everything imported as references to secrets.yaml's.
Then the remaining problem is how Helmfile should get the URL of a secrets.yaml.
Let's say you have a "source" repo that contains helmfile.yaml and secrets.yaml where helmfile.yaml looks like:
releases:
- name: myapp
  chart: mychart
  secrets:
  - secrets.yaml

helmfile build --name myapp would produce the below:
apiVersion: helmfile.example.com/v1alpha1
kind: ReleaseSet
metadata:
  name: myapp
spec:
  releases:
  - name: myapp
    chart: mychart
    set:
    - valuesFrom:
        sops:
           url: github.com/yourorg/yourrepo//yourenv@secrets.yaml?ref=<commit id>

Can we safely assume that helmfile build is always run within a Git repository with a origin correctly set so that Helmfile is able to run git remote show origin to get the github.com/yourorg/yourrepo/ part of the url and git rev-parse --short HEAD to get the <commit id>?

I do envision that, after this enhancement, helmfile can be used in combination with other templating tool based on jsonnet, Lua, ECMAScript or anything else.

Wait... so you are going to remove all templating capabilities in helmfiles? that would be bad as I really really utilize this feature. Its a huge reason I am using this tool

My assumption was that the ReleaseSet would be store in Kubernetes as a CR and my recommendation was for the secrets to be treated the same way as a CR inside of Kubernetes so the github repo would not need to be referenced.

Can we safely assume that helmfile build is always run within a Git repository with a origin correctly set so that Helmfile is able to run git remote show origin to get the github.com/yourorg/yourrepo/ part of the url and git rev-parse --short HEAD to get the ?

Not necessarily, i think this should be pretty simple as you laid out in the top post.
doesntMatterWhereThisIsRunFrom$> helmfile -e prod -f helmfile.yaml.gotmpl build > [any destination i want on the filesystem]

Where doesntMatterWhereThisIsRunFrom$> is anywhere on my laptop, may or may not be in git, i just want to process a helmfile and write the generated releases: to a file. (i.e. this proposed ReleaseSet yaml file.
Keep it super simple and don't build in any use-case specific assumptions.

Wait... so you are going to remove all templating capabilities in helmfiles?

No. I'll keep maintaining the helmfile.yaml template. I'll remain the off-the-shelf solution to make your helmfile release sets DRY. I do want to make sure that people can "optionally" use another templating method when necessary.

OR... are u just saying that by now being able to consume this new ReleaseSet format, people will be free to generate that using any tool they wish externally of helmfile itself... or stick with helmfile's built in golang templating engine

That's it!

Where doesntMatterWhereThisIsRunFrom$> is anywhere on my laptop, may or may not be in git, i just want to process a helmfile and write the generated releases: to a file. (i.e. this proposed ReleaseSet yaml file.

You need two sets of things in Git:

In repo1,  helmfile build input: helmfile.yaml templates, secrets.yaml and values.yaml files
In repo2, helmfile built output: flattened and rendered helmfile.yaml containing release sets(with perhaps references to secrets.yaml?

In your model, how would helmfile build produce the flattened helmfile.yaml from the template? Especially, turning secrets.yaml paths in repo1 to the references in helmfile.yaml in the repo2 seems impossible without knowledge about where the repo1 is.

My assumption was that the ReleaseSet would be store in Kubernetes as a CR and my recommendation was for the secrets to be treated the same way as a CR inside of Kubernetes so the github repo would not need to be referenced.

Yep, that makes sense. But how would you version-control and install secrets into your cluster that are referenced from the CR then?

No. I'll keep maintaining the helmfile.yaml template. I'll remain the off-the-shelf solution to make your helmfile release sets DRY. I do want to make sure that people can "optionally" use another templating method when necessary.

Perfect.

In your model, how would helmfile build produce the flattened helmfile.yaml from the template? Especially, turning secrets.yaml paths in repo1 to the references in helmfile.yaml in the repo2 seems impossible without knowledge about where the repo1 is.

I don't uses secrets.yaml, but I generate and --set-state-values with extremely short lived single use tokens which are used to bootstrap an app and then are useless. I don't care if those tokens end up in the output of build anyways as I would use build as more of an audit record for reference purposes.
I get what you are describing though and I can see the references being a relevant concern. However, that said, all I am saying is that if build is added as a feature, it should have two modes of operation.

build mode1: Literally just process the helmfile and give me the ReleaseSet in stdout. Don't do any special reference maintenance or impose any requirements about being in a git repo etc. Just give the operator the literal output of the helmfile. Keep it raw and literal. Let me deal w/ responsibility and risks of that output.

build mode2: Something more sophisticated that has the reference maintenance features you are describing around secrets and git repo references etc.

I don't uses secrets.yaml, but I generate and --set-state-values with extremely short lived single use tokens which are used to bootstrap an app and then are useless. I don't care if those tokens end up in the output of build anyways as I would use build as more of an audit record for reference purposes.

Okay. So the purpose of sops-encrypted secrets embedded in helmfile build output is that you can avoid comitting helmfile build output containing cleartext secrets in Git.
You've mentioned two modes of helmfile build but I see it doesn't affect your use-case anyway.
If you have no secrets sections in your helmfile.yaml template, there will be no path-to-reference translation needed hence no access to the local Git repo is needed. Just give values via --set-state-values and helmfile build will work like the mode1.

I totally understand, however I'd still advocate for a literal output build mode even if someone has secrets.yaml references.  At least give users this option, even w/ the risk someone might shoot themselves in the foot with it, point being one can't predict how people might want to use build; whether it be for gitops flows or simply being able to cleanly get generated release output for debugging purposes. I just like being able to get literal clean output of any templating system of whats generated without any manipulation. Just my 2cents

@bitsofinfo Thanks. That makes sense :)
I would start with the mode 1 only then.
One reason is that mode 2 can be implemented on top of that without affecting overall helmfile architecture, and another is without helmfile build automatically translating paths to references, GitOps users can always version-control secrets.yaml files in their own respository and references to them can be included in helmfile.yaml templates in the first place.

You could take the helm approach and version things incrementally or you could take the kops approach and only keep the latest, but be able to regenerate it from the original git repo.

You could take the helm approach and version things incrementally

Do you mean that helmfile build or its related helmfile command deploys decrypted secrets onto the cluster before deploying the releaseset, where the secrets are versioned like helm releases in the cluster?

the kops approach and only keep the latest, but be able to regenerate it from the original git repo.

Do you mean keeping only the latest decrypted secrets in the cluster?

I'll add flags to allow setting annotations and labels in the resulting resource. You'll use annotations and labels to propagate any metadata from CI to CD.
Perhaps it will look like helmfile build --annotate name1=val1 --label label1=val1

Those options args look good to me. So if we want to specify multiple annotations/labels does the operator declare the arguments multiple times or comma separated kv pairs?
What about an option to capture the "build" command arguments to add as an additional annotation?

@bitsofinfo Probably I'd start with a simpler implementation which is likely to require a flag per one k-v pair.
Extending it it accept comma-separated kvs later should not be hard. A flag to capture build command in an annotation sounds like a great addition :)

You could take the helm approach and version things incrementally

Do you mean that helmfile build or its related helmfile command deploys decrypted secrets onto the cluster before deploying the releaseset, where the secrets are versioned like helm releases in the cluster?

the kops approach and only keep the latest, but be able to regenerate it from the original git repo.

Do you mean keeping only the latest decrypted secrets in the cluster?

I've been looking for a way to implement a gitops workflow using helmfile so I was very excited to see this issues thread. However IMO storing the secrets in K8S would be less than ideal in many scenarios. One of the things I love about the helmfile/helm secrets approach is it's completely agnostic in-terms of the backend store. Many folks have compliance issues storing secrets in K8S base64 and therefore use other backends like vault or the like. The current agnosticism of helmfile where it leaves it up to the helm chart to ultimately determine where those secrets are stored is something I think should be maintained.
While using a git reference resolves the above issue,  I can't say I'm super fond of this approach either. One of the benefits of using sops is that it enables tracking of secrets. I think it would be ideally when a PR is created in the gitops repo, we should be able to see in that PR whether a secret was changed. Perhaps there is way to accomplish this with a git ref by only updating the ref when the secret value changed? (Although this doesn't seem super elegant to point to old shas). Further, requiring git access to the original repo and requiring git to run helmfile seems a bit bloated IMO.
Personally I would be fond of baking the necessary sops into the rendered file along with the encrypted secret. Even if the secret is gonna be long in certain scenarios, the simplicity, traceability and transparency of this approach is something I personally would really appreciate.
I can't thank you enough for this amazing tool @mumoshu!

@aweis89 Hey! Thanks a lot for your detailed feedback.
I fully agree and I think that's where evolving Helmfile's vals integration helps.
vals(https://github.com/variantdev/vals) is the underlying go library that provides native Vault, SSM secrets manager, SOPS supports to Helmfile. The current usage can be seen at #906.
Regardless of the vals integration, as you pointed out, you still get raw secrets in helmfile build output today.
But the original vals has an alternative mode that allows you to "format safe K8s secrets resources where its data values is vals urls" and pipe vals before kubectl apply run by Flux or ArgoCD.
https://github.com/variantdev/vals#helm
So, I believe we only need two things done to fully support GitOps in Helmfile.

Enhance Helmfile to support leaving vals refs(=references in secrets, not secrets themselves) in K8s secrets resources rendered by helmfile build. E.g. helmfile build --keep-secret-refs-as-is-for-gitops.

Inject vals eval -f FILE before kubectl apply run by Flux or ArgoCD. For ArgoCD you can use the configuration management plugin feature https://argoproj.github.io/argo-cd/user-guide/config-management-plugins/. Flux also has a similar feature.

2. https://argoproj.github.io/argo-cd/user-guide/config-management-plugins/

It definitely seems like vals would solve all my gitops issues with secrets so thanks for bringing that to my attention. It seems like helmfile template combined with vals would solve all my deployment workflow needs. I was planning on putting helmfile.yamls in several application repos, each would have the values for all envs the application gets deployed, including vals =refs+ for any secrets. The application's CI job would write the helmfile -e ENV template output to an envs' gitops repo, creating a PR if there are changes. That env gitops repo's CI jobs would use vals to convert =refs+ to their actual values before running kubectl apply.  Sorry for the ramble but I'm just wondering if there's an issue with this workflow that helmfile build is designed to solve.

Your view seems definitely correct 😉


Op-21723:CVE-fixes with java-17 and spring-boot upgrade-v4.1

Jira :- https://devopsmx.atlassian.net/browse/OP-21936
Op-21881:CVE-Fixes with java-17 and spring-boot upgrade-V4.1
Removed Fiat service code from gate Service

@vivek-opsmx raise a new pr as this was created by me just to see changes.


Draft of non-primitive object type spec

I'm not sure I got everything here.
I also updated the null and undefined sections to reflect the behaviour with strict null checks on. We need to create a section that covers nothing but strict-null-checks differences.

#20677 is the final version of this that merged into spec-update, so this PR is no longer needed.


[map] [mobile] add new list view of users on map

Current:

Desired situation

Notes:

@bakeiro the list should scroll when it's closed? or it should show only the first result?

@bakeiro the list should scroll when it's closed? or it should show only the first result?

that's actually a good question... I think scrollable it's better ;)

that's actually a good question... I think scrollable it's better ;)

x or y?

that's actually a good question... I think scrollable it's better ;)

x or y?

Y please :)
[X is how is right now, which (IMHO is weird) plus I think will be confusing for users to have x while it's minimized, and Y while is open, so always Y I think it's better :) ]

@bakeiro This looks way better, thanks for making that change! Way less confusing!
Just that one comment about the arrow then, would it be possible to make the arrow a bit smaller? Especially the up arrow seems huge and on mobile it's usually thinner and smaller.


Live Video using Ionic / Capacitor

Will the live video scan work with Ionic / Capacitor?

I have absolutely no idea what that is, specifically, but my guess would be that if you can find a way to get the image data into a dataURI or a canvas, then you can pipe it into quagga!
noting to look at this product in the future, no idea when i might get to it


Timelapse not viewable on iOS devices

Environment

Camera:  RPi HQ
OS: Bullseye
Allsky version: v2023.05.01_03
Raspberry Pi 4B 4GB

Bug Description

I have not been able to view timelapses created by AllSky on my iOS devices. All timelapses are generated properly, there are no errors in the log, Windows and Android devices are able to load the timelapses without issue, but iOS devices fail to load the video. I'm assuming this is a codec issue? Even though I thought h264 was playable by iOS devices.
I have been able to reproduce this issue on several iOS devices (one iPhone 14 Pro and two different iPad Pros) and browsers (Chrome and Safari).
The issue occurs on a fresh install with no changes to AllSky.
Screenshots:

iPhone
Android

iPhone timelapse on load screen
Android timelapse on load screen

iPhone timelapse playing screen
Android timelapse playing screen

Timelapse Settings in config.sh
########## Timelapse
# Set to "true" to generate a timelapse video at the end of each night.
TIMELAPSE="true"

# Set the resolution in pixels of the timelapse video.
TIMELAPSEWIDTH=0
TIMELAPSEHEIGHT=0

# Bitrate of the timelapse video.
TIMELAPSE_BITRATE="2000k"

# Timelapse video Frames Per Second.
FPS=25

# Encoder for timelapse video.
VCODEC="libx264"

# Pixel format.
PIX_FMT="yuv420p"

# Amount of information displayed while creating a timelapse video.
FFLOG="warning"

# Set to "true" to keep the list of files used in creating the timelapse video.
KEEP_SEQUENCE="false"

# Any additional timelapse parameters.  Run "ffmpeg -?" to see the options.
TIMELAPSE_EXTRA_PARAMETERS=""

# Set to "true" to upload the timelapse video to your website at the end of each night.
UPLOAD_VIDEO="false"

# Set to "true" to upload the timelapse video's thumbnail to your website at the end of each night.
TIMELAPSE_UPLOAD_THUMBNAIL="true"

Log / configuration files
No logs to upload, as there are no warnings or errors to report.

David
Do you know what version of IOS is on each of the devices? If you can get the versions then I can run a few tests to see if I can see whats going wrong
Thanks
Alex

Alex,
I'm using iPadOS 16.2 and iOS 17.1.2

I have the same problem.. The latest ios is installed everywhere

Since this only happens on a specific iOS release I don't think it's related to Allsky.
Can you try copying a video to your phone, then playing it from your phone to see if the problem is with the web or something else?

I too was experiencing this, so did a bit of digging - apparently this has 'been a thing' on ios for almost a decade.. so, I ended up adding this to my config, and now my iphone friends say they can view the videos!
TIMELAPSE_EXTRA_PARAMETERS="-level 3.1 -refs 11"

@flewid. Thanks a bunch.  I will add this to the Troubleshooting documentation.
Do you see artifacts when not using those settings?
If so, can you please try using just one of the settings, then just the other to see which one gets rid of the artifacts?
cd ~/allsky/images/DATE
# replace DATE with the date of a video you know had artifacts
# change config.sh to just one setting
timelapse.sh --output allsky-DATE-SETTING1_NAME.mp4 DATE
# change to the other setting
timelapse.sh --output allsky-DATE-SETTING2_NAME.mp4 DATE
# change to NO setting
timelapse.sh --output allsky-DATE-no-settings.mp4 DATE

You will end up with 4 videos in that directory - the original with both settings plus two with one setting each plus one with neither setting.
If you can find a frame with artifacts and do a screenshot of all 4 files that would be fantastic, and if you don't mind I can include them in the documentation.
Thanks

I too was experiencing this, so did a bit of digging - apparently this has 'been a thing' on ios for almost a decade.. so, I ended up adding this to my config, and now my iphone friends say they can view the videos!
TIMELAPSE_EXTRA_PARAMETERS="-level 3.1 -refs 11"
Also I should note, I also no longer have any artifacts in the video, so this was a double win.

This works❗, thank you very much

Do you see artifacts when not using those settings? If so, can you please try using just one of the settings, then just the other to see which one gets rid of the artifacts?

Sure thing @EricClaeys , so I did some experimentation today and exported all four videos, including the 'time' command so we could see if any were quicker than the others (rpi 4, 4gb ram).
One thing to also note - playing these videos on desktop - (linux, firefox) - there are no artifacts on any of them, however, as noted below, playing on Android I do see the artifacts, using firefox as well. Not sure what's up there, but adding both options does appear to fix that for me completely.
Also when the artifacts do appear, it seems to be during a 'light change' -- day to night, or, where I have it placed for testing it gets light out of my front windows if they are on, and it APPEARS to happen whenever the lights are flipped on/off  - on the ones that display artifacts.

NEITHER OPTION USED:
real	        19m11.669s
user	65m20.155s
sys		0m16.074s
file size: 116MB / 121377605 bytes
artifacts: yes
plays on ios: no

JUST "-refs 11":
real	        43m26.804s
user	153m26.065s
sys		0m24.467s
file size: 116 MB / 121417932 bytes
artifacts: yes
plays on ios: no

JUST "-level 3.1":
real	       14m18.680s
user	47m57.623s
sys		0m13.232s
file size: 116 MB / 121078268 bytes
artifacts: none !
plays on ios: yes

BOTH "-level 3.1 -refs 11":
real	        43m39.879s
user	154m27.676s
sys		0m23.461s
file size: 116 MB / 121442286 bytes
artifacts: none !
plays on ios: yes

And here are some screens of the artifacts I'm seeing on Android

@flewid, thanks a TON!
It looks like just -level 3.1 is needed?  What operating system do you have on the Pi?  Before I add it to the next release I want to make sure it works on Buster, Bullseye, and Bookworm.
FYI, many programs that create videos add a full image followed by some number of images that only contain changes to the last full image, followed by a full image, then changes, etc.  This can drastically decrease the file size for scenes that don't change a lot.
When someone turns on a light the whole scene changes so the next few "difference" frames are huge.  I don't know exactly how this works but I suspect it's related to what you are seeing.  Slower devices like phones and non-optimized browsers may have trouble keeping up so you see artifacts - usually blocking ss since the software often breaks images into blocks and works on them.

Forgot to say, I can't find any documentation on the -refs or -level options to ffmpeg.  How did you find out about them?

Forgot to say, I can't find any documentation on the -refs or -level options to ffmpeg. How did you find out about them?

I actually found it from this post from 2013 -- i only added the refs because he said he needed it to work, but yeah I've been using just level 3.1 and appears fine this way, and much quicker to generate the videos :)
I'm on raspbian - Debian 12.5 (bookworm?)
Makes sense on the frame stuff and the scene change, that's kind of what i was figuring myself, thanks for the explanation :)
So far, so good with these changes on my system, and android, ios, ipados, and desktop are all working with no artifacts !

Thanks.  That's Bookworm.  I have Buster and Bullseye so will test there and hopefully put in the next release as a standard option so people don't need to add to TIMELAPSE_EXTRA_PARAMETERS.
Thanks again for your help.

@aaronwmorris, do you see any problems with using -level 3.1 with ffmpeg?  The default is level 4 but Apple devices don't support that.

Thanks. That's Bookworm. I have Buster and Bullseye so will test there and hopefully put in the next release as a standard option so people don't need to add to TIMELAPSE_EXTRA_PARAMETERS. Thanks again for your help.

no problem at all, let me know if you need me to test anything else, happy to help :)

@flewid Thanks! This fix worked perfectly.

@David-Jackson, are you using just -level 3.1 ?  And what operating system (run cat /etc/os-release)?
Eric

I do not see any isses with -level 3.1 other than perhaps the limits on resolution.
According to https://en.wikipedia.org/wiki/Advanced_Video_Coding Level 3.1 is limited to the following
<EMAIL_ADDRESS>(13)
<EMAIL_ADDRESS>(11)
<EMAIL_ADDRESS>(5)

The limits in the spec may not be hard limits, but I will admit I do not have a lot of media encoding experience.

I'm not sure if this is related, but since I made this change, now the star trail images seem to just be a singular image, not a star trail? kind of odd.

@flewid, it's highly unlikely they are related. The -level 3.1 option doesn't change the images, just the video created by the images.
See the documentation page on Startrails for ideas on how to "fix" the startrails problem.  It could be that your images are slightly brighter with the full moon and they are now over the threshold.

thanks you're right, was due to the moon !

Closing Issue.  Using -level 3.1 is documented in the next Allsky release.


tablet: Do not allow healthchecker to be started with target type

"master".
@alainjobart
I ran into this problem on Friday where I accidentally set the target type to "master".
This didn't work anyway because every tablet starts as "spare" and
healthcheck isn't allowed to transition a "spare" to "master".
(Healthcheck is only responsible for returning "replica" and "rdonly"
tablets which caught up while in "spare".)
(This was caught in topo.IsTrivialTypeChange().)

LGTM


Presence sensor doesn't expose light sensor to homekit

Device Infomation JSON File
"category": "hps",
    "product_id": "ztc6ggyl",
    "product_id": "sxm7l9xa",

Detail Informations
Some presence sensors also have light sensor that measure illuminance_value
however, in homekit, only the presence_state is exposed and not the lux value.

Note if I override the category to luminance sensor (ldcg), then the sensor works perfectly as a luminance sensor, but stops working as a presence sensor.
Is there a way to have both presence detection and luminance detection?
            "category": "ldcg",
            "schema": [
                {
                    "code": "illuminance_value",
                    "newCode": "bright_value"
                }
            ]

It’s been a year since this was opened, but it would be really great to have illuminance exposed in HomeKit.


Enhancement/Add 'to' to error message.

Coverage remained the same at 88.276% when pulling ae343c4cabdcf359e1631adc55418c79f3fb6274 on MainShayne233:master into a2b365fb6f562aaa1cadd62ae58afd0b767b8754 on technicalpickles:master.


[GLUTEN-1662][VL] feat: Support InsertIntoHiveDirCommand in velox parquet write

What changes were proposed in this pull request?
This patch adds support for InsertIntoHiveDirCommand in velox parquet write
(Fixes: #1662)
related: https://github.com/oap-project/gluten/issues/1636
How was this patch tested?
spark.sql("insert overwrite directory 's3a://tpch/etl' STORED AS parquet select l_returnflag, l_linestatus, sum(l_quantity) as sum_qty, sum(l_extendedprice) as sum_base_price, sum(l_extended
price * (1 - l_discount)) as sum_disc_price, sum(l_extendedprice * (1 - l_discount) * (1 + l_tax)) as sum_charge, avg(l_quantity) as avg_qty, avg(l_extendedprice) as avg_price, avg(l_discoun
t) as avg_disc, count(*) as count_order from lineitem where l_shipdate <= '1998-12-01' group by l_returnflag, l_linestatus order by l_returnflag, l_linestatus").show

spark.hadoop.fs.s3a.impl           org.apache.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3a.aws.credentials.provider org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider

#spark.sql.sources.outputCommitterClass=com.netflix.bdp.s3.S3DirectoryOutputCommitter
spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version 2
spark.hadoop.fs.s3a.committer.name directory
spark.hadoop.fs.s3a.committer.magic.enabled false
spark.hadoop.fs.s3a.commiter.staging.conflict-mode replace
spark.hadoop.fs.s3a.committer.staging.unique-filenames true
spark.hadoop.fs.s3a.committer.staging.abort.pending.uploads true
spark.hadoop.mapreduce.outputcommitter.factory.scheme.s3a org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory

note: this patch will do an override the HiveFormat class from Spark


Return empty string instead of nil in HGETALL, HVALS, HRANGEBYLEX responses

Fix issue #1311.
Additionally, fix the same misbehavior of the HVALS and HRANGEBYLEX commands.

Thanks @torwig , merging...

Thanks @torwig , merging...


[Feat] Implement Newsletter Functionality

Description
Issue Description: Implement Newsletter Functionality
Overview
We need to implement a newsletter subscription functionality on our platform, Ezyshop. This feature will allow users to enter their name and email to subscribe to our newsletter. Upon successful subscription, users will receive a confirmation email.
Requirements

Subscription Form:

Create a form where users can enter their name and email address.
Include a "Subscribe" button to submit the form.

Email Confirmation:

Upon subscription, users should receive a confirmation email from Ezyshop.
Use a third-party email service (e.g., SendGrid, Mailgun) for sending emails.

Database Storage:

Store user details (name and email) in our MongoDB database to track subscriptions.
Ensure that the database schema is updated accordingly to handle subscriptions.

Duplicate Subscription Handling:

If a user tries to subscribe again using an already registered email, a popup message should appear stating: "Already subscribed to our newsletter."
Ensure that the frontend and backend properly handle this scenario to prevent duplicate entries in the database.

Additional Notes

Please provide guidance on the best practices for implementing email confirmation and subscription tracking.
Ensure that the code is well-documented and follows our coding standards.

Acceptance Criteria

A functional subscription form that captures user input.
Confirmation emails are sent successfully upon subscription.
User details are stored in MongoDB without duplicates.
The user receives a friendly error message when trying to subscribe with an existing email.

Let me know if you can implement this functionality and if you need any additional information or resources to get started.
Screenshots
No response
Checklist

[X] I have checked the existing issues
[X] I have read the Contributing Guidelines
[X] I'm a GSSoC'24-Extd contributor
[X] I am willing to work on this issue (optional)

@ShivanshPlays You are assigned.
Try to implement the newsletter functionality, so that when a user enters their name and email and clicks the subscribe button, they receive an email from Ezyshop, Additionally, can the user details be stored in my MongoDB database to track subscriptions, If a user tries to subscribe again with an already registered email, a popup should appear saying, Already subscribed to our newsletter. Let me know if you can implement this.

Checkout the below demo video for better understand 👇

https://github.com/user-attachments/assets/a7833364-4b0d-4821-85e4-dd20ac3df8b6

okay understood


zfp binary CUDA support broken?

I can't figure out how to test CUDA support on my system.
I generated a simple test file: python3 -c 'import struct; import math; open("test.bin", "wb").write(struct.pack("<256f", *list(math.sin(v/128*6.28)*1000 for v in range(256))))'.
Then I tried to compress it:
zfp -x cuda -i test.bin -z testz.bin -f -1 256 -r 10 -s -h
Which reports the following info:
type=float nx=256 ny=1 nz=1 nw=1 raw=1024 zfp=16 ratio=64 rate=0.5 rmse=707.3 nrmse=0.3536 maxe=1000 psnr=3.01
Sometimes it reports nan values or random large floats, despite being given the same input data, but the statistics suggest it's not actually compressing and just storing zeros (related to #105 ?). If I remove the -h flag then it just tells me "compression failed". Everything works fine with the other execution modes (serial and omp).
I'm trying to build and package the 1.0.0 release, which doesn't come with tests, so I'm not sure if CUDA is broken in general or this is specific to the command line tool. I am building against CUDA 11.6.

I'm surprised that zfp -x cuda -h even works for you as the CUDA backend does not yet handle headers (-h) correctly (see #77).  Using your command line, I'm getting this message:
% zfp -x cuda -i test.bin -z testz.bin -f -1 256 -r 10 -s -h
incorrect or missing header

If you drop -h, you should be seeing this:
% zfp -x cuda -i test.bin -z testz.bin -f -1 256 -r 10 -s
type=float nx=256 ny=1 nz=1 nw=1 raw=1024 zfp=320 ratio=3.2 rate=10 rmse=0.9085 nrmse=0.0004542 maxe=2.733 psnr=60.83

which matches what you get with the serial backend:
% zfp -i test.bin -z testz.bin -f -1 256 -r 10 -s
type=float nx=256 ny=1 nz=1 nw=1 raw=1024 zfp=320 ratio=3.2 rate=10 rmse=0.9085 nrmse=0.0004542 maxe=2.733 psnr=60.83

I'm assuming that you have built with -DZFP_WITH_CUDA=ON; you should be getting another error message if you haven't.
Let me also add that it may be worthwhile running ctest -R Cuda on the develop branch, which runs a few dozen CUDA unit tests.  Those tests are not included on the release branches.
The next zfp release will focus on improving CUDA and HIP support, including new/missing capabilities, performance improvements, as well as bug fixes.

Thank you for information on running the tests. They betrayed the root cause of my observed failures: Encode : 222 the provided PTX was compiled with an unsupported toolchain. Turns out I had built against CUDA 11.6 but the CUDA my system's driver was designed for was 11.4. I rebuilt against 11.4 and now the tests pass and the responses I get when I run the zfp commands I posted match yours.
This is usually not a problem due to CUDA's binary compatibility guarantees. However, this requires that the CUDA kernels be compiled to binary form for each extant compute capability, plus a PTX for future capabilities. zfp must only compile a PTX in the default case. I imagine a curated HPC software environment is much less likely to encounter this type of mismatch, but I would say it's pretty common in regular PC applications and scenarios especially when binaries are published.
Perhaps you can address this (along with improved error handling and propagation) as part of your planned GPU improvements.

I agree that improved error reporting is a high priority.  We've thought about something akin to errno, but for deep call stacks such single-error codes may not reveal much about the root cause of failure.  Some kind of hierarchical error reporting would be preferable, but I'm not sure how to best achieve that in C.  I'd welcome any suggestions.
Other than that, I'm not sure what zfp could/should do about issues external to the library itself, such as CUDA incompatibilities.  Are you suggesting embedding information on what CUDA version zfp was built against and then checking for compatibility?  That could be quite onerous in practice given dependencies on OpenMP, CUDA, HIP, SYCL, Cython, NumPy, etc., many of which are evolving faster than zfp itself.  But I'm open to suggestions.
One thing that might help that we have partially implemented on a feature branch is GPU warmup, where a simple kernel is launched when the execution policy is set.  If this fails, so does the zfp_stream_set_ececution() call.  The primary purpose of this is to avoid lazy initialization by the CUDA runtime and to perform other expensive initialization calls only once (see, #135, for instance), but it could perhaps also be used for additional sanity checks.

The big thing on zfp's side in my book is to build multiple binary versions plus a PTX version of each CUDA kernel  as described on that NVidia page I linked. That means that distributed zfp packages will be compatible with many graphics cards and driver versions. I'm not sure how to do that in your build system, but it's how the big CUDA-dependent packages (e.g. ML frameworks) are set up and works well. There shouldn't be any need to do runtime detection or selection, the driver and CUDA runtime will handle that if you make available appropriate versions.

I must admit to not being familiar with the issues, and it would take some time to go through the NVIDIA documentation to figure out what needs to be done.  Do you know of any projects where this is handled that we could use as a template?

I did some research and very recent versions of CMake know how to handle this automatically: https://cmake.org/cmake/help/v3.24/prop_tgt/CUDA_ARCHITECTURES.html#prop_tgt:CUDA_ARCHITECTURES . I understand requiring these recent versions is not an attractive proposition but it is a reference for the proper logic in CMake which is license-compatible. I would encourage defining CUDA_ARCHITECTURES=all by default. Defining more architectures allows more target-specific optimizations and increased performance, defining less reduces binary size and compilation time (though neither is at all significant now compared to the big ML frameworks!). If not enough are defined users simply won't be able to use zfp (and current versions will act quite mysteriously).
For now, it is sufficient to use something like -DCMAKE_CUDA_FLAGS= to pass the required flags to the CUDA compiler. There are various online references such as here (though I think its description of the -arch flag is wrong); I confirmed that adding all the -gencode options listed under "Sample flags for generation on CUDA 11.0" allows my scenario of building against CUDA 11.6 but running against driver version 470, which shipped with CUDA 11.4, to work correctly.
The defaults for these flags and the allowed and suggested combinations depend on the CUDA runtime built against and what capabilities your code needs. CMake's new options know how to handle these factors best. Since you support a wide variety of CUDA versions, I'm not sure it would be wise to explicitly list these flags. Maybe it would be good for now to simply reference some information in the documentation to allow people to select for best performance in their particular environment. I imagine an HPC administrator for instance would want to select all the architectures for hardware known to be installed in their machines to get maximum performance, instead of letting the CUDA compiler pick arbitrarily and constantly paying JIT costs.
Once you are comfortable mandating CMake 3.23, then the default experience can be nicer as a good set of architectures will be selected by default and the interested administrator can easily list compute capabilities instead of having to cross-reference the compiler documentation and manage supported flags directly.

Thanks for sharing.  We will be taking a closer look over the coming weeks.  I don't think we're ready to mandate CMake 3.23 yet, but we could perhaps support this via CMake conditionals.
Just to be sure I understand the use case, compiling for different CUDA architectures makes sense for distributing binaries (e.g., RPMs, Spack packages) and when building and installing zfp on file systems shared by multiple architectures.  If you're building zfp from source for a particular architecture, then you'd end up paying a penalty by inflating binaries and startup time for no benefit, so the default ought to be to build only for the current architecture.
When does the JIT compilation occur?  At load time or when the first CUDA kernel is launched?  All gets baked into the same binary, right?  I imagine something similar would be needed for HIP and SYCL.

NVIDIA's documentation here answers these questions.  They say all kernel versions are baked into the same binary, and object loading/possible JIT compilation occurs when a particular CUDA kernel is first launched. Not sure how different kernels are treated. Never used HIP or SYCL so I can't provide advice there, sorry.
What seems to be the case but they don't say there (and what was the root cause of this bug) is that PTX versions are specific to the CUDA library version. If the PTX compiled by a newer CUDA library version needs to be loaded by a driver shipped with an older library version, the JIT may fail (though the other way around will always work). CMake's native setting compiles real architecture code for the current GPU(s) so this is not a problem, but it is worth mentioning.
But I do want to say that depending on host details like that upsets reproducibility and makes me unhappy as a packager. I understand the motivations for making native the default, though if you do so, this fact should be called out loudly in the documentation.
I did some testing and compared to using whatever the CUDA compiler defines as the default architecture, passing the 9 architecture flags I linked earlier inflates compilation time by 2.5x (21s to 54s) and the size of libzfp.so by 4.4x (1.2MB to 5.3MB). There is no runtime cost for adding more architectures. Compared to the big ML frameworks where building only the native architecture saves hours and hundreds of megabytes, and compared to the risk of someone trying to move binaries across machines and things breaking unexpectedly and the benefit of reproducibility, making native the default is IMHO a false economy. A developer rebuilding the package constantly can easily set the CMake flags to native to save their time if necessary.

@tpwrules Sorry this took so long to address, but because of several CUDA and HIP branches simultaneously in flight, it took some nontrivial effort to get everything merged into a reasonably working state.  The result now appears on the staging branch, which still needs some additional work to get variable-rate decompression working again, but at least you should now be able to specify CUDA architectures using the CMAKE_CUDA_ARCHITECTURES macro.  Please let me know if this addresses your needs.  As you suggested, we had to bump the minimum CMake version to 3.23, but only when CUDA is enabled.


"Humdrum files" directory/folder name

Having spaces in directory/file names usually causes problems when processing on the command line.  Can you change this to either Humdrum or Humdrum_files?

Edited with commit cd17220


Use mock data and mock models in tests

Codecov Report

Merging #135 (d63c405) into master (def2512) will decrease coverage by 39.05%.
The diff coverage is 47.02%.

@@             Coverage Diff             @@
##           master     #135       +/-   ##
===========================================
- Coverage   58.58%   19.52%   -39.06%
===========================================
  Files          60       64        +4
  Lines        2115     2100       -15
===========================================
- Hits         1239      410      -829
- Misses        876     1690      +814

Impacted Files
Coverage Δ

KD_Lib/KD/vision/CSKD/sampler.py
0.00% <0.00%> (ø)

KD_Lib/KD/vision/attention/attention.py
54.54% <ø> (-18.19%)
:arrow_down:

KD_Lib/KD/vision/noisy/utils.py
66.66% <0.00%> (ø)

...D_Lib/Quantization/dynamic/dynamic_quantization.py
62.50% <ø> (-37.50%)
:arrow_down:

KD_Lib/models/shallow.py
28.57% <0.00%> (-71.43%)
:arrow_down:

KD_Lib/Pruning/common/iterative_base_class.py
13.00% <13.00%> (ø)

KD_Lib/Pruning/lottery_tickets/lottery_tickets.py
42.10% <42.85%> (+0.77%)
:arrow_up:

...uning/weight_threshold/weight_threshold_pruning.py
54.54% <54.54%> (ø)

KD_Lib/KD/vision/DML/dml.py
12.28% <75.00%> (-63.16%)
:arrow_down:

KD_Lib/KD/__init__.py
100.00% <100.00%> (ø)

... and 56 more


wire up telemetry for acuqire token silent

Add instrumentation for Acquire token silent.
This will be to help debug perf, but future instrumentation will be added

Coverage increased (+0.1%) to 76.154% when pulling e1174e7f6ec0fb4b8aeee8ee4b52a3fc6df0f004 on telemetry-ats into 6e93f24442a37426698fd5cf62f875ce5a2d9b80 on dev.