pulseaudio degraded (latency & popping) when running as brew service [only] - monterrey m1

hi, first off - I love this project and thank you for all for you wonderful/incredible work
root question:
do brew services run with less priority than manually executing a daemon from terminal?
I'm experiencing clipping & latency when running pulseaudioas a brew service on my [nearly maxed out m1 macbook pro], ie:
brew services start pulseaudio

grepping the process
❯ ps aux | grep pulse
tmccall          45712   0.0  0.0 408505488   1504 s002  R+   11:09AM   0:00.00 grep --color=auto pulse
tmccall          45609   0.0  0.0 408686304  13200   ??  S    11:09AM   0:00.06 /opt/homebrew/opt/pulseaudio/bin/pulseaudio --exit-idle-time=-1 --verbose

but zero clipping and latency when running the command direclty
/opt/homebrew/opt/pulseaudio/bin/pulseaudio --exit-idle-time=-1 --verbose

grepping the process
❯ ps aux | grep pulse
tmccall          43811   2.8  0.0 408799648  13904 s001  S+   10:58AM   0:17.62 /opt/homebrew/opt/pulseaudio/bin/pulseaudio --exit-idle-time=-1 --verbose
tmccall          45428   0.2  0.0 408645776   1808 s002  S+   11:08AM   0:00.00 grep --color=auto pulse

system info
❯ neofetch
                    'c<EMAIL_ADDRESS>                 ,xNMM.          -----------------------
               .OMMMMo           OS: macOS 12.3.1 21E258 arm64
               OMMM0,            Host: MacBookPro18,4
     .;loddo:' loolloddol;.      Kernel: 21.4.0
   cKMMMMMMMMMMNWMMMMMMMMMM0:    Uptime: 16 hours, 32 mins
 .KMMMMMMMMMMMMMMMMMMMMMMMWd.    Packages: 354 (brew)
 XMMMMMMMMMMMMMMMMMMMMMMMX.      Shell: bash 5.1.16
;MMMMMMMMMMMMMMMMMMMMMMMM:       Resolution: 1800x1169
:MMMMMMMMMMMMMMMMMMMMMMMM:       DE: Aqua
.MMMMMMMMMMMMMMMMMMMMMMMMX.      WM: Quartz Compositor
 kMMMMMMMMMMMMMMMMMMMMMMMMWd.    WM Theme: Blue (Dark)
 .XMMMMMMMMMMMMMMMMMMMMMMMMMMk   Terminal: Apple_Terminal
  .XMMMMMMMMMMMMMMMMMMMMMMMMK.   Terminal Font: RobotoMonoNerdFontComplete-Regular
    kMMMMMMMMMMMMMMMMMMMMMMd     CPU: Apple M1 Max
     ;KMMMMMMMWXXWMMMMMMMk.      GPU: Apple M1 Max
       .cooc,.    .,coo:.        Memory: 5616MiB / 65536MiB

m1 bottle info
❯ brew info pulseaudio
pulseaudio: stable 14.2 (bottled), HEAD
Sound system for POSIX OSes
https://wiki.freedesktop.org/www/Software/PulseAudio/
/opt/homebrew/Cellar/pulseaudio/14.2 (168 files, 7.4MB) *
  Poured from bottle on 2022-04-07 at 10:19:03
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/pulseaudio.rb
License: GPL-2.0-or-later and LGPL-2.1-or-later and BSD-3-Clause
==> Dependencies
Build: pkg-config ✔
Required: json-c ✔, libsndfile ✔, libsoxr ✔, libtool ✔<EMAIL_ADDRESS>✔, speexdsp ✔
==> Options
--HEAD
	Install HEAD version
==> Caveats
To restart pulseaudio after an upgrade:
  brew services restart pulseaudio
Or, if you don't want/need a background service you can just run:
  /opt/homebrew/opt/pulseaudio/bin/pulseaudio --exit-idle-time=-1 --verbose

I also experience this issue when building the formula from source, so I don't think it's bottle specific, ie:
brew reinstall pulseaudio -s

brew service plist
❯ bat ~/Library/LaunchAgents/homebrew.mxcl.pulseaudio.plist
───────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: /Users/tmccall/Library/LaunchAgents/homebrew.mxcl.pulseaudio.plist
       │ Size: 658 B
───────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ <?xml version="1.0" encoding="UTF-8"?>
   2   │ <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
   3   │ <plist version="1.0">
   4   │ <dict>
   5   │     <key>KeepAlive</key>
   6   │     <true/>
   7   │     <key>Label</key>
   8   │     <string>homebrew.mxcl.pulseaudio</string>
   9   │     <key>ProgramArguments</key>
  10   │     <array>
  11   │         <string>/opt/homebrew/opt/pulseaudio/bin/pulseaudio</string>
  12   │         <string>--exit-idle-time=-1</string>
  13   │         <string>--verbose</string>
  14   │     </array>
  15   │     <key>RunAtLoad</key>
  16   │     <true/>
  17   │     <key>StandardErrorPath</key>
  18   │     <string>/opt/homebrew/var/log/pulseaudio.log</string>
  19   │     <key>StandardOutPath</key>
  20   │     <string>/opt/homebrew/var/log/pulseaudio.log</string>
  21   │ </dict>
  22   │ </plist>
───────┴─────────────────────────────────────────────────────

unable to duplicate on intel machine
the interesting part is that if I run pulseaudio as a brew service on an older arch (intel)
brew services start pulseaudio

and
/usr/local/opt/pulseaudio/bin/pulseaudio --exit-idle-time=-1 --verbose

BOTH work perfectly without clipping/latency on older intel
❯ neofetch
                    'c.          tdmccall@the-beast
                 ,xNMM.          ------------------
               .OMMMMo           OS: macOS Catalina 10.15.7 19H1824 x86_64
               OMMM0,            Host: MacPro5,1
     .;loddo:' loolloddol;.          Packages: 475 (brew)
 XMMMMMMMMMMMMMMMMMMMMMMMX.      Shell: bash 5.1.16
;MMMMMMMMMMMMMMMMMMMMMMMM:       Resolution: 2560x1440, 2560x1440
:MMMMMMMMMMMMMMMMMMMMMMMM:       DE: Aqua
.MMMMMMMMMMMMMMMMMMMMMMMMX.      WM: Rectangle
 kMMMMMMMMMMMMMMMMMMMMMMMMWd.    Terminal: iTerm2
 .XMMMMMMMMMMMMMMMMMMMMMMMMMMk   Terminal Font: RobotoMonoNerdFontComplete-Regular 12
  .XMMMMMMMMMMMMMMMMMMMMMMMMK.   CPU: Intel Xeon W3690 (12) @ 3.47GHz
    kMMMMMMMMMMMMMMMMMMMMMMd     GPU: Radeon RX 580
     ;KMMMMMMMWXXWMMMMMMMk.      Memory: 20065MiB / 49152MiB
       .cooc,.    .,coo:.

intel bottle info
❯ brew info pulseaudio
pulseaudio: stable 14.2 (bottled), HEAD
Sound system for POSIX OSes
https://wiki.freedesktop.org/www/Software/PulseAudio/
/usr/local/Cellar/pulseaudio/14.2 (168 files, 7MB) *
  Poured from bottle on 2022-04-05 at 13:46:42
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/pulseaudio.rb
License: GPL-2.0-or-later and LGPL-2.1-or-later and BSD-3-Clause
==> Dependencies
Build: pkg-config ✔
Required: json-c ✔, libsndfile ✔, libsoxr ✔, libtool ✔<EMAIL_ADDRESS>✔, speexdsp ✔
==> Options
--HEAD
	Install HEAD version
==> Caveats
To restart pulseaudio after an upgrade:
  brew services restart pulseaudio
Or, if you don't want/need a background service you can just run:
  /usr/local/opt/pulseaudio/bin/pulseaudio --exit-idle-time=-1 --verbose

would love any thoughts/input!
I can run pulse with nohup if needed for now
ty again!

Since this is about configuration questions I'm moving this to the discussions

Oh, I actually can't. You'll have to re-open this as a discussion then since all brew services does is wrap launchctl to launch services. It doesn't actually set anything specific for ARM or Intel (it doesn't even have a concept of architectures)

@SMillerDev this answered my question, and confirmed my fears! ty
appreciate all your hard work.


Try spread pods in HA to different nodes on single zone deployments

Description
Add soft antiaffinity with topology key kubernetes.io/hostname with a weight of 50 to still give more priority to zone antiaffinity if multiple zones available
Issue reference
Please reference the issue this PR will close: https://github.com/dapr/dapr/issues/7665
Checklist
Please make sure you've  completed the relevant tasks for this PR, out of the following list:

[ ] Code compiles correctly
[ ] Created/updated tests
[ ] Unit tests passing
[ ] End-to-end tests passing
[ ] Extended the documentation / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]
[ ] Specification has been updated / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]
[ ] Provided sample for the feature / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]

@filintod, the kube-scheduler already scores based on Node locality for a replica set. Setting this value to me seems like we are just adding more processing time to the scheduling process. Same with the exiting zone affinity rule- do we actually need this?

@JoshVanL was seeing some pods for the same dapr system to be running on the same nodes in some customers, even though it could also be there was a single node, but seem strange for prod systems.   Where do you see this default node anti-affinity for replicasets, as that was not expected in the past, not sure nowadays.

@filintod I think we should do some experimenting with also removing the zone affinity rules. Scheduler also takes this into account by default. By adding custom rules we are causing changes to the default scoring which might be having unintended consequences and fighting sane defaults.
I would have thought having better pod priority to be the thing to focus on to ensure uptime of the control plane.

one thing is that you cannot say HA and have things running on the same node

I said uptime not HA 🙂 I see these as two separate things where uptime is a subset property of achieving HA, but HA also incorporating the idea of replication.
The Dapr control plane is not sensitive to churn like an application serving business traffic needing network failover would be. Single replica Dapr control plane can handle plenty, it just needs to be up, somewhere, and needs have higher priority of being up over consuming Dapr apps.

yes, I meant for you cannot say in term of dapr cannot say as we have here https://github.com/dapr/dapr/blob/master/charts/dapr/values.yaml#L29 HA  and people might expect fault tolerance and spreading on different nodes/az is part of it.
Uptime is for sure important and we should find ways to make these services increase in priority for when one of them is going to be kick out to not be the one of the last in the list. but probably different to this PR.

btw, actually thought I had seen the replica set node locality you mentioned but actually it was different.  https://kubernetes.io/docs/reference/scheduling/config/ . So lmk if you can point to me where you see it. From the default ones, you could get balance if all nodes have somewhat balanced load allocatable, but overtime that is usually not feasible without some sort of rebalancing (that there are tools around). If that is the case, we do need to have antiaffinity to tilt the balance on not having all pods on the same node if possible.


chore: update copyright dates

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Stage 2 - Support KaTex.js for math formulas

I think this issue is pretty straight-forward. @algeboy I want to confirm that we want to use KaTeX.js to display math formulas.
This is how this would work:

You would use the KaTeX syntax to create the formulas like so using the official API:

{% katex.html formula="KaTeX string" %}

The library will render the formula on the page.

awesome.  we can try that,  we can make such content from stackedit.io which outputs KaTeX

Alright sounds great. I'll create the component that you can use in blog posts. I'll create the documentation in a separate issue.

@algeboy I knocked this feature out today. Below is a screen shot of KaTex working and how you'd use it in a blog post.

To use KaTex, simply use a component I created like so:
{% include katex.html exp="YOUR_KATEX_EXPRESSION" %}

Let me know if there's anything else that should be changed about this feature.

can this be done in raw markdown?  E.g. just using $exp$, e.g. $\sqrt{2}{a^2+b^2}$ instead of  the more verbose {% include katex.html exp="YOUR_KATEX_EXPRESSION" %}?

@algeboy let me do some research to see if this can be done

@algeboy I was able to make writing math formulas easier in markdown. Below is an image of how this looks:

If you look closely, you can now write math without the {%%} statement. For example, you can use this:
$$
Your MathJax goes here
$$

instead of
{% include katex.html src="YOUR_KATEX" %}

Note: I removed Katex and replaced it with MathJax to make this work. I did this because MathJax works very well with kramdown. It will take a bit longer to load and the syntax is different than katex.

Hmm, I understand the impulse.  Actually MathJax is sort of spotty on matching fonts with the rest of the document.  Katex fixed alot of this.  If its a hassle we can just include both and let folks decide.  I just had the feeling that there would be a way to make javascript or something like it that would let one write $f(x)=\sqrt[2]{x-5}$ and it would replace it with something like
{% include katex.html src = "f(x)=\sqrt[2]{x-5}" %}  that way you write in usual markdown but you get the more glamorous fonts of katex.

Alright. Let me see if I can make it work with Katex, that way there aren’t
two math libraries floating around.
On Fri, Oct 18, 2019 at 10:38 AM James B. Wilson<EMAIL_ADDRESS>wrote:

Hmm, I understand the impulse. Actually MathJax is sort of spotty on
matching fonts with the rest of the document. Katex fixed alot of this. If
its a hassle we can just include both and let folks decide. I just had the
feeling that there would be a way to make javascript or something like it
that would let one write $f(x)=\sqrt[2]{x-5}$ and it would replace it
with something like
{% include katex.html src = "f(x)=\sqrt[2]{x-5}" %} that way you write in
usual markdown but you get the more glamorous fonts of katex.
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/thetensor-space/thetensor-space.github.io/issues/25?email_source=notifications&email_token=AG25BJJPRXSRUJCREEHD6Y3QPHRADA5CNFSM4I3JBL4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBVCNWQ#issuecomment-543827674,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AG25BJP7NDSVZDCHWSBXVGTQPHRADANCNFSM4I3JBL4A
.

--
Alexander Cannon | Front End Software Engineer
T<PHONE_NUMBER> |<EMAIL_ADDRESS>https://alex-cannon.github.io/portfolio
https://www.linkedin.com/in/alexander-lee-cannon/

I will mention an example that I like is how stackedit.io works.  You can edit things in the inline latex style $f(x)=\sqrt[2]{x-4}$ and display equation form $$f(x)=\sqrt[2]{x-4}$$ way, but it converts the
HTML of hte markdown into katex.
Here's an explicit example from that system:

Hello World
Here is an inline math equations $f(x)=\sqrt[2]{x-4}$ and here is one in display mode
$$\int_0^19 \frac{dx}{x}$$

Written with StackEdit.

it creates something well spaced and nice looking all in katex.  You can see the page source it makes.

Alright, This works now. See #33


Extension moves valid bookmarks to trash

Using version 1.0.0 with Chrome 98.0.4758.102. The extension has moved bookmarks with valid URLS to trash, e.g. https://www.gimmesomeoven.com/chocolate-chip-cookies/, or http://thepeteplan.wordpress.com/

Thanks for reporting (and for the cookie recipe) @raggleton; will look into this.

Just FYI for anyone who experiences the same thing, Chrome keeps a backup copy of your bookmarks, so it's possible to restore them: https://www.reddit.com/r/chrome/comments/9r18dj/comment/e8ean9b/

To add to that, the extension is non-destructive, so your bookmarks will never be deleted, they'll just be moved to a bookmark folder called Trash.


Audio Update for LWJGL3

Audio now runs music in a separate thread on LWJGL3. Also synchronized to allow to access music from separate threads without crashes.
This fixes issue https://github.com/libgdx/libgdx/issues/6450.

I'm not familiar with Desktop backend so I let the review to somebody else more knowledgeable, just a couple of comments.
Running update() on a separate thread will cause OnCompletionListener code to be executed on the audio thread (called from OpenALMusic.update()) potentially causing concurrency issues. One option would be to execute the listener as a Runnable on the main loop thread (there might be a slight extra delay though for some use cases such as chaining audios).
The dipose mechanism is a bit hacky IMO. I think it would be more elegant and consistent with the rest of the clean up and dispose logic to register it as a LifecycleListener and implement dispose() in a similar way TimerThread does.
If problems arise with this change, there's always the option to create it as a separate new Audio implementation as we did on Android with AsynchronousAndroidAudio, each of them with their own advantages.

Running update() on a separate thread will cause OnCompletionListener code to be executed on the audio thread...

Good point. I made it so results are returned via render-thread.

The dispose mechanism is a bit hacky IMO. I think it would be more elegant and consistent with the rest of the clean...

There is nothing hacky about it. I could change audioThread to a boolean if that's better? I also am happy to add a lifecycle listener. It just seems to just add more code?

If problems arise with this change, there's always the option to create it as a separate new Audio implementation...

Sure, I support whoever wants to do that.

There is nothing hacky about it. I could change audioThread to a boolean if that's better? I also am happy to add a lifecycle listener. It just seems to just add more code?

My point is that Audio implementation details should not surface on Application so that the Audio can be replaced by a different implementation. Ideally, thread handling, disposing, etc... should be handled inside the Audio implementation and the Application should only see the interface and work with it. With the current proposal, the Application holds a reference to a Thread, what if somebody wants to replace it with a non thread based implentation?

@obigu I see. Also considering MockAudio, it makes sense to move it into OpenALLwjgl3Audio. Let change things around...

I moved the music-thread into OpenALLwjgl3Audio which can be turned on/off via configuration.
Also, after more careful analysis, the music player still has to be called via render-thread at the moment. There is a bunch of list in OpenALLwjgl3Audio which aren't properly synchronized. I guess all this PR solves is allowing to run music on a separate thread which should resolve issues when the the render-loop is slow preventing crackles when playing music files while e.g. assets are loaded.

I'm not sure whether it is a good idea to execute all completion listeners on the render thread. This may help to prevent concurrency issues, but can also be done by the users themselves if it is documented in the changelog. (Sidenote: are there other platforms where this is already the case, Android maybe?) On the other hand, as @obigu mentioned, this change can lead to slight extra delays making it impossible to do audio chaining.

@crykn I think you were looking at an older version. I already changed that back. It's not only returned via render-thread if music is played via separate thread (which can be turned on/off via LWJGL configuration).

I don't think marking all music methods synchronized is the way to go, we should only synchronize critical sections (where we want to protect shared resources if needed). Or probably use defensive copy for some cases to avoid congestion and live locking.

I agree with that. However, 90% of those methods would have to be synchronized anyway. Too many of the variables in OpenALMusic are both used in the render-loop and the music thread. Also, I am not sure if intermixing calls to alSourceUnqueueBuffers, alSourceQueueBuffers, etc. would work.

Final thought, i'm not sure this is the right approach, if we want to avoid locking render thread, maybe we should consider deferring all the calls using a blocking message queue. I'm afraid that current change will bring more trouble than what we have right now.

I agree with the message queue. However, calls to OpenALLwjgl3Audio from OpenALMusic would still need to be from the render-thread. That said, this requires a rather major rework not just for OpenALMusic but also OpenALSound.
I am aware, having most of the methods in OpenALMusic now synchronized might not what we want? However, any other way would require a major rewrite of the audio library to properly work with a separate music thread. The proposed message queue would probably be what we want. I personally don't have time to undertake that endeavor. If that's what everyone wants, I drop this PR. No harm done. Just bring your down-votes. :)

Calling audio stuff from a user thread is beyond the purpose of this PR, we shouldn't address both use cases in the same PR.

I agree, I took that back. This is just about threaded music and nothing else.

Also, I am not sure if intermixing calls to alSourceUnqueueBuffers, alSourceQueueBuffers, etc. would work.

Calling those AL functions is supposed to be thread-safe. In fact, in my own implementation, I rely on that, at least for calls on "fire & (almost) forget" audio samples. They don't carry any important extra state I need to take care of.

I agree with the message queue

This is what I do for audio streams - a simple message queue. Through the public interface, the user app can issue commands, which are then processed during the next update loop. At a reasonable frame/update rate, delays caused by that are pretty much unrecognizable.

@noblemaster i will try another approach and make a proposal to discuss. #6450 being related to a misuse of the API (call non thread-safe code from another thread), i think the issue can be closed.

I tried something that looks promising to me, my first tests are conclusive. It still a work in progress, there are few bugs and i'll need more time to make it clean and perfectly working (a lot of manual tests are needed).
You can see a preview here : https://github.com/mgsx-dev/libgdx/commit/aa05abbd9adde09fed0295fe9261bba9dc8cb30f
The diff is hard to read, sorry about that, i had to move a lot of code in a separated class in order to isolate code run in the background thread from code run in the render thread.
To summarize things up :

The approach i choose was to use a bunch of message queues: some for requesting the audio thread, some for requesting the render thread. (bi-directional communication between both threads).
i split OpenALMusic in 2 classes for clarity : one called by the render thread only, the other (OpenALMusicAsync) called from audio thread only.
this way, the only synchronization points are queues offer and poll, which is the best we can do i think.

Note that i was wrong about Thread.yield, it's not necessary and consume a lot of resources for nothing, Thread.sleep is better. I set it temporarily to 100ms but i think we can find an appropriate value based on buffer size / latency.
Let me know what do you think, if maintainers are OK with this change i could invest more time in it.

@mgsx-dev That looks like a much better approach than what I came up with. I'll drop my pull request if you finish yours.
If it were me, I would probably combine OpenALMusic and OpenALMusicAsync given they call each other. Haven't looked that close, but other than that it seems to be fine.


New feature request

Hello,
I Love the App. I would just like to see if it would be possible to expand the apps where the  ">" is rather then at end of menu apps, in the scenario I would prefer to place ">" towards far right of the menu bar. I just fill having that would save even more time rather than having to move the cursor to the end of other menu apps.
Thank you for your attention.

You can move the “<“ icon to the right, next to Notification Center icon.
I don’t use the recent version of macOS but in previous versions it was possible to disable the Notification Center completely so the icon could be placed in the top right corner too ;)

Yes you are correct you can move it, but my problem is that when you collapse the hidden apps by pressing the "<" they still collapse at the end of the menu extras rather than beside the now ">." My question/request is if there is a way to pin the collapsed apps, so when HiddenBar is collapsed they reveal themselves beside the ">" rather then automatically at the end.

@rsv87 the idea is great and is how the app should be really implemented. However, not possible in the way the app is implemented.

Hi @rsv87, sorry for the late response 😢 , as @s-valent mentioned, your idea is great and on the user side, I would also love it. But on the developer side, it's basically not able to accomplish because of the technical things.
Because of that, I have to close this issue. Sorry guys.


Fix GitHub action errors

Issue: https://issues.redhat.com/browse/AMW-215

For some reason this CI is failing, although it was passing yesterday - https://github.com/RanabirChakraborty/rhbk-datagrid-aws/actions/runs/10816078663/job/30006439565, using the same galaxy token. There is probably some issue with galaxy itself.


*: remove "strings.Compare == 0"

Make more readable.

Codecov Report

Merging #9667 into master will decrease coverage by 0.07%.
The diff coverage is 27.27%.

@@            Coverage Diff             @@
##           master    #9667      +/-   ##
==========================================
- Coverage   68.77%   68.69%   -0.08%
==========================================
  Files         373      373
  Lines       34346    34346
==========================================
- Hits        23621    23594      -27
- Misses       8994     9004      +10
- Partials     1731     1748      +17

Impacted Files
Coverage Δ

etcdctl/ctlv3/command/printer_simple.go
73.82% <0%> (+1.34%)
:arrow_up:

etcdctl/ctlv3/command/user_command.go
61.64% <0%> (ø)
:arrow_up:

auth/simple_token.go
86.99% <100%> (ø)
:arrow_up:

auth/store.go
73.4% <33.33%> (ø)
:arrow_up:

rafthttp/probing_status.go
46.66% <0%> (-17.78%)
:arrow_down:

pkg/adt/interval_tree.go
83.18% <0%> (-9.31%)
:arrow_down:

pkg/fileutil/purge.go
65.9% <0%> (-6.82%)
:arrow_down:

proxy/grpcproxy/watcher.go
85.71% <0%> (-4.09%)
:arrow_down:

raft/progress.go
94.17% <0%> (-1.95%)
:arrow_down:

lease/lessor.go
85.62% <0%> (-1.6%)
:arrow_down:

... and 21 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b8e8d75...48f28b9. Read the comment docs.


Do not need to pull registry:2.5.0 and nginx:1.9 when user using make install to build locally.

We only need to pull the image when building for offline installer.
For development scenario, if the image does not exist ,"docker-compose up" will pull the images, and "make install" should not try to connect docker hub if not needed.

#958 fixed


show user location is not working

<MapView
						provider={MapView.PROVIDER_GOOGLE}
						initialRegion={this.state.init}
						showCompass={true}
						rotateEnabled={false}
						showUserLocation={true}
						showsMyLocationButton={true}
						style={css.map}>

There is no blue dot on the map. am I missing something?
Also when I click the show location button, nothing seem to happen.

Do you have location permission granted on the Device?

it's xcode code iphone x simmulator.
I think I did click allow the location prompt when first open the map.
How can I check that I actually did?

does this mean location permission granted for expo apps?

this is more like bug. reported

You wrote showUserLocation insteadOf showsUserLocation


docker-yaktor auto creation has some quirks

"latest" tag is not being created correctly.
grunt release-minor seems broken every other time it's invoked
"npm publish" is happening from local machine as part of "grunt release-*", when I think the npm publish should be happening during the travis build by default.
https://travis-ci.org/SciSpike/yaktor/builds/148358226#L807 because alpine is using ash not bash
(continued) can't automatically release yaktor/yaktor images based on patch releases until that's fixed
1. make it use bash
1. or make it ash friendly

#4 has been addressed as of make shell portable see POSIX Basic Regular Expressions

#21 will remove the need to do this completely. Perhaps we will deprecate yaktor/yaktor?

closed by #21


An unhandled exception was thrown by the Error Prone static analysis plugin.

The following error was found when building a new target by bazel.
error-prone version: 2.1.1
Stack Trace:
com.sun.tools.javac.code.Symbol$CompletionFailure: class file for com.sun.jna.Structure not found
bazel version:
Build label: 0.7.0-homebrew
Build target: bazel-out/darwin_x86_64-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Oct 19 09:12:48 2017<PHONE_NUMBER>)
Build timestamp:<PHONE_NUMBER>
Build timestamp as int:<PHONE_NUMBER>

Can you share a repro?  Can you also try building with the bazel flag --java_classpath=off?

I suspect this is google/error-prone#827, especially if the flag @eaftan mentioned makes a difference.

Thank you, I will try it with --java_classpath=off to see if it is gone, and, regarding the repro, let me see if I can sever the code to get a repro w/o leaking any sensitive info.
For google/error-prone#827, I am not sure I understand why it may be the same cause as mine as my java file is not a generated one, and the error message from both cases look different to me.

I wasn't thinking - the error messages are related and both issues affect Bazel, but the other one is windows-only.

@RNAi can you check if disabling FunctionalInterfaceClash with -Xep:FunctionalInterfaceClash:OFF makes a difference?


fix(store): stop doing header verification in GetRangeByHeight

Overview
We don't need to verify headers on every GetRangeByHeight 'cause verification already happened in Append. No tests failed, added TestStore_Append_BadHeader to prevent bugs in the future.
Fixes #135

Thanks friends, waiting for Rene to take a look.

@renaynay kindly ping :)


Building Gitea will build go-sqlite3 with or without the sqlite build tag

Gitea version (or commit ref): master
Git version: 2.8.1
Operating system: Windows 10
Database:

[ ] PostgreSQL
[ ] MySQL
[ ] MSSQL
[ ] SQLite
[x] Not relevant

Can you reproduce the bug at https://try.gitea.io:

[ ] Yes (provide example URL)
[ ] No
[x] Not relevant

Log gist:

[x] Not relevant

Description
Even without specifying "sqlite" as a build tag, the command go build main.go will also build the vendor go-sqlite3, which requires gcc.
Screenshots

Note
Currently, two of Gitea's files import this vendor: models/models_sqlite.go and models/unit_tests.go. The former has the sqlite build constraint, while the latter does not. This is probably the root of this issue.

The sqlite tag only touches the final product not the tests. One could argue to add a different command for building the tests. But this an enhancement not a bug.

@go2sh if it is not possible to build gitea without gcc than it is a bug


Cop idea: redundant presence validation on association

Is your feature request related to a problem? Please describe.
class Book < ApplicationRecord
  belongs_to :user
  validates :user, presence: true

  # or
  belongs_to :user
  validates :user_id, presence: true

  # or
  belongs_to :author, foreign_key: :user_id
  validates :user_id, presence: true
end

Since Rails 5.0 the defaults (https://guides.rubyonrails.org/configuring.html) for belongs_to is optional: false unless config.active_record.belongs_to_required_by_default is explicitly set to false. The presence validator is added automatically, and the explicit presence validation is redundant.
See https://github.com/rails/rails/blob/061bf3156fb90ac6b8ec255dfa39492cf22d7b13/activerecord/lib/active_record/associations/builder/belongs_to.rb#L126
Describe the solution you'd like
Add a cop RedundantPresenceValidationOnBelongsTo.
For projects that are deliberately on config.load_defaults 4.2 or prefer not to load_defaults at all (😱), or explicitly set belongs_to_required_by_default to false, a reasonable option is to Enable: false the cop in the project configuration.

This issue has been implemented by #594.


[MOB-9520]: enable tree-shaking

JIRA Ticket(s) if any

MOB-9520

Description
Enables tree shaking to resolve this issue
Test Steps

I'm definitely not an expert with JS packages and bundle sizes but I would've expected changes in our webpack config files to reduce bundle size and for tree shaking. The changes here seem more like bundle stat info, is this just preliminary work?

@mprew97 -- one question -- are we importing any pollyfills or css?  (not including things like style-components)

@mprew97 Unless I'm doing something wrong, this hasn't changed the bundle size :(  index.js remains at 353kb.


Failed to activate the pathfinder package

[Enter steps to reproduce below:]

Open Atom

Atom Version: 1.2.4
System: Mac OS X 10.11.1
Thrown From: Atom Core
Stack Trace
Failed to activate the pathfinder package
At Cannot read property 'command' of undefined

TypeError: Cannot read property 'command' of undefined
    at Object.module.exports.activate (<anonymous>:3:23)
    at Package.module.exports.Package.activateNow (/Applications/Atom.app/Contents/Resources/app.asar/src/package.js:170:19)
    at /Applications/Atom.app/Contents/Resources/app.asar/src/package.js:150:32
    at Package.module.exports.Package.measure (/Applications/Atom.app/Contents/Resources/app.asar/src/package.js:92:15)
    at /Applications/Atom.app/Contents/Resources/app.asar/src/package.js:143:26
    at Package.module.exports.Package.activate (/Applications/Atom.app/Contents/Resources/app.asar/src/package.js:140:34)
    at PackageManager.module.exports.PackageManager.activatePackage (/Applications/Atom.app/Contents/Resources/app.asar/src/package-manager.js:512:21)
    at /Applications/Atom.app/Contents/Resources/app.asar/src/package-manager.js:495:29
    at Config.module.exports.Config.transact (/Applications/Atom.app/Contents/Resources/app.asar/src/config.js:311:16)
    at PackageManager.module.exports.PackageManager.activatePackages (/Applications/Atom.app/Contents/Resources/app.asar/src/package-manager.js:490:19)

Commands

Config
{
  "core": {},
  "editor": {
    "fontSize": 11,
    "invisibles": {}
  }
}

Installed Packages
# User
auto-indent, v0.5.0
pathfinder, v0.4.0

# Dev
No dev packages

@robertlilje This looks like an issue with the pathfinder package - I'd suggest you open an issue on that repo, but it looks like it's been taken down: https://atom.io/packages/pathfinder


Add GSSError, inheriting from KrbError, and use it everywhere KrbError is currently used.

See https://github.com/requests/requests-kerberos/pull/75

It looks like pykerberos uses KrbError for server side exceptions (along with GSSError in a few places), but uses GSSError for all client side exceptions. Since WinKerberos only implements the client side, we'll raise GSSError everywhere.
This change should be completely backward compatible since GSSError inherits from KrbError.


[eslint] - fix plugin rule list

It looks like ts-package-json-sdktype rule entry was duplicated in #18597 -
which casues the "number of rules should match the expected value" to report an
error.
This PR fixes the issue by removing the duplicated entry. I also went ahead and
sorted the entries to make these things easier to spot in the future (although
there is no guarantee sort will be preserved).

Nice catch!


DM pop-up's z-index value is too low

activate windows, then i'll fix it /j

activate windows, then i'll fix it /j

Too bad I deleted system32


Any ideas on why my publish is failing

https://ci.appveyor.com/project/GaryEwanPark/githubreleasemanager/build/0.1.0-unstable.54 (Build 17)#L374
I haven't dug into it yet, and this is the first time that I have tried the upload, so likely I am doing something wrong, but just wanted to ask for any obvious troubleshooting tips.
Thanks

As an update...
I have just tried running the same command locally on my machine, with all the variables populated, and I get the same "Not Found" error.  It seems as though there is a delay, while it is uploading the file, and then it returns the error message.
Any ideas?
Thanks!

Ok, I have found the issue, although I am not sure why I would have to do this, as it is surely working for you...
I was trying to use the argument -r gep13/GitHubReleaseManager but this wasn't working.  I have just downloaded the source code for this project, and removed the requirement of the repository name to contain a slash, and I have instead used -r GitHubReleaseManager and this works without any issues.  The zip file is uploaded, and the build sent to the queue.
Do you have any ideas as to why you would require to use the full name of the repository, including your username, and I don't?
When I check my dashboard, this is what I see:

Which led me to try with a non qualified repository name.
Cheers!

Thanks @gep13 for letting me know. I'll do some more investigation into why they are different but looks like I just need to remove any constraints on the name.
I think I need to do some work to make it clearer what the reasons for failure are too.

Yip, that is essentially what I did in my local version.  Removed the if statement around there being a / in the repo name.
Let me know if you want me to check anything, looking forward to your next release!

Release version 0.10.0 is published with these changes: https://www.nuget.org/packages/PublishCoverity


Animated node with tag 98 does not exists

I'm using material-bread version 0.2.10 in a brand new React Native (v0.63.2) project with react-native-reanimated v1.13.0, react-native-gesture-handler v1.7.0, react-native-screens v2.10.1 and react-native-svg v12.1.0.
I've imported TextField to a screen and when I click in TextField to fill it with some data, I get the following error:
Animated node with tag 98 does not exists
The project is still in the beginning, so I don't have any other library on it.
Did I miss any step?

Hey shimatai! I am running into a very similar issue. Did you find a fix for this or did you end up pivoting to something else?


Appstudio update test-component-pac-dimz

Pipelines as Code configuration proposal

Pipelines as Code CI/test-component-pac-dimz-on-pull-request has successfully validated your commit.

StatusDurationName

✅ Succeeded
7 seconds
init

✅ Succeeded
16 seconds
clone-repository

✅ Succeeded
1 minute
build-container

✅ Succeeded
23 seconds
sanity-inspect-image

✅ Succeeded
21 seconds
deprecated-base-image-check

✅ Succeeded
3 minutes
clair-scan

✅ Succeeded
44 seconds
clamav-scan

✅ Succeeded
15 seconds
sbom-json-check

✅ Succeeded
18 seconds
sanity-label-check

✅ Succeeded
19 seconds
sanity-optional-label-check

✅ Succeeded
5 seconds
show-summary

Pipelines as Code CI/test-component-pac-dimz-on-pull-request has successfully validated your commit.

StatusDurationName

✅ Succeeded
6 seconds
init

✅ Succeeded
12 seconds
clone-repository

✅ Succeeded
25 seconds
build-container

✅ Succeeded
10 seconds
deprecated-base-image-check

✅ Succeeded
12 seconds
sanity-inspect-image

✅ Succeeded
33 seconds
clamav-scan

✅ Succeeded
9 seconds
clair-scan

✅ Succeeded
7 seconds
sbom-json-check

✅ Succeeded
7 seconds
sanity-optional-label-check

✅ Succeeded
7 seconds
sanity-label-check

✅ Succeeded
4 seconds
show-summary


Skale auto Faucet function

Create function to call before all interaction on Skale chains :

Test receiver balance
If receiver balance < Skale Faucet contract Threshold : use Skale Proof Of Work function to calculate "gasPrice" value wich is used for for authentication
Send transaction using "function pay(address payable receiver) external"

done in v1.6.4


[Feature request] GLSL code formatting feature

I like to look at the twigl code on Twitter.
However, much of that code is obfuscated to keep it short.
How about a formatting feature to make the obfuscated code easier to read?
I think js-beautify can be used to format the code.
https://github.com/beautify-web/js-beautify
The following is an implementation image.
before:

after:

I think js-beautify can be called as follows.
<script src="https://cdnjs.cloudflare.com/ajax/libs/js-beautify/1.13.13/beautify.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/js-beautify/1.13.13/beautify-css.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/js-beautify/1.13.13/beautify-html.js"></script>

beautify.addEventListener('click', () => {
    const opts = {
        indent_size: 4
    };
    editor.setValue(js_beautify(editor.getValue(), opts));
}, false);

wow! this is a good idea.
However, would you mind if I change the position and design of the bracket icons by myself?

Yes. I'll leave it to you to decide how to implement it.

BTW, I found one problem with JS Beautifier.
In the case of #define, when Beautifier is executed, unexpected line breaks may occur.
before)
#define X(s,l)d=s*sqrt(h);if(h>0.&&d>m&&sin(atan(d,P.y)+P.x*3.+t*l*4.)>.9)m=d;

after)
#define X(s, l) d = s * sqrt(h);
if (h > 0. && d > m && sin(atan(d, P.y) + P.x * 3. + t * l * 4.) > .9) m = d;

Unfortunately, I don't know how to solve this. I hope someone can help us.

That possibility occurred to me right away.
So I haven't started on this issue, imagining that I would probably have to make the parser myself.
When I get some time, I might give it a shot!

I haven't tried it, but the following GLSL parser may be helpful.
https://github.com/stackgl/glsl-parser

Here's one: http://evanw.github.io/glslx/

This could certainly be useful.

BTW, In the future, AI may help make code easier to read.
https://twitter.com/rianflo/status/1618386347810779136


Page templates

Possibly utilising #28 custom elements, another repository for reusing template HTML elements, without all the web component features.
src/page/shop/item.html
<!doctype html>
<meta charset="utf-8" />
<title>Example shop item</title>
<template-header/>
<main>
	<template-shop-item />
</main>

Would load src/template/shop-item.html in place of the template element, but should it also come with optional CSS, PHP and JavaScript? If so, this is already covered by #28 ...
just an idea, putting it here while it's on my mind.

Out of scope in this repo, tracked here: https://github.com/PhpGt/DomTemplate/issues/3


Problems about Continuous treatment

Hi,
Based on your detailed intructions, I try to reproduce the results on IHDP datasets and have some problems.
Under the setting Extrapolation (h=2), the results of TransTEE in Table 2 is 0.2745+/-0.14976, while I get 0.4783+/-0.3629. My results are aslo computed with 100 datasets under default parameters.
Then I adjust the epochs from the default 1500 to 2000 and 2500, the results are 0.3418+/-0.3019 and 0.2551+/-0.2370 separately. Although 0.2551 is close to your result, the standard deviation is much larger than yours.
In that case, could you give me some guidance about the result reproduction?

Hi, thanks for reproducing. The variance is high as the sample size is very small. It is normal that your variance is 0.2370 compared to the 0.14976 we report as your mean is lower..
We didn't do aggressive hyperparameter tunings beyond the default one.

Thanks for your quick reply~

Hi,
I'm so sorry to bother you again. I tried other settings as shown in your Table 2 (IHDP), but the results are quite different from yours. Even I tune n_epochs to 5000, I obtain 0.2629+/-0.4208 for vanilla h=1, 0.3246+/-0.5065 for vanilla h=5  and 0.2314 +/- 0.2218 for extrapolation h=5.
I used 100 subdatasets and each was used under one specific seed. I find the variances among different repeats are large. So I wonder do you only use one subdataset and repeat 100 times with different seeds?
I'm really really confused about the results. I would be so grateful if you could give me some advices.

I also want to know whether the results in Table 2 are computed by def curve(model, test_matrix, t_grid, targetreg=None) in eval.py. Thanks so much!


Warning on rule AA0072 triggered on app.json file

Describe the bug
I have an issue with rule AA0072 which is triggered on app.json and I don't understand why
See screen shoot
To Reproduce
I don't know.. I can send you my app as example.
Expected behavior
Don't have this type of warning.
Screenshots

5. Versions:

AL Language: v6.1.397164
Business Central: 17.3

According to https://github.com/microsoft/AL/issues/6246 it can be because you have that many codeunits with TableNo properties set. Is that the case for you? If so, it's a duplicate.

Hi @titofffffff could you try again with the latest version (https://github.com/microsoft/AL#al-developer-preview-builds) ?

This issue has been closed because it needs more information and has not had recent activity. Please open a new issue if you believe this has not been resolved and reference the current issue.

This issue has been closed because it needs more information and has not had recent activity. Please open a new issue if you believe this has not been resolved and reference the current issue.


Cloning instructions are wrong.

Thanks for this!
Your README.md needs updating though to use the https URL when cloning and not ssh:
git clone https://github.com/cgerikj/binary-greedy-meshing --recursive

Also it's better to generate the CMake detritus into a build sub-folder:
> mkdir build && cd build
> cmake .. -G"Visual Studio 15 2017 Win64"
> start binaryMesher.sln

Thanks, fixed


Return a global authenticated user variable

Instead of using this everywhere $this->auth->user(), is it possible to assign it to a variable like this:
$authUser = $this->auth->user();

And use it globally? I tried adding it to a __construct method of the BaseController but doesn't work when calling the $authUser from any controller that extends it. What would be the correct way to do this?

You can just use $this->user() as a shortcut. That's a part of the helpers
trait.
On Thu, 9 Jul 2015 16:45 MovingGifts<EMAIL_ADDRESS>wrote:

Instead of using this everywhere $this->auth->user(), is it possible to
assign it to a variable like this:
$authUser = $this->auth->user();
And use it globally? I tried adding it to a __construct method of the
BaseController but doesn't work when calling the $authUser from any
controller that extends it. What would be the correct way to do this?
—
Reply to this email directly or view it on GitHub
https://github.com/dingo/api/issues/510.

Yup that works - thanks! Just out of curiosity, if I were to assign it to a variable, how would I do that (not to assign the same variable in every controller of course) but have global access to it in all controllers, just like we have global access to $this->user(). Any pros/cons in doing that?
Thanks for the lightning fast response btw :)

Is it discouraged to create an Alias for it? Like we had the API::user() alias in v0.8?

That alias was a facade. I'll most likely be adding facades back at some point in time but I'm not sure if it's all going to be under the API facade or if I'm going to have API\Auth and so on. Still something I'm thinking about.
@MovingGifts I'm not sure what you mean exactly. If your controllers use the trait you have access to it. Otherwise, you can use app('Dingo\Api\Auth\Auth')->user() or app('api.auth')->user() to also fetch the authentication user.


Missing button text for PostScript

Describe the issue
Icon and button text are missing for the PostScript file downloads.
Examples:

https://datatracker.ietf.org/doc/draft-ietf-tsvwg-quickstart/07/
https://datatracker.ietf.org/doc/draft-ietf-rmt-bb-webrc/04/

Screenshot:

Code of Conduct

[X] I agree to follow the IETF's Code of Conduct

I thought we had removed Postscript a long time ago?

We removed the ability for people to upload new postscript, but we are still showing postscript people had provided in years past I thought.

Ah. I think I misunderstood this and ripped these buttons out a long time ago. I wonder if we can just stop showing PS that we have now?

I believe there are some documents where the PS mattered (the txt referred to the PS for a diagram). But nobody's complained - we could stop showing the link until someone asks and provide it on demand.

But that is a policy decision that has not been made, and is probably an IESG level decision, no?

Yes.

@larseggert - followup?

The IESG discussed this but I am blanking on the outcome. CC @evyncke

Found back the discussion on the private https://iesg.privatewikis.ietf.org/notes/2024-01-11
The bottom line is literally Sure, kill it. (talking about the buttons)
I.e., there is a need to remove the textless button though.

Should be fixed in #7200


Small visual bug when switching units

When switching between cm and in, there's a single frame in which the canvas is redrawn incorrectly. This is low priority as I don't image people would be switching units like this, but it's a little jarring and should be fixed regardless.

Seems to be happening because the canvas dedraws a frame in which ptDensity has been updated, but the localPerspective hasn't been yet. Fix will involve making sure localPerspective and ptDensity always agree with eachother in the draw function.

Are both ptDensity and localPerspective listed as dependencies on the draw function? It's possible that that function has too many dependencies causing unnecessary re-renders. I've seen that in at least one other place and corrected it. We may just need to trim the dependency list (on useEffect/useCallback/useMemo, whichever applies).

You are so fast! I'll check out the fix now :) Thank you!


[BUG] Calling FaceAPI.PersonGroups.create with null CreatePersonGroupsOptionalParameter argument results in HTTP 400

Describe the bug
PersonGroups.create can be called with a null value for the CreatePersonGroupsOptionalParameter argument. createWithServiceResponseAsync then converts the null argument into null values for the name and userData parameters that are sent to the service.
However, name is not optional when calling Person Group - Create.
Therefore, you must create a CreatePersonGroupsOptionalParameter object with the name field set and pass it to PersonGroups.create or you will get a 400 response.
Exception or Stack Trace
Status code 400, {"error":{"code":"BadArgument","message":"Request body is invalid."}}
To Reproduce
Note: to use FaceAPIManager, I am referencing azure-cognitiveservices-faceapi-1.0.2-beta.jar, which I compiled from https://github.com/Azure/azure-sdk-for-java. For more information see: #3640.
Code Snippet
import com.microsoft.azure.cognitiveservices.vision.faceapi.*;
import com.microsoft.azure.cognitiveservices.vision.faceapi.models.*;

public class Test {
    public static void main(String[] args) {
        try {
            AzureRegions region = AzureRegions.WESTUS;
            FaceAPI client = FaceAPIManager.authenticate(region, "INSERT API KEY HERE");
            personGroupCreate(client);
        } catch (Exception e) {
            System.out.println(e.getMessage());
            e.printStackTrace();
        }
    }

    public static void personGroupCreate(FaceAPI client) {
        final String PERSON_GROUP_ID = "INSERT PERSON GROUP ID HERE";
        try {
            client.personGroups().create(PERSON_GROUP_ID, null);
            System.out.println("Person group created");
        } catch (Exception e) {
            System.out.println(e.getMessage());
        }
    }
}

Expected behavior
Output: Person group created
Screenshots
N/A
Setup (please complete the following information):

OS: Win 10 Pro 1803
IDE: None
Version of the Library used: azure-cognitiveservices-faceapi-1.0.2-beta.jar

Additional context
To call the PersonGroup.create method correctly:
CreatePersonGroupsOptionalParameter param = new CreatePersonGroupsOptionalParameter ().withName ("name");
client.personGroups().create(PERSON_GROUP_ID, param);

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

[x] Bug Description Added
[x] Repro Steps Added
[x] Setup information Added

Thank you for opening this issue! We are routing it to the appropriate team for follow up.

Has this issue been corrected in the new Java SDK? I know that one is unreleased still. The one I am testing with is azure-cognitiveservices-faceapi-1.0.2-beta, and this issue is still there. Thanks.
@dfulcer @kurtzeborn

@milismsft @ryogok


No supertype allowed in handler return

The following program gives the annotated error:
interface Done {
  def done(): Nothing
}

def main() = {
  try {
    do done()
  } with Done {
    def done() = ()   // Error: Expected Nothing but got Unit.
  }
}

Is this the behaviour we want?
Note that this also disallows, e.g., a hole instead of do done().

What would you expect, otherwise? The answer type of the handler is Nothing unless you upcast it there.


Refactor type to string in Utils.

Extracted the switch case into a separate function to cleanup getMangledTypeStr a bit.

Of course, if you feel like doing the same cleanup upstream and then porting it here, that would be fine.

Okay! Thanks. :)


[BUG] Backup marked as "completed" cannot be restored, gzip: invalid header

Describe the bug
I have 3 months of backups marked as "completed" without issue, which can't be restored due to what appears to be a problem during backup creation. The error message on restore is "gzip: invalid header".
Problems creating a backup should raise an error, and the resulting backup should not be marked as "Completed".
See #7670 for the specific case. I thought this was due to my confusion around the engine image version used for the volume, but testing with a variety of versions in the restore-backup-to-file.sh script indicates it must be a problem in image creation. Regardless, backups should not depend on assumptions about the longhorn context... all information required to restore the backup should be included in the backup metadata.
time="2024-01-14T12:03:42Z" level=info msg="Running restore to file command: backup-url=nfs://warehouse.vert:/mnt/storage/.docker/longhorn-backups?backup=backup-8417093ca8854143&volume=pvc-2925f092-964d-4f62-9c4f-941f39605413 backing-file= output-file=/tmp/restore/mariadb.raw output-format=raw" func=cmd.RestoreToFileCmd.func1 file="restore_to_file.go:63"
time="2024-01-14T12:03:42Z" level=info msg="Output file path=/tmp/restore/mariadb.raw" func=cmd.restoreToFile file="restore_to_file.go:138"
time="2024-01-14T12:03:42Z" level=info msg="Start to restore nfs://warehouse.vert:/mnt/storage/.docker/longhorn-backups?backup=backup-8417093ca8854143&volume=pvc-2925f092-964d-4f62-9c4f-941f39605413 to backup.img" func=cmd.restoreToFile file="restore_to_file.go:142"
time="2024-01-14T12:03:42Z" level=info msg="Attempting mount for nfs path warehouse.vert:/mnt/storage/.docker/longhorn-backups with nfsvers 4.2" func="nfs.(*BackupStoreDriver).mount" file="nfs.go:126" pkg=nfs
time="2024-01-14T12:03:42Z" level=info msg="Mounting NFS share nfs://warehouse.vert:/mnt/storage/.docker/longhorn-backups on mount point /var/lib/longhorn-backupstore-mounts/warehouse_vert/mnt/storage/.docker/longhorn-backups with options [nfsvers=4.2 actimeo=1 soft timeo=300 retry=2]" func="nfs.(*BackupStoreDriver).mount" file="nfs.go:137" pkg=nfs
time="2024-01-14T12:03:42Z" level=info msg="Loaded driver for nfs://warehouse.vert:/mnt/storage/.docker/longhorn-backups" func=nfs.initFunc file="nfs.go:91" pkg=nfs
time="2024-01-14T12:03:42Z" level=info msg="Saving config in backupstore" func=backupstore.SaveConfigInBackupStore file="config.go:79" filepath=backupstore/volumes/a7/51/pvc-2925f092-964d-4f62-9c4f-941f39605413/locks/lock-e0dd61273bea4e88.lck kind=nfs object=config pkg=backupstore reason=start
time="2024-01-14T12:03:42Z" level=info msg="Saved config in backupstore" func=backupstore.SaveConfigInBackupStore file="config.go:90" filepath=backupstore/volumes/a7/51/pvc-2925f092-964d-4f62-9c4f-941f39605413/locks/lock-e0dd61273bea4e88.lck kind=nfs object=config pkg=backupstore reason=complete
time="2024-01-14T12:03:42Z" level=info msg="Stored lock backupstore/volumes/a7/51/pvc-2925f092-964d-4f62-9c4f-941f39605413/locks/lock-e0dd61273bea4e88.lck type 1 on backupstore" func=backupstore.saveLock file="lock.go:190" pkg=backupstore
time="2024-01-14T12:03:44Z" level=info msg="Loading config in backupstore" func=backupstore.LoadConfigInBackupStore file="config.go:54" filepath=backupstore/volumes/a7/51/pvc-2925f092-964d-4f62-9c4f-941f39605413/locks/lock-e0dd61273bea4e88.lck kind=nfs object=config pkg=backupstore reason=start
time="2024-01-14T12:03:44Z" level=info msg="Loaded config in backupstore" func=backupstore.LoadConfigInBackupStore file="config.go:65" filepath=backupstore/volumes/a7/51/pvc-2925f092-964d-4f62-9c4f-941f39605413/locks/lock-e0dd61273bea4e88.lck kind=nfs object=config pkg=backupstore reason=complete
time="2024-01-14T12:03:44Z" level=info msg="Loaded lock backupstore/volumes/a7/51/pvc-2925f092-964d-4f62-9c4f-941f39605413/locks/lock-e0dd61273bea4e88.lck type 1 on backupstore" func=backupstore.loadLock file="lock.go:171" pkg=backupstore
time="2024-01-14T12:03:44Z" level=info msg="Trying to acquire lock backupstore/volumes/a7/51/pvc-2925f092-964d-4f62-9c4f-941f39605413/locks/lock-e0dd61273bea4e88.lck" func="backupstore.(*FileLock).canAcquire" file="lock.go:65" lock="{ volume: pvc-2925f092-964d-4f62-9c4f-941f39605413, name: lock-e0dd61273bea4e88, type: 1, acquired: false, serverTime: 2024-01-14 12:03:42.44230808 +0000 UTC }" pkg=backupstore
time="2024-01-14T12:03:44Z" level=info msg="backupstore volume pvc-2925f092-964d-4f62-9c4f-941f39605413 contains locks [{ volume: , name: lock-e0dd61273bea4e88, type: 1, acquired: false, serverTime: 2024-01-14 12:03:42.44230808 +0000 UTC }]" func="backupstore.(*FileLock).canAcquire" file="lock.go:66" pkg=backupstore
time="2024-01-14T12:03:44Z" level=info msg="Acquired lock backupstore/volumes/a7/51/pvc-2925f092-964d-4f62-9c4f-941f39605413/locks/lock-e0dd61273bea4e88.lck type 1 on backupstore" func="backupstore.(*FileLock).Lock" file="lock.go:114" pkg=backupstore
time="2024-01-14T12:03:44Z" level=info msg="Saving config in backupstore" func=backupstore.SaveConfigInBackupStore file="config.go:79" filepath=backupstore/volumes/a7/51/pvc-2925f092-964d-4f62-9c4f-941f39605413/locks/lock-e0dd61273bea4e88.lck kind=nfs object=config pkg=backupstore reason=start
time="2024-01-14T12:03:44Z" level=info msg="Saved config in backupstore" func=backupstore.SaveConfigInBackupStore file="config.go:90" filepath=backupstore/volumes/a7/51/pvc-2925f092-964d-4f62-9c4f-941f39605413/locks/lock-e0dd61273bea4e88.lck kind=nfs object=config pkg=backupstore reason=complete
time="2024-01-14T12:03:44Z" level=info msg="Stored lock backupstore/volumes/a7/51/pvc-2925f092-964d-4f62-9c4f-941f39605413/locks/lock-e0dd61273bea4e88.lck type 1 on backupstore" func=backupstore.saveLock file="lock.go:190" pkg=backupstore
time="2024-01-14T12:03:44Z" level=info msg="Loading config in backupstore" func=backupstore.LoadConfigInBackupStore file="config.go:54" filepath=backupstore/volumes/a7/51/pvc-2925f092-964d-4f62-9c4f-941f39605413/volume.cfg kind=nfs object=config pkg=backupstore reason=start
time="2024-01-14T12:03:44Z" level=info msg="Loaded config in backupstore" func=backupstore.LoadConfigInBackupStore file="config.go:65" filepath=backupstore/volumes/a7/51/pvc-2925f092-964d-4f62-9c4f-941f39605413/volume.cfg kind=nfs object=config pkg=backupstore reason=complete
time="2024-01-14T12:03:44Z" level=info msg="Loading config in backupstore" func=backupstore.LoadConfigInBackupStore file="config.go:54" filepath=backupstore/volumes/a7/51/pvc-2925f092-964d-4f62-9c4f-941f39605413/backups/backup_backup-8417093ca8854143.cfg kind=nfs object=config pkg=backupstore reason=start
time="2024-01-14T12:03:44Z" level=info msg="Loaded config in backupstore" func=backupstore.LoadConfigInBackupStore file="config.go:65" filepath=backupstore/volumes/a7/51/pvc-2925f092-964d-4f62-9c4f-941f39605413/backups/backup_backup-8417093ca8854143.cfg kind=nfs object=config pkg=backupstore reason=complete
time="2024-01-14T12:03:44Z" level=info msg="Restoring delta block backup" func=backupstore.RestoreDeltaBlockBackup file="deltablock.go:719" backup_url="nfs://warehouse.vert:/mnt/storage/.docker/longhorn-backups?backup=backup-8417093ca8854143&volume=pvc-2925f092-964d-4f62-9c4f-941f39605413" event=restore object=snapshot original_volume=pvc-2925f092-964d-4f62-9c4f-941f39605413 pkg=backupstore reason=start snapshot=backup-8417093ca8854143 volume_dev=backup.img
time="2024-01-14T12:03:44Z" level=info msg="Saving config in backupstore" func=backupstore.SaveConfigInBackupStore file="config.go:79" filepath=backupstore/volumes/a7/51/pvc-2925f092-964d-4f62-9c4f-941f39605413/locks/lock-e0dd61273bea4e88.lck kind=nfs object=config pkg=backupstore reason=start
time="2024-01-14T12:03:44Z" level=info msg="Saved config in backupstore" func=backupstore.SaveConfigInBackupStore file="config.go:90" filepath=backupstore/volumes/a7/51/pvc-2925f092-964d-4f62-9c4f-941f39605413/locks/lock-e0dd61273bea4e88.lck kind=nfs object=config pkg=backupstore reason=complete
time="2024-01-14T12:03:44Z" level=info msg="Stored lock backupstore/volumes/a7/51/pvc-2925f092-964d-4f62-9c4f-941f39605413/locks/lock-e0dd61273bea4e88.lck type 1 on backupstore" func=backupstore.saveLock file="lock.go:190" pkg=backupstore
 0 / 100    0.00%time="2024-01-14T12:03:45Z" level=error msg="Failed to delta restore volume pvc-2925f092-964d-4f62-9c4f-941f39605413 backup backup-8417093ca8854143" func=backupstore.RestoreDeltaBlockBackup.func2 file="deltablock.go:759" error="gzip: invalid header"
time="2024-01-14T12:03:45Z" level=info msg="Removed lock backupstore/volumes/a7/51/pvc-2925f092-964d-4f62-9c4f-941f39605413/locks/lock-e0dd61273bea4e88.lck type 1 on backupstore" func=backupstore.removeLock file="lock.go:180" pkg=backupstore
 0 / 100    0.00% 2s
time="2024-01-14T12:03:46Z" level=error msg="Failed to remove tmp file=backup.img.converted" func=cmd.CleanupTempFiles file="restore_to_file.go:223" error="remove /backup.img.converted: no such file or directory"
time="2024-01-14T12:03:46Z" level=error msg="Failed to remove tmp file=backing.img.cp" func=cmd.CleanupTempFiles file="restore_to_file.go:223" error="remove /backing.img.cp: no such file or directory"
time="2024-01-14T12:03:46Z" level=fatal msg="Error running restore to file command" func=cmd.RestoreToFileCmd.func1 file="restore_to_file.go:66" error="gzip: invalid header"

To Reproduce

Create a scheduled backup, observe "completed" status
???
Attempt to restore and observe "gzip: invalid header" error.

Expected behavior
Completed backups should restore.
Support bundle for troubleshooting
Not relevant (and honestly confusing on my dev cluster, which has had many versions of longhorn installed). I can email it upon request. But the backup metadata is
{
  "Name": "pvc-2925f092-964d-4f62-9c4f-941f39605413",
  "Size": "12884901888",
  "Labels": {
    "KubernetesStatus": "{\"pvName\":\"pvc-2925f092-964d-4f62-9c4f-941f39605413\",\"pvStatus\":\"Bound\",\"namespace\":\"default\",\"pvcName\":\"datadir-mariadb-0\",\"lastPVCRefAt\":\"\",\"workloadsStatus\":[{\"podName\":\"mariadb-0\",\"podStatus\":\"Running\",\"workloadName\":\"mariadb\",\"workloadType\":\"StatefulSet\"}],\"lastPodRefAt\":\"\"}",
    "RecurringJob": "restored-recurring-job-fe3ec17624033ffa",
    "VolumeRecurringJobInfo": "{\"backup\":{\"jobSpec\":{\"name\":\"backup\",\"groups\":[\"default\"],\"task\":\"backup\",\"cron\":\"0 2 * * *\",\"retain\":7,\"concurrency\":1},\"fromGroup\":[\"default\"],\"fromJob\":false},\"restored-recurring-job-b942e1756afafbd1\":{\"jobSpec\":{\"name\":\"restored-recurring-job-b942e1756afafbd1\",\"groups\":[\"default\"],\"task\":\"snapshot\",\"cron\":\"38 */12 * * *\",\"retain\":2,\"concurrency\":1},\"fromGroup\":[\"default\"],\"fromJob\":false},\"restored-recurring-job-d492cb5c84a7b09e\":{\"jobSpec\":{\"name\":\"restored-recurring-job-d492cb5c84a7b09e\",\"groups\":[\"default\"],\"task\":\"backup\",\"cron\":\"0 2 * * *\",\"retain\":7,\"concurrency\":1},\"fromGroup\":[\"default\"],\"fromJob\":false},\"restored-recurring-job-fe3ec17624033ffa\":{\"jobSpec\":{\"name\":\"restored-recurring-job-fe3ec17624033ffa\",\"groups\":[\"default\"],\"task\":\"backup\",\"cron\":\"0 2 * * *\",\"retain\":7,\"concurrency\":1},\"fromGroup\":[\"default\"],\"fromJob\":false},\"snapshot\":{\"jobSpec\":{\"name\":\"snapshot\",\"groups\":[\"default\"],\"task\":\"snapshot\",\"cron\":\"38 */12 * * *\",\"retain\":2,\"concurrency\":1},\"fromGroup\":[\"default\"],\"fromJob\":false}}",
    "longhorn.io/volume-access-mode": "rwo"
  },
  "CreatedTime": "2023-10-12T02:32:38Z",
  "LastBackupName": "backup-8417093ca8854143",
  "LastBackupAt": "2024-01-13T09:04:34Z",
  "BlockCount": "5711",
  "BackingImageName": "\"\"",
  "BackingImageChecksum": "\"\"",
  "CompressionMethod": "\"gzip\"",
  "StorageClassName": "\"longhorn\""
}

Environment

Longhorn version: 1.6.0-dev at the moment, though I've tried restoring the backup from 1.4.0, 1.4.4, 1.5.3 as well.
Impacted volume (PV): pvc-2925f092-964d-4f62-9c4f-941f39605413
Installation method: kubectl
Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: k3s  v1.28.5+k3s1

Number of control plane nodes in the cluster: 3
Number of worker nodes in the cluster: 4

Node config

OS type and version: ubuntu 22.04
Kernel version: 5.15.0
CPU per node: varies
Memory per node: varies
Disk type (e.g. SSD/NVMe/HDD): HDD
Network bandwidth between the nodes (Gbps): 1

Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): Bare metal
Number of Longhorn volumes in the cluster: 14

Additional context
It's a dev cluster with mixed arm64/amd64, though everything is on ubuntu.

Please provide a support bundle, volume and backup config files of the volume. I think it is due to the volume backup is compressed by lz4.
BTW, what's the purpose of o using restore_to_file?

the issue impacts every backup for this volume since I moved to 1.6.0-dev in October. Backups from other volumes or from this volume before the move can restore fine.

This is due to that the backup is incremental. If one of backup files corrupted, backups using the corrupted file will fail to restore. I've created a ticket for the issue https://github.com/longhorn/longhorn/issues/7070.
I think it is not due to NFS and gzip. The issue is usually due to the silent data corruption caused by the underlying storage related to memory, disk and so on.

Personally I think it would be helpful to inspect one of the broken backups directly; I'm happy to take any further steps you think would be helpful here or even (privately) link to a broken backup for you to inspect.

Sure. I will check the information you provided and prepare a script for helping find and inspect the problematic backup file if necessary.

Posting a support bundle here.
supportbundle_115e29e8-9a7a-4725-b9f6-1d336837655a_2024-04-16T16-09-56Z.zip
As directed by @derekbit
Do we know if the data is retrievable at all? I really need it ASAP.

@thejaykobe Can you reach out to me on Rancher or CNCF slack channel to speed up the troubleshooting process? Thank you.

@ohthehugemanatee
Sorry for the late reply.
Does the problematic volume and backup still exist?

@ohthehugemanatee
Can you try to restore the problematic backup and volume by the restore-backup-to-file.sh by specifying --version v1.5.x-head? The usage is in the doc https://longhorn.io/docs/1.7.0/advanced-resources/data-recovery/recover-without-system/.
I expect the restoration will fail and please send us the log of the restore process. Thank you.

I went through @ohthehugemanatee's support bundle again.
Just realized the engine of the volume pvc-2925f092-964d-4f62-9c4f-941f39605413 is v1.4.0. All backups and the failed restoration are run using v1.4.0. v1.4.0 only supports gzip, and the issue is not related to lz4 introduced in v1.5.0.

Hi @thejaykobe
If your backupstore is using s3
Can you download the backupvolume to the local environment
The path pattern should be like
<backup store work directory>/backupstore/volumes/
    <2 random characters>/<2 random characters>/<backup volume name>/
        volume.cfg
        backups/
            backup1.cfg
            backup2.cfg
            ......
        blocks/
            <2 random characters>/<2 random characters>/
                <backup data, 2MB block, the name is the checksum>.blk
            <2 random characters>/<2 random characters>/
                <backup data, 2MB block, the name is the checksum>.blk
            ......

After downloading to the local, the folder structure should be like
$ ll {VOLUME_NAME}/
- backups/
- blocks/
- volume.cfg

Then, install lz4  and gzip in your local environment
Now, you can run the script here to detect the corrupted file.
By analyzing the corrupted file we might find out how to fix the compression gzip: invalid header issue
Thanks!

I worked with @thejaykobe during these two weeks and have successfully restored the volume.
What I found

The original volume is gone, so the details are gone
The compression method in the volume config is gzip. However, according to the information provided by @thejaykobe, the volume is created in Longhorn v1.5.3. The compression method should be lz4.
The backup blocks of the volume are mostly compressed by gzip, but few blocks are compressed by lz4. The backup resources indicate all existing backups are compressed by gzip.

Currently, I'm unable to reproduce the issue in my environment and need more input and further investigation.

Pre Ready-For-Testing Checklist

[x] Where is the reproduce steps/test steps documented?
The reproduce steps/test steps are at:

We are still investigating the root cause of the issue.
The PRs improve the resilience in the decompression path.
The following test cases need to be tested in S3, Azure and NFS backupstore.

Test case 1

Create a volume in LH v1.4.x, write data and take backup multiple times
Update to v1.6.x, restore the backups. The restoration should succeed and then verify data.

Test case 2

Create a volume in LH v1.6.x, write data and take backup multiple times
Update to v1.6.x, restore the backups. The restoration should succeed and then verify data.

Test Case 3

The test cases in e2e related to backup/restore should pass

Test Case 4

Create a new 10GiB volume and write 5GiB data by dd
Backup volume and restore it. Verify the checksum.
Login to the backupstore. Update compressionMethod in volume.cfg and backup_*.cfg to gzip.
Restore the volume and verify the checksum.
Also check if there is any memory leak in instance-manager pods by kubectl -n longhorn-system top pods.

Test case 5

Test backing images and restore as the steps in Test Case 4

Test Case 6

Set global setting backup-compression-method to gzip
Create a new 10GiB volume and write 5GiB data by dd
Backup volume and restore it. Verify the checksum.
Login to the backupstore. Update compressionMethod in volume.cfg and backup_*.cfg to lz4.
Restore the volume and verify the checksum.
Also check if there is any memory leak in instance-manager pods by kubectl -n longhorn-system top pods.

Test case 7

Test backing images and restore as the steps in Test Case 6

[x] Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
The PR is at

https://github.com/longhorn/backupstore/pull/167
https://github.com/longhorn/longhorn-engine/pull/1076
https://github.com/longhorn/backupstore/pull/172

[ ] Which areas/issues this PR might have potential impacts on?
Area: backup and restore
Issues

Verified on master-head 20240520

longhorn master-head https://github.com/longhorn/longhorn/commit/5a3c410218e4152c6302ed12cde1cfa6ed1ec446
longhorn-engine master-head https://github.com/longhorn/longhorn-engine/commit/e39b7f0313b22d5c435ce57d1442800999a0f4ac
backupstore master-head https://github.com/longhorn/backupstore/commit/3bce6e69af1514165c17e6949f8736d9f69aabf2

The test steps
https://github.com/longhorn/longhorn/issues/7687#issuecomment-2103744408
Result Passed

[x] Test case 1

Create a volume in LH v1.4.x, write data and take backup multiple times
Update to v1.5.x and then to v1.6.x and then to master, restore the backups. The restoration succeeded.

[x] Test case 2

Create a volume in LH v1.6.1, write data and take backup multiple times
Update to master-head, restore the backups. The restoration succeeded.

[x] Test case 3

Back up backing images and restore it

[x] Test case 4

[x] Test case 5

[x] Test case 6

[x] Test case 7

For cases 4-7, we didn't observe any memory leaks in the instance-manager pods on master-head.
For cases 4-7, we could use the below scripts in the MinIO or NFS pod bash mode

echo '#!/bin/sh

# Check if a file path parameter is provided
if [ -z "$1" ]; then
  echo "Please provide a file path as a parameter."
  exit 1
fi

# File path
file_path="$1"

# Read the file content and replace
cat "$file_path" | sed "s/lz4/gzip/g" > /tmp/tempfile

# Write the replaced content back to the original file
cat /tmp/tempfile > "$file_path"

# Remove the temporary file
rm /tmp/tempfile

echo "Replacement completed: $file_path"' > replace_lz4_with_gzip.sh

echo '#!/bin/sh

# Check if a file path parameter is provided
if [ -z "$1" ]; then
  echo "Please provide a file path as a parameter."
  exit 1
fi

# File path
file_path="$1"

# Read the file content and replace
cat "$file_path" | sed "s/gzip/lz4/g" > /tmp/tempfile

# Write the replaced content back to the original file
cat /tmp/tempfile > "$file_path"

# Remove the temporary file
rm /tmp/tempfile

echo "Replacement completed: $file_path"' > replace_gzip_with_lz4.sh


Add domain api support

Ready, basic domain management implementation.
I didn't understand how to properly configure so that the configuration is generated via poetry run python mkdocs, please help with this
p.s.
and do not judge strictly this is my first pull request to someone else's project

Domains API released in v2.4.0
Upgrade with:
pip install -U twc-cli

Doc: twc domain


Add "unattended-upgrades" of debian as system option?

https://www.linode.com/docs/guides/how-to-configure-automated-security-updates-debian/#install-unattended-upgrades

When done - also see: #2018


Add support to send alarm events to Alarm Company over standarized alarm protocols.

Proposal
Are there plans for adding support to send alarm events to "third party" alarm centrals?
Like adding support for SIA DC-09, or Contact ID?
I would LOVE to get rid of my duplicate sensor setup since the alarm system needs access to it's own set of sensors, and my lighting etc. can't easily get fast enough access to these over the slow cloud. (Especially for motion this doesn't work).
So being able to skip the entire dedicated alarm system and have Alarmo set up in HA and configure it so it sends events to the Alarm company/central over the already standardized protocols that exists for the alarm sector would be awesome.
I know Alarmo aims for a more run your-own alarm, but the added safety of a 3rd party acting on events when I'm not within cell coverage is important to me (and I can't be the only one?)
Additional info
Perhaps there is a need for a way to include/exclude what sensors should be sent to the external system too? If not all sensors should be acted upon. Like I have some water sensors that isn't really important that the alarm company gets notified about, but it's nice to get notified by Alarmo/HA when they're triggered so I can empty the container it monitors.

Hi. I need ability to send alarm events to "third party" alarm centrals too.
Please add this feature!!!

Hi. Are you planning to add this feature?

@dpbarinov

Hi. Are you planning to add this feature?

No. I have no idea about such protocols and I don't think this platform is reliable enough to automatically send reports to an security company.
I would assume such companies require a certified alarm system, rather than some DIY solution with Home Assistant running on a Raspberry Pi.

I've been doing a bit more research around it.
And I see that HA has a SIA addon that we maybe can pick some code from.
This HA integration is listen only, to get sensor data into HA. But it contains a lot of the "codes" and what each code means etc.
So in my mind it would be a great starting point since it most likely contains a lot of logic that could be used.
https://www.home-assistant.io/integrations/sia/
https://github.com/home-assistant/core/tree/dev/homeassistant/components/sia
Home Assistant Core is Apache License 2.0, so shouldn't be an issue to snip out some code from here. But not sure what license alarmo is under, can't find any info about it in this repo? @nielsfaber
SIA is as I understand it the "newest" protocol, so I guess it would be fine to just implement this, and drop ContactID (older protocol).

No. I have no idea about such protocols and I don't think this platform is reliable enough to automatically send reports to an security company. I would assume such companies require a certified alarm system, rather than some DIY solution with Home Assistant running on a Raspberry Pi.

Good point. I have little to no knowledge about alarm system certifications.
I just know I can set up where my current alarm system should send event through my alarm panel (IP, subscription id, etc. etc.)
Not sure if the SIA protocol has any identifying info about what system it is (brand, model etc.)
I'll keep reading up on the protocol.

@dpbarinov

Hi. Are you planning to add this feature?

No. I have no idea about such protocols and I don't think this platform is reliable enough to automatically send reports to an security company. I would assume such companies require a certified alarm system, rather than some DIY solution with Home Assistant running on a Raspberry Pi.

Security on the HA is a completely reliable solution, in addition, the raspberry can be upgraded (adding a GSM modem and backup power) to meet the requirements of security companies

@dpbarinov

Security on the HA is a completely reliable solution, in addition, the raspberry can be upgraded (adding a GSM modem and backup power) to meet the requirements of security companies

Since any user can interconnect any kind of hardware and software (including this integration), the reliability of the system is very much arguable.
@ruant

And I see that HA has a SIA addon that we maybe can pick some code from.

If an addon already exist, why not use that instead? I guess it would be easier (and more generic) to focus on some 'glue layer' to have Alarmo interact with that.

@ruant

And I see that HA has a SIA addon that we maybe can pick some code from.

If an addon already exist, why not use that instead? I guess it would be easier (and more generic) to focus on some 'glue layer' to have Alarmo interact with that.

The integration I mentioned is LISTEN/READONLY.
For example my existing alarm system, I can configure to send the SIA events to the SIA HA integration.
In other words, the SIA HA integration is the same as the alarm company. Just receiving the events.
So basically it's the exact opposite of what I want.
I want to be able to send the SIA events, so the alarm company receives events when alarms are triggered in alarmo.


feat(core): allow plugins to customize/override Webpack devServer config

Motivation
We never documented this, but with Webpack 4 there was the ability to just write config.devServer
Since Webpack 5, it does not work automatically anymore with the Node imperative API, and users can't customize easily the devServer.
Instead of adding a new API, I'm merging the user/plugin-provided devServer config to the default one to allow this niche use-case.
Note: Docusaurus users integrating with an API on another domain may find this useful to write a plugin such as:
function ProxyPlugin(context) {
  return {
    name: 'proxy-plugin',

    configureWebpack() {
      return {
        devServer: {
          proxy: {
            '/hey': {
              target: 'https://google.com',
              pathRewrite: {'^/hey': ''},
            },
          },
        },
      };
    },
  };
}

Have you read the Contributing Guidelines on pull requests?
yes
Test Plan
local for now, but we should find a way to test the start/devServer workflow better
Fixes https://github.com/facebook/docusaurus/discussions/6100

@slorber Could this mean that we would be able to test things like ganalytics plugins in dev? By using an Express middleware? Didn't go into WDS deeply, just some wild thoughts

Hmm 🤔 maybe but this means we'll need to re-route GA calls to localhost to log them? Not sure how to do that
This can become quite verbose so not sure it should be enabled by default.
This API could be useful: https://webpack.js.org/configuration/dev-server/#devserveronaftersetupmiddleware
However I'm not sure the merge() fn will merge 2 plugins providing their own onAfterSetupMiddleware impl, and one might override the other 🤷♂
This seems technically possible but not so easy to do it right, and maybe a bit overkill. How users would use this? Is the purpose to just verify that a ga(event) fn is called? IMHO the errors are usually related to bad config, so if we create our own dev-only ga function, that wouldn't help users much debug real problems

Is the purpose to just verify that a ga(event) fn is called?

Not really—the purpose is to let people actually enable the plugin during dev. Something like this: https://github.com/verdaccio/verdaccio/pull/2772/files could lead to unexpected things leaking into production. Having an identical dev environment would work much better.

What does enabling the plugin in dev mean? How do users would know?
I see, we should probably add a warning for usage of themeConfig.googleAnalytics

So here there are two things:

We should probably check with 'googleAnalytics' in themeConfig to prevent even a value of undefined (a very minor improvement though);
More importantly is the ability to fully know if ganalytics is running properly during dev. If we can both (a) enable the plugin in dev and (b) not let dev navigations pollute prod analytics by proxying the requests to a local receiver, we reach a true isomorphic behavior.

More importantly is the ability to fully know if ganalytics is running properly during dev. If we can both (a) enable the plugin in dev and (b) not let dev navigations pollute prod analytics by proxying the requests to a local receiver, we reach a true isomorphic behavior.

You can't really know unless you really see the events in the GA console 🤷♂ if you have a proxy, then it might work with the proxy but fail once we send the analytics to the real service.

Mmm, that's true. The only case this could happen though (that I can think of) is that user has a bad tracking ID, which we can't protect against. But yeah, it was just a wild thought, probably not much useful


[move-lang] Remove sender arg. Add new module address syntax

Removed sender argument to compiler

Added inline syntax for addresses, i.e. 'module 0x1::M'

It is a very large, but mechanical PR. I tried to break up the PR into many different commits using fixup, which should make reviewing a bit easier. Let me know if this helps and I will do it more often.

Motivation

Not having an address come in from the command line will make nominal addressing a bit easier as the space is less complex
nominal addresses will be able to be set in different files and used for the entire compilation unit, which will effectively replace this feature

Test Plan

new test
updated old test

/land


[IOTDB-565] MQTT Protocol Support

MQTT Protocol
MQTT is a machine-to-machine (M2M)/"Internet of Things" connectivity protocol.
It was designed as an extremely lightweight publish/subscribe messaging transport.
It is useful for connections with remote locations where a small code footprint is required and/or network bandwidth is at a premium.
The MQTT v3.1(an OASIS Standard) protocol support for IoTDB. This module includes the MQTT broker that allows remote devices send messages into IoTDB server directly.

MQTT Broker
The MQTT Broker provide the ability of direct connection to IoTDB through MQTT. It listen the publish messages from MQTT clients and then
write the data into local or remote IoTDB server immediately.
The messages payload can be format to events by PayloadFormatter which loaded by java SPI, and the default implementation is JSONPayloadFormatter.
The configuration of the broker locate ${IOTDB_HOME}/${IOTDB_CONF}/iotdb-mqtt-broker.properties, and the startup scripts locate${IOTDB_HOME}/sbin/start-mqtt-broker.sh (.bat for windows platform)

MQTT Broker Configuration
The MQTT Broker load configuration from local IoTDB server ${IOTDB_HOME}/${IOTDB_CONF}/iotdb-mqtt-broker.properties by default.
If the local config file not exists, then load from classpath /iotdb-mqtt-broker.properties. Otherwise, load from the system defaults.
Configurations are as following:

NAME
DESCRIPTION
DEFAULT

mqtt_broker_host
the broker server address Required
<IP_ADDRESS>

mqtt_broker_port
the broker port  Required
1883

mqtt_broker_user
the broker username Required
root

mqtt_broker_password
the broker password Required
root

mqtt_broker_handler_pool_size
the publish handler pool size
1

mqtt_broker_payload_formatter
the message payload formatter
json

iotdb_host
the iotdb address Required
<IP_ADDRESS>

iotdb_port
the iotdb port Required
6667

iotdb_user
the iotdb username Required
root

iotdb_password
the iotdb password Required
root

Examples
The following is an example which a mqtt client send events to MQTT broker.
       MQTT mqtt = new MQTT();
       mqtt.setHost("<IP_ADDRESS>", 1883);
       mqtt.setUserName("root");
       mqtt.setPassword("root");

       BlockingConnection connection = mqtt.blockingConnection();
       connection.connect();

       Random random = new Random();
       for (int i = 0; i < 10; i++) {
           Map<String,Object> tuple = new HashMap();
           tuple.put("device", "root.sg.d1");
           tuple.put("timestamp", System.currentTimeMillis());
           tuple.put("measurements", "s1");
           tuple.put("values", random.nextDouble());

           String payload = JSON.toJSONString(tuple);
           connection.publish("root.sg.d1.s1", payload.getBytes(), QoS.AT_LEAST_ONCE, false);

           Thread.sleep(1000);
       }

       connection.disconnect();
   }

@qiaojialin @jixuan1989 Could you take a look? thanks.

Good work! The only one we may need to discuss is whether the MQTT broker should be launched as an internal service of iotdb server, just like the JDBC or the HTTP service.

Hi, I have not reviewed the codes, but I think  we can add a switch to enable or disable some features like new API when starting an IoTDB instance.
Actually, I am considering whether we can enable or disable some features when compiling IoTDB. Then we can get an embedded version for edge-side.

Thanks @qiaojialin @jixuan1989
I agree with you that MQTT broker should be launched as an internal service of iotdb server, and we can  add a switch in the server config file. I will refactor the code moving the logic to server module later.

Thanks @qiaojialin @jixuan1989
I agree with you that MQTT broker should be launched as an internal service of iotdb server, and we can add a switch in the server config file. I will refactor the code moving the logic to server module later.

Hi, I'd like to explain more about the idea:

if you are using session API to send MQTT data to IoTDB instance, then the module looks like a middle-ware. If so, it plays a similar role with IoTDB-Grafana-Connector. Then, do not integrate it into the server module.

If you are using TsServiceImpl API to handle MQTT data directly, then put it into the server module.

Thanks @qiaojialin @jixuan1989

I agree with you that MQTT broker should be launched as an internal service of iotdb server, and we can add a switch in the server config file. I will refactor the code moving the logic to server module later.

Hi, I'd like to explain more about the idea:

if you are using session API to send MQTT data to IoTDB instance, then the module looks like a middle-ware. If so, it plays a similar role with IoTDB-Grafana-Connector. Then, do not integrate it into the server module.

If you are using TsServiceImpl API to handle MQTT data directly, then put it into the server module.

It is not necessary to use the session when we decide to launch the mqtt as a internal service of iotdb server.

Hi, I have read the codes.

If we are using session API, I think we can keep it as a middleware, just like IoTDB-Grafana-Connector.
As a middleware, maybe the backend IoTDB instance is restarted, while the middleware is alive as usual. In this case, the session connection may be broken. So, an auto-retry will be a good experience for users because after restarting the IoTDB instance, they do not need to restart this middleware.
You can use a simple SessionWrapper org.apache.iotdb.session.pool.SessionPool to implement it.

And I have no other opinions. The code is beautiful.

These days I am in the hospital to take care of my new born child, and I will reply later.

These days I am in the hospital to take care of my new born child, and I will reply later.

Congratulations! Take care of everything.

I have completed the code refactoring. MQTT is a built-in service of the iotdb server.
Why as a built-in service?

As an IOT database, mqtt should be supported more natively (as a built-in service). IoT Native

As a built-in service, it can reduce the network and serialization overhead caused by session.

Two scenarios:
Scenario 1: mqtt accesses the local iotdb. In this case, it is unnecessary to use session.
Scenario 2: mqtt accesses the remote iotdb. In this case, the client can directly connect to the mqtt service of the remote iotdb. There is no need to transit through the session.

docs updated:
MQTT Protocol
MQTT is a machine-to-machine (M2M)/"Internet of Things" connectivity protocol.
It was designed as an extremely lightweight publish/subscribe messaging transport.
It is useful for connections with remote locations where a small code footprint is required and/or network bandwidth is at a premium.
IoTDB supports the MQTT v3.1(an OASIS Standard) protocol.
IoTDB server includes a built-in MQTT broker that allows remote devices send messages into IoTDB server directly.

Built-in MQTT Service
The Built-in MQTT Service provide the ability of direct connection to IoTDB through MQTT. It listen the publish messages from MQTT clients
and then write the data into storage immediately.
The messages payload can be format to events by PayloadFormatter which loaded by java SPI, and the default implementation is JSONPayloadFormatter.

MQTT Configurations
The IoTDB MQTT service load configurations from ${IOTDB_HOME}/${IOTDB_CONF}/iotdb-engine.properties by default.
Configurations are as following:

NAME
DESCRIPTION
DEFAULT

enable_mqtt_service
whether to enable the mqtt service
true

mqtt_host
the mqtt service binding host
<IP_ADDRESS>

mqtt_port
the mqtt service binding port
1883

mqtt_handler_pool_size
the handler pool size for handing the mqtt messages
1

mqtt_payload_formatter
the mqtt message payload formatter
json

Examples
The following is an example which a mqtt client send messages to IoTDB server.
       MQTT mqtt = new MQTT();
       mqtt.setHost("<IP_ADDRESS>", 1883);
       mqtt.setUserName("root");
       mqtt.setPassword("root");

       BlockingConnection connection = mqtt.blockingConnection();
       connection.connect();

       Random random = new Random();
       for (int i = 0; i < 10; i++) {
           Map<String,Object> tuple = new HashMap();
           tuple.put("device", "root.sg.d1");
           tuple.put("timestamp", System.currentTimeMillis());
           tuple.put("measurements", "s1");
           tuple.put("values", random.nextDouble());

           String payload = JSON.toJSONString(tuple);
           connection.publish("root.sg.d1.s1", payload.getBytes(), QoS.AT_LEAST_ONCE, false);

           Thread.sleep(1000);
       }

       connection.disconnect();
   }

@qiaojialin @jixuan1989  Please take a look again.

So... when the server started, only one Formatter instance will take effect. As this MQTT server is for write only, this is ok.

@qiaojialin @jixuan1989 Thanks for your detailed code review.

Hi, please merge the master to make ci more stable.

@qiaojialin  Done.

Besides, I see there are some CI failed because of the mqtt dependency is not found from maven repo.
It is strange that the "IoTDB-Flink Examples" failed because of that issue.
However, it inspires me that you should claim the mqtt maven repo address on mqtt-example module, too. It is because some users may just compile the example module (and skip compiling the server module). Then, On their local repo folder, there is no such a jar, and error will occurs.

@jixuan1989 I think have fixed the ci error, but it's strange that the ci build not automatic triggered.

However, it inspires me that you should claim the mqtt maven repo address on mqtt-example module, too. It is because some users may just compile the example module (and skip compiling the server module). Then, On their local repo folder, there is no such a jar, and error will occurs.

mqtt-client jar in the maven central repo, but moquette jar is not in (which is in https://jcenter.bintray.com).
It's unnecessary to add moquette maven repo address on mqtt-example module, because the mqtt-example is only dependent on mqtt-client. So it is OK even with users just build the example module.

@jixuan1989 I think have fixed the ci error, but it's strange that the ci build not automatic triggered.

Yeah  it is quite strange...
I manually trigger it and let's wait for the result.

@jixuan1989
Now there is a tsfile test error which is unrelated to mqtt after I merged the master to current branch.
Maybe someone should fix it first. And we should take effort to keep the master branch always passing the CI.

[ERROR] Tests run: 13, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.039 s <<< FAILURE! - in org.apache.iotdb.tsfile.write.writer.RestorableTsFileIOWriterTest
[ERROR] testBadHeadMagic(org.apache.iotdb.tsfile.write.writer.RestorableTsFileIOWriterTest)  Time elapsed: 0.001 s  <<< ERROR!
java.lang.Exception: Unexpected exception, expected<java.io.IOException> but was<java.lang.AssertionError>
at org.apache.iotdb.tsfile.write.writer.RestorableTsFileIOWriterTest.testBadHeadMagic(RestorableTsFileIOWriterTest.java:71)

Hi, yes I noticed the error.

@jixuan1989 I merged the master again, and all tests passed.

oh, exists some conflicting files due to flink-tsfile merged to master. I will update.


Add openqa_cli property to xml

The idea is that some projects can put custom wrapper instead of openqa-cli calls.
That wrapper can manage generated parameters as needed, make additional calls, etc.

Do not merge until the plugin is adjusted as well

This might be a good alternative to https://github.com/os-autoinst/openQA/pull/5757 (which is rather messy anyway, at least in its current form). So I guess the related ticket is https://progress.opensuse.org/issues/164388.

Do not merge until the plugin is adjusted as well

@andrii-suse what else is needed here? or you mean that some changes in another repo are needed?

@andrii-suse what else is needed here? or you mean that some changes in another repo are needed?

I've checked - the other repo should be fine, so this is good to go, feel free to merge

Let's try then :)