[SUPPORT]Failed to update metadata（hudi 0.15.0）

Environment Description

Hudi version : 0.15.0

Spark version : 3.5.1

Hive version : no

Hadoop version : 3.1.3

Storage (HDFS/S3/GCS..) : HDFS

Running on Docker? (yes/no) : no

Stacktrace
Add the stacktrace of the error.
24/07/08 15:02:16 ERROR HoodieStreamingSink: Micro batch id=5377 threw following exception:
org.apache.hudi.exception.HoodieException: Failed to update metadata
at org.apache.hudi.client.BaseHoodieClient.writeTableMetadata(BaseHoodieClient.java:262)
at org.apache.hudi.client.BaseHoodieWriteClient.commit(BaseHoodieWriteClient.java:294)
at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:239)
at org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:108)
at org.apache.hudi.HoodieSparkSqlWriterInternal.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:1082)
at org.apache.hudi.HoodieSparkSqlWriterInternal.writeInternal(HoodieSparkSqlWriter.scala:508)
at org.apache.hudi.HoodieSparkSqlWriterInternal.write(HoodieSparkSqlWriter.scala:187)
at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:125)
at org.apache.hudi.HoodieStreamingSink.$anonfun$addBatch$3(HoodieStreamingSink.scala:141)
at scala.util.Try$.apply(Try.scala:210)
at org.apache.hudi.HoodieStreamingSink.$anonfun$addBatch$2(HoodieStreamingSink.scala:133)
at org.apache.hudi.HoodieStreamingSink.retry(HoodieStreamingSink.scala:237)
at org.apache.hudi.HoodieStreamingSink.addBatch(HoodieStreamingSink.scala:132)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runBatch$17(MicroBatchExecution.scala:732)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:125)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:201)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:108)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:900)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:66)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runBatch$16(MicroBatchExecution.scala:729)
at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken(ProgressReporter.scala:427)
at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken$(ProgressReporter.scala:425)
at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:67)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runBatch(MicroBatchExecution.scala:729)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$2(MicroBatchExecution.scala:286)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken(ProgressReporter.scala:427)
at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken$(ProgressReporter.scala:425)
at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:67)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$1(MicroBatchExecution.scala:249)
at org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor.execute(TriggerExecutor.scala:67)
at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runActivatedStream(MicroBatchExecution.scala:239)
at org.apache.spark.sql.execution.streaming.StreamExecution.$anonfun$runStream$1(StreamExecution.scala:311)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:900)
at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:289)
at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.$anonfun$run$1(StreamExecution.scala:211)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
at org.apache.spark.JobArtifactSet$.withActiveJobArtifactState(JobArtifactSet.scala:94)
at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:211)
Caused by: java.lang.IllegalStateException
at org.apache.hudi.common.util.ValidationUtils.checkState(ValidationUtils.java:62)
at org.apache.hudi.metadata.HoodieMetadataPayload.lambda$createPartitionFilesRecord$2(HoodieMetadataPayload.java:335)
at java.util.HashMap.forEach(HashMap.java:1289)
at org.apache.hudi.metadata.HoodieMetadataPayload.createPartitionFilesRecord(HoodieMetadataPayload.java:332)
at org.apache.hudi.metadata.HoodieTableMetadataUtil.lambda$convertMetadataToFilesPartitionRecords$6(HoodieTableMetadataUtil.java:425)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1699)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
at org.apache.hudi.metadata.HoodieTableMetadataUtil.convertMetadataToFilesPartitionRecords(HoodieTableMetadataUtil.java:428)
at org.apache.hudi.metadata.HoodieTableMetadataUtil.convertMetadataToRecords(HoodieTableMetadataUtil.java:356)
at org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.lambda$updateFromWriteStatuses$22(HoodieBackedTableMetadataWriter.java:920)
at org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.processAndCommit(HoodieBackedTableMetadataWriter.java:861)
at org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.updateFromWriteStatuses(HoodieBackedTableMetadataWriter.java:918)
at org.apache.hudi.client.BaseHoodieClient.writeTableMetadata(BaseHoodieClient.java:257)

The writing program goes through multiple deltacommit processes and finally, by observing the Spark UI, it is noted that the data flow is stalled and no further calculations are being performed.

These errors occurred when switching from version 0.14.1 to 0.15.0, and the Hudi write configurations have not changed.

Thanks for the feedback, the error stacktrace is not very detailed, it would be very helpful if you can past the "caused by" part, @nsivabalan , not sure if it is related with the rollback fixes for mor table.

Thanks for the feedback, the error stacktrace is not very detailed, it would be very helpful if you can past the "caused by" part, @nsivabalan , not sure if it is related with the rollback fixes for mor table.

The text above contains "Caused by".

Thanks for the feedback, the error stacktrace is not very detailed, it would be very helpful if you can past the "caused by" part, @nsivabalan , not sure if it is related with the rollback fixes for mor table.

By checking with timeline show active --limit 1000 --with-metadata-table, there is no record of any INFLIGHT or similar behavior under the metadata-table corresponding to the deltacommit that has been in INFLIGHT status, and it shows all as '-'.
When the deltacommit has been in INFLIGHT status for a long time, why is there no rollback but instead the next deltacommit is performed? I completely do not understand this.
I have tried other attempts: disabling multi-write and enabling single-write, but the same exception error still occurs.

Thanks for the feedback, the error stacktrace is not very detailed, it would be very helpful if you can past the "caused by" part, @nsivabalan , not sure if it is related with the rollback fixes for mor table.

When switching from 0.14.1 to 0.15, a significant number of rollbacks were observed in the timeline, and another table experienced continuous rollbacks yesterday, preventing data from being written.

It does have a file size check in branch 0.15.0: , but the check is removed from master in commit: https://github.com/apache/hudi/pull/9765, and not sure it is related with the rollback fix: https://github.com/apache/hudi/pull/11187, cc @codope @nsivabalan , can you check whether it is related with the fix?

Looks like it is related with another issue: https://github.com/apache/hudi/issues/11594, they should belongs to the same culprits.

@MrAladdin Is it possible for you to zip and share the complete .hoodie directory? We need to inspect the commit metadata and see how come 0-byte files being added to the files metadata.

In the meantime, you can disable metadata table to unblock yourself
hoodie.metadata.enable=false

In the meantime, you can disable metadata table to unblock yourself hoodie.metadata.enable=false

It uses the record_index index, so it cannot be closed.

@MrAladdin Is it possible for you to zip and share the complete .hoodie directory? We need to inspect the commit metadata and see how come 0-byte files being added to the files metadata.

After compression, the .hoodie file is 550MB in size, which might be inconvenient to provide. The issue at https://github.com/apache/hudi/issues/11594 appears to be part of a series of related problems. You can find my specific production configuration there, which is the same as the 0.14.1 version, and 0.14.1 works correctly.

hey @MrAladdin : are you in hudi slack. we can connect and investigate faster. can you post a msg there and tag me (shivnarayan) and sagar (sagar sumit)

also, do you think you can use global simple in the mean time while we try to find the root cause and get a fix out?

also, do you think you can use global simple in the mean time while we try to find the root cause and get a fix out?

Due to my business scenario involving a large number of upsert operations (public opinion data), other index types did not perform well in previous tests. Only dynamic bucket and the newly released record_index in version 0.14 met the requirements. I have always wanted to find an index that doesn't require manual intervention or adjustment, so record_index has attracted my attention and interest. This test is mainly focused on the performance improvements of record_index in version 0.15. I can wait for this to be fixed before conducting tests. Actually, there is another issue mentioned in https://github.com/apache/hudi/issues/11567. When the amount of already stored data is huge, it is also a maddening issue. You can pay attention to this as well.

hey @MrAladdin : are you in hudi slack. we can connect and investigate faster. can you post a msg there and tag me (shivnarayan) and sagar (sagar sumit)

I'm really sorry, but due to certain reasons, I am unable to help you.

@MrAladdin Or maybe a quick google meeting sync, that would be helpful.

@MrAladdin Is it possible for you to zip and share the complete .hoodie directory? We need to inspect the commit metadata and see how come 0-byte files being added to the files metadata.

Hello, it seems like I've discovered something new, possibly related to the hoodie.allow.empty.commit parameter. There appears to be a conflict with the metadata check mechanism in version 0.15.0. I have disabled it in the code, and so far, it's been running for 4 hours without any issues. I'll monitor the situation for a while longer before providing further feedback.

@MrAladdin Or maybe a quick google meeting sync, that would be helpful.

Sure, here is the translation:
Thank you for your positive attitude, thumbs up to you, hahaha. My spoken English isn’t very good. I just provided some feedback on the test results above. I am also looking into the source code and continuously trying to resolve the issue. I will do my best to identify the problem.

@nsivabalan @ad1happy2go @danny0405 @codope  After reviewing the source code and observing the long-running task following the closure of the hoodie.allow.empty.commit parameter, it has been preliminarily confirmed that the anomaly is indeed caused by the empty commit.

@nsivabalan @ad1happy2go @danny0405 @codope  There are a large number of metadata-related jobs missing from the Spark UI due to an exceptional deltacommit that has been consistently in the INFLIGHT state. Furthermore, using the command commit showfiles --commit exception_deltacommit_id does not reveal any file write information. Additionally, the .hoodie/.temp/ directory contains folders and data corresponding to these exception_deltacommit_ids.
exception ：

normal ：

Here's a question for everyone: Can anyone tell me why there are INFLIGHT deltacommit but no rollback? These deltacommit should be failures, right?

I guess the rollback strategy might be condifured as lazy, these are data table delta commits right?

Here's a question for everyone: Can anyone tell me why there are INFLIGHT deltacommit but no rollback? These deltacommit should be failures, right?

I guess the rollback strategy might be condifured as lazy, these are data table delta commits right?

I will study and understand lazy rollback later.  These are data table delta commits （timeline show active --limit 1000 --with-metadata-table.  Do you have any other discoveries or thoughts about this exception?

Change the version back to 0.14.1 again, all parameters remain consistent with 0.15.0, and there are no issues with the program running.  emo~

@nsivabalan @ad1happy2go @danny0405 @codope
The reason has been found: spark.speculation conflicts with 0.15. Everything works fine when spark.speculation is disabled.


Move the 'box2d' recipe.

Move the recipe from the old github repo into this one and create a recipe license file for it.
Tasks:

[ ] Move the recipe inside here. (update if necessary)
[ ] Create the recipe license file.

IceShard is moving away from box2d in favor of Chipmunk2d (as of today).
This package will no longer be kept up to date.


find a bug when using text_frame.fit_text(), not deal the situation when returned None

when using
text_frame.fit_text('Arial', max_size=18)
it crushed, the log for this part is as follows.

And I enter in the line here text, remainder = self._break_line(line_source, point_size), find the problem is in find_max(),

It thinks this will return a object has this two attribute text, remainder , but when it returned None, this line will crush.

What is the size of the textbox and what text are you trying to fit into it? Also, in future, please post traceback as text and code as links to locations in this repo to ease navigating to it.


Importing hydra.experimental results with an ImportError

Hey!
Firstly, thanks for creating and maintaining the package!
In my project I use both fairseq & hydra (hydra-core) packages. Importing hydra's experimental components results with an ImportError while using hydra's version_base at least 1.2.
IMO, the problematic line in your code:
https://github.com/facebookresearch/fairseq/blob/34973a94d09ecc12092a5ecc8afece5e536b7692/fairseq/dataclass/utils.py#L19
In my case, the ImportError in hydra-core is being raised here.
My suggestion in to remove the .experimental .
Thanks in advance!

same issue here. have to manually remove to get it to work. maybe add it as
try:
    from hydra.experimental import compose, initialize
except ImportError:
    from hydra import compose, initialize


Fix docker compose file

Current Behavior
The docker compose up command currently fails during the startup of the indexer-setup container. As a result, the relayer container do not run, leading to incorrect system operation. This failure occurs because the indexer-setup container attempts to deploy contracts that already exist in initial state.
New Behavior
Remove the indexer-setup container entirely from the system.
Breaking Changes
No breaking changes are expected.

As discussed, the issue in this case is the volumes aren't cleared on each run, so the NEAR CLI data already exists even though the localnet is reset. Closing this for now and I should open a new PR soon.


Combined with onnx.load(<filename>) to run onnxinfo

Present implementation requires another read_onnx(<filename>) to get the ONNX file, but python and ONNX users usually import onnx, I think it is better not to read the same file two times.

Finding that the type of model in onnx.load and my own _onnxinfo.read_onnx is different.
_onnxinfo.read_onnx: <class '_onnxinfo.ModelProto'> onnx.load: <class 'onnx.onnx_ml_pb2.ModelProto'>


Remove redundant 502 & 504 response codes

Already handled by the 500..599 range.
From PR chat in https://github.com/rinsed-org/rb-snowflake-client/pull/45/files#r1675761722.

for the record: @alexstoick and I chatted and made this change: https://github.com/rinsed-org/rb-snowflake-client/pull/47
so that we don't need to always bump the version.


Just rebuild to see if anything stalls

some runs stall upon exit from pytest in
https://github.com/conda-forge/pyout-feedstock/pull/7
want to see if we see that here without adding 3.9/rerendering (if feasible)

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

yeap , stalled on OSX so it is more likely due to either changes in conda setup or dependencies than due to changes in pyout


Remove remains of jupyter bundlerextension

If you install or update jupyter_server after notebook the executable jupyter-bundlerextension will be overwritten because of the entrypoint defined in setup.py and the conda-forge-recipe. You will get the following error if you execute the following example command jupyter bundlerextension list
Traceback (most recent call last):
  File "/Users/USER/opt/anaconda3/bin/jupyter-bundlerextension", line 7, in <module>
    from jupyter_server.bundlerextensions import main
ModuleNotFoundError: No module named 'jupyter_server.bundlerextensions'

This can be fixed by reinstalling notebook with conda install notebook --force-reinstall, however this PR removes the remaining parts (left of https://github.com/jupyter/jupyter_server/pull/47) of the bundler from the codebase.

:+1: This is fine for now.
I think we're going to keep the bundler in Jupyter Server (though there hasn't been any discussion around it, yet). That said, we'll need to rename the entrypoint and avoid colliding with notebook for a little while. If we do keep the bundler app in Jupyter Server, Notebook will lose the bundlerextensions entrypoint in favor of JS—or it at least it be shimmed to Jupyter Server;s bundler app.
In the meantime, removing the entrypoint here is fine. We'll need to create a new entrypoint in a future PR.

Thanks, @m-rossi! Merging.


First line of code block in Tutorials is not indenting

First line of code block is not indenting in the Tutorials.
In this example, all lines are indented the same but the first line gets compiled differently. I also tried switching the indent format from tabs to spaces and it made no difference.

Fixed, thanks @Robert-Frampton!


Fix the fake descriptor to present a more realistic method name.

This is extracted from https://github.com/googleapis/gax-java/pull/613/files#diff-13d1e54be780cb9d5ec8bbe6a8f0ec9f.
It allows for easier testing of future opencensus integration

Codecov Report

Merging #631 into master will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #631   +/-   ##
=========================================
  Coverage     74.94%   74.94%
  Complexity      935      935
=========================================
  Files           177      177
  Lines          4099     4099
  Branches        328      328
=========================================
  Hits           3072     3072
  Misses          874      874
  Partials        153      153

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1d492dc...3637a38. Read the comment docs.


Add numberpad for phone input

This is Gboard example

TODO:

[ ] keyboard needs to activate when the keyboard is inflated on a phone input

Nice to have's

letter display on keys
larger text for numbers

TODO:

[ ] keyboard needs to activate when the keyboard is inflated on a phone input

Nice to have's

letter display on keys
larger text for numbers


[feat] add benchmark + some tiny chores

remove some unnecessary syntax
add benchmark to check whether sequitur is indeed linear
add some things to .gitignore

I have also got some ideas for possible performance improvements and I also have a nix setup stashed, so let me know if you'd like these upstreamed.

I also wanted to implement this algorithm a while ago but I didn't find the time. What you did is pretty much what I started doing as well. It's a pity that there's no more "functional" datastructure with the same properties as the HashTable + doubly linked list combination.

I can also add the speedscope results but they're not so useful as everything inlines into encode and add, so you end up with something like this:

where the little fizzles are calls to Hashable

Thank you for the PR!

I have also got some ideas for possible performance improvements and I also have a nix setup stashed, so let me know if you'd like these upstreamed.

Of course, performance improvements are welcome.
The only improvement I can think of now is to use PrimVar.
I'm not a nix user and can't maintain a nix setup, but it's ok to add if you want.

I also wanted to implement this algorithm a while ago but I didn't find the time.

I had been the same. Then this year's ICFP programming contest came in. In some of its problems, data compression may improve the score. I noticed the possibility of using the sequitur algorithm. Although I did not have time to implement it during the contest, I began implementing it after the contest ended.
By the way, do you have any applications you would like to use this implementation for?

By the way, do you have any applications you would like to use this implementation for?

in fact, I'm using it right now because I'm writing my BSc. thesis about calculating the longest common prefix of context free grammars, not only is an SLP a nice grammar to test roundtripping on, but the algorithm also crucially depends on the grammar at hand to not be expanded, i.e. the languages we look at should be present as a set of compressed strings.

in fact, I'm using it right now because I'm writing my BSc. thesis about calculating the longest common prefix of context free grammars, not only is an SLP a nice grammar to test roundtripping on, but the algorithm also crucially depends on the grammar at hand to not be expanded, i.e. the languages we look at should be present as a set of compressed strings, so maybe I will use it to improve some things there.

Sorry for the late reply.
This looked like an interesting project.
I would be delighted if the code in this repository is helpful.


[Doc] bucketing is applicable to all file-based data sources

What changes were proposed in this pull request?
Starting Spark 2.1.0, bucketing feature is available for all file-based data sources. This patch fixes some function docs that haven't yet been updated to reflect that.
How was this patch tested?
N/A

Test build #70399 has started for PR 16349 at commit c8f1b42.

Test build #70399 has finished for PR 16349 at commit c8f1b42.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70399/
Test PASSed.

Merging in master/branch-2.1.


Spaceship::Portal raising UnexpectedResponse instead of ProgramLicenseAgreementUpdated

New Issue Checklist

[x] Updated fastlane to the latest version
[x] I read the Contribution Guidelines
[x] I read docs.fastlane.tools
[x] I searched for existing GitHub issues

Issue Description

When using Spaceship::Portal, it is raising Spaceship::UnexpectedResponse exception if there are pending agreements instead of Spaceship::ProgramLicenseAgreementUpdated.
The reason behind that is, the code:
https://github.com/fastlane/fastlane/blob/b4f34772f48e5ad17604e831192b2157af3a2d5b/spaceship/lib/spaceship/client.rb#L741-L743
is looking for Program License Agreement in resultString field, while the error message is in userString. Error I get:
Spaceship::UnexpectedResponse ({
  "responseId"=>"78ad8f20-7806-4afb-9130-1d6c276068a0",
  "resultCode"=>4550,
  "resultString"=>"Access Unavailable",
  "userString"=>"You currently don't have access to this membership resource. To resolve this issue, your team's Account Holder, John Doe, must agree to the latest Program License Agreement.",
  "creationTimestamp"=>"2019-07-02T19:34:13Z",
  "protocolVersion"=>"QH65B2",
  "userLocale"=>"en_US",
  "requestUrl"=>"https://developer.apple.com/services-account/QH65B2/account/ios/identifiers/listAppIds.action",
  "httpCode"=>200
})

Command executed

bundle console
require 'spaceship'
Spaceship.login
Spaceship::Portal.client.team_id = team_id
Spaceship::Portal::App.find('com.company.appid')

Please check the error message. If you don’t have access to program resources it looks like you cannot execute that command.

i have updated my post to include all commands i am running to reproduce this issue

Should be related to https://github.com/fastlane/fastlane/issues/14880#issuecomment-509367895 Can we close this?

I dont think this is related to that.

Agreed, this is a different issue. As said in the original post, the code only looks at resultString, but the response from the API that includes "Program License Agreement" is in userString.

Would appreciate if we can reopen discussion on this as this bug is still prevalent. So far i've had to patch Spaceship::Client.detect_most_common_errors_and_raise_exceptions(body) with the suggested fix by @salmanasiddiqui to be able to raise the appropriate ProgramLicenseAgreementUpdated exception when there are pending agreements.

Make a pull request with your patch, so we can review it and integrate it.

@max-ott here's the pull request: https://github.com/fastlane/fastlane/pull/15193


Some fixes in markdown documents

Changes in GitHub's markdown parser caused the lists to render incorrectly. See https://github.com/blog/2333-a-formal-spec-for-github-flavored-markdown
I also fixed some other issues I noticed like heading sizes and broken links.

Cheers! :beers:


complete the gentoo init

This works for me...

The frabjous overlay have a really simple and working well init script, why not using this one?
#!/sbin/openrc-run
# Copyright 1999-2017 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

description="Gogs, a self-hosted Git service"

: ${GOGS_USER:=git}
: ${GOGS_GROUP:=git}
: ${GOGS_LOG:="/var/log/${SVCNAME}/${SVCNAME}.log"}
: ${GOGS_OPTS:="--config /var/lib/gogs/conf/app.ini"}

command="/usr/bin/gogs"
command_args="web ${GOGS_OPTS}"
command_background=yes
pidfile="/run/${SVCNAME}/${SVCNAME}.pid"
start_stop_daemon_args="-u ${GOGS_USER}:${GOGS_GROUP} \
	-e GOGS_CUSTOM=/var/lib/gogs -e USER=${GOGS_USER} \
	-1 ${GOGS_LOG} -2 ${GOGS_LOG}"

depend() {
	need localmount net
}

start_pre() {
	checkpath -d -m 0755 -o "${GOGS_USER}":"${GOGS_GROUP}" "${pidfile%/*}"
}

Hmm... @acidwars  any updates to the review comments made by @csmk?

@acidwars please, consider this revision I made based on the one distributed in my overlay.
#!/sbin/openrc-run
# Copyright 1999-2017 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

description="Gogs, a self-hosted Git service"

: ${GOGS_BIN:="$(which gogs)"}
: ${GOGS_USER:=gogs}
: ${GOGS_HOME:="/var/lib/gogs"}
: ${GOGS_LOG:="/var/log/gogs/${RC_SVCNAME}.log"}
: ${GOGS_CONFIG:="${GOGS_HOME}/conf/app.ini"}
: ${SSD_OPTS:="--wait 1000 -1 ${GOGS_LOG} -2 ${GOGS_LOG}"}

name="${GOGS_NAME:-"Gogs"}"
command="${GOGS_BIN}"
command_args="web --config ${GOGS_CONFIG} ${GOGS_OPTS}"
command_user="${GOGS_USER}"
command_background=true
pidfile="/run/${RC_SVCNAME}.pid"
required_files="${GOGS_CONFIG}"
retry=${GOGS_TERMTIMEOUT:-"TERM/60/KILL/5"}
start_stop_daemon_args="${SSD_OPTS} \
	-e \"GOGS_CUSTOM=${GOGS_HOME}\" \
	-e \"USER=${GOGS_USER}\""

depend() {
	need localmount net
}

start_pre() {
	checkpath -d -m 0700 -o ${GOGS_USER} "${GOGS_HOME}" "${GOGS_LOG%/*}"
	checkpath -f -m 0600 -o ${GOGS_USER} "${GOGS_CONFIG}"
}

As you can see, instead of using "hardcoded" variables, the user can easily changes it by just creating a file using the same name from the init script, which must be located in /etc/conf.d (e.g. /etc/conf.d/gogs). This way we can keep the init script untouched.
Note: If the user wants to define a group that is not the primary one, he/she just needs to add a colon between username and group in the variable GOGS_USER (e.g. myuser:myanothergroup).

I also suggest to change the name of folder "gentoo" in Gogs's repo, to "openrc", because the init script above can also be used by other OpenRC-based distros, like Alpine Linux and Artix.

As I am not using Gogs on Gentoo anymore, maybe there should be a new pull request with the change from @csmk


Aop.logging

AOP: Logging.
@sergey-moiseyenko this PR is going to be rebased after #7 is merged.

@sergey-moiseyenko PR is available for review, thx

everything is fine


100% CPU usage when not dropping OutputStream, in Ubuntu Linux, using ALSA

The following program uses 100% of a CPU, according to htop, in release mode, whether any key on the keyboard is pressed or not.
use winit::{
    event::{Event, WindowEvent},
    event_loop::{EventLoop, ControlFlow},
    window::WindowBuilder,
    event::ElementState,
};

fn main() {
    let (sender, receiver) = std::sync::mpsc::channel();

    std::thread::spawn(move || {
        let output = match rodio::OutputStream::try_default() {
            Ok(output) => output,
            Err(_) => return,
        };

        while let Ok(_message) = receiver.recv() {
            // In a real application play a sound selected by the message
        }
    });

    let event_loop = EventLoop::new();

    let window = WindowBuilder::new()
        .build(&event_loop)
        .unwrap();

    event_loop.run(move |event, _, control_flow| {
        match event {
            Event::WindowEvent {
                event: WindowEvent::CloseRequested,
                window_id,
            } if window_id == window.id() => *control_flow = ControlFlow::Exit,
            Event::WindowEvent {
                event: WindowEvent::KeyboardInput{
                    input: winit::event::KeyboardInput {
                        state: ElementState::Released,
                        ..
                    },
                    ..
                },
                window_id,
            } if window_id == window.id() => {
                let _ = sender.send(());
            }
            Event::MainEventsCleared => {
                std::thread::sleep(std::time::Duration::from_millis(16));
            }
            _ => (),
        }
    });
}

If I comment out these lines, then the CPU usage drops down to near 0% :
let output = match rodio::OutputStream::try_default() {
    Ok(output) => output,
    Err(_) => return,
};

Am I using the library in a strange way? If so, I don't understand what the preferred way is to mix together an unknown number of sounds, starting at different times, in response to user input, given that the sounds stop when the OutputStream is dropped,
Some versions that may be relevant:
[dependencies]
rodio = "0.15.0"
winit = "0.26.1"

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.4 LTS
Release:	20.04
Codename:	focal
$ cat /proc/asound/version
Advanced Linux Sound Architecture Driver Version k5.13.0-48-generic.
$ aplay --version
aplay: version 1.2.2 by Jaroslav Kysela<EMAIL_ADDRESS>
Possibly related CPAL issue

I've since had the chance to try this program on some other machines running Linux. On those other machines the CPU usage was much lower. But on the original machine, (hereafter "Machine A") the 100% CPU behaviour persists.
I've also tried this CPAL example, (extending the time so I could get a CPU measurement) and found it also used 100% CPU on Machine A. CPAL relies on the alsa crate. I additionally tried the PCM example from the alsa docs, (again extending the time,) and found that it did not use 100% CPU, but instead used a much more reasonable amount.
Combining the previous two paragraphs, I am lead to the conclusion that some weird combination of what Rodio/CPAL is doing, and Machine A's sound hardware/driver is causing the issue.

Have you considered using receiver.try_recv() with a sleep if the receiver is empty? recv() blocks the thread until an object is received.

The goal I had that lead me to run into this issue was to have a dedicated thread to play sounds in response to messages. I wanted the thread to be running for as long as the entire program was running, so I can just send a message without blocking the main thread, or waiting for the thread to start up before the sound is played.
So, I'd need to keep the thread alive somehow. Given that, I'm not sure how using try_recv would help, since the only way I can think of to keep the thread alive with try_recv would be to loop around and try it again, maybe after a sleep. But if it sleeps, then that introduces latency, and if it doesn't sleep, isn't that just busy-waiting, which would then again use 100% of a CPU?
Somewhat frustratingly, I've now tried the above program on Machine A, and it no longer consumes as much CPU. I guess my audio driver must have been updated automatically or something? I do not recall doing anything that seems like it should have affected this myself. As of this writing, Machine A is still using Rust 1.62.0, for example.
So, it's mysterious to me why this problem appeared and then disappeared, but I suppose it has disappeared.


Varrany stops working in Windows 11

I am trying to get my Varanny / Radiomail setup working.
I am having issues with Varanny starting then I connect with Radiomail than Varanny disconnects with a an error message:
[WASAPI] Data Discontinuity (possible overrun). Attempting recovery.
2024/10/26 21:32:51 Client closed the connection
2024/10/26 21:32:51 Cleaning up after closing connection
Stopping monitoring...
Then the black command screen just stays open.
I am running Varrany in a Lenovo Thinkcentre M700 with Windows 11 installed. I have followed the instructions the best that I can.
Vara has been installed on this PC as well as Winlink Express and they both work with no issues. I also installed MyPublicWIFI and set it up per your instructions. I can connect to PC with my IPhone. I am using a digirig and a DRA30 as sound cards and both have been tested and working properly with Winlink. I also purchased the display port emulator plugs (this pc has display ports instead of HDMI) for use as a headless computer when it finally works.
I edited the json file as suggested in the instructions. Had some problems editing but it finally worked.
I am a radio technician and an experienced ham repairing and putting together hardware and antennas but I am not an IT professional. I know my way around computers through experience (that is how I've gotten this far) but I am not an expert.
Any suggestions on getting my setup working will be greatly appreciated.

Screen shot of json file and of my failed “build from source”.

Looks like you're running into a problem when using the audio monitor from RadioMail? From what I can tell, seems that varanny is running into an exception when trying to connect to the sound card.
I have not tested Windows 11 so, not sure if this is a problem with that or perhaps that particular sound card.
This functionality is not required to get VARA to work. Have you tried to establish a connection in RadioMail? It should launch VARA for you.

Thank you very much for your reply.
When I try to connect, once I choose which radio to use, the connection is
established and For the short period it is working (only a few seconds) I
can check the audio monitor and it is working. Shortly after that the
connection is lost. The program does not give me time to send an email.
What should I try now?
Felipe Guilfuchi Ramos
WP4DMD
On Sat, Nov 2, 2024 at 1:41 PM Georges Auberger @.***>
wrote:

Looks like you're running into a problem when using the audio monitor from
RadioMail? From what I can tell, seems that varanny is running into an
exception when trying to connect to the sound card.
I have not tested Windows 11 so, not sure if this is a problem with that
or perhaps that particular sound card.
This functionality is not required to get VARA to work. Have you tried to
establish a connection in RadioMail? It should launch VARA for you.
—
Reply to this email directly, view it on GitHub
https://github.com/islandmagic/varanny/issues/18#issuecomment-2453058922,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BMPCIZY6JA7JT3ZTFA2IWQTZ6UFEFAVCNFSM6AAAAABQWMFNICVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINJTGA2TQOJSGI
.
You are receiving this because you authored the thread.Message ID:
@.***>

Don't go into the Audio Input Monitor in RadioMail. Just select the radio and go back to the main screen. From there, try to connect to a station.

Thanks for the information.  I will also test it out with the DRA30 souncard and let you know
how it goes.
On Sat, Nov 2, 2024 at 2:01 PM Georges Auberger @.***>
wrote:

Don't go into the Audio Input Monitor in RadioMail. Just select the radio
and go back to the main screen. From there, try to connect to a station.
—
Reply to this email directly, view it on GitHub
https://github.com/islandmagic/varanny/issues/18#issuecomment-2453064040,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BMPCIZ7PZ6P5XDOFVASAJZ3Z6UHPRAVCNFSM6AAAAABQWMFNICVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINJTGA3DIMBUGA
.
You are receiving this because you authored the thread.Message ID:
@.***>

Tested with DRA30 sound card and same issues. Ran Varanni without checking audio monitor. It did not generate overrun. I tried to reconnect to do more tests but could not find my modems / radios. I had to reset pc for more tests. My public wifi won’t let me reset pc until I change wifi connection on my iphone.

I finally made a connection and sent an email. Yes, the problem is with the audio monitor. I will do more tests with the DRA30 sound card and my base radio. I will post the results. I am still getting used to using Radiomail and Varanni. I am more familiar with Winlink.

Connection made with DRA30. Both souncards work. I did not touch the Audio Monitor.

Had problems uploading images from my iphone. will try later.

Image of Varanni connecting with DRA30 soundcard which did not upload yesterday.

Image of succesful connection which did not uplod yesterday.

Great to hear you got it working.

I am still running tests. It is working despite the error message. I have
been testing with both soud cards and have been using the audio monitor on
purpose and have been able to reconnect and send emails. I still have to
setup Varanni to run at startup. I downloaded a tutorial on setting this
up. I will post some more screen shots so you can study them.
On Sun, Nov 3, 2024 at 9:37 PM Georges Auberger @.***>
wrote:

Great to hear you got it working.
—
Reply to this email directly, view it on GitHub
https://github.com/islandmagic/varanny/issues/18#issuecomment-2453686911,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BMPCIZZJBLIVZBEQPUV2YQLZ63FUTAVCNFSM6AAAAABQWMFNICVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINJTGY4DMOJRGE
.
You are receiving this because you authored the thread.Message ID:
@.***>

Varrani is working despite error message. Tested with both soundcards, digirig and dra30.

Oh,…and the audio monotor IS working!

So, everything is working then?

I closed the issue by accident and then reopened it.
Things look like they are working. Should the error message be ignored?
Will there be an update? Are there any other suggestions to make the
instalation and configuring of Varanni a bit more troublefree for us none
coders?
On Sun, Nov 3, 2024 at 10:10 PM Georges Auberger @.***>
wrote:

So, everything is working then?
—
Reply to this email directly, view it on GitHub
https://github.com/islandmagic/varanny/issues/18#issuecomment-2453709431,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BMPCIZ2QR2X23DQR5GVWLWTZ63JPRAVCNFSM6AAAAABQWMFNICVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINJTG4YDSNBTGE
.
You are receiving this because you modified the open/close state.Message
ID: @.***>

Yeah the messages are mostly from other libraries so nothing to worry about.

VARA stores your license key in the .ini file. If you have created specific .ini files for a configuration that varanny uses, make sure that .ini file has the key in it too.

I just received your reply. How is the Vara .ini file edited to add this information?
I also have an issue with My Public Wifi. When computer boots up, after My Public Wifi starts, the setup screen opens up and there information page opens up in the background. I can shut down the information page but not the setup screen. This happens intermitantly.

Original Vara .ini file

Varafm.digirig.ini file

Varafn.dra30.ini file

Varafm.dra30.ini

I deleted your screen shot because I don't think it's a good idea to post your license information on public boards. You can edit .ini file with a text editor. I'm afraid I can't spend more time troubleshooting your installation.

I am sorry for posting license information. Did it unintentionaly. Just one last comment / question before closing this issue. All three .ini files had the license and activation code. Is there smething else I should verify to correct this problem?


Add includeLocalCovers false to /movies endpoint Radarr Calls

https://github.com/Radarr/Radarr/pull/8832
don't need the local media covers file paths.....let's be more efficient
prior to fixup; massive speed improvements for posterity
UI call: 25s and 55s
List Call with includeLocalCovers=False: sub 1sec
2023-07-17 21:55:03.3|Trace|Http|Req: 26 [GET] /radarr/api/v3/movie (from 73.*.*.12 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36)
2023-07-17 21:55:48.0|Trace|Http|Req: 77 [GET] /radarr/api/v3/movie?includeLocalCovers=False (from <IP_ADDRESS> Radarr/<IP_ADDRESS>57 (ubuntu 22.04))
2023-07-17 21:55:48.0|Debug|Radarr.Http.Authentication.ApiKeyAuthenticationHandler|AuthenticationScheme: API was successfully authenticated.
2023-07-17 21:55:48.6|Trace|Http|Res: 77 [GET] /radarr/api/v3/movie?includeLocalCovers=False: 200.OK (590 ms)
2023-07-17 21:55:48.8|Trace|Http|Req: 78 [GET] /radarr/api/v3/movie?includeLocalCovers=False (from <IP_ADDRESS> Radarr/<IP_ADDRESS>57 (ubuntu 22.04))
2023-07-17 21:55:48.8|Debug|Radarr.Http.Authentication.ApiKeyAuthenticationHandler|AuthenticationScheme: API was successfully authenticated.
2023-07-17 21:55:49.4|Trace|Http|Res: 78 [GET] /radarr/api/v3/movie?includeLocalCovers=False: 200.OK (594 ms)
2023-07-17 21:55:54.1|Trace|Scheduler|Pending Tasks: 0
2023-07-17 21:55:54.2|Trace|Http|Req: 79 [GET] /radarr/api/v3/movie?includeLocalCovers=False (from <IP_ADDRESS> Radarr/<IP_ADDRESS>57 (ubuntu 22.04))
2023-07-17 21:55:54.2|Debug|Radarr.Http.Authentication.ApiKeyAuthenticationHandler|AuthenticationScheme: API was successfully authenticated.
2023-07-17 21:55:54.8|Trace|Http|Res: 79 [GET] /radarr/api/v3/movie?includeLocalCovers=False: 200.OK (625 ms)
2023-07-17 21:55:59.1|Trace|Http|Res: 73 [GET] /radarr/api/v3/movie: 200.OK (25131 ms)
2023-07-17 21:55:59.2|Trace|Http|Res: 26 [GET] /radarr/api/v3/movie: 200.OK (55865 ms)

this is now in radarr master - the call to add is excludeLocalCovers=True


more of a question

I was trying to adapt the sine wave example to play the Mario tune, but my timing is all off...
Would you mind taking a look and advising me on where I've gone wrong?
The following code is partially adapted from https://www.princetronics.com/supermariothemesong/
import os
import sequtils
import strformat

import rapid/audio/samplers/osc
import rapid/audio/device

const
  NOTE_B0   = 31.float
  NOTE_C1   = 33.float
  NOTE_CS1  = 35.float
  NOTE_D1   = 37.float
  NOTE_DS1  = 39.float
  NOTE_E1   = 41.float
  NOTE_F1   = 44.float
  NOTE_FS1  = 46.float
  NOTE_G1   = 49.float
  NOTE_GS1  = 52.float
  NOTE_A1   = 55.float
  NOTE_AS1  = 58.float
  NOTE_B1   = 62.float
  NOTE_C2   = 65.float
  NOTE_CS2  = 69.float
  NOTE_D2   = 73.float
  NOTE_DS2  = 78.float
  NOTE_E2   = 82.float
  NOTE_F2   = 87.float
  NOTE_FS2  = 93.float
  NOTE_G2   = 98.float
  NOTE_GS2  = 104.float
  NOTE_A2   = 110.float
  NOTE_AS2  = 117.float
  NOTE_B2   = 123.float
  NOTE_C3   = 131.float
  NOTE_CS3  = 139.float
  NOTE_D3   = 147.float
  NOTE_DS3  = 156.float
  NOTE_E3   = 165.float
  NOTE_F3   = 175.float
  NOTE_FS3  = 185.float
  NOTE_G3   = 196.float
  NOTE_GS3  = 208.float
  NOTE_A3   = 220.float
  NOTE_AS3  = 233.float
  NOTE_B3   = 247.float
  NOTE_C4   = 262.float
  NOTE_CS4  = 277.float
  NOTE_D4   = 294.float
  NOTE_DS4  = 311.float
  NOTE_E4   = 330.float
  NOTE_F4   = 349.float
  NOTE_FS4  = 370.float
  NOTE_G4   = 392.float
  NOTE_GS4  = 415.float
  NOTE_A4   = 440.float
  NOTE_AS4  = 466.float
  NOTE_B4   = 494.float
  NOTE_C5   = 523.float
  NOTE_CS5  = 554.float
  NOTE_D5   = 587.float
  NOTE_DS5  = 622.float
  NOTE_E5   = 659.float
  NOTE_F5   = 698.float
  NOTE_FS5  = 740.float
  NOTE_G5   = 784.float
  NOTE_GS5  = 831.float
  NOTE_A5   = 880.float
  NOTE_AS5  = 932.float
  NOTE_B5   = 988.float
  NOTE_C6   = 1047.float
  NOTE_CS6  = 1109.float
  NOTE_D6   = 1175.float
  NOTE_DS6  = 1245.float
  NOTE_E6   = 1319.float
  NOTE_F6   = 1397.float
  NOTE_FS6  = 1480.float
  NOTE_G6   = 1568.float
  NOTE_GS6  = 1661.float
  NOTE_A6   = 1760.float
  NOTE_AS6  = 1865.float
  NOTE_B6   = 1976.float
  NOTE_C7   = 2093.float
  NOTE_CS7  = 2217.float
  NOTE_D7   = 2349.float
  NOTE_DS7  = 2489.float
  NOTE_E7   = 2637.float
  NOTE_F7   = 2794.float
  NOTE_FS7  = 2960.float
  NOTE_G7   = 3136.float
  NOTE_GS7  = 3322.float
  NOTE_A7   = 3520.float
  NOTE_AS7  = 3729.float
  NOTE_B7   = 3951.float
  NOTE_C8   = 4186.float
  NOTE_CS8  = 4435.float
  NOTE_D8   = 4699.float
  NOTE_DS8  = 4978.float

  melody = @[
    NOTE_E7, NOTE_E7, 0, NOTE_E7, 0, NOTE_C7, NOTE_E7, 0, NOTE_G7, 0, 0,  0, NOTE_G6, 0, 0, 0, NOTE_C7, 0, 0, NOTE_G6,
    0, 0, NOTE_E6, 0, 0, NOTE_A6, 0, NOTE_B6, 0, NOTE_AS6, NOTE_A6, 0, NOTE_G6, NOTE_E7, NOTE_G7, NOTE_A7, 0, NOTE_F7, NOTE_G7,
    0, NOTE_E7, 0, NOTE_C7, NOTE_D7, NOTE_B6, 0, 0, NOTE_C7, 0, 0, NOTE_G6, 0, 0, NOTE_E6, 0, 0, NOTE_A6, 0, NOTE_B6,
    0, NOTE_AS6, NOTE_A6, 0, NOTE_G6, NOTE_E7, NOTE_G7, NOTE_A7, 0, NOTE_F7, NOTE_G7, 0, NOTE_E7, 0, NOTE_C7, NOTE_D7, NOTE_B6, 0, 0
    ]

  tempo = @[
    12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
    9, 9, 9, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
    9, 9, 9, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
  ]

var
  dev = newRAudioDevice()
  oscl = newROsc(oscSine)
dev.attach(oscl)
dev.start()

for (n, t) in zip(melody, tempo):
  var
    noteDuration = 1000 / t
    # delayValue = int((1_000_000 / n) / 8)
    # delayBetween = int(noteDuration * 1.3)
    numCycles    = int(n * noteDuration / 1000)
  echo(fmt"{noteDuration}  {numCycles}")

  oscl.play(n)
  sleep(numCycles)

  oscl.play(0)
  # sleep(delayBetween)

# sleep(1000)

oscl.stop()

Sorry, I'm not able to test this right now. The latest version of Nimterop introduced an ABI break, and causes rapid/audio to segfault (because field order matters! and Nim does not seem to order fields the same way C does). I'm contacting the author of the library right now to see if they can do something about this.
It probably works on your machine because you're using an older version of Nimterop, so don't update. Meanwhile, can you send me a sample of how it sounds and how it should sound?

Rapid-Attempt_at_Mario_Bros_Tune.zip
It should sound like the main Mario Bros theme song. --> https://youtu.be/nYwM_OFiFDY?t=21
These are the libraries and versions I have installed:
ajax  [0.1.0]
c2nim  [0.9.14]
cligen  [0.9.45]
gifenc  [0.1.1]
glm  [1.1.1]
gr  [0.1.0]
hmac  [0.1.9]
html5_canvas  [1.3]
nico  [0.2.0, 0.2.1]
nimaws  [0.3.2]
nimPNG  [0.2.6]
nimpy  [0.1.0]
nimSHA2  [0.1.1]
nimterop  [0.4.4]
pylib  [0.1.0]
rapid  [0.1.0]
regex  [0.14.1]
sdl2  [2.0.2]
sdl2_nim  [<IP_ADDRESS>]
segmentation  [0.1.0]
sha1  [1.1]
sndfile  [0.1.0]
unicodedb  [0.9.0]
unicodeplus  [0.6.0]
webaudio  [0.2.0]

Just so you know, oscl.play(0) does not stop playback. It sets the frequency to 0 which probably causes some weird math related issues during sound generation, so don't actually do this :P
Use oscl.stop() instead.
In the for loop, the sleep duration is determined using this formula: n * noteDuration / 1000. However, looking at the for loop definition, I can see that n is actually the note you want to play, and not the duration you want to play it for! You probably meant to use t here.
But anyways, here's an improved version of your code. You can adjust how long the note plays by altering the NoteLength const.
# NOTEs omitted

const
  melody = @[
    NOTE_E7, NOTE_E7, 0, NOTE_E7, 0, NOTE_C7, NOTE_E7, 0, NOTE_G7, 0, 0,  0, NOTE_G6, 0, 0, 0, NOTE_C7, 0, 0, NOTE_G6,
    0, 0, NOTE_E6, 0, 0, NOTE_A6, 0, NOTE_B6, 0, NOTE_AS6, NOTE_A6, 0, NOTE_G6, NOTE_E7, NOTE_G7, NOTE_A7, 0, NOTE_F7, NOTE_G7,
    0, NOTE_E7, 0, NOTE_C7, NOTE_D7, NOTE_B6, 0, 0, NOTE_C7, 0, 0, NOTE_G6, 0, 0, NOTE_E6, 0, 0, NOTE_A6, 0, NOTE_B6,
    0, NOTE_AS6, NOTE_A6, 0, NOTE_G6, NOTE_E7, NOTE_G7, NOTE_A7, 0, NOTE_F7, NOTE_G7, 0, NOTE_E7, 0, NOTE_C7, NOTE_D7, NOTE_B6, 0, 0
  ]

  lengths = @[
    12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
    15, 15, 15, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
    15, 15, 15, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
  ]

var
  dev = newRAudioDevice()
  oscl = newROsc(oscSine)
  mix = newRMixer()
  osclTrack = mix.add(oscl)

osclTrack.volume = 0.2

dev.attach(mix)
dev.start()

# we sleep for 500 millis to prevent the oscillator from playing before the device starts
sleep(500)

const NoteLength = 0.8

for (note, time) in zip(melody, lengths):
  var
    duration = float(time * 10)
    onTime = duration * NoteLength
    offTime = duration - onTime
  echo (note: note, time: time)

  if note != 0:
    oscl.play(note)
    sleep(onTime.int)

    oscl.stop()
    sleep(offTime.int)
  else:
    sleep(duration.int)

oscl.stop()

One piece of advice I can give to you is to not use cryptic names like (n, t) when what the variable stores is clear. It will save you a lot of headaches!

Thank you for your advice. I am still having trouble getting it to play right. I'm on a windows machine.
I've tried changing the timing to this:
while true:
  for (note, time) in zip(melody, lengths):
    var
      timing = time / 12
      onTime = int((2000 / timing) / 10)
      offTime = int(onTime * 1.3)
    echo (note: note, time: time, onTime: onTime, offTime: offTime)

    if note != 0:
      oscl.play(note)
      sleep(onTime)

      oscl.stop()
      sleep(offTime)
    else:
      # prevent frequency 0, because god knows what happens when you use that
      oscl.stop()
      sleep(offTime)

The notes seem to play in the right order, but if I try to tighten it up, then it seems like maybe the oscilator doesn't have a chance to actually initialize or change it's tone in time before the next instruction comes in, and it only plays a few random notes.
I don't know what I'm doing wrong. I guess I'm just not using it right...

The problem is that sleep can only handle precision up to milliseconds, so some values get rounded down to 0, so basically some notes are played and then immediately stopped. Nim would need a more precise sleep proc that can handle nanoseconds, as far as I know there's sleepAsync, but I haven't tested if it does actually handle nanoseconds or not. Looking at the source code, I'd say it does, so try replacing the sleep(time)s with waitFor sleepAsync(time), and remove all the rounding to ints.

hmm. I'll have to give that a try. I was just trying to get this working as
something fun for my kids as well as to understand how to generate music
for a game using tones, rather than pre-recorded music files.
On Tue, May 5, 2020 at 11:07 AM lqdev<EMAIL_ADDRESS>wrote:

The problem is that sleep can only handle precision up to milliseconds,
so some values get rounded down to 0, so basically some notes are played
and then immediately stopped. Nim would need a more precise sleep proc that
can handle nanoseconds, as far as I know there's sleepAsync, but I
haven't tested if it does actually handle nanoseconds or not. Looking at
the source code, I'd say it does, so try replacing the sleep(time)s with waitFor
sleepAsync(time), and remove all the rounding to ints.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/liquid600pgm/rapid/issues/20#issuecomment-624147106,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB62P6TUTQIKSVOBEK7MGTLRQA2VBANCNFSM4MZERDOQ
.

That's cool!
Maybe you can implement this as a precise sampler, here's the gist:
Samplers are actually what generate or process audio, so every ROsc, RWave, RMixer is a sampler. This is the usual pattern for implementing a custom sampler:
import rapid/audio/sampler

type
  SongPlayer* = ref object of RSampler

method sample*(player: SongPlayer, dest: var SampleBuffer, count: int) =
  # this is a barebones sampler that outputs silence
  dest.add(0.0, count * 2)

proc initSongPlayer*(player: SongPlayer) =
  discard  # init logic goes here

proc newSongPlayer*(): SongPlayer =
  result = SongPlayer()
  result.initSongPlayer()

In the sample method, your custom sampler must output samples for left and right channels interleaved (so the order should be [L, R, L, R, L, R, …]). The number of samples you output must always be a multiple of 2. A sampler can also call back to other samplers, so for example you may store a voice: ROsc in your SongPlayer and call player.voice.sample(dest, count) to defer the actual sample generation to the oscillator. You can also call player.voice.sample(dest, 1) to generate only one audio sample, and then use it in a loop. For example:
type
  SongPlayer* = ref object of RSampler
    voice: ROsc
    time: uint

method sample*(player: SongPlayer, dest: var SampleBuffer, count: int) =
  for _ in 0..<count:
    player.voice.sample(dest, 1)
    inc(player.time)

Then you can store your song in a different format, like this:
import std/tables

type
  NoteCommand = object
    on: bool
    freq: float
  Song = Table[uint, NoteCommand]

Then alter the song player like this:
type
  SongPlayer* = ref object of RSampler
    song: Song
    voice: ROsc
    time: uint

method sample*(player: SongPlayer, dest: var SampleBuffer, count: int) =
  for _ in 0..<count:
    if player.time in player.song:
      let note = player.song[player.time]
      if note.on:
        player.voice.play(note.freq)
      else:
        player.voice.stop()
    player.voice.sample(dest, 1)
    inc(player.time)

You'd then have to store the notes as a sequence of on|off commands, maybe even create a file format for that. Or maybe even a nice editor :) the sky is the limit.
But this way your note playback would be precise to the single audio sample.
Here are some things to keep in mind:

The sample callback must be really fast. Otherwise you may get stuttering audio due to underflows. You can make it go faster by wrapping it in a {.push stacktrace: off.} pragma, but then you lose on debugging capabilities.
Don't do system calls like memory allocation inside the sample method. They're very slow.
The sample method runs on a separate thread, so be cautious while accessing global state. Preferably store everything inside of the sampler object itself to avoid headaches.

At this point I'm starting to think I should turn this into a tutorial. Anyways, happy coding!

Wow! Thank you for the very thorough write up. I'll have to go over it at least one more time to fully digest it. XD


chore: diap amends

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 86.39%. Comparing base (270aa9e) to head (4a4f1ba).
Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #325      +/-   ##
==========================================
- Coverage   90.50%   86.39%   -4.11%
==========================================
  Files         347      347
  Lines        5159     5159
  Branches      753      753
==========================================
- Hits         4669     4457     -212
- Misses        475      671     +196
- Partials       15       31      +16

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Centos zmq

related to https://github.com/hyperledger/indy-sdk/issues/2215
Please also see: https://github.com/zeromq/libzmq/security/advisories
I haven't checked the Debian/Ubuntu versions indy uses
https://software.opensuse.org/download.html?project=network%3Amessaging%3Azeromq%3Arelease-stable&package=libzmq3-dev

Hi @AxelNennker,
we had to re-write master history to fix DCO there. So now this PR should be updated as well.
Can I force-push to your branch or would you like to do it by yourself or to create a new one?

Please force push. The other issue, too.


Register

@JuliaRegistrator register()

Registration pull request created: JuliaRegistries/General/53753
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.1.1 -m "<description of version>" 6018bc89a5fb9190974ff8badc8f638a6b94dd38
git push origin v0.1.1

Also, note the warning: This looks like a new registration that registers version 0.1.1.
Ideally, you should register an initial release with 0.0.1, 0.1.0 or 1.0.0 version numbers
This can be safely ignored. However, if you want to fix this you can do so. Call register() again after making the fix. This will update the Pull request.

register()

@JuliaRegistrator register()

Error while trying to register: "Tag with name v0.1.1 already exists and points to a different commit"


Removed files not showing up in PR review panel

Issue Description
Type: bug report or feature request
Describe what happened (or what feature you want)
When using Octo for reviewing a PR, I noticed that removed files doesn't show up on the bottom panel of the review layout, where all other modified files do appear. Unless I'm missing something, this makes impossible to know if you're missing some changes (i.e., the files removed completely) from the PR you're reviewing through Octo interface.
Describe what you expected to happen
I would expect the removed files appear on the bottom panel and once selected the upper right panel were empty while the upper left displayed the old (removed) version of the file.
How to reproduce it (as minimally and precisely as possible)

Take a PR with deleted files among its changes.
:Octo review start (actually, I did it resuming a review created from github web)
Deleted files won't show in the bottom panel.

Tell us your environment
Nvim 0.10.0-dev inside WSL2
Anything else we need to know?

Just tested with https://github.com/pwntester/octo-tests/pull/21 and I can see the removed file (test1.lua) in the changes panel:

Is your PR public so I can take a look?


Speedup find_and_fix_disjoint_nets by replacing net_name setter with direct API call

Massimo,
I do not now how to access the branch to test the code.
I hope you will be able to show me the process on Monday such I can do it in time


Null controllerPointer crash in BasePointerRenderer

Environment
Master branch, current release.
Unity 5.5.2p2
Steps to reproduce
Use the BenzierPointerRenderer & VRTK Pointer, likely a problem with other pointer renderers.
Expected behavior
Not crash.
Current behavior
In this line:
https://github.com/thestonefox/VRTK/blob/master/Assets/VRTK/Scripts/Pointers/PointerRenderers/VRTK_BasePointerRenderer.cs#L291
... it is referencing controllingPointer.customOrigin, but on the following line, checks to see if controllingPointer is not null. If it was null, which happened in my testing, the function will bomb out with a NullReference exception on line 291.
I have a VTRK_Pointer on the same gameObject.
I wasn't able to test this pointer functionality at all due to this crash. Has it been working, or is it still an experimental change?

Are you setting the VRTK_Pointer Pointer Renderer field to the Beizier Pointer Render script?
If you look at one of the example scenes you'll see it working.

Nope, fixed that.
However, it still looks a bit silly in the code, since why would be you be checking if controllerPointer is not null on line 292, if it would crash on 291 if that was the case?

So are you saying now that the pointer is working for you, but you think it shouldn't crash if it's not been set up correctly?

I have yet to test if it works (which it probably does), but trying to trace back the problem led me to code that isn't structured correctly, which makes it appear a problem with VRTK, not my setup. If controllerPointer should never be null, then it is confusing when the code sometimes checks for null status improperly.

If you look at one of the pointer examples, you should see that they work without crashing. Which would suggest that if they are set up as expected then no crash would occur.

Yes, I get that. I should be setup fine now. The code is still sloppy in that section (e.g. doing a null check after referencing it). You are even setting the gameObject active depending on if it is null or not, which if should always be setup properly & non-null, makes me wonder why you are doing it regularly in FixedUpdate.

if you have thoughts on a better way and can raise a pull request then that would be good.

The things you see are because that code is moved over from the old Simple/BezierPointer. If you happen to have time to clean stuff up or fix it would be great if you can do a PR.

Setting the game object active or disabling it is what I've done in the old pointer scripts and should probably be added to the new ones. The idea is to only add and enable the smoothing helper game objects if needed.

I don't fully understand the pointer system as a whole, like you guys do, to feel confident in making the changes myself. I did make this change in my local project:
`        protected virtual void UpdatePointerOriginTransformFollow()
{
pointerOriginTransformFollow.enabled = controllingPointer != null;
if (pointerOriginTransformFollow.enabled) {
pointerOriginTransformFollow.gameObjectToFollow = (controllingPointer.customOrigin == null ? transform : controllingPointer.customOrigin).gameObject;
pointerOriginTransformFollowGameObject.SetActive(true);
            pointerOriginTransformFollow.smoothsPosition = pointerOriginSmoothingSettings.smoothsPosition;
            pointerOriginTransformFollow.maxAllowedPerFrameDistanceDifference = pointerOriginSmoothingSettings.maxAllowedPerFrameDistanceDifference;
            pointerOriginTransformFollow.smoothsRotation = pointerOriginSmoothingSettings.smoothsRotation;
            pointerOriginTransformFollow.maxAllowedPerFrameAngleDifference = pointerOriginSmoothingSettings.maxAllowedPerFrameAngleDifference;
        } else if( pointerOriginTransformFollowGameObject != null ) {
            pointerOriginTransformFollowGameObject.SetActive(false);
        }
    }`

... which should clean up the null checking, but I'm not sure if it is a good solution for the whole process. I have still yet to test.

Fixed with https://github.com/thestonefox/VRTK/commit/4e483f5512fdd517b6951c8f7d9a17b03f540111


Migration of PKISetup Service from Security-Secret-Store

Fix #1499

Please see commit log on this branch for detailed information.
This PR is based on the current state of master in the https://github.com/edgexfoundry/security-secret-store repository.
There is a security-secret-store PR https://github.com/edgexfoundry/security-secret-store/pull/69 currently ongoing with 144 comments on it. I'm not going comb through all of that to see what needs to be applied here. Once that PR is merged (hopefully soon) I will review the diff. Subsequent edits to the PKISetup service can be made to this codebase, if applicable.

Codecov Report

Merging #1500 into master will increase coverage by 1.79%.
The diff coverage is 73.22%.

@@            Coverage Diff             @@
##           master    #1500      +/-   ##
==========================================
+ Coverage   18.84%   20.63%   +1.79%
==========================================
  Files          89       98       +9
  Lines        9100     9410     +310
==========================================
+ Hits         1715     1942     +227
- Misses       7235     7299      +64
- Partials      150      169      +19

Impacted Files
Coverage Δ

internal/pkg/config/loader.go
0% <ø> (ø)
:arrow_up:

internal/security/setup/directory.go
0% <0%> (ø)

internal/security/setup/init.go
0% <0%> (ø)

internal/security/setup/certificates/interface.go
100% <100%> (ø)

internal/security/setup/certificates/crypto.go
100% <100%> (ø)

internal/security/setup/certificates/file.go
50% <50%> (ø)

internal/pkg/config/x509.go
61.11% <61.11%> (ø)

internal/security/setup/certificates/root.go
74.6% <74.6%> (ø)

internal/security/setup/certificates/tls.go
78.66% <78.66%> (ø)

internal/security/setup/seed.go
94.73% <94.73%> (ø)

... and 9 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a478f95...66c8438. Read the comment docs.

Also, nice test coverage!

LGTM

recheck


Elasticsearch Transport Couchbase for the 5.x versions

Dear contributors,
I really love your couchbase transport plugin!
Since I couldn't get the alpha version for ES 5.x to work properly, I still use ES 2.x and thus would really love to see the plugin being released for 5.x as a production version.
Can you tell me when or if you're planning to finish the 5.x plugin?
Thank you very much!
Best regards,
woltob

We now have a pre-release version of the plugin for ES 5.2.1 available here: https://github.com/couchbaselabs/elasticsearch-transport-couchbase/releases/tag/v.<IP_ADDRESS>
It's fully functional, but hasn't made it through the full QA process at Couchbase yet.


Expose getTestMode() in TestBase class.

Exposes TestMode in the TestBase class. This is useful for Test classes that extend from TestBase. In my case, the Live tests not only depend on having the correct TestMode set, but also that there is a connection string set. Having this exposed removes the need for me to re parse the same environment variable and reason about it.

Can one of the admins verify this patch?


only one times use tween x/y

Describe your bug here.
so when i use any tween x/y it wont tweening again after using tween
idk if this a bug or anything else
Command Prompt/Terminal logs (if existing)
No response
Are you modding a build from source or with Lua?
Lua
What is your build target?
Windows
Did you edit anything in this build? If so, mention or summarize your changes.
No.

you have to specify what the problem is, would you mind showing the modchart code?

Note that tweens don't add to the original value, but set it to the value given instead so unless you give it a formula it will stay at the value given

Note that tweens don't add to the original value, but set it to the value given instead so unless you give it a formula it will stay at the value given

^

ah i see so the modchart is limited anyways thanks

you CAN do this: getProperty('fakeobject.x') + 50 and put that where the target variable is

you CAN do this: getProperty('fakeobject.x') + 50 and put that where the target variable is

like how?

doTweenX('tweenX', 'object', getProperty('object.x')+50, 1);
this makes the tween read the current value and add 50 to it, rather than just setting it to 50

how do i exactly use that function?

same way you were using it already, just replace object with your object tag

same way you were using it already, just replace object with your object tag

how ? i want to use this for the note/modchart

I really can't make it any simpler than that, you'll have to figure it out on your own
It is not that hard


refactor(versions): refactor versions helper

Refactor versions helper to use lodash filters, instead of
nested if statements.

Copying this from my work in #139 as it's not related, and will help reduce the size of the changes contained in #139

Codecov Report

Merging #143 into master will decrease coverage by 0.11%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #143      +/-   ##
==========================================
- Coverage   69.09%   68.98%   -0.12%
==========================================
  Files          19       19
  Lines         809      806       -3
  Branches      130      130
==========================================
- Hits          559      556       -3
  Misses        250      250

Impacted Files
Coverage Δ

lib/helpers/versions.js
100% <100%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9486547...6dcf783. Read the comment docs.


Zapier missing Firebase "Add Child Record" trigger used in Firebase-Sendgrid tutorial

https://sendgrid.com/docs/Integrate/Tutorials/Zapier/sending_when_new_firebase_child_records_added.html
Zapier seems to have removed the "Add Child Record" Firebase trigger so this tutorial doesn't work.
https://zapier.com/zapbook/firebase/
https://web.archive.org/web/20150324221128/https://zapier.com/zapbook/firebase/
Now there's only the "Update value at this path" trigger.

Oh, I wrote that too soon. If I go directly to https://zapier.com/zapbook/firebase/sendgrid/29752/firebase-added-a-child-record-to-sendgrid-send-email/ it's still available.

This options is not available anymore, i do not know why Zapier removed it but looks like when you are composing the zap there are not triggers available for firebase

I got an email from Zapier a few days ago with some more detail (I've highlighted important bits):

We're reaching out because one of your Zaps is using a Firebase Trigger. Unfortunately, these Triggers may not work going forward.
The Firebase team no longer officially supports the webhooks that Zapier used to set Firebase Zaps into motion. Because we cannot guarantee that Zaps using Firebase Triggers will function correctly, these options have been hidden for new Zaps.
However, we did not turn off any existing Zaps that are using Firebase Triggers. These Zaps should be used at your own risk. You can see which of your Zaps are using Firebase by visiting your dashboard.
If you're searching for a Firebase alternative, take a look at our integrations for Knack and Airtable, two popular database apps. We're also building our own short-term storage app called Storage by Zapier—it's in beta, but you can test it out here.
We're truly sorry about any headaches this might cause. The Firebase team is working on an API that would give us more reliable access to trigger data, but there's no timetable at this point. When they launch the new API we'll look at rebuilding these Firebase Triggers.
If you have any questions about using Firebase with Zapier, or setting up an alternative, don't hesitate to reply to this email—our support team is happy to lend a hand.

Bryan Helmig, CTO of Zapier

FYI an official answer from Firebase as well: https://groups.google.com/forum/#!topic/firebase-talk/-6lZEuoO52U
I guess the internal Google Cloud solution might be the future way to integrate to Zappier, a kind of "Google Cloud Functions" for Firebase - but it's in pre-alpha at the moment.

Hello @sbrown345,Thanks for your suggestion or question! In an attempt to clear out our backlog,                                              we are closing out issues over 90 days old.                                             If this is still an issue, please feel free to reopen it.                                             Team SendGrid DX-Docs


[Bug Report]: formatN should not output "0" when no input is provided

Environment
Carbone Version: 2.1.0
Node Version: 10.20.1
Desktop OS: Debian Stretch
LibreOffice: <IP_ADDRESS>
Expected Behaviour
null:formatN() xml output should be an empty string
Actual Behaviour
null:formatN() output "0"
Steps to reproduce
Try to format null or undefined:
null:formatN()
Other comments
Is there a workaround? I am playing with Conditioned output but I can't make it work.
{$my_number:ifEmpty()} // OK: value is not printed if null
{$my_number:ifEmpty():formatN()} // KO: '0' is printed if null

I suppose and understand that everyone is busy here.
Anyone, this one is a stop blocker for my project. Any help to find a workaround would be appreciated.
Thanks!

Hi,
Do you use the last version of Carbone (v2.1.0). I try to reproduce your bug but I didn't succeed...
{$my_number:ifEmpty():formatN()} // Print nothing for me

Or can you send here your template and a sample of your JSON data so we can take a look at it?

Many thanks @Dobby85 for your reply.

Do you use the last version of Carbone (v2.1.0)?

Yes, I do.

Or can you send here your template and a sample of your JSON data so we can take a look at it?

Sure, here are the template and the result files: issue-templates.zip
Use it with the following model:
{
    my_number: 5,
    my_number2: null,
    my_number3: undefined,
}

TEMPLATE:

OUTPUT:

Thanks,
By using formatN in an ODS template, we remove the formatter and ask libre office to format the cell for us. By doing this, Libre Office set a 0 as it is a numeric cell...
I come back to you if I find a workaround to your problem

I see...
ifEmpty should be a way to conditionally "ask libre office to format the cell for us".
I would be happy to help, not sure if it is easy to enter into the code.
If you think so, could you please give me some pointers to the code related to the behavior you describe?.

Hello @Toub, sorry for the delay!
After discussing with the team, we got the same problem internally. Our main priority is to improve the XLSX report generation and then we will be able to fix this issue.
In the meantime, you can take a look at the preprocessing process where the formatN is behaving differently than other formatters for ODS/XLSX files: https://github.com/Ideolys/carbone/blob/37355e9e789320a5d50013ec94be50092e33b9e6/lib/preprocessor.js#L212
If a formatN formatter is found, the string cell is converted to a number cell. As it is a preprocessing function, this stage doesn't know if the injected value exists. To fix the issue, the cell conversion should be done during or after the building process.

Thanks for your reply.
I switch back to exceljs for complex spreedsheet generation.
But I will give an other try to carboneJS soon in order to provide to my applications users a simple way customize basic non-spreadsheet documents templates.


Update 0042-LIQF-setting_fees_and_rewarding_lps.md

@vega-paul
I have fixed the liniting / spelling. Just this review comment to address:

https://github.com/vegaprotocol/specs/pull/1598#discussion_r1116749182


Added picture tag to tags list.

See: http://www.w3.org/html/wg/drafts/html/master/semantics.html#the-picture-element


Fix aws_demos, aws_tests project with RDPv125

Fix aws_demos, aws_tests project with RDPv125Description
Fixed aws_demos, aws_tests
Checklist:

[x] I have tested my changes. No regression in existing tests.
[x] My code is Linted.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

I would like to close this PR because the build of aws_tests is failed. I will resubmit another PR later. Sorry for the inconvenience


Fix Platform typings

In our logs I've found a situation when we received userAgent string:
[FBAN/FB4A;FBAV/2<IP_ADDRESS>.120;FBBV/231020957;FBDM/{density=2.75,width=1080,height=2150};FBLC/en_GB;FBRV/0;FBCR/Jio 4G;FBMF/Xiaomi;FBBD/xiaomi;FBPN/com.facebook.katana;FBDV/Redmi Note 6 Pro;FBSV/8.1.0;FBOP/1;FBCA/armeabi-v7a:armeabi;]
Which was parsed as { name: 'Xiaomi Browser', version: null,}
From Platform interface  it's known that version?: string;can not be null

The platform.js typings are currently not maintained by us. Although I might decide to shift them into this repo in the future, for now this should be opened at DefinitelyTyped. I'll look into that sometime soon.


pre-trained weights for Inbreast's classifier

Hi, could you please provide the pre-trained weights for Inbreast's PHYSEnet+classifier?

Hi, they are already available, they are the ones with pretraining mode "Patches + CBIS".


Tapo P110 devices connect failed

Version of the integration
2.13.0
Configuration
Tapo P110, Firmware 1.3.0
Describe the bug
Unable to add all 3 tapo devices with the same Connection error.
IP is correct and in the same network of HA.
Pinging 1 of 3 (just for example) devices is OK:
userland@localhost:~$ ping <IP_ADDRESS>
PING <IP_ADDRESS> (<IP_ADDRESS>) 56(84) bytes of data.
64 bytes from <IP_ADDRESS>: icmp_seq=1 ttl=255 time=76.4 ms
64 bytes from <IP_ADDRESS>: icmp_seq=2 ttl=255 time=19.5 ms
64 bytes from <IP_ADDRESS>: icmp_seq=3 ttl=255 time=11.4 ms
HA is core running under Debian in Android Userland
Debug log
2024-01-12 16:02:19.282 ERROR (MainThread) [custom_components.tapo.config_flow]
Failed to setup cannot connect
Traceback (most recent call last):
...skipping 1 line
return await self.loop.create_connection(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1099, in create
connection
raise exceptions[0]
File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1081, in create_
connection
sock = await self._connect_sock(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/asyncio/base_events.py", line 984, in _connect
_sock
await self.sock_connect(sock, address)
File "/usr/local/lib/python3.12/asyncio/selector_events.py", line 647, in sock
_connect
return await fut
^^^^^^^^^
File "/usr/local/lib/python3.12/asyncio/selector_events.py", line 687, in _soc
k_connect_cb
raise OSError(err, f'Connect call failed {address}')
OSError: [Errno 113] Connect call failed ('<IP_ADDRESS>', 80)
The above exception was the direct cause of the following exception:
...skipping 1 line
File "/home/userland/.homeassistant/custom_components/tapo/config_flow.py", li
ne 192, in _get_first_data_from_api
(await tapo_client.get_device_info())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/userland/hass/lib/python3.12/site-packages/plugp100/api/tapo_clien
t.py", line 92, in get_device_info
return await self.execute_raw_request(get_info_request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/userland/hass/lib/python3.12/site-packages/plugp100/api/tapo_clien
t.py", line 77, in execute_raw_request
await self._initialize_protocol_if_needed()
File "/home/userland/hass/lib/python3.12/site-packages/plugp100/api/tapo_clien
t.py", line 71, in _initialize_protocol_if_needed
await self._guess_protocol()
File "/home/userland/hass/lib/python3.12/site-packages/plugp100/api/tapo_clien
t.py", line 221, in _guess_protocol
response = await self.get_component_negotiation()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/userland/hass/lib/python3.12/site-packages/plugp100/api/tapo_clien
t.py", line 81, in get_component_negotiation
return (await self.execute_raw_request(TapoRequest.component_negotiation()))
.map(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...skipping 1 line
return (await self._protocol.send_request(request)).map(lambda x: x.result)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/userland/hass/lib/python3.12/site-packages/plugp100/protocol/passt
hrough_protocol.py", line 41, in send_request
response = await self._send_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/userland/hass/lib/python3.12/site-packages/plugp100/protocol/passt
hrough_protocol.py", line 61, in _send_request
await self._login_with_version(self._credential)
File "/home/userland/hass/lib/python3.12/site-packages/plugp100/protocol/passt
hrough_protocol.py", line 80, in _login_with_version
session_or_error = await self._passthrough.handshake(self._url)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/userland/hass/lib/python3.12/site-packages/plugp100/protocol/secur
epassthrough_transport.py", line 70, in handshake
response = await self._http.async_make_post(url, json=request_body)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/userland/hass/lib/python3.12/site-packages/plugp100/common/utils/h
ttp_client.py", line 18, in async_make_post
async with self.session.post(
File "/home/userland/hass/lib/python3.12/site-packages/aiohttp/client.py", lin
e 1187, in aenter
self._resp = await self._coro
...skipping 1 line
File "/home/userland/hass/lib/python3.12/site-packages/aiohttp/client.py", lin
e 574, in _request
conn = await self._connector.connect(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/userland/hass/lib/python3.12/site-packages/aiohttp/connector.py",
line 544, in connect
proto = await self._create_connection(req, traces, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/userland/hass/lib/python3.12/site-packages/aiohttp/connector.py",
line 911, in _create_connection
_, proto = await self._create_direct_connection(req, traces, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/userland/hass/lib/python3.12/site-packages/aiohttp/connector.py",
line 1235, in _create_direct_connection
raise last_exc
File "/home/userland/hass/lib/python3.12/site-packages/aiohttp/connector.py",
line 1204, in _create_direct_connection
transp, proto = await self._wrap_create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/userland/hass/lib/python3.12/site-packages/aiohttp/connector.py",
line 1000, in _wrap_create_connection
raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host 192.168.1
...skipping 1 line
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/userland/.homeassistant/custom_components/tapo/config_flow.py", li
ne 119, in async_step_user
device_data = await self._get_first_data_from_api(tapo_client)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/userland/.homeassistant/custom_components/tapo/config_flow.py", li
ne 199, in _get_first_data_from_api
raise CannotConnect from error
custom_components.tapo.errors.CannotConnect

Same issue applies to the p115

I have also the issue with my p110's

Hi, I am having the same issue with P110 but not all of them. 3/7 have this issue.

I'm having the same issues - have reinstalled the tapo integration through HACs as well as readding devices in the Tapo app itself.

Update:
one of the 3 has connected. all other are still on the retry loop.

I am having the same issue, the device had previously worked well. Now I have readded the device on the Tapo app with another name and then trying to add it to HA on the same previous IP address and I am unable to connect
I get this error, I have no idea what to do next
Logger: custom_components.tapo.config_flow
Source: custom_components/tapo/config_flow.py:119
Integration: TP-Link Tapo ([documentation](https://github.com/petretiandrea/home-assistant-tapo-p100), [issues](https://github.com/petretiandrea/home-assistant-tapo-p100/issues))
First occurred: 6:04:35 PM (1 occurrences)
Last logged: 6:04:35 PM

Failed to setup cannot connect
Traceback (most recent call last):
  File "/config/custom_components/tapo/config_flow.py", line 194, in _get_first_data_from_api
    .get_or_raise()
     ^^^^^^^^^^^^^^
  File "/config/lsiopy/lib/python3.11/site-packages/plugp100/common/functional/tri.py", line 159, in get_or_raise
    raise self.value
Exception: Server response doesn't match our challenge on ip ('http', '<IP_ADDRESS>', 80)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/config/custom_components/tapo/config_flow.py", line 119, in async_step_user
    device_data = await self._get_first_data_from_api(tapo_client)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/tapo/config_flow.py", line 199, in _get_first_data_from_api
    raise CannotConnect from error
custom_components.tapo.errors.CannotConnect

I got new Tapo P110s this week and none of them can be connected to by Home Assistant but I can ping them from Home Assistant.

Same here, new P110's home assistant (all latest versions) cannot connect. "Failed to connect" error.
Logger: custom_components.tapo.config_flow
Source: custom_components/tapo/config_flow.py:119
Integration: TP-Link Tapo (documentation, issues)
First occurred: 21:15:45 (3 occurrences)
Last logged: 21:17:38
Failed to setup cannot connect
Traceback (most recent call last):
File "/config/custom_components/tapo/config_flow.py", line 194, in _get_first_data_from_api
.get_or_raise()
^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/plugp100/common/functional/tri.py", line 159, in get_or_raise
raise self.value
Exception: Server response doesn't match our challenge on ip ('http', '<IP_ADDRESS>', 80)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/config/custom_components/tapo/config_flow.py", line 119, in async_step_user
device_data = await self._get_first_data_from_api(tapo_client)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/tapo/config_flow.py", line 199, in _get_first_data_from_api
raise CannotConnect from error
custom_components.tapo.errors.CannotConnect

Same here, 2.3.0.
1x P110 was working other 3 not yet connected. Now connected all 4 and updated firmware on all 4 to 1.3.0.
All 4: "Failed to connect".

Same here with all P110 😞

I don't know what happened, just gived a "desperated" try and one device was added successfully. Then the other two, ok with them, too!
Is there something I can do to provide useful data?

HomeAssistant restart and issue appeared again

Same issue here "Failed to Connect"
Although I will add that I have some tapo cameras, and having the same issue..........
searched high and low for answer on how to get these working again, but getting to the point where I may just go to a different brand, any suggestions?

Hi all, not sure if this is helpful and a bit of a ramble I'm afraid but I have 23 x P110 connected via the integration and they have been flawless so can't help thinking it may not be the integration but something else within the system?   I did encounter problems when ticking the box 'track by MAC address (experimental) so I now leave this unchecked.    I also assign static IPs to all my P110's.   I also had some issues with the KASA and TAPO apps (still have a few HS100s) but it turned out they each had different email addresses associated with them and as soon as I changed this those issues disappeared.   My process to add is to connect them via the Tapo app - they are all called 'Socket: xxxx' - then update the firmware (the last one I did last night after the integration update as it shows Hardware version 1, Firmware 1.3.0 Build 230905 Ref 152200.   Take a note of the MAC/IP address and assign this as static on the router and then add into HA.   I'm running HA on a RP4.  I also have them alll on their own IOT network.    Although the logs might suggest otherwise based on the above I can't help thinking there may be a conflict elsewhere - as an example I've had all sorts of problems recently with HA 'randomly' restarting and each log was different, I tried disabling integrations, add-ons etc but ultimately the problem was that the PSU for my PI wasn't sufficient - upgraded this and no random restarts anymore.

Hi all,
So, I still cannot get this to work and agree that it is probably not the integration.
I have just deleted all previous Tapo integrations on my HA.
I have set up a new Tapo account on a different email, (all lowercase) and added a P110 plug device - successful.
I have just deleted all previous Tapo integrations on my HA,
I have downloaded and installed from HACS again

Started the process of adding a device
Input the correct IP (have pinged and all ok)
Correct user email
Correct password
Failed to connect.........

Does anyone have suggestions on what to do next?

Hiya - that does seem strange - you can add one but no another would immediately suggest there is something one with the only thing that is different, i.e. the plug - perhaps the second one is faulty - reconfigure using the app and try again?  (What did you call the first plug and is the second one of a similar name - perhaps it doesn't like the name you used in the setup?  Firmware versions on the plug the same?    Sorry nothing specific but a few things to try....

I was having the same issue with this after a recent update. One thing that stood out for me was the IP it was trying to connect with wasn't the IP of the device (I would use <IP_ADDRESS> yet the logs show <IP_ADDRESS>).
It felt like it was cached somewhere, my suspicion was that something relating to mac addresses prior to the update.To verify the underlying code I tested this and was able to connect successfully using a simple test script
I can't be certain which change was finally responsible but I this was fixed through one of these changes

static IPs for the plugs
remove all tapo devices
remove the tapo controller in HACs altogether and re-download / install again
Using a different browser (not the app) re-add the integration

Once I did this for a single plug the others were auto discovered and the integration is now operational again

Found out there was a capital letter In the email when I signed up, tappo does not show you the capital letter. Created a new account with out using any caps and it works fine now.

Hey all,
Going to give my Tapo one last try before moving on to maybe TUYA.
What's the best way to remove all previous Tapo integration and data so I am starting afresh?
I have a feeling that deleting the integration would not do this.....happy to be proved wrong.
Do I need to also Create a new Tapo account and register my plugs there?
Then, with a clean sheet, the best way to install 4 x P110 plugs?
Happy to follow suggestions to get the best outcome, ie Tapo integration working!
I am a beginner, so please feel free to write your comments as if I am an idiot, (maybe I am, as I cannot get the current integration to work!)in simple terms so that other beginners like me will also be able to benefit from your knowledge.
Appreciate anyone who takes the time to respond to this!
Thanks
J

I'm getting a similar issue that's described in this ticket. All my Tapo devices were working fine until a few days ago.
My username and password as all fine, I'm getting any errors with those.
It looks like others are having the same issue: https://community.home-assistant.io/t/tapo-devices-fail-to-connect/675178/8
I'm running the following if that helps:

HA Hardware: Raspberry Pi 3 B+
HA Core: 2024.5.5
HA Supervisor: 2024.05.1
HA Operating System: 12.3
Tapo Control Integration: 3.1.2

The devices I'm trying to connect are:

P100 plugs x4
L530B blubs x2

Here is the latest error I get in my logs when trying to add a device:
This error originated from a custom integration.

Logger: custom_components.tapo.config_flow
Source: custom_components/tapo/config_flow.py:143
integration: TP-Link Tapo (documentation, issues)
First occurred: 11:39:02 (1 occurrences)
Last logged: 11:39:02

Failed to setup cannot connect
Traceback (most recent call last):
  File "/config/custom_components/tapo/config_flow.py", line 314, in _async_get_device
    device = await connect(config=config, session=session)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/plugp100/new/device_factory.py", line 43, in connect
    protocol = await _get_or_guess_protocol(config, session)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/plugp100/new/device_factory.py", line 65, in _get_or_guess_protocol
    return await _guess_protocol(config, session)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/plugp100/new/device_factory.py", line 102, in _guess_protocol
    raise InvalidAuthentication(config.host, config.device_type)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/plugp100/new/errors/invalid_authentication.py", line 3, in __init__
    super(f"Invalid authentication error for ${host}, ${device_type}")
TypeError: super() argument 1 must be a type, not str

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/config/custom_components/tapo/config_flow.py", line 143, in async_step_user
    device = await self._async_get_device(user_input)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/tapo/config_flow.py", line 322, in _async_get_device
    raise CannotConnect from error
custom_components.tapo.errors.CannotConnect

Similar issue as @blackthorn87, 2 out of 4 P100 devices stopped working (I didn't notice this until today, so not sure when exactly). Tried downgrading & upgrading the plugin and removing and readding the devices, but gettimg the same tapo.errors.CannotConnect error.
What's interesting is that the 2 other plugs from the same package of 4 with the same firmware version work.

Similar issue as @blackthorn87, 2 out of 4 P100 devices stopped working (I didn't notice this until today, so not sure when exactly). Tried downgrading & upgrading the plugin and removing and readding the devices, but gettimg the same tapo.errors.CannotConnect error.
What's interesting is that the 2 other plugs from the same package of 4 with the same firmware version work.

I figured out my issue, it was user error (https://github.com/petretiandrea/home-assistant-tapo-p100/issues/774).

Similar issue as @blackthorn87, 2 out of 4 P100 devices stopped working (I didn't notice this until today, so not sure when exactly). Tried downgrading & upgrading the plugin and removing and readding the devices, but gettimg the same tapo.errors.CannotConnect error.
What's interesting is that the 2 other plugs from the same package of 4 with the same firmware version work.

I managed to get one of the non-functional P100s integrated back. Not sure which of these steps is significant:

Remove plug from HA
Downgrade Tapo plugin to some older version
Restart HA
Upgrade Tapo plugin back to 3.1.2
Change account password
Update plug firmware
Dismiss the auto-discovered plug (I'd bet this is the significant one)
Add the switch manually

Similar or actually same issue (and pretty well exact log content) as this ticket. Any updates, pointers or ideas on a possible resolution?
As new to the Tapo range - first test device - I have never seen a working install and if this is going to be an issue I guess I will have to go to another brand.
As a workaround meanwhile I set up a Smart Things account (something I do not want to use in production) and integrated the P110 there - then used the HA stock Smart Things integration.
This however has no future as the energy sensors are not fully implemented in ST as far as I can judge and is therefore nothing more useful than an on/off switch in HA :-(
Any feedback greatly appreciated. Thx.
Config:
Tapo P110 / FW: 1.3.1
HA Core 2024.8.1
Tapo integration 3.1.2/3.1.3/3.1.4

I updated the firmware on all my P110s today which disconnected them all. They don’t have access to the internet and I temporarily let them access the internet for the upgrade. I fixed it by removing and readding them to the integration.

I had issue with connection Tapo P110
`Failed to setup cannot connect
Traceback (most recent call last):
File "/usr/local/lib/python3.13/site-packages/plugp100/protocol/klap/klap_protocol.py", line 71, in send_request
if response := await self._send_request(request, retry):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/plugp100/protocol/klap/klap_protocol.py", line 84, in _send_request
self._klap_session = await self.perform_handshake()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/plugp100/protocol/klap/klap_protocol.py", line 122, in perform_handshake
if seeds := await self.perform_handshake1():
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.13/site-packages/plugp100/protocol/klap/klap_protocol.py", line 207, in perform_handshake1
raise Exception(
f"Server response doesn't match our challenge on url {self._base_url}"
)
Exception: Server response doesn't match our challenge on url http://<IP_ADDRESS>:80/app`
Working solution:

removed from Tapo App
added again

Not it's working good


Rejections inside notifications do not revert the tables state

Description
Rejections in notifications do not revert the changed tables back to their initial values.
Example: when there's a check(false, ...) inside an atomicassets notification, the assets are not returned to the sender (i.e. the assets table state is not reverted back)
Step To Reproduce
Atomicassets contract is used to demonstrate

Create a contract that listens on transfer notifications for some collection from atomicassets (and rejects them)

[[eosio::on_notify("atomicassets::logtransfer")]]
void testcontract::handle_logtransfer(name from, name to, vector<uint64_t> asset_ids, string memo)
{
    check(false, "transfers are closed");
}

Send an asset from that collection from one account to another (alice -> bob)
Check the atomicassets table assets under the alice scope

Current Behavior
The asset should still be in alice's inventory
Expected Behavior
The asset stays in bob's inventory

@benjiewheeler could you kindly add a minimal repro repo / gist, would speed up the fix

@jafri I created this repo with a sample contract and a simple mocha test
https://github.com/benjiewheeler/vert-issue-repro

Try<EMAIL_ADDRESS>