[HUDI-91][HUDI-12]Migrate to spark 2.4.4, migrate to spark-avro library instead of databricks-avro, add support for Decimal/Date types

Sending this PR out early to get feedback. Have not yet looked into what changes are required for tests. But in general these changes have been working for us on AWS EMR without any issues so far. This PR implements the following:

Migrates Hudi to Spark 2.4.4
Migrates Hudi to use spark-avro module instead of the deprecated databricks-avro
Adds support for Decimal/Date types correctly through Hive
Timestamp still needs to be supported as it is blocked on https://issues.apache.org/jira/browse/HUDI-83

Sending this PR out early to get feedback. Have not yet looked into what changes are required for tests. But in general these changes have been working for us on AWS EMR without any issues so far. This PR implements the following:
* Migrates Hudi to Spark 2.4.4

* Migrates Hudi to use spark-avro module instead of the deprecated databricks-avro

* Adds support for Decimal/Date types correctly through Hive

* Timestamp still needs to be supported as it is blocked on https://issues.apache.org/jira/browse/HUDI-83

@n3nash : Can you and Modi review this PR and see if it is aligned with what you have done internally  ?

Hi Udit! Thanks for making this PR!
I've been working on upgrading HUDI to Spark 2.4 internally at Uber! So I'll list out a few things that I had to do, so that you're not trying to re-discover these things yourself :)

Some of the create functions in HoodieWrapperFileSystem don't fully work with Parquet 1.10+. See here.  It might be causing unit tests failure as it may introduce an older version of avro into the calsspath. We're currently trying to figure out how to address this. Let us know if you have any suggestions!

Btw - I also went to UIUC! Great to meet new Illini!

Hi,
any plans to migrate Hudi to Scala 2.12? I don't see any Jira issue regarding this.
I would volunteer after this PR is merged. I already looked around, it seems that the biggest problem will be migrating to spark-streaming-kafka-0-10_2.12, since there is no spark-streaming-kafka-0-8 version for Scala 2.12. And the tests in hudi-utilities use the low level Kafka API that is gone in version spark-streaming-kafka-0-10.

@ezhux I have migrated to spark-streaming-kafka-0-10_2.11 internally at my organisation. Let me know if I can help you in any way. :)

@ezhux https://issues.apache.org/jira/browse/HUDI-238 tracks this I think.. yes we want to put up a  2.12 bundle as well. Please engage on the ticket for timelines

Hi Udit! Thanks for making this PR!
I've been working on upgrading HUDI to Spark 2.4 internally at Uber! So I'll list out a few things that I had to do, so that you're not trying to re-discover these things yourself :)
1. Some of the `create` functions in `HoodieWrapperFileSystem` don't fully work with Parquet 1.10+. See [here](https://github.com/apache/incubator-hudi/blob/b19bed442d84c1cb1e48d184c9554920735bcb6c/hudi-common/src/main/java/org/apache/hudi/common/io/storage/HoodieWrapperFileSystem.java#L146). We'll need to make sure that all the `create` functions correctly call `wrapOutputStream`.

2.  It might be causing unit tests failure as it may introduce an older version of avro into the calsspath. We're currently trying to figure out how to address this. Let us know if you have any suggestions!

Btw - I also went to UIUC! Great to meet new Illini!

@modi95 good to hear from a fellow Illini !
About the points you raised:

I guess I am not quite sure yet, where this comes into the picture. Did you run into an actual issue ? Is there an easy way to reproduce this ?

Yeah I need to look into the test failures. Not sure of the reasons yet. But first thing I guess I need to do is upgrade the docker images right, to use spark 2.4.4 and probably hive 2.3.6 ?

Change looks good. But we need to get the tests working reliably and think about changes to bundling.. Our docker images are still 2.3x . Should we upgrade and push them as well ? @bvaradar do you want to shepherd this since you are working closely with @modi95 as well?

Yeah any guidance on this front would be appreciated. On how we want to go about it for getting the tests working. I will look into the docker setup.

@umehrot2 I believe we can first push spark 2.4 docker images and get current master working (since com.databricks/spark-avro is shaded, it wil work).. @bhasudha or @bvaradar have a handle on this.

Once you resolve the conflicts and the unit-tests passes, we can merge

Mistakenly added a comment for another PR here. Deleted it.

@umehrot2 : If you want to give it a shot, its better to open a new PR, You would need to update docker.spark.version in pom files  below docker/hadoop/... and also update spark version in Dockerfile in these directories : spark_base, sparkadhoc, sparkworker and sparkmaster.
You can then run docker/build_local_docker_images.sh to build new docker images locally and then run integration tests. We would have to push these docker images so that travis integration can pick it up (we will help on this).
@umehrot2 : If you need help, let us know. Either me or @bhasudha  can get the docker images built and pushed.

@umehrot2 : If you want to give it a shot, its better to open a new PR, You would need to update docker.spark.version in pom files below docker/hadoop/... and also update spark version in Dockerfile in these directories : spark_base, sparkadhoc, sparkworker and sparkmaster.
You can then run docker/build_local_docker_images.sh to build new docker images locally and then run integration tests. We would have to push these docker images so that travis integration can pick it up (we will help on this).
@umehrot2 : If you need help, let us know. Either me or @bhasudha can get the docker images built and pushed.

@bvaradar thanks for the suggestions. Will give it a shot tomorrow, and reach out in case of any doubts.

Re-kicked Integration tests

Looks like the integration tests are failing with dependency version mismatches.

Looks like the integration tests are failing with dependency version mismatches.

@bvaradar yeah have been looking into them. Like you mentioned there are multiple dependency related issues going on. Working on a solution.

@modi95 @bvaradar
I was able to fix the integration test dependency issues on my local atleast.  To give an overview, there were 3 major failures happening:

The ITTestHoodieSanity tests were failing firstly becuase of this error:

17:15:31.995 [pool-21-thread-2] ERROR org.apache.hudi.io.HoodieCreateHandle - Error writing record HoodieRecord{key=HoodieKey { recordKey=98ea14b7-b318-4b0b-9f14-0115900a10e0 partitionPath=2016/03/15}, currentLocation='null', newLocation='null'}

java.lang.NoSuchMethodError: org.apache.parquet.io.api.Binary.fromCharSequence(Ljava/lang/CharSequence;)Lorg/apache/parquet/io/api/Binary;

	at org.apache.parquet.avro.AvroWriteSupport.fromAvroString(AvroWriteSupport.java:371) ~[hudi-spark-bundle-0.5.1-SNAPSHOT.jar:0.5.1-SNAPSHOT]

	at org.apache.parquet.avro.AvroWriteSupport.writeValueWithoutConversion(AvroWriteSupport.java:346) ~[hudi-spark-bundle-0.5.1-SNAPSHOT.jar:0.5.1-SNAPSHOT]

	at org.apache.parquet.avro.AvroWriteSupport.writeValue(AvroWriteSupport.java:278) ~[hudi-spark-bundle-0.5.1-SNAPSHOT.jar:0.5.1-SNAPSHOT]

	at org.apache.parquet.avro.AvroWriteSupport.writeRecordFields(AvroWriteSupport.java:191) ~[hudi-spark-bundle-0.5.1-SNAPSHOT.jar:0.5.1-SNAPSHOT]

	at org.apache.parquet.avro.AvroWriteSupport.write(AvroWriteSupport.java:165) ~[hudi-spark-bundle-0.5.1-SNAPSHOT.jar:0.5.1-SNAPSHOT]

	at org.apache.parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:121) ~[hive-exec-2.3.1.jar:1.10.1]

	at org.apache.parquet.hadoop.ParquetWriter.write(ParquetWriter.java:288) ~[hive-exec-2.3.1.jar:1.10.1]

	at org.apache.hudi.io.storage.HoodieParquetWriter.writeAvroWithMetadata(HoodieParquetWriter.java:91) ~[hudi-spark-bundle-0.5.1-SNAPSHOT.jar:0.5.1-SNAPSHOT]

	at org.apache.hudi.io.HoodieCreateHandle.write(HoodieCreateHandle.java:101) ~[hudi-spark-bundle-0.5.1-SNAPSHOT.jar:0.5.1-SNAPSHOT]

	at org.apache.hudi.io.HoodieWriteHandle.write(HoodieWriteHandle.java:150) ~[hudi-spark-bundle-0.5.1-SNAPSHOT.jar:0.5.1-SNAPSHOT]

	at org.apache.hudi.func.CopyOnWriteLazyInsertIterable$CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteLazyInsertIterable.java:142) ~[hudi-spark-bundle-0.5.1-SNAPSHOT.jar:0.5.1-SNAPSHOT]

	at org.apache.hudi.func.CopyOnWriteLazyInsertIterable$CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteLazyInsertIterable.java:125) ~[hudi-spark-bundle-0.5.1-SNAPSHOT.jar:0.5.1-SNAPSHOT]

	at org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:38) ~[hudi-spark-bundle-0.5.1-SNAPSHOT.jar:0.5.1-SNAPSHOT]

This is happening because in Hudi even for bits running through Spark we are using Hive 2.3.1 which is not really compatible with Hive. So, hive-exec 2.3.1 ends up in HoodieJavaApp classpath while running the example, and that has its own shaded parquet version which is old and conflicts with parquet 1.10.1.
What I propose here, is that we should use version of Hive that is compatible with Spark, atleast for the bits running inside Spark so that compatible versions of Hive end up in class paths. Now hive-exec 1.2.1.spark2 does not cause this issue as it does not shade parquet. Also, we have removed Hive shading in master now, so anyways we are dependent on runtime Hive version which is Spark's Hive version. So, from code's perspective also I think it makes sense to depend on Spark's Hive version for the code which is running inside of Spark to avoid such issues.

Post that ITTestHoodieSanity all the _rt tests were failing because now that our code is using Avro 1.8.2 and Hive is still on older versions, we need to shade avro in hudi-hadoop-mr-bundle which we had done internally for EMR through an optional profile. Now that we are migrating Hudi itself to Avro 1.8.2 we need to always shade Hive to get around this issue. More details on https://issues.apache.org/jira/browse/HUDI-268

Finally some tests were failing because spark-avro was not being passed while starting the spark-shell, and it was not finding the classes. So, I switched over to downloading spark-avro instead of databricks-avro

By making the above changes, the integration tests work now. Let me know your thoughts about these changes, if there are concerns.

@bvaradar can we re-trigger the tests ? I think this time it failed due to flaky timeouts

@bvaradar can we re-trigger the tests ? I think this time it failed due to flaky timeouts

@umehrot2 : I re-triggered anyways but looks like the log length reached max limits

@bvaradar can we re-trigger the tests ? I think this time it failed due to flaky timeouts

@umehrot2 : I re-triggered anyways but looks like the log length reached max limits

@bvaradar you are right, it failed again because of log length. Any suggestions ?

@umehrot2 : Comparing this run's logs with that of #1009 , I can see that somehow spark logging level became INFO with this PR (hence, so much logging).  You can look at the logs corresponding to org.apache.hudi.integ.ITTestHoodieSanity. Both runs use the same docker image (spark-2.4.4). Can you check how Spark log level became INFO. I didn't dig further than that but let me know if you need help.

@umehrot2 : Regarding your comment regarding moving to spark.hive version for hudi-spark-bundle and hudi-utilities-bundle, I am ok as long as hive sync works with hiveserver/HMS 2.x
Regarding realtime queries, if we shade avro in hudi-hadoop-mr-bundle,  we would need to follow same shading policy in user's jar when they use custom HoodieRecordPayload.  AFAIK, there is no good solution here  to resolve this dependency hell. If we go this approach, we would need to document this caveat and make it easy for users to perform this shading (with boiler-plate pom)
@vinothchandar : Any suggestions ?
@modi95 (cc @n3nash) : Please note that there will be similar issues with realtime queries and spark-2.4.4 when  we eventually migrate RT tables to Spark 2.4.4

On RecordPayload and avro bundling, I expect most people (esp with if you have record level indexes as we plan to)  would be happy with OverwriteWithLatest payload, which is internal to the project? I think it may be fair to simplify things to supporting these default payloads well, while providing a guide for authoring Payloads.. (Need a JIRA for documenting such a guide).. Longer term, I am wondering if we should move away from Avro as the standard object (I did not want to invent our own object for obvious reasons of incurring additional conversion cost and code maintenance) or allow Payload to be written using all or some of Row/Avro/ArrayWritable? I know its not a clear answer, but its as concrete as I could get;
@umehrot2 is shading needed to avoid conflicts with Hive's avro? hows does this fit into this PR?

@umehrot2 : Sorry for the back-and-forth on this.  Issue 1 (as mentioned in  https://github.com/apache/incubator-hudi/pull/1005#issuecomment-554089712)  is due to fat jar hive-exec. @n3nash proposed a solution in Uber which wont require moving to spark-hive.  Instead of the test dependency : hive-exec,  can you try depending on the non-fat version of the jar called : hive-exec-core.  Hopefuly, we can control parquet/avro versions getting loaded for the tests.

@umehrot2 : Sorry for the back-and-forth on this. Issue 1 (as mentioned in #1005 (comment)) is due to fat jar hive-exec. @n3nash proposed a solution in Uber which wont require moving to spark-hive. Instead of the test dependency : hive-exec, can you try depending on the non-fat version of the jar called : hive-exec-core. Hopefully, we can control parquet/avro versions getting loaded for the tests.

@bvaradar That's fine, we should take time and solve the right way.
In hudi-utilities it is a test dependency, but not in hudi-spark. hudi-spark depends on hive-service which is bringing in hive-exec as a transitive dependency.  And that is the reason its ending up in tests classpath. We would have to get rid of hive-service if we were to do that.
Also I don't see any artifact like hive-exec-core. Maven does not recognize it. But besides that, when we are depending on runtime Spark's version of Hive, is there any reason why we are wanting to build it with hive 2.x instead ? May be hudi-hive in itself since that is an independent module makes sense to build with hive 2.x. But anything running within spark, why are we inclined to building with version of hive not supported by spark ?

@vinothchandar Yes updating quick start page makes sense. Will you guys be doing that ?

Thanks @n3nash  for your thoughts.
@umehrot2 : If it is possible to achieve spark 2.4 upgrade cleanly without moving to spark-hive version, it makes sense to me to retain native hive version. I think it is better to not get locked down on spark-version of hive.
As we are using custom code (non-spark) to do hive syncing, theoretically speaking - we may run into some hive issues which would need upgrade but as the issue is not seen in spark, they may be unwilling to patch their hive jars.  we can use spark-hive as a last resort if we cannot upgrade to Spark 2.4 any other way  :)
In that spirit, To your concern related to transitive dependencies in hudi-spark module - As maven honors dependency ordering, can we list hive-exec (with classifier as "core") in the dependency section before hive-service and add exclusions in the dependency section for hive-service to exclude hive-exec. I am not sure if this would work but don't have time to try this out myself.
Something along the lines of :

 <groupId>${hive.groupid}</groupId>

 <artifactId>hive-exec</artifactId>

 <version>${hive.version}</version>

 <classifier>core</classifier>

${hive.groupid}
hive-service
${hive.version}

 <exclusions>

   <exclusion>

     <groupId>${hive.groupid}</groupId>

     <artifactId>hive-exec</artifactId>

   </exclusion>

 </exclusions>

@umehrot2 : If we cannot make it to work any other way, I am ok with using spark-hive.

What balaji suggested makes sense to me. spark.hive version has its own issues, but we can live with this if there is no other way.

Want to know the progress of this PR now. I think not every user uses Spark2.4, Can we combine the two methods (databricks-avro and spark-avro) and simplify them as our own internal implementation, which can be compatible with most versions of spark2?

@umehrot2 are you still driving this? We would like to merge this asap, giving us enough time for the next release to be cut..
cc @leesf

@umehrot2 are you still driving this? We would like to merge this asap, giving us enough time for the next release to be cut..
cc @leesf
@vinothchandar just got back from my time off a couple of days back. Let me catch up on this PR and try to get it merged soon. When are we targeting for next release to be cut ?

My suggestion is to freeze code by 15th, test the RC for a week and cut one by jan last/feb first week. @leesf is the release manager though.. So he can share plans..

@umehrot2 are you still driving this? We would like to merge this asap, giving us enough time for the next release to be cut..
cc @leesf
@vinothchandar just got back from my time off a couple of days back. Let me catch up on this PR and try to get it merged soon. When are we targeting for next release to be cut ?

My thought is to get code ready by 15th to get some buffer, and release at the end of this month.

Ack working with that deadline in mind.

@vinothchandar @bvaradar @n3nash I have updated the PR to now use the hive-exec with core classifier to solve the unit test issues that were occuring becuase of Hive. Removed the usage of spark.hive.version as desired. Let me know if this looks good.

@bvaradar Created a JIRA to track documentation of Avro shading caveat https://issues.apache.org/jira/browse/HUDI-519

About documentation of --packages are you guys going to take care of that ?

@umehrot2 typically the author of the PR also does the doc changes to keep things in sync.. Are you able to make the changes in docs? Mainly, it should be the quickstart, demo, writing_data pages

Glad to see this PR can be merged, does it mean we need to use spark2.4 and avro 1.8 in hudi 0.5.1 finally?

@vinothchandar Glad to see this PR can be merged, does it mean we need to use spark2.4 and avro 1.8 in hudi 0.5.1 finally?
I think I should close my PR about spark avro before later.

Yes, It is merged and should be available in 0.5.1

@bvaradar
OK~ I will close my PR https://github.com/apache/incubator-hudi/pull/770 right now.
The problem about timestamp type maybe can be discussed and solved in other JIRA in later version.


Fixed example code

The comma was at the end of a comment instead of the end of the line of code, so pasting it into an editor would throw errors.

Well spotted, thanks!


remove FF for built-in scanners, Wiz, default baselines, Jira integra…

…tion STO-7735
Preview links

STO release notes > Version 1.99.0

Removed feature flag notes from all linked topics

Early Access features > STO features promoted to GA

Merge requirements
PRs must meet these requirements to be merged:

[ ] Successful preview build.
[ ] Code owner review.
[ ] No merge conflicts.
[ ] Release notes/new features docs: Feature/version released to at least one prod environment.

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://667345e970b58f938b9eb3c8--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://667346f3ab5bad91b47b02c9--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://667348cd235f9b9984a5c336--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://6673495fb8e00c9a199862e1--harness-developer.netlify.app


[ci skip] [cf admin skip] NO_CI adding bot automerge

Hi! This is the friendly automated conda-forge-webservice.
I've added bot automerge as instructed in #7.
Merge this PR to enable bot automerging.
Fixes #7

I'm actually removing automerge from most of the Python feedstocks I maintain. Things that do not update the deps often is fine but I'm finding myself fixing automerge erros more and more and merging them myself is getting easier.
Note that grayskull already has a solution for that. We just need to implement it in the bot to fix automerge.


EVENTS: Safety message page creation as a sub page from an event

This is a breakout from Events Epic: #2039
Requirement:
We need to have a safety message page to set up an event that would have prompts like:
" Does this event require a safety message?" If Yes is selected, the following information would need to be entered:

Start Date
End Date
Start Time
End Time:
number of Volunteers required for it
Type of event.
If no is selected, the information does not appear.

Definition of Done:

The Safety message page would be created with the items mentioned above.

Additional Information:

Comment from @stevejgordon : PARTIALLY COMPLETE. We have everything except the safety message requirement.

Technical expectations:  we need to add a safety message to the Event and add the message (if it exists) to any Event page where users can sign up or admins can edit it.


Add default billing mode option

In order to help the UI to decide which billing mode must be selected by default, we must add a new optional field to show the default billing mode.
1.Extend Billing data model with new field defaultBillingMode.
2. When provided, the value must be one of the values available in the billingModes field.
3. Add description - This is  used only for display purposes. When provided, this billingMode will be selected  in the frontends by default.

let me do it


Update selector in dragonslair scraper

Helping to keep the scraper up to date.
DragonsLair updated their selector a lil bit 😄

hehe, it was my plan to update it this weekend, thanks a bunch <3


[question] time series

Hello there,
I want to know if you ran any tests on time series and how it performs in such data. Does it provide a reasonable explanation about an anomaly? Have you seen the alibi-detect packages, and if so how this compares to them?
Thanks for your time, great post on TDS tho.

Yes, I cover time series data in my book, Outlier Detection in Python,
https://www.manning.com/books/outlier-detection-in-python.
It's quite possible to do outlier detection with time series. Counts
Outlier Detector is designed for tabular data, which is a bit different,
though time series data can be converted to tabular format.
I have used Alibi-Detect and cover that in the book as well. In general, it
works well, but all outlier detectors are a bit different, and all
emphasize, and all miss, different types of anomalies.
Brett
On Fri, Jun 21, 2024 at 1:06 AM Lucas Kaniak @.***>
wrote:

Hello there,
I want to know if you ran any tests on time series and how it performs in
such data. Does it provide a reasonable explanation about an anomaly? Have
you seen the alibi-detect packages, and if so how this compares to them?
Thanks for your time, great post on TDS tho.
—
Reply to this email directly, view it on GitHub
https://github.com/Brett-Kennedy/CountsOutlierDetector/issues/4, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AUHC56RDJVSCP3INE34AAVTZIOYEHAVCNFSM6AAAAABJVFTOSCVHI2DSMVQWIX3LMV43ASLTON2WKOZSGM3DKNZSGI2DANQ
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>


adds pinterest - today page and single photo

Description

I added support to Pinterest, get the today page as well as a extracting images from the page without creating an account.

Resolves: #850
Checklist

Before submitting this pull request, kindly verify that the ensuing checkpoints have been reached.

[x] Have you adhered to the repository's defined coding convention rules?
[x] Have you updated the 'documentation.md' file with the method/function documentation?
[x] Have you sent a message along with the result or response?
[x] Have you used the try-catch technique?
[x] Has the method/class been added to the documentation (md file)?

Screenshots

Uploading a screenshot illustrating the approach you developed for validation is required.

Additional Notes/Comments

Any additional remarks or suggestions concerning this pull request are welcome.

I certify that I have carried out the relevant checks and provided the requisite screenshot for validation by submitting this pull request.
I appreciate your contribution.

Go ahead @samejima-san
Note

Create a separate module for this, as per the folder and project structure
Do not use the `selenium web driver as it is incompatible with all devices and cloud platforms.
Before making any changes, please check whether the module you want to add exists. If yes, then you can just add your functionality as a method only make a separate module and class for it.

@nikhil25803 no worries good luck with school!
I can help maintain this project if you need another set of eyes on it.

Sure @samejima-san, will let you know :))


latest version sidekiq fixes for papyrus

When importing inventory adjust we supose to be running a consolidation spool job. infortunately this is not happening anymore
If possible change the Papyrus::FormatParams to a class method
Papyrus::ObjectConverter.serialize(someobj)

Thank you for review!
Fixed second one, I guess first one its already fixed in import by you!


Document 6.0.0 may break FactoryBot.lint

In a sufficiently large codebase, the automatic definition of traits for Active Record enum  (#1380) may result in more combinations being checked by FactoryBot.lint. This breaking change may not be obvious because the first impression of the error messages may be "these factories were passing before the upgrade but now failing ?", while the truth is that they don't exist before the upgrade. It took me a while to realize this so I hope adding this note could help people be aware of this situation when upgrading.

Thanks!


feat(frontend): ConvertAmountDisplay component

Motivation
Desktop:

Mobile:

@peterpeterparker it's actually "free", here is how the whole picture looks like (screenshot from figma):


Uncaught error and manifest file not working

Hi Loran
The extension is not working for me. I get the errors below. I am a complete noob and a sad one at that!
ui.js:formatted:438 Uncaught (in promise) TypeError: Cannot read property '1' of null
at readMore (ui.js:formatted:438)
readMore @ ui.js:formatted:438
NodeParser.js:163 Uncaught (in promise) Error: invalid token 151
at NodeParser.getToken (NodeParser.js:163)
at NodeParser.readString (NodeParser.js:124)
at NodeParser.readAttributes (NodeParser.js:177)
at NodeParser.readNode (NodeParser.js:85)
at NodeParser.readList (NodeParser.js:185)
at NodeParser.readNode (NodeParser.js:91)
at Main.js:71
NodeParser.getToken @ NodeParser.js:163
NodeParser.readString @ NodeParser.js:124
NodeParser.readAttributes @ NodeParser.js:177
NodeParser.readNode @ NodeParser.js:85
NodeParser.readList @ NodeParser.js:185
NodeParser.readNode @ NodeParser.js:91
(anonymous) @ Main.js:71
Main.js:567 Uncaught DOMException: Failed to read the 'cssRules' property from 'CSSStyleSheet': Cannot access rules
at getCSSRule (chrome-extension://dmojkdggbdjlhgmakakajjbbdibkjkgj/core/Main.js:567:39)
at HTMLDocument. (chrome-extension://dmojkdggbdjlhgmakakajjbbdibkjkgj/core/Main.js:89:3)
getCSSRule @ Main.js:567
(anonymous) @ Main.js:89
NodeParser.js:163 Uncaught (in promise) Error: invalid token 151
at NodeParser.getToken (NodeParser.js:163)
at NodeParser.readString (NodeParser.js:124)
at NodeParser.readAttributes (NodeParser.js:177)
at NodeParser.readNode (NodeParser.js:85)
at NodeParser.readList (NodeParser.js:185)
at NodeParser.readNode (NodeParser.js:91)
at Main.js:71
NodeParser.getToken @ NodeParser.js:163
NodeParser.readString @ NodeParser.js:124
NodeParser.readAttributes @ NodeParser.js:177
NodeParser.readNode @ NodeParser.js:85
NodeParser.readList @ NodeParser.js:185
NodeParser.readNode @ NodeParser.js:91
(anonymous) @ Main.js:71
Promise.then (async)
wsHook.after @ Main.js:66
(anonymous) @ WebScoketInterception.js:88
Also I downloaded the zip file and tried to unpack it but got a error message saying that manifest file is missing.
Please help. Thanks so much.

Did you follow these steps?

Click 'clone or download' and choose to download the ZIP file
Extract the folder from the ZIP file
Open up chrome://extensions/
Enable developer mode at the top of the screen by clicking the slider
Click 'load unpacked extension'
Select the dist/ folder from step 2
The extension should appear at the top of the list
Refresh your WhatsApp web
An orange slider should appear at the top of the screen

Please direct your attention to step 6 to make sure you point Chrome to the correct folder.

Just tested it on Chrome 68/Win10 and it seems to work fine.
Where do you see that error? Can you make a screenshot? You can ctrl-c/v it in these comments.

Hi Loran
Chrome version is 68.0.3440.106.
I am getting the error below:
Failed to load extension
File
~\Downloads\WhatsAllApp-master (3)\WhatsAllApp-master
Error
Manifest file is missing or unreadable
Could not load manifest.

You have to select the /dist folder as it says in step 6.

You're welcome. Please check the FAQ at the README.md for these questions.

Thanks so  much <3


perf: don't estimate datum size for struct and list

If struct and list are deeply nested, it will be expensive to estimate the size of the serialized hash key.
https://github.com/risingwavelabs/risingwave/pull/9556#issuecomment-1536355186.
Mentioned by @TennyZhuang.

Good first issue for contributors interested in perf.

Write a benchmark which has nested struct and list.
Compare w/o estimate size for it, compare performance from benchmark.

Some Struct with specific schema have a fixed size and we can estimate it 🤔

That's a good point. I guess currently we don't do that: https://github.com/risingwavelabs/risingwave/blob/4222535a02517a35409cb5d909be7bddb288e4ed/src/common/src/util/value_encoding/mod.rs#L164-L181.
If struct is fixed size we can estimate it on the chunk level rather than per datum. Let me extend this issue to cover that too.
If struct has variably sized field, we can just estimate the fixed sized parts.


[TextInput] lineHeight in multiline TextInput

It would be great if we can set lineHeight  in a TextInput which is set multiline={true}

So I looked into this...and at first it seems easy...you can override NSLayoutManager's lineSpacingAfterGlyphAtIndex method and set the lineHeight (new property) on RCTTextView (used for multiline).
- (CGFloat)layoutManager:(NSLayoutManager *)layoutManager lineSpacingAfterGlyphAtIndex:(NSUInteger)glyphIndex withProposedLineFragmentRect:(CGRect)rect
{
  return [_lineHeight floatValue];
}

The problem is that, the caret is too large.

The rect can be adjusted with caretRectForPosition BUT you need to subclass UITextView and in the current implementation, RCTTextView is a subclass of RCTView.
It looks like this one won't be feasible without a reorganization of RCTTextView.

@dsibiski do you know if this gets fixed by any of the current PRs for multiline support and such?

@browniefed Hmm, no, I don't know for sure. I'd have to hunt down some of those PRs...are there any others that you know of that tried to address line height?

I also encountered the same issue :-)
@ide @brentvatne ideas?

Experiencing the same issue :+1:

Hi there! This issue is being closed because it has been inactive for a while.
But don't worry, it will live on with ProductPains! Check out its new home: https://productpains.com/post/react-native/textinput-lineheight-in-multiline-textinput
ProductPains helps the community prioritize the most important issues thanks to its voting feature.
It is easy to use - just login with GitHub.
Also, if this issue is a bug, please consider sending a PR with a fix.
We're a small team and rely on the community for bug fixes of issues that don't affect fb apps.

Still an issue


Adapt notifications

What is this feature about? (1 sentence)
This removes some unnecessary notifications inside the Gnosis Safe Multisig web interface and reduces the duration of others.
Why is it needed? What is the value? For whom do we build it?

When using the interface, some notifications are quite intrusive. This is something we notice ourselves and some users have complaints about this.
Especially when there are multiple txs to sign/execute, it gets messy.

High-level overview of the feature

An overview of the notifications can be found here: https://docs.google.com/spreadsheets/d/1gcnqzOhntt_ulqZ-kCsGnJZGy_Zl5DBTpU0X1KeGilc/edit#gid=0

The "Proposal" column contains what to do with each notification.

"A pending transaction requires your confirmation!" – a transaction is not really pending when this notification comes up, it's awaiting. Pending is when execution has been submitted.
"Please sign the settings change" – for some reason there's a separate notification for settings change, but not for other types of transactions.
Some notifications could be inline messages (Safe name changed, owner name changed, entry created, entry saved).

"A pending transaction requires your confirmation!" – a transaction is not really pending when this notification comes up, it's awaiting. Pending is when execution has been submitted.

true, how about just removing "pending" then, i.e. call it "A transaction requires your confirmation."?

"Please sign the settings change" – for some reason there's a separate notification for settings change, but not for other types of transactions.

Yes, don't remember why we did it that way. Perhaps to emphasize the importance of the tx? I would keep it as is though for now.

Some notifications could be inline messages (Safe name changed, owner name changed, entry created, entry saved).

Agree, long term that's the way to go.

true, how about just removing "pending" then, i.e. call it "A transaction requires your confirmation."?

Sounds ok.

It seems to me that some of the notification changes from the doc aren't yet implemented. I asked on Slack about it.

All good here. I mixed 2 PRs up.

looks good to me


App crashing when trying to import Generic JSON file from Coldcard

Was trying to import a generic JSON file using an SD card. The SD card was empty, without any other files.  The file was properly written onto the card. When I put the card into the phone and clicked on Import after selecting "via file". The App crashed.
https://github.com/bithyve/bitcoin-keeper/assets/68264080/5651092b-38c9-49f5-85d3-b2a3247c685f

@cakesoft-utkarsh Please Check this


FIX: MovingAverage from_config, get_config

get_config correctly serializes the wrapped optimizer using tf.keras.optimizers.serialize()

from_config to deserialize the config correctly, including the wrapped optimizer

Wrapper optimizer can be specified using keras optimizer strings like: adam, rmsprop, adagrad, etc.

Add tests for above new functionality

Minor refactoring in moving_average.py

How does this optimizer affect checkpointing?
If we store the model, will it take a moving average value?
In TF1, both original value and average value are stored and we need to apply EMA when we load the model for inference. How does it work here?

@saberkun It's doesn't save or checkpoint the moving average of the weights. The user must call opt.assign_average_vars(model.variables) before saving the model.
When the model is loaded, it only stores a single set of weights, either the original or the moving average based on whether the weights were assigned before saving to disk.
This won't work well with using model.fit, but it should be fine if you're using a custom training loop.
I'll create an issue to save the moving average weights when saving the MovingAverage optimizer. This will be closer to TF1.x behavior.


Dev simplification

We could simplify a lot the "cleaning environment" task of the cli by removing the bind between the local folder and the engine's container.
Like this, when the container is deleted, any data is also deleted.
It will help to keep the dev env always clean and reproducible!

fixed by https://github.com/liteflow-labs/liteflow-js/pull/72


/present-proof/send-request 403 error

Hi, I'm running agents in v.0.7.1 on an Azure VM, connected to Sovrin's BuilderNet.
Trying to call /present-proof/send-request on my verifier agent with body:
{
  "comment": "string",
  "connection_id": "cb679c76-467e-4580-af3b-a6299a150d7d",
  "proof_request": {
    "name": "Proof request",
    "non_revoked": {
      "to":<PHONE_NUMBER>
    },
    "nonce": "1234567890",
    "requested_attributes": {
      "additionalProp1": {
        "name": "org"
      }
    },
    "requested_predicates": {
      "additionalProp2": {
        "name": "age",
        "p_type": ">=",
        "p_value": 10,
        "restrictions": [
          {}
        ]
      }
    },
    "version": "1.0"
  },
  "trace": false
}

But got a 403: Connection c5cd3e9b-31c6-4a12-aa5d-a387f5520c55 not ready error in Postman, and aries_cloudagent.core.conductor WARNING Cannot queue message for delivery, no supported transport in CLI, could anyone support with this please?
For context, the same body was used successfully in the exact same setup except using a local VON-Network.

Hi @teyahdavis , this error could be caused by lots of factors.  There could be connection issues between aca-py and the ledger, could be a TAA issue (I think with buildernet you have to accept TAA but not with a local VON network), or "other".
Check the aca-py log files for any error messages, there could be an error during the startup/initialization which is affecting this operation

Thanks @ianco !
In my verifier aca-py logs, i'm just seeing:
2021-11-09 11:59:21,442 aries_cloudagent.admin.server ERROR Handler error with exception: Connection 78c62e39-d922-4216-9455-00ef46d23723 not ready

=================
Traceback (most recent call last):
  File "/home/indy/aries_cloudagent/admin/server.py", line 162, in ready_middleware
    return await handler(request)
  File "/home/indy/aries_cloudagent/admin/server.py", line 199, in debug_middleware
    return await handler(request)
  File "/home/indy/.pyenv/versions/3.6.13/lib/python3.6/site-packages/aiohttp_apispec/middlewares.py", line 45, in validation_middleware
    return await handler(request)
  File "/home/indy/aries_cloudagent/admin/server.py", line 366, in setup_context
    return await task
  File "/home/indy/aries_cloudagent/protocols/present_proof/v1_0/routes.py", line 540, in presentation_exchange_send_free_request
    raise web.HTTPForbidden(reason=f"Connection {connection_id} not ready")
aiohttp.web_exceptions.HTTPForbidden: Connection 78c62e39-d922-4216-9455-00ef46d23723 not ready
2021-11-09 11:59:21,444 aiohttp.access INFO <IP_ADDRESS> [09/Nov/2021:11:59:21 +0000] "POST /present-proof/send-request HTTP/1.1" 403 274 "-" "PostmanRuntime/7.28.4"

and there don't seem to be any problems with the startup - it seems like the problem is with the verifier agent because my issuer and holder were able to communicate and issue/store a credential etc.

It looks like the connection between your verifier and holder/prover isn't in active state.  Try the /connections endpoint and see what state your connection is in

when my verifier creates a connection request and my holder receives it, the state I can see is: request
generally I don't have to do anything other than receive it for it to be active?

On the verifier side, you are creating an invitation.  On the holder side, you have to receive and then request the connection.  And then on the verifier side you have accept the connection request.  (There is another "acknowledge connection" step but this one is optional and not properly specified in the RFC)
Depending on what "auto" flags you specify on aca-py startup, some of these steps may happen automatically.
So, check the startup parameters on each of your aca-py instances, you may have auto flags set on one but not another, and this would mean you need a different set of manual steps to get the connections setup between different agents.

each of my agents have --auto-accept-invites \ --auto-accept-requests \ flags, is there something else it might be?

These should work, what do you get from the /connections endpoint?

verifier:
{
            "invitation_key": "6fRr85NVxdufZ5XAcXyhVRGYvBDuizoYp6brFztGwUnH",
            "created_at": "2021-11-09 16:35:51.537150Z",
            "accept": "auto",
            "updated_at": "2021-11-09 16:35:51.537150Z",
            "connection_id": "867add77-1773-44cf-8c46-14fa6fa5bf77",
            "their_role": "invitee",
            "routing_state": "none",
            "invitation_mode": "once",
            "rfc23_state": "invitation-sent",
            "connection_protocol": "connections/1.0",
            "state": "invitation"
        }

holder:
{
            "invitation_key": "6fRr85NVxdufZ5XAcXyhVRGYvBDuizoYp6brFztGwUnH",
            "created_at": "2021-11-09 16:36:03.484147Z",
            "request_id": "5a8f3a02-0c5b-49a4-8c93-a560c8450a7d",
            "their_label": "Buyer",
            "state": "request",
            "connection_protocol": "connections/1.0",
            "invitation_msg_id": "1b60baaa-ae74-4d7d-b7eb-1f9b517936ce",
            "updated_at": "2021-11-09 16:36:03.540631Z",
            "their_role": "inviter",
            "accept": "auto",
            "my_did": "3W2PXB2iP3r98W7SDv6XdM",
            "routing_state": "none",
            "connection_id": "3a287f56-8346-48b1-8b2e-1605b63c6604",
            "rfc23_state": "request-sent",
            "invitation_mode": "once"
        }

Looks like the holder has sent the connection request, but the verifier hasn't received it (status is still invitation)
Possibly the invitation is getting sent with the wrong endpoint?  (should be the endpoint of the verifier)   In which case the holder thinks they've sent the connection request but it never made it to the verifier

the steps I'm taking to make the connection are:
verifier - /connections/create-invitation
holder - /connections/receive-invitation
I'm copying the invitation {} body from the response from the verifier and literally just sending it in the holder receive invitation.
I've double-checked the endpoints and they seem to be correct, just confused as this is what I've done for the issuer and holder and that worked as expected?
also thank you so much, I appreciate your help!

If the holder isn't able to connect to the issuer I'd expect some kind of error message in the log file.
Can you past the "banner" that's displayed on startup from each agent, as well as the contents of your invitation?

invite contents:
{
        "@type": "did:sov:BzCbsNYhMrjHiqZDTUASHg;spec/connections/1.0/invitation",
        "@id": "1644b969-a897-473e-9988-fbbdbf188d21",
        "serviceEndpoint": "http://<IP_ADDRESS>:8000/",
        "label": "Buyer",
        "recipientKeys": [
            "6rzyWQudN5y9vTK46zbREN6Xay3KQPCDP5SFPjGewrdW"
        ]
    }

by banner do you mean

or something else?

wait I've found the problem, it was a copy/paste mistake in the verifier's startup command!

Aha!  Good work!


adds port to titlebar

CI seems to be broken, the changes look good so I'll merge it anyways

I meant to get back to this after the last smod release but multiadmin can't be built as a single platform independent binary anymore, so just have to split it into a windows and linux one.

Latest version in the 3.4.X branch uses AOT, CI just needs to be updated, check #260

Ah ok, I might just not have found the way to do it last time then.


Missing documentation for PhoneFactorProtocolProvider

The PhoneFactorProtocolProvider does not appear to be documented within the Reference area of the B2C documentation. There is an article in this section related to Azure Multi-Factor authentication, but no information is provided as to how to select one vs the other. Also, the preferred Phone Factor Protocol article should be linked from the the "Phone factor provider " entry in the "Types of Technical Profiles" section above (https://docs.microsoft.com/en-us/azure/active-directory-b2c/technical-profiles-overview#type-of-technical-profiles).

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: fe70bb3e-884f-a8db-eee7-ae98cf653b93
Version Independent ID: d8647921-91df-9c37-8cd9-d44ce938dd9d
Content: Overview of technical profiles in custom policies - Azure AD B2C
Content Source: articles/active-directory-b2c/technical-profiles-overview.md
Service: active-directory
Sub-service: b2c
GitHub Login: @mmacy
Microsoft Alias: marsma

@dotnetgator thanks for this. The content author is reviewing your feedback.

#reassign:mmacy

@yoelhor do you have details on these PhoneFactorProtocolProvider details?

@v-shils  for what reason was this closed? The TP is still included in the Starter Pack Policies, and remains undocumented. It seems to have been closed without explanation or action item.

@dotnetgator Hi John,
When a PR in the private repo (azure-docs-pr) is merged, and that PR fixes an issue (includes a "Fixes URLTOTHEISSUE" comment), the issue in azure-docs is automatically closed when the PR reviewer (in this case @v-shils) merges it.
It takes some time to make it from master > live, and then there are two publishing events per day - 10AM PST and 3PM PST - after which the change should be published lived. Thus, there's always a bit of delay prior to the changes showing up on the live site.

@dotnetgator Ah, interesting - without perms in the upstream repo that contains the PR fix, GitHub doesn't display the link.
With perms in upstream repo:

Without perms:

Thanks! It was just unusual compared to other updates that I had received, and looked like it had been "just closed". my apologies for the confusion.

@dotnetgator Oh, and one other tidbit - when you see PRMerger has closed an issue, that means that the edit on the doc was minimal enough that it satisfied the automerge criteria and was automerged by the PRMerger bot when the author signed-off on the PR. When you see a v-dash or other GitHub use has merged the PR, that means it's a large enough edit that it's gone through a human PR review.


Failed register interface application mapping for service xxx

Dubbo version: 3.2.5
Java version: 17

dubbo服务启动报错 Failed register interface application mapping for service xxxx.....

在源码中找到相关逻辑地方在ServiceConfig

发现没有地方可以设置关掉的开关，我的项目是按应用注册，不需要接口映射这些过渡老版本的问题

Service name mapping is a required option for app level service discovery even though you do not want to compact with 2.7.x or interface level service discovery.

Service name mapping is a required option for app level service discovery even though you do not want to compact with 2.7.x or interface level service discovery.

在我的项目里是完完全全的应用级服务发现，不需要接口级的。我觉得还是有必要有开关可以关掉，即使为了兼容老用户让这个开关默认开启，但也不能让没有历史负担的新项目承担这个兼容问题

This is the requirement of application level service discovery.

No news is good news. Please feel free to create a new issue if you have any question.


simplify mac build instructions

turn 3 lines into 1 that does the same thing

Huh, neat.

Pretty sure you can do the same mkdir on linux.


UGT + UA-Intercom incorrectly changing vehicle gate to open

Describe the bug
When a UA-Intercom is connected directly to a UGT, the update messages from the intercom are causing the integration to falsely claim the gate is open.
From tracing the code, it appears this is happen because the UA-Intercom is sending access.data.device.update messages, but the message doesn't contain a data["configs"][x]["key"] = "input_state_dps" entry.  Eventually the open status clears, but I don't know yet if that is because a new message comes through or if the integration occasionally polls for status (I need to do more debug logging after it falsely reports open).
My plan to fix it is bail out of the _handle_config_update call if the appropriate key isn't present, instead of assuming the door / gate is open.
Which Hub are you using
Unifi Gate Hub + UA-Intercom
Versions:

Home Assistant version 2024.12.5
Unifi Access version 2.5.3
Unifi Access Integration version 1.2.5

Home Assistant DEBUG logs (MANDATORY)
Logs will follow in the next few days (I need to go through and make sure redactions aren't needed).

Actually looks like this is a dup of #84 and isn't specific to the combination of UGT and Intercom.  Closing as a dup, and I'll test the branch mentioned on that issue.


kubevirt: 0.52.0 -> 0.53.0

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

I think this can go in, even though we're in ZHF. It's unlikely that this update introduces a regression, and the package didn't exist in the previous stable release, anyway.


/var/log/php dir not writeable by webapp user

During my time testing this configuration (its working great btw, thanks for this) I found that the configured www-error.log file was not being created in /var/log/php/7.1
This PR chown's /var/log/php to be owned by the webapp user so this file is created

Thanks for the fix Jason.


Cut release and produce docker image: JHU Package Provider

This is complete, with release 0.0.1 of the package providers.  Includes the JHU J10P provider, a generic BagIt provider, and the NIHMS provider.


Error build TypeScript (tsc)

What happened?
Steps to Reproduce
npm run build

Expected Result
Builds all dependencies
Actual Result
Fail to build module @opentelemetry/sdk-logs
Additional Details
OpenTelemetry Setup Code
// Import downloaded modules: winston
import Transport from "winston-transport";
import type TransportStream from "winston-transport";

// Import downloaded modules: Open Telemetry
import { Logger } from "@opentelemetry/api-logs";
import { OTLPLogExporter } from "@opentelemetry/exporter-logs-otlp-grpc";
import { Resource } from "@opentelemetry/resources";
import {
  LoggerProvider,
  SimpleLogRecordProcessor,
} from "@opentelemetry/sdk-logs";
import { SemanticResourceAttributes } from "@opentelemetry/semantic-conventions";

const OpenTelemetryLogLevel = {
  FATAL: 21,
  ERROR: 17,
  WARN: 13,
  INFO: 9,
  DEBUG: 5,
  TRACE: 1,
} as const;

// we use default winston level https://github.com/winstonjs/winston#logging-levels
// OTL have different level, we need to do a mapping, so customer have final log in OTLP
const mapLogLevelFromNpmToOpenTelemetry = (
  level: string,
): keyof typeof OpenTelemetryLogLevel => {
  switch (level) {
    case "error":
      return "ERROR";
    case "warn":
      return "WARN";
    case "info":
      return "INFO";
    case "verbose":
    case "debug":
    case "silly":
      return "DEBUG";
    default:
      return "INFO";
  }
};

const mapNpmLogLevelToOpenTelemetrySeverityNumber = (
  level: string | undefined,
): (typeof OpenTelemetryLogLevel)[keyof typeof OpenTelemetryLogLevel] => {
  switch (level) {
    case "trace":
      return 1;
    case "verbose":
    case "debug":
      return 5;
    case "info":
      return 9;
    case "warn":
      return 13;
    case "error":
      return 17;
    case "fatal":
      return 21;

    default:
      return 9;
  }
};

export class OpenTelemetryWinstonTransport extends Transport {
  loggerOTL: Logger;

  constructor(opts?: TransportStream.TransportStreamOptions) {
    super(opts);
    const logExporter = new OTLPLogExporter();
    const loggerProvider = new LoggerProvider({
      resource: new Resource({
        [SemanticResourceAttributes.SERVICE_NAME]:
          process.env["OTEL_SERVICE_NAME"],
      }),
    });

    loggerProvider.addLogRecordProcessor(
      new SimpleLogRecordProcessor(logExporter),
    );
    this.loggerOTL = loggerProvider.getLogger("default");
  }

  /* eslint-disable @typescript-eslint/no-explicit-any */
  override log(info: any, next: () => void) {
    /* eslint-disable @typescript-eslint/no-explicit-any */
    setImmediate(() => {
      const logRecord = {
        severityNumber:
          info?.severityNumber ||
          mapNpmLogLevelToOpenTelemetrySeverityNumber(info?.level),
        severityText:
          mapLogLevelFromNpmToOpenTelemetry(info?.level || "") || "",
        body: info?.message || "",
      };

      this.loggerOTL.emit(logRecord);
      next();
    });
  }
}

package.json
{
  "scripts": {
    "clean": "rimraf dist",
    "build": "npm run clean && npx tsc",
  },
  "dependencies": {
    "@opentelemetry/api-logs": "^0.48.0",
    "@opentelemetry/exporter-logs-otlp-grpc": "^0.48.0",
    "@opentelemetry/resources": "^1.21.0",
    "@opentelemetry/sdk-logs": "^0.48.0",
    "@opentelemetry/semantic-conventions": "^1.21.0",
  }
}

Relevant log output
npm run build

><EMAIL_ADDRESS>build
> npm run clean && npx tsc

><EMAIL_ADDRESS>clean
> rimraf dist

node_modules/@opentelemetry/sdk-logs/build/src/export/NoopLogRecordProcessor.d.ts:5:5 - error TS2416: Property 'onEmit' in type 'NoopLogRecordProcessor' is not assignable to the same property in base type 'LogRecordProcessor'.
  Type '(_logRecord: ReadableLogRecord) => void' is not assignable to type '(logRecord: LogRecord, context?: Context | undefined) => void'.
    Types of parameters '_logRecord' and 'logRecord' are incompatible.
      Type 'LogRecord' is not assignable to type 'ReadableLogRecord' with 'exactOptionalPropertyTypes: true'. Consider adding 'undefined' to the types of the target's properties.
        Types of property 'severityText' are incompatible.
          Type 'string | undefined' is not assignable to type 'string'.
            Type 'undefined' is not assignable to type 'string'.

5     onEmit(_logRecord: ReadableLogRecord): void;
      ~~~~~~

node_modules/@opentelemetry/sdk-logs/build/src/LogRecord.d.ts:9:22 - error TS2420: Class 'LogRecord' incorrectly implements interface 'ReadableLogRecord'.
  Types of property 'severityText' are incompatible.
    Type 'string | undefined' is not assignable to type 'string'.
      Type 'undefined' is not assignable to type 'string'.

9 export declare class LogRecord implements ReadableLogRecord {
                       ~~~~~~~~~

Found 2 errors in 2 files.

Errors  Files
     1  node_modules/@opentelemetry/sdk-logs/build/src/export/NoopLogRecordProcessor.d.ts:5
     1  node_modules/@opentelemetry/sdk-logs/build/src/LogRecord.d.ts:9

@Clement-Stauner thanks for reaching out. I've tried reproducing this (https://github.com/pichlermarc/repro-4489) but I'm not sure I'm missing something. Would you mind trying to remove node_modules and package-lock.json and trying again?

I managed to solve the issue by setting the TS compiler skipLibCheck option to true.
{
  "compilerOptions": {
    "skipLibCheck": true
  }
}

But there are still two native type safety bugs in the library that the TS compiler caught while trying to compile, it you set strong type checking in the tsconfig.json file, you will get the error.
Here here my tsconfig.json file that caught the two bugs with skipLibCheck option to false.
{
  "compilerOptions": {
    "baseUrl": "./", // Add this line
    "paths": {
      // Add or adjust this section
      "*": ["node_modules/*", "src/*"]
    },
    "module": "CommonJS", // Ensure this is set for Node.js compatibility
    "rootDirs": ["src", "dist"], // Add this line if you want to include multiple root directories

    "outDir": "dist",

    "noEmitOnError": true,
    "forceConsistentCasingInFileNames": true,
    "esModuleInterop": true,
    "isolatedModules": true,
    "moduleResolution": "node",

    /* Type Checking */
    "strict": true /* Enable all strict type-checking options. */,
    "noImplicitAny": true /* Enable error reporting for expressions and declarations with an implied 'any' type. */,
    "strictNullChecks": true /* When type checking, take into account 'null' and 'undefined'. */,
    "strictFunctionTypes": true /* When assigning functions, check to ensure parameters and the return values are subtype-compatible. */,
    "strictBindCallApply": true /* Check that the arguments for 'bind', 'call', and 'apply' methods match the original function. */,
    "strictPropertyInitialization": true /* Check for class properties that are declared but not set in the constructor. */,
    "noImplicitThis": true /* Enable error reporting when 'this' is given the type 'any'. */,
    "useUnknownInCatchVariables": true /* Default catch clause variables as 'unknown' instead of 'any'. */,
    "alwaysStrict": true /* Ensure 'use strict' is always emitted. */,
    "noUnusedLocals": true /* Enable error reporting when local variables aren't read. */,
    "noUnusedParameters": true /* Raise an error when a function parameter isn't read. */,
    "exactOptionalPropertyTypes": true /* Interpret optional property types as written, rather than adding 'undefined'. */,
    "noImplicitReturns": true /* Enable error reporting for codepaths that do not explicitly return in a function. */,
    "noFallthroughCasesInSwitch": true /* Enable error reporting for fallthrough cases in switch statements. */,
    "noUncheckedIndexedAccess": true /* Add 'undefined' to a type when accessed using an index. */,
    "noImplicitOverride": true /* Ensure overriding members in derived classes are marked with an override modifier. */,
    "noPropertyAccessFromIndexSignature": true /* Enforces using indexed accessors for keys declared using an indexed type. */,
    "allowUnusedLabels": true /* Disable error reporting for unused labels. */,
    "allowUnreachableCode": true /* Disable error reporting for unreachable code. */,

    /* Completeness */
    "skipDefaultLibCheck": true /* Skip type checking .d.ts files that are included with TypeScript. */,
    "skipLibCheck": true
  },
  "include": ["src/**/*"]
}

Ah I see. This is actually reproducible with "exactOptionalPropertyTypes": true, which is not included in strict by default (because turning it on does cause a lot of churn for existing code bases to adhere to it).
My guess is that adding that will actually cause a huge diff though because we're not using optional properties like that. Not sure what the implications for this would be (if that's a breaking change in TypeScript, would have to try that out). It might be a good idea to do this in SDK 2.0 at least to maximize compatibility.
I'll bring it up in the SIG meeting this week, looks like this issue is a duplicate of #3713

Closing as duplicate of #3713.
We've discussed it in the SIG meeting last week, and we're generally in favor of changing to "exactOptionalPropertyTypes": true. PRs for this are welcome, for further prograss-tracking, please use #3713 :slightly_smiling_face:


change input API

Is your feature request related to a problem? Please describe.
As a input to extraction_runner, specify the github repo.
Describe the solution you'd like

read in the repo-id, and run find operation to search the .c and .h ebpf files
save the commit info
Describe alternatives you've considered
None.
Additional context
None.

related to #41

@sdsen Do we still need this issue in this repo or move to annotator repo?


Choose Quality

Describe your suggested feature
Can Choose Quality of the anime that will downloaded
Other details
No response
Acknowledgements

[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open issue.
[X] I have written a short but informative title.
[X] If this is an issue with an extension, I should be opening an issue in the extensions repository.
[X] I have updated the app to version <IP_ADDRESS>.
[X] I will fill out all of the requested information in this form.

#444


Integrate image-md-check.py from SovereignCloudStack/Docs

https://github.com/SovereignCloudStack/Docs/blob/main/Design-Docs/tools/image-md-check.py

It should be possible to check the metadata of images.

Will be done by @sihensel as part of the next scs sprint


Remove the $_COOKIE reference from Image::renderImage()

Consider moving it to ImageController.

Dupe of #176


Decrease bundle size

Work in progress.

I went against your wishes and got rid of low hanging fruit: async and ethereumjs-utils from index.js and some subproviders. The commits above, when only including the provider engine (and no subproviders), bundle size is reduced by more than half a Mb.
Again, this is low hanging fruit. More work to do. All tests pass up until this point.

relevant to https://github.com/MetaMask/provider-engine/issues/20
could you not make them class methods? they are stateless util methods, they should just live in the util dir, and that way we can reuse them

Good point. Sure.

Functions have been moved. Tests pass.

L:eyes:ks g:eyes:d!


[Diag] Don't attempt 'fixOverrideDeclarationTypes' when indices count doesn't match for subscript

Fixes a crasher.
Simplified repro:
protocol A {
  subscript(x: Int, y: Int) -> Int {get}
}

class C: A {
  subscript(x: Int) -> Int { return 1 }
}

@swift-ci Please smoke test

CC: @jrose-apple (This code was introduced in 7bfdd4a20b23bdaa76cae3ab30dfd212160aa861)


Use NWXCMake version info

PR Type

[ ] Breaking change
[ ] Feature
[x] Patch

Brief Description
Use the version info from NWXCMake.
Not In Scope
PR Checklist

[x] Is documented
[x] Is tested
[x] Adheres to applicable organization standards

🚀 [bumpr] Bumped! New version:v1.0.3 Changes:v1.0.2...v1.0.3


Grid component flex related props aren't working correctly

[ x] This is not a v0.x issue.
[ x] I have searched the issues of this repository and believe that this is not a duplicate.

Expected Behavior 🤔

alignItems prop should center elements horizontally.
Current Behavior 😯

Code below isn't able to center items vertically. Everyting is wrapper in <Grid container></Grid> component.
<Grid item xs={10} sm={10} alignItems={'center'}>
     <Button
        variant={'contained'}
        color={'primary'}
        onClick={() => sampleFunction(searchValue)}
    >
        Search
    </Button>
</Grid>

Context 🔦

I'm trying to center content vertically AND I had found a solution to this problem.
I appears that MuiGrid-align-items-xs-center CSS class don't have display: flex property.
When such property is applied to the component it works correctly and it centers elements vertically as it should. Solution below is working as expected.
.MuiGrid-align-items-xs-center {     align-items: center;     display: flex; }
Your Environment 🌎

Tech
Version

Material-UI
<EMAIL_ADDRESS>

React
16.8.6

Browser
Chrome 74.0.3729.169

TypeScript
nope

@MMaicki An item can be a container at the same time: <Grid item container>. I hope it helps.


UniProt gene resolution error in data_pipeline

Hi, I noticed that the code in data_pipeline that resolves UniProt ids to Ensembl ids has a bug.  Specifically, I'm referring to Evidences.py#L232  in the 19.11 release.  That code references the field 'is_reference' several times but it doesn't appear like that field should ever exist.  It's not defined in the gene-data ES mapping file and it's not set somewhere in GeneData either.  I think that's also probably why this separate bug in the code has never actually been encountered, starting from Evidences.py#L249:
try:
    # *** Evaluates to boolean ***
    reference_target_list = luts.available_genes.get_gene(ensembl_id)['is_reference'] is True
except KeyError:
    reference_target_list = []

if reference_target_list:
    # *** If reference_target_list is boolean, reference_target_list[0] will throw "Not subscriptable" error
    target_id = 'http://identifiers.org/ensembl/%s' % reference_target_list[0]
else:
    target_id = ensembl_id

I'm not sure how critical this is, but it looks like at the very least it would make it impossible for a gene meeting the "nonref_ensembl_xref_for_uniprot_entry" or "missing_target_id_for_protein" conditions to be caught.
Should this be referring to the "is_ensembl_reference" field instead?  That field seems somewhat useless though since it's always true due to the fact that ensembl.py._clean_non_reference_genes is removing non-reference genes before they're insterted into ES.
Lastly, is it true that the idea of "reference" or "nonreference" captured by the "is_ensembl_reference" field the same as the concept appealed to in the use of the genes_with_non_reference_ensembl_ids.tsv csv?  I'd be happy to help make some contributions on this, but I wanted to first get a sense of what "nonreference" really means in the pipeline and make sure there aren't multiple meanings for it.
p.s. The function to load genes_with_non_reference_ensembl_ids.tsv is defined twice with slightly different implementations, though the second function does not appear to be used: LookupHelpers.py#L57 and EvidenceString.py#L517.

Digging a little more on this, I found something else that was a bit confusing -- unless I'm missing something it looks like the code for doing the UniProt to Ensembl id resolution while processing evidence records is:
From normalise_target_id:
# If the evidence has a UniProt target, convert to Ensembl:
if target_id.startswith(GeneData.UNI_ID_ORG_PREFIX):
    if '-' in target_id:
        target_id = target_id.split('-')[0]
    uniprotid = target_id.split(GeneData.UNI_ID_ORG_PREFIX)[1].strip()
    # Query gene-data to get the ensembl_gene_id value for the uniprotid
    ensemblid = available_genes.get_uniprot2ensembl(uniprotid)
    # Resolve the Ensembl id to a reference gene if it is nonreference (otherwise use as-is):
    new_target_id = self.get_reference_ensembl_id(ensemblid,
                                                             available_genes=available_genes,
                                                             non_reference_genes=non_reference_genes)

That makes sense, except that the call to get_reference_ensembl_id first checks to see if the Ensembl id is present in gene-data, which can't be false given that it just came from there:
From get_reference_ensembl_id:
    def get_reference_ensembl_id(ensemblid, available_genes, non_reference_genes):
        # Check to see if the ensemblid exists in gene-data even though that's where it just came from:
        if ensemblid not in available_genes:
            ensemblid = EvidenceManager._map_to_reference_ensembl_gene(ensemblid, non_reference_genes) or ensemblid
        return ensemblid

I haven't done any digging yet to see how many more evidence records could be mapped if that was working correctly, but I thought I'd point it out in case anyone else knows off-hand how important that is.

As an update on this, it is resulting in the loss of 484 gwas_catalog genetic_association evidence records (as of the 19.11 release) all of which could be integrated if the UniProt -> Non-reference -> Ensembl ID translation was working correctly.  It is not affecting any other sources though.


Clipboard: Removed phantomjs dependency

Phantomjs is an unmaintained project, removed phantomjs dependency from package.json and karma.conf.js.Added chrome headless support in package.json and karma.conf.js.
The dist/clipboard.js and dist/clipboard.min.js file got updated whenever we install and test the package.
Signed-off-by: ossdev<EMAIL_ADDRESS>

Hi,
Added custom launcher support for headless chrome.
-> Reason for using --no-sandbox flag:
For aarch64 chrome binary is not present, we are using chromium binary to launch the Chrome.
Security is one of the most important goals for Chromium, and --no-sandbox flag provides that.
When trying to use chromium as root user like in docker container then only the requirement of --no-sandbox flag is coming
Without the flag, chromium gives error as root user-
[28769:28769:0925/163702.505790:ERROR:zygote_host_impl_linux.cc(89)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.

Thus, we need to give the --no-sandbox flag when we launch chrome with chromium binary.

Hey @ossdev07! I love that idea, thanks for sending this PR.
Looks like Travis is complaining about it, could you make it work on Travis before I merge? Thanks!

@ossdev07 Will you update so it can be merged?

Hi @zenorocha
I have been working on the failing 3 test cases of the package and would really appreciate if you may guide me through some of the aspects of the test cases.
    describe('#static isSupported', () => {
        it('should return the support of the given action', () => {
            assert.equal(Clipboard.isSupported('copy'), false);
            assert.equal(Clipboard.isSupported('cut'), false);
        });

        it('should return the support of the cut and copy actions', () => {
            assert.equal(Clipboard.isSupported(), false);
        });
    });

The above test case is causing the 2 assertion issue in the package, in your view can we modify the test case or skip it if it is not essential for the package.

window.Clipboard is a native part of current browsers so this is expected to fail imo.
Should be renamed to ClipboardJS.

@ossdev07 and @zenorpcha what is status? Are we coming to an end?

This branch has conflicts that must be resolved
Only those with write access to this repository can merge pull requests.
Conflicting files
dist/clipboard.js
dist/clipboard.min.js
package.json

Hi
I have updated the src/clipboard.js and src/clipboard-action.js files to resolve the issue of failing 3 test cases with headless chrome, apparently, the phantomjs and chrome work differently that is why I have updated the file.
Also, for the clear selection function, it was not taking the focus off selected element so I have added to blur the active element.
The dist/clipboard.js and dist/clipboard.min.js get updated itself whenever we run install and test the package if you want I can remove these files from the pull request for better clarity.

That does not really answer my question.

@DanielRuf
The changes which I did regarding the check for false inside src/clipboard.js is because of the following reason:

Actually in phantomjs document.queryCommandSupported gives output as function where as !!document.queryCommandSupported gives as false.

25 03 2019 14:19:25.046:INFO [PhantomJS 2.1.1 (Linux 0.0.0)]: Connected on socket tn_0845WgY2jdzdqAAAA with id 49411648
LOG: 'document.queryCommandSupported = function queryCommandSupported() {
    [native code]
}'
LOG: '!document.queryCommandSupported = false'
LOG: '!!document.queryCommandSupported = true'
LOG: '!!document.queryCommandSupported(copy) = false'
LOG: 'document.queryCommandSupported = function queryCommandSupported() {
    [native code]
}'
LOG: '!document.queryCommandSupported = false'
LOG: '!!document.queryCommandSupported = true'
LOG: '!!document.queryCommandSupported(cut) = false'
LOG: 'document.queryCommandSupported = function queryCommandSupported() {
    [native code]
}'
LOG: '!document.queryCommandSupported = false'
LOG: '!!document.queryCommandSupported = true'
LOG: '!!document.queryCommandSupported(copy) = false'
LOG: '!!document.queryCommandSupported(cut) = false'
PhantomJS 2.1.1 (Linux 0.0.0): Executed 26 of 26 SUCCESS (0.052 secs / 0.018 secs)
TOTAL: 26 SUCCESS

But in chrome document.queryCommandSupported gives output as function but !! document.queryCommandSupported gives as true.

25 03 2019 14:44:07.672:INFO [HeadlessChrome 73.0.3683 (Ubuntu 0.0.0)]: Connected on socket KVb6qeAIzV4c6AaOAAAA with id 19968799
LOG: 'document.queryCommandSupported = function queryCommandSupported() { [native code] }'
LOG: '!document.queryCommandSupported = false'
LOG: '!!document.queryCommandSupported = true'
LOG: '!!document.queryCommandSupported(copy) = true'
LOG: 'document.queryCommandSupported = function queryCommandSupported() { [native code] }'
LOG: '!document.queryCommandSupported = false'
LOG: '!!document.queryCommandSupported = true'
LOG: '!!document.queryCommandSupported(cut) = true'
LOG: 'document.queryCommandSupported = function queryCommandSupported() { [native code] }'
LOG: '!document.queryCommandSupported = false'
LOG: '!!document.queryCommandSupported = true'
LOG: '!!document.queryCommandSupported(copy) = true'
LOG: '!!document.queryCommandSupported(cut) = true'
HeadlessChrome 73.0.3683 (Ubuntu 0.0.0): Executed 26 of 26 SUCCESS (0.039 secs / 0.022 secs)
TOTAL: 26 SUCCESS

Regarding the use of TinyEmitter, I have not additionally added this in the file as I have already mentioned this file gets updated by itself whenever we try to test the package, hence this is being introduced by itself in the file and I have not added it from my end.

I can remove the dist/clipboard.js and dist/clipboard.min.js files from the pull request as the changes in those files are not suggested from my end and are introduced itself by testing and running the package.

Please revert if something is still not clear I will happy to explain this again.

Actually in phantomjs document.queryCommandSupported gives output as function where as !!document.queryCommandSupported gives as false.

See https://developer.mozilla.org/en-US/docs/Web/API/Document/queryCommandSupported

document.queryCommandSupported gives as true.

Please do not commit the dist files. Remove them in another commit and squash both commits.

Conflicting files
dist/clipboard.js
dist/clipboard.min.js
package.json

Hi @zenorocha @Ruffio
The Travis-ci checks are now passing could you please have a look at it, and check it at your end. Please do revert if something is not clear I will be happy to help.
Waiting for the phantomjs dependency to be removed from the package.
Thanks again for your support and patience.

In the meantime you can use https://github.com/DanielRuf/phantomjs-mocked

I would like to see the PhantomJS dependency removed. Not only is it unmaintained, but PhantomJS is also not available on AArch64 - so the test suite does not pass on my AArch64 platform.

Ping @Ruffio @zenorocha
Please, could you accept this PR?

Ping @zenorocha @Ruffio could you please have a look at this PR?

@zenorocha @Ruffio @DanielRuf Can someone have a look at this PR, I will be happy to update the patch if you have any suggestions.

I'm not a collaborator or member so I can not do any review or merge this.

I'm not a collaborator or member so I can not do any review or merge this.

@zenorocha can this PR be accepted? Else write to @ossdev07 what should be changed for it to get merged ;-)

Thanks @ossdev07 and everyone else in this thread for your patience.


Set permissions in code for page and table extensions

For a Table or Page it's possible to set the Permissions property via code if applicable.
Example: Permissions = tabledata TableName = RD;
The Permissions property should also be available for page- and table extensions. For example specific validation code in this case shouldn't be placed in codeunits to make it possible to give permissions via code.

AL Extension Version: 3.0.118998
Version: Platform 14.0.32600.33003 + Application 33107 (W1 14.1)

Great idea! Go ahead and post this to our Ideas forum at https://aka.ms/BusinessCentralideas
We're constantly monitoring top Ideas and will consider them for a future release.


Edit rule page styling

Edit rule page styling doesn't go to well, fix it

Fixed and merged in commit: https://github.com/sanoma/django-arctic/commit/6e983cc36c4d6f3c7886f5114b4803ce8a912784


CTGUI Recipe Issues

1.12.1
Furnace recipes input using CTGUI 2 aren't working.  I input recipe, reload game, recipe doesn't work.  Syntax looks fine, especially since it was put in by CTGUI 2.
Also, how are we supposed to designate an oreDictionary item in the CTGUI 1 interface, the checkboxxes at the left of the screen seem to just be cosmetic, and don't actually let you check them.

Use the issue template.

You couldn't even answer my question about the ore Dictionary? Your template doesn't even have the option for questions to be asked.

Well don't try and combine 2 issues / questions into a single issue...

LOL... whatever dude, ill look elsewhere for assistance.


[Question] Potential issue with multiple runner labels?

Hi, thanks for this awesome project.
Because I was working on a runner manager on my own, I'm wondering how you solved following issue.
How do you prevent using requesting an random runner if someone accidentally or by purpose just use the label runs-on
  runs-on:
    - runs-on

My outcome was that you can not prevent this, therefore my recommendation or request would be to move to runners with a single label e.g.,

runs-on: 2cpu-linux-x64@runs-on::${{ github.run_id }}
runs-on: 2cpu-linux-x64@runs-on::run_id=${{ github.run_id }}
runs-on: 2cpu-linux-x64@runs-on?run_id=${{ github.run_id }}

./config.sh --url https://github.com/qoomon/sandbox --token XXXXXXXXXXXX --no-default-labels --labels '2cpu-linux-x64@runs-on::123456789'

Hi @qoomon, if someone assigns a subset of another job's labels to a job, then yes there is a risk that this job will steal the runner of that other job. But this is why the run-id label is recommended for inclusion, so that the set of labels are unique across workflow runs.
Using a single label is already supported:
# this works in most (all?) repos
runs-on: "runs-on,runner=2cpu-linux-x64,etc"

But was deprecated because GitHub seemed have introduced a change to reject jobs with commas. That change seems to have been rejected by now, but to be safe I recommend the array syntax now.
If I had to add single label support to RunsOn, that would probably be with the safer -- separator:
runs-on: runs-on--runner=2cpu-linux-x64--run-id=${{ github.run_id }}--etc

How did you handle that for your own runner manager?

My labels looks like this
runs-on: ubuntu-latest@2cpu-linux-x64@self-hosted

So what does your runner manager have that RunsOn doesn't (yet :))? Always interested in exchanging ideas.

I'm working on idle instance pooling for faster job starts.

@qoomon pooling is hard to get right at scale, and the 30s delay is not that much an issue now. But let me know about your progress!


RaadKomik: Update domain

Closes #2634
Checklist:

[ ] Updated extVersionCode value in build.gradle for individual extensions
[x] Updated overrideVersionCode or baseVersionCode as needed for all multisrc extensions
[x] Referenced all related issues in the PR body (e.g. "Closes #xyz")
[ ] Added the isNsfw = true flag in build.gradle when appropriate
[x] Have not changed source names
[ ] Have explicitly kept the id if a source's name or language were changed
[x] Have tested the modifications by compiling and running the extension through Android Studio
[ ] Have removed web_hi_res_512.png when adding a new extension

Does.
The default mangaUrlDirectory is /manga, and substring(1) is used when addPathSegement is used.
Example:
source-extensions
Sample: https://github.com/keiyoushi/extensions-source/blob/1233aa8816188a887eb30dcd517371b39581e4bf/lib-multisrc/mangathemesia/src/eu/kanade/tachiyomi/multisrc/mangathemesia/MangaThemesia.kt#L93


Added YOLOv5 serverless function for auto labeling

Motivation and context
Integration of YOLOv5 as a serverless nuclio function that can be used for auto-labeling. Ultralytics has released YOLOv5 a while ago and it's one of the most used computer vision libraries and therefore it would make sense to support it in CVAT. The integration is quite simple into CVAT as Ultralytics maintains a Pytorch Hub model (https://github.com/ultralytics/yolov5/issues/36) and a docker image (https://hub.docker.com/r/ultralytics/yolov5).
Related issues from ultralytics repo: https://github.com/ultralytics/yolov5/issues/5427

How has this been tested?
Automatic annotation was run using YOLOv5 on a custom dataset.
The serverless function was deployed using
nuctl deploy --project-name cvat \
  --path serverless/pytorch/ultralytics/yolov5/nuclio \
  --volume `pwd`/serverless/common:/opt/nuclio/common \
  --platform local

Then using the 'Automatic annotation' action the function was tested and the auto-generated labels were controlled to check that no coordinates misfit is happening.

Checklist

[ x] I submit my changes into the develop branch
[ x] I have added a description of my changes into CHANGELOG file
[ x] I have updated the documentation accordingly
[ x] I have added tests to cover my changes
[ x] I have linked related issues (read github docs)
[ x] I have increased versions of npm packages if it is necessary (cvat-canvas,
cvat-core, cvat-data and cvat-ui)

License

[ x] I submit my code changes under the same MIT License that covers the project.
Feel free to contact the maintainers if that's a concern.
[x ] I have updated the license header for each file (see an example below)

# Copyright (C) 2022 Intel Corporation
#
# SPDX-License-Identifier: MIT

@wartek69 , thanks for the contribution! Could you please add information about the model into README as well? https://github.com/openvinotoolkit/cvat#deep-learning-serverless-functions-for-automatic-labeling

@nmanovic I've added the model to the readme!

I would like to add that if you want to deploy a custom model you should:

Copy the weights.pt file to /serverless/common
Edit main.py file with :

#Line 11
model_path = "/opt/nuclio/common/weights.pt"
model = torch.hub.load('ultralytics/yolov5', 'custom', path = model_path)  # or yolov5m, yolov5l, yolov5x, custom

It's important that the classes names defined in function.yaml file match with the classes defined when you train the model.

@simaiden Wow, thank you it works! From docs, it seemed I had to download weights into docker image or something, this was much nicer. @nmanovic It would be useful to put this into docs, usually people would want to run YOLO with custom weights without doing anything with docker/nuclio.

@martinerk0 , could you please send us a PR with necessary updates for documentation?

Motivation and context
Integration of YOLOv5 as a serverless nuclio function that can be used for auto-labeling. Ultralytics has released YOLOv5 a while ago and it's one of the most used computer vision libraries and therefore it would make sense to support it in CVAT. The integration is quite simple into CVAT as Ultralytics maintains a Pytorch Hub model (ultralytics/yolov5#36) and a docker image (https://hub.docker.com/r/ultralytics/yolov5). Related issues from ultralytics repo: ultralytics/yolov5#5427
How has this been tested?
Automatic annotation was run using YOLOv5 on a custom dataset. The serverless function was deployed using
nuctl deploy --project-name cvat \
  --path serverless/pytorch/ultralytics/yolov5/nuclio \
  --volume `pwd`/serverless/common:/opt/nuclio/common \
  --platform local

Then using the 'Automatic annotation' action the function was tested and the auto-generated labels were controlled to check that no coordinates misfit is happening.
Checklist

[ x] I submit my changes into the develop branch
[ x] I have added a description of my changes into CHANGELOG file
[ x] I have updated the documentation accordingly
[ x] I have added tests to cover my changes
[ x] I have linked related issues (read github docs)
[ x] I have increased versions of npm packages if it is necessary (cvat-canvas,
cvat-core, cvat-data and cvat-ui)

License

[ x] I submit my code changes under the same MIT License that covers the project.
Feel free to contact the maintainers if that's a concern.
[x ] I have updated the license header for each file (see an example below)

# Copyright (C) 2022 Intel Corporation
#
# SPDX-License-Identifier: MIT

hi
how did you rund nuctl deploy --project-name cvat \   --path serverless/pytorch/ultralytics/yolov5/nuclio \   --volume pwd/serverless/common:/opt/nuclio/common \   --platform local this comman.
i cannot run it on windows 10. It throws an errors like this:
`C:\Users\livan\cvat>nuctl deploy --project-name cvat \
Error - exec: "/bin/sh": file does not exist
/nuclio/pkg/cmdrunner/shellrunner.go:96
Call stack:
stdout:
stderr:
/nuclio/pkg/cmdrunner/shellrunner.go:96

Failed to get docker version
/nuclio/pkg/dockerclient/shell.go:729
No docker client found
/nuclio/pkg/dockerclient/shell.go:83
Failed to create docker client
.../pkg/containerimagebuilderpusher/docker.go:31
Failed to create containerimagebuilderpusher
/nuclio/pkg/platform/local/platform.go:88`

Я хотел бы добавить, что если вы хотите развернуть пользовательскую модель, вам следует:

Скопируйте weights.pt файл в / бессерверный / общий
Редактировать main.py файл с :

torch
= model "/opt/nuclio/common/weights.pt"
= model_path #Строка 11.hub.load('ultralytics/yolov5', 'пользовательский', путь  = model_path) # или yolov5m, yolov5l, yolov5x, пользовательский

Важно, чтобы имена классов были определены в функции.файл yaml соответствует классам, определенным при обучении модели.

Hi, how i can copy my weights to /serversless/common?
docker cp?

Я хотел бы добавить, что если вы хотите развернуть пользовательскую модель, вам следует:

Скопируйте weights.pt файл в / бессерверный / общий
Редактировать main.py файл с :

torch
= model "/opt/nuclio/common/weights.pt"
= model_path #Строка 11.hub.load('ultralytics/yolov5', 'пользовательский', путь  = model_path) # или yolov5m, yolov5l, yolov5x, пользовательский

Важно, чтобы имена классов были определены в функции.файл yaml соответствует классам, определенным при обучении модели.

Hi, how i can copy my weights to /serversless/common? docker cp?

Just copy in your os filesystem, cvat mount the entire folder into the docker container

How to clean up disk space by automatically annotating whether it will occupy a large amount of disk space


Fix spurious proto: tag has too few fields warning

Fixes #22261.

In response to a cherrypick label: new pull request created: #22335


Support for all Instagram versions

Hi, I know a better way to enable the developer/experiment options that supports all the versions ( past, present, future*) and the cherry on top is, this method doesn't require to modify the apk file

Hi, open a pull request then!
We would really appreciate your contribution.

Unfortunately I'm not good with xposed module, but here is the method I was talking about.
Basically Instagram stores all the user's developer/experiments overrides in data/data/com.instagram.android/files/mobileconfig/mc_overrides.json in json format.
The data will be of the form {":":[": : "],....."qe_overrides":[]}
28538 is the feature code for "Employee Mode" . All we need to do is check if the key "28538:":["0: : true"],  is present in the file, if it is not present, add it in the file and if the file itself isnt present, create the file and add the key in the above mentioned format

Hey, we checked this method.
unfortunately, with each new update the class which holds this json file changes.
so it would be the same problem.

Class ?? What I'm trying to say is We don't need to modify anything in the app, no need to inject anything.
All we need to do is just check if the json is present in the folder .../files/mobileconfig. If it doesn't exist create a new json file, and If it exists check if the above mentioned key pair exists

I get you but, how do we do that?
Because xposed modules works only by hooking into a method and changing it.
The thing you want can't be achieved with a xposed module...

I positive we can. I made a test version (unfortunately I dont have the code with me).
I hooked MainActivity onload function and then checked if the file exists and so on.

Ok thx for tip, we will try that.


Add withdraw/redeem calls

Requires merging https://github.com/glifio/go-pools/pull/30

Included in #112


planetary network - Handle not finding peers correctly.

When you don't find peers, just give some context.
"No connections were found, please try again later"
I think right now it crashes the app instead.

Disabled for now


feature/performance

@gmlwhdtjd we can speed up here together

add performance check script

./run-performance.sh
...
34.1.ms blah blah
...

Now, this pr seems to be working on the build speed and runtime speed of our package. Of course, that is also a very important part. But what I'm really interested in is the part about whether the build speed of the user project is not lowered due to type estimation when using our package.

Now, this pr seems to be working on the build speed and runtime speed of our package. Of course, that is also a very important part. But what I'm really interested in is the part about whether the build speed of the user project is not lowered due to type estimation when using our package.

I understand. anyway, some parts of codes being very dramatically speed up.


Extension slow.. then completely stopped working..

Describe the bug
问题描述
A clear and concise description of what the bug is.
{
"detail": "Hmm...something seems to have gone wrong."
}
To Reproduce
如何复现
Steps to reproduce the behavior:

Go to '...duckduckgo'
Click on '....'click on sidebar ChatBoxGPt
Scroll down to '....'
See error

{
"detail": "Hmm...something seems to have gone wrong."
}
Expected behavior
期望行为
A clear and concise description of what you expected to happen.
Screenshots
截图说明
If applicable, add screenshots to help explain your problem.
Please complete the following information):
请补全以下内容

OS: [e.g. Windows]   Linux 22

Browser: [e.g. chrome, safari]  Version 123.0.6312.86 (Official Build) (64-bit)

Extension Version: [e.g. v2.0.2]

Additional context
其他
Add any other context about the problem here.

intermittent


Incorporate wasmedge-quickjs node crypto

The PR is almost done, perhaps we just merge it into our own branch and see if it works instead of using crypto browserify, as it is noticeably slow with ethersjs random wallet creation

[ ] Document this new requirement: sudo apt install lld

This is going to be quite a large project. The best way to go about this is to have ic-wasi-polyfill impilement the WASI Crypto functions...then we just have to update that dependency (and maybe wasi2ic), turn on the nodejs_crypto features in wasmedge_quickjs in Cargo.toml, and hopefully we'd be good to go.


pod install fails when using latest Swinject, SwinjectStoryboard and SwinjectPropertyLoader beta versions

Since SwinjectPropertyLoader 1.0.0-beta.1 has a dependency on Swinject 2.0.0-beta.1, cocoa pods is confused when also having Swinject 2.0.0-beta.2 as a dependency in your project.
Here's an excerpt from the podfile:
pod 'Swinject', '2.0.0-beta.2'
pod 'SwinjectStoryboard', '1.0.0-beta.2'
pod 'SwinjectPropertyLoader', '1.0.0-beta.1'

and  this is the output when trying pod install
Analyzing dependencies
[!] Due to the previous naïve CocoaPods resolver, you were using a pre-release version of
`Swinject`, without explicitly asking for a pre-release version, which now leads to a conflict.
Please decide to either use that pre-release version by adding the version requirement to
your Podfile (e.g. `pod 'Swinject', '= 2.0.0-beta.2'`) or revert to a stable version by running
`pod update Swinject`.

neither of the suggestions work BTW. pod update Swinject gives the following (expected) outcome:
Analyzing dependencies
[!] Unable to satisfy the following requirements:

- `Swinject (= 2.0.0-beta.2)` required by `Podfile`
- `Swinject (= 2.0.0-beta.1)` required by `SwinjectPropertyLoader (1.0.0-beta.1)`
- `Swinject (= 2.0.0-beta.2)` required by `SwinjectStoryboard (1.0.0-beta.2)`

SwinjectPropertyLoader 1.0.0-beta.1 can be used with Swinject 2.0.0-beta.1, which supports Xcode 8 beta 4. Migration to Xcode 8 (GM) is in progress in pull request #3.

ok, thanks


Adding support for choosing specific auth backends

Currently there is no way to choose a specific auth backend other than defining a custom TokenObtainPairSerializer with an overridden validate method and making a custom TokenObtainPairView from that.
It is not too much work but it would be great if there was a setting that let us choose one or multiple specific authentication backends to be used instead of all the ones defined in the auth backend settings.
My specific use case for such a feature is the following:
On top of checking for username and password I want to check that the device that attempts to login is registered as a verified device in my backend. However I still want to have access to my simple email/password login through login.
If you think this is a valid issue I wouldn't mind working on it.

Does #279 solve this? Used to check only is_active , but now we can use a custom callable to check.

It sure does! Thank you.


[AndroidTV] Elementum does not start downloading

I have a Mi Box S with AndroidTV Oreo. I installed a SDCard as Internal Storage and configured Kodi with Elementum and Alfa add-ons.
When I search for torrents in Alfa addon and try to use Elementum to stream it it starts loading but never shows the downloading message even after several minutes waiting, no error messages either .
I tried same torrent file in a different machine with Windows and it downloads without issue, so I suspect is something related to elementum.
I have kodi log and elementum settings, I will attach those next comment.

kodi.log

elementum-settings.xml.txt

2019-12-27 21:23:33.583 T:29469  NOTICE: [plugin.video.elementum] INFO  bittorrent   ▶ AddTorrent       Adding new torrent item with url: magnet:?xt=urn:btih:595bf281fa42d23e7e9a22ef6c1c9f7033f173ba&dn=%5Bzooqle.com%5D+The+Mandalorian+S01E01+720p+WEBRip+x264+AAC+5+1+MSubs+-+LOKiHD+-+Telly&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80&tr=http%3A%2F%2Fbt1.archive.org%3A6969%2Fannounce&tr=http%3A%2F%2Fbt2.archive.org%3A6969%2Fannounce&tr=http%3A%2F%2Fopen.acgtracker.com%3A1096%2Fannounce&tr=http%3A%2F%2Ftracker.tntvillage.scambioetico.org%3A2710%2Fannounce&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969&tr=http%3A%2F%2Ftracker1.itzmx.com%3A8080%2Fannounce&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80&tr=http%3A%2F%2Fexplodie.org%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969

In your kodi log libtorrent cannot resolve the magnet.
You can search Elementum issues for alfatorrents. There were issues like this one.


Changelog

Changelog for 0.0.2 release.
I think it covers most of the changes, but look at changelog and change as necessary.

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@ef50fa4). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #139   +/-   ##
=========================================
  Coverage          ?   85.21%
=========================================
  Files             ?       23
  Lines             ?     1393
  Branches          ?        0
=========================================
  Hits              ?     1187
  Misses            ?      206
  Partials          ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ef50fa4...1e59b23. Read the comment docs.


Install mlflow

Contribution for netcups AI tutorial  contest. I show how to set up MLFlow for experiment tracking on a VPS, including HTTP Basic auth and a simple usage example.

Thank you very much for your contribution. We have just finished reviewing your entry and I have forwarded your contact information to our team, so you should receive your coupon in the coming days.


Disallow static array with N > 10000

Code generation of large static arrays is very inefficient. There's no reason to use them instead of a regular array.
Resolves #4452

Well, it looks like #12507 is a reason to use such large StaticArrays on the stack

The specific value was taken from @asterite's suggestion in https://github.com/crystal-lang/crystal/issues/4452#issuecomment-304655355 which is already 5 years old. And we're several LLVM versions ahead now.
So I suppose we could just increase it to a useful and manageable value.

I measured the codegen time for the following program with different static array sizes:
n = uninitialized StaticArray(UInt8, N)

N
Time for Codegen (bc+obj)

1_000
00:00:00.669502576

2_500
00:00:00.365869809

5_000
00:00:01.185747601

10_000
00:00:05.297291184

20_000
00:00:25.333701219

32_768
00:01:22.311464825

40_000
00:02:16.271377868

Oddly though, static arrays with size DEFAULT_BUFFER_SIZE = 32_768 in IO don't seem to be affected by that.
Codegen time for the program IO.copy(IO::Memory.new, IO::Memory.new) is 00.282783051. 😕

The slow path occurs when those StaticArrays are passed by value. For some reason this includes the implicit return of the last expression of the top-level scope, because __crystal_main doesn't have Nil as its return type. This doesn't crash:
x = uninitialized UInt8[1048576]
1

Oh, I never figured the main code had a return value. Seems a bit odd... 🤔
But I suppose this change doesn't make sense then.
If at all we'd need to restrict the size of types that can be passed by value.


[Usage]: release notes，best practice, abort-actively

Your current environment
Do you have release notes when release new version such as v0.5.3, v0.5.4? indicating the optimization/bug fixing/new feature supporting in the higher version.
and, could you give some examples/best practices to guide users to deploy models with higher performance?

and one more request, has vllm realized early-stopping? that is, abort the request when the computation/memory resource is not available.
I noticed there is timeout-abort func in vllm v0.5.4, but does vllm can abort request actively without needing user to quit the request?
the feature is realized in <Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving>.

one more request, I noticed there is timeout-abort func in vllm v0.5.4, but does vllm can abort request actively without needing user to quit the request?