Taking over this repo

Hey, I've been using wrecker for a while and I really like it as I think it has a lot of potential.
I've been slowly adding a few features to my fork and have also built a web interface for it as a separate package. I'd like to make all that public, but it looks to me that this repo is not being watched by any maintainers.
Would it be possible to get ownership of it or maybe be made a contributor?

@jfischoff sorry for mentioning you directly, but I know you were a past contributor to this repo and I saw you were active and thought you might have an idea about this

I don't this is being actively maintained by skedge anymore. I'm happy to
do what I can to help you maintain it.
On Thu, Jul 13, 2017 at 3:02 PM José Lorenzo Rodríguez <
<EMAIL_ADDRESS>wrote:

@jfischoff https://github.com/jfischoff sorry for mentioning you
directly, but I know you were a past contributor to this repo and I saw you
were active and thought you might have an idea about this
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/skedgeme/wrecker/issues/9#issuecomment-315214674, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAKSO0hfT48um3AzJwjEvoKvhrFS_F1Kks5sNpQJgaJpZM4OXmHR
.

I cleaned up the work I was doing in secret and pushed all the improvements I made to my fork:

Re-implemented the Statistics using a tdigest, this allows to calculate percentiles in the stats with fixed memory consumption
Added a record-query options to the CLI: randomising the query string is a common thing to do, and sometimes you are not interest in every single URL, just the path is important
Added an option to lost the tests that are present in the executable
Made it compile with stack's lts-9.0
Used hindent, as I'm lazy about having my own formatting standard.

@jfischoff Do you think you could give me maintainer access to hackage so I can push new package versions?
Otherwise, the alternative would be to publish my fork under another name, which would be a bit sad.

What is your username on Hackage?

BTW TDigest looks dope.
It would be nice to be able to measure the stages of the request, like connection time and download time. The parsing overhead of http-client is large, having these details would make wrecker more accurate. This would require patches to http-client however.

@jfischoff my hackage name is lorenzo
It sounds like a good idea to account for the library overhead and also track connection + download time, but I'm not sure how much effort that is patch http-client to do that.

The other options is to move to uhttpc which I think would be more likely to accept those sort of patches. I'll add you shortly.

@lorenzo I've added you as a maintainer. Thanks!

Thanks @jfischoff
I'll look into uhttpc too


[BUG] x-body-name violates the OAS3 spec by being required to be within the schema field of media tye

Description
The docs and tests promote a schema that is not valid OAS3.
This problem means tools can't work together and forces hacks in other libraries and user code.
For instance this can't work wit the swagger tools or openapi generator.
requestBody:
  content:
    application/json:
      schema:
        x-body-name: pet
        $ref: '#/components/schemas/Pet'

A media type's "schema" field may be either a single "$ref" OR a full blown schema so you can't has an extension AND a ref.
The current impl alternatively permits the extension to be the referred to schema nit this pollutes the datatype with info that relates only to the usage within a requrst.
So can we support the approach below in addition to whatever you currently have ...
requestBody:
  content:
    application/json:
       x-body-name: pet
      schema:
        $ref: '#/components/schemas/Pet'

Expected behaviour
The tool should follow the spec.
Actual behaviour
x-body-name requires schemata to violate te spec OR pollute datatypes.
Found in
Current master and cli 2.10.0

See https://github.com/zalando/connexion/pull/1453

I've added a feature and updated the docs.


[chip_sw, sysrst_ctrl] Test ulp wakeup of sysrst_ctrl

This is a simple test case for ulp wakeup. The test passes on dvsim, and I could use some suggestions to improve it.

The sysrst_ctrl IOs are incorrectly connected to random pins in the chip. Plus, the existing sysrst ctrl tests employed forces for controlling the pins. We have now refactored the logic of how we connect external interfaces to the chip IOs, which eliminates the need to use forces. Please hold off if possible, on merging this PR, until the rest of the sysrst ctrl tests are updated to use the new flow (which will give you the means to properly develop this test) and are connected to the right chip pins.

Thanks @ballifatih for adding this!
I have added @weicaiyang and @sriyerg to help take a look, since they may have some additional feedback regarding the DV portion.
We should probably extend this testpoint and also put the chip into sleep mode before forcing the lid-open pin. After all, this is testing a wakeup feature and we would like to make sure that the communication with the power manager is working correctly. @sriyerg, @weicaiyang WDYT?
This should not be too hard to implement as it would mainly need some additional power manager initialization steps, similar to what is done in the the chip_sw_adc_ctrl_sleep_debug_cable_wakeup and chip_sw_sysrst_ctrl_reset tests.

Agreed - it will be good to update the testplan and the test to also verify wake up from deep sleep state.

@weicaiyang I initially tried with sysrst_ctrl_if but I couldn't get it to work. I can make the transition to chip_if when @sriyerg merges the pin configuration update. Thanks for the suggestion and clarification.
Meanwhile, I think we can continue updating this PR. I have made a small change that also puts the chip into deep sleep. It would be great if someone can have a look, because I didn't fully grasp the interaction between pwrmgr and rstmgr. @msfschaffner
This is what I'm doing with the test after the update:

Set lid_open = 0
Check there is no wake up event
Enter deep sleep mode
Start glitching lid_open for a short duration (less than debounce timer) and then stablize on '1'
Wait sw to wake up
Check if the z3_wakeup pin is raised

@weicaiyang WDYT, should we iterate over the wakeup procedure multiple times in this test?

We didn't mention that in the testplan. So either way is fine to me.

LGTM. defer to @msfschaffner for approval.

LGTM. defer to @msfschaffner for approval.

OK sounds good to me (I believe we cover multiple iterations at the block-level already so this should be fine).

The test is currently failing on dvsim, because I am not able to drive tb.dut.XYZ pins after rebasing on master (if I interpret my waveforms correctly). Since we will already switch to sysrst_ctrl_if to drive IO pads anyway, I will wait for other IO re-configurations to be completed @sriyerg . At the moment, I see some sysrst_ctrl tests failing on the HW dashboard.

The test is currently passing by using sysrst_ctrl_if, but we can wait #15811 to merge for another round of reviews.


merge from konsum-frontend-svelte

package.json

chore(deps): add ^8.3.12 remove ^8.3.11 (devDependencies.npm-pkgbuild)

:tada: This PR is included in version 1.31.14 :tada:
The release is available on:

v1.31.14
npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Enable yamllint and make the repo yamllintable

Make this repo passing the yamllint test and add/update the tox target
to harmonize with what other components within ansible namespace are
doing.
Namely:

ansible/ansible-runner
ansible/awx
ansible-network/ansible_collections.vyos.vyos
ansible/workshops

Signed-off-by: Yanis Guenane<EMAIL_ADDRESS>

This is a great start to creating some consistency in the CI and QE processes across everything, I'm all for it. 👍

@cutwater up


Disable test decl-with-definition-irgen.swift

The test is causing errors in https://ci.swift.org/job/oss-swift_tools-R_stdlib-RD_test-simulator/
Original PR : https://github.com/apple/swift/pull/33451
Disable the test until the rdar is fixed.

preset=buildbot,tools=R,stdlib=RD
@swift-ci Please test with preset macOS

@swift-ci smoke test

@swift-ci smoke test

preset=buildbot,tools=R,stdlib=RD
@swift-ci Please test with preset macOS

CC @scentini


Add background running information to FAQ

Where to find the issue
No information in FAQ about running app in background is needed
Describe the issue
In the FAQ is no information about if the app must be started and run in background to work or the exchange of the keys is managed by operation system and the app do the analysis of "infected" keys only. (maybe the wrong wording "infected" sorry)
Suggested change
Add information to FAQ or better display the information in the app directly

Hi @Clauswgh
thanks. Could you be a bit more specific about the question you would like to include? We assume that people install and open the app at least once. As soon as they have completed the basic setup when starting the app for the first time, the exchange of random IDs happens in the background. This is also stated within the app.
In the FAQs, we collect questions that many users have as well as troubleshooting tips if something isn't working like it should. Examples:

Will exposure logging still work when I close the app?
Apple: My risk status hasn't been updated for over a day. An internet connection was available, what's wrong?

So which would be the question you would like to include?

@OlMue OK, the question is answered in "Will exposure logging still work when I close the app?" - but the information is missing in the FAQ inside the APP:
Path to FAQ inside the App: Startscreen scroll down to "Häufige Fragen" tab on it to visit "www.bundesregierung.de/breg-de/themen/corona-warn-app/corona-warn-app-faq-1758392"
There I can't find anything about closing the app :-(

Thanks for the clarification. This then seems to be more along the lines of corona-warn-app/cwa-documentation#289. Hence, we'll close it here in the Website repository (where the FAQ already is available) and work towards consolidating FAQs better in the overall Apps + Websites ecosystem.


hasManyThrough soft delete does not work due to ambiguous field list

Hi.
I implemented a hasManyThrough soft delete and got this error message
SQLSTATE[23000]: Integrity constraint violation: 1052 Column
'deleted_at' in field list is ambiguous (SQL: update
`phase_user` inner join `phases` on `phases`.`id` =
`phase_user`.`phase_id` set `deleted_at` = 2014-07-05
03:03:27, `updated_at` = 2014-07-05 03:03:27 where
`phase_user`.`deleted_at` is null and `phases`.`contract_id`
= 385)

I can confirm that if I qualify (is that the correct term?) the deleted_at and updated_at fields that the soft delete will work as expected.
I reckon this is not-too-obscure a use case so it would be great if this could be looked at.
Many thanks.

I hit this too, my query gets:
set `mutable`.`deleted_at` = 2014-07-05 03:03:27 `updated_at` = 2014-07-05 03:03:27

I think the problem is happening somewhere around:
$this->grammar->compileUpdate($this, $values);

With stack trace starting from:
/vendor/laravel/framework/src/Illuminate/Database/Connection.php 725

The gist of it is that if I soft delete a belongsToMany() relation, so like:
$model->relatedModels()->delete()

It's not prepending the table name to the updated_at portion.
This is pretty devastating to the REST API I'm working on because I allow the user to order by created_at, updated_at, etc but now have to swizzle those params to pivot_created_at, pivot_updated at internally.  Then I set:
const CREATED_AT = 'pivot_created_at';
const UPDATED_AT = 'pivot_updated_at';
const DELETED_AT = 'pivot_deleted_at';
I'm in the process of trying this and am running into a lot of problems so my feeling is that belongsToMany() is not really easy to use (as of Laravel 5.2.41), even though it's conceptually correct.  Kinda frustrating :-(


[Bug]: Cleanup caikit ootb runtime

Is there an existing issue for this?

[X] I have searched the existing issues

Deploy type
Installing Dashboard directly (eg. v2.12.0, commit xyz, branch name)
Version
main
Current Behavior
A little bit of a cleanup still needed for the caikit-ootb.yaml.
Expected Behavior
Add an additional env var in caikit-ootb.yaml
Expected output:
        env:
          - name: RUNTIME_LOCAL_MODELS_DIR
            value: /mnt/models
          - name: RUNTIME_GRPC_SERVER_THREAD_POOL_SIZE
            value: 64

Remove ports name field in caikit-ootb.yaml as this will cause some issues with KServe.
Steps To Reproduce
Go to Settings => Serving Runtimes & view the YAML for the ootb caikit runtime
Workaround (if any)
No response
What browsers are you seeing the problem on?
No response
Anything else
No response

cc @lucferbux @Xaenalt


More minor documentation updates

In the case of the NetworkX random graphs link, I think it would be the most clear to link directly to the documentation for barabasi_albert_graph (https://networkx.org/documentation/stable/reference/generated/networkx.generators.random_graphs.barabasi_albert_graph.html#networkx.generators.random_graphs.barabasi_albert_graph) instead of the random graphs section, but I'm not sure if that works with the github.io URL format or if there is a reason for using that instead of networkx.org.

This is ready to merge?

Yes, ready to merge, thanks


Leap years

ERA data has leap years, and right now we select data based on the time.dt.dayofyear
This means that the 1st of june has two different dayofyear depending on leap year or not

The current fix is adapting the data by doing
new_da = old_da.convert_calendar('noleap')


[SPARK-31159][SQL] Rebase date/timestamp from/to Julian calendar in parquet

What changes were proposed in this pull request?
The PR addresses the issue of compatibility with Spark 2.4 and earlier version in reading/writing dates and timestamp via Parquet datasource. Previous releases are based on a hybrid calendar - Julian + Gregorian. Since Spark 3.0, Proleptic Gregorian calendar is used by default, see SPARK-26651. In particular, the issue pops up for dates/timestamps before 1582-10-15 when the hybrid calendar switches from/to Gregorian to/from Julian calendar. The same local date in different calendar is converted to different number of days since the epoch 1970-01-01. For example, the 1001-01-01 date is converted to:

-719164 in Julian calendar. Spark 2.4 saves the number as a value of DATE type into parquet.
-719162 in Proleptic Gregorian calendar. Spark 3.0 saves the number as a date value.

According to the parquet spec, parquet timestamps of the TIMESTAMP_MILLIS, TIMESTAMP_MICROS output type and parquet dates should be based on Proleptic Gregorian calendar but the INT96 timestamps should be stored as Julian days. Since the version 3.0, Spark conforms the spec but for the backward compatibility with previous version, the PR proposes rebasing from/to Proleptic Gregorian calendar to the hybrid one under the SQL config:
spark.sql.legacy.parquet.rebaseDateTime.enabled

which is set to false by default which means the rebasing is not performed by default.
The details of the implementation:

Added 2 methods to DateTimeUtils for rebasing microseconds. rebaseGregorianToJulianMicros() builds a local timestamp in Proleptic Gregorian calendar, extracts date-time fields year, month, ..., second fraction from the local timestamp and uses them to build another local timestamp based on the hybrid calendar (using java.util.Calendar API). After that it calculates the number of microseconds since the epoch using the resulted local timestamp. The function performs the conversion via the system JVM time zone for compatibility with Spark 2.4 and earlier versions. The rebaseJulianToGregorianMicros() function does reverse conversion.
Added 2 methods to DateTimeUtils for rebasing days. rebaseGregorianToJulianDays() builds a local date from the passed number of days since the epoch in Proleptic Gregorian calendar, interprets the resulted date as a local date in the hybrid calendar and gets the number of days since the epoch from the resulted local date. The conversion is performed via the UTC time zone because the conversion is independent from time zones, and UTC is selected to void round issues of casting days to milliseconds and back. The rebaseJulianToGregorianDays() functions does revers conversion.
Use rebaseGregorianToJulianMicros() and rebaseGregorianToJulianDays() while saving timestamps/dates to parquet files if the SQL config is on.
Use rebaseJulianToGregorianMicros() and rebaseJulianToGregorianDays() while loading timestamps/dates from parquet files if the SQL config is on.
The SQL config spark.sql.legacy.parquet.rebaseDateTime.enabled controls conversions from/to dates, timestamps of TIMESTAMP_MILLIS, TIMESTAMP_MICROS, see the SQL config spark.sql.parquet.outputTimestampType.
The rebasing is always performed for INT96 timestamps, independently from spark.sql.legacy.parquet.rebaseDateTime.enabled.
Supported the vectorized parquet reader, see the SQL config spark.sql.parquet.enableVectorizedReader.

Why are the changes needed?

For the backward compatibility with Spark 2.4 and earlier versions. The changes allow users to read dates/timestamps saved by previous version, and get the same result. Also after the changes, users can enable the rebasing in write, and save dates/timestamps that can be loaded correctly by Spark 2.4 and earlier versions.
It fixes the bug of incorrect saving/loading timestamps of the INT96 type

Does this PR introduce any user-facing change?
Yes, the timestamp 1001-01-01 01:02:03.123456 saved by Spark 2.4.5 as TIMESTAMP_MICROS is interpreted by Spark 3.0.0-preview2 differently:
scala> spark.read.parquet("/Users/maxim/tmp/before_1582/2_4_5_ts_micros").show(false)
+--------------------------+
|ts                        |
+--------------------------+
|1001-01-07 11:32:20.123456|
+--------------------------+

After the changes:
scala> spark.conf.set("spark.sql.legacy.parquet.rebaseDateTime.enabled", true)

scala> spark.read.parquet("/Users/maxim/tmp/before_1582/2_4_5_ts_micros").show(false)
+--------------------------+
|ts                        |
+--------------------------+
|1001-01-01 01:02:03.123456|
+--------------------------+

How was this patch tested?

Added tests to ParquetIOSuite to check rebasing in read for regular reader and vectorized parquet reader. The test reads back parquet files saved by Spark 2.4.5 via:

$ export TZ="America/Los_Angeles"

scala> spark.conf.set("spark.sql.session.timeZone", "America/Los_Angeles")
scala> val df = Seq("1001-01-01").toDF("dateS").select($"dateS".cast("date").as("date"))
df: org.apache.spark.sql.DataFrame = [date: date]
scala> df.write.parquet("/Users/maxim/tmp/before_1582/2_4_5_date")

scala> val df = Seq("1001-01-01 01:02:03.123456").toDF("tsS").select($"tsS".cast("timestamp").as("ts"))
df: org.apache.spark.sql.DataFrame = [ts: timestamp]

scala> spark.conf.set("spark.sql.parquet.outputTimestampType", "TIMESTAMP_MICROS")
scala> df.write.parquet("/Users/maxim/tmp/before_1582/2_4_5_ts_micros")

scala> spark.conf.set("spark.sql.parquet.outputTimestampType", "TIMESTAMP_MILLIS")
scala> df.write.parquet("/Users/maxim/tmp/before_1582/2_4_5_ts_millis")

scala> spark.conf.set("spark.sql.parquet.outputTimestampType", "INT96")
scala> df.write.parquet("/Users/maxim/tmp/before_1582/2_4_5_ts_int96")

Manually check the write code path. Save date/timestamps (TIMESTAMP_MICROS, TIMESTAMP_MILLIS, INT96) by Spark 3.1.0-SNAPSHOT (after the changes):

$ export TZ="America/Los_Angeles"

scala> spark.conf.set("spark.sql.session.timeZone", "America/Los_Angeles")
scala> spark.conf.set("spark.sql.legacy.parquet.rebaseDateTime.enabled", true)
scala> spark.conf.set("spark.sql.parquet.outputTimestampType", "TIMESTAMP_MICROS")
scala> val df = Seq(("1001-01-01", "1001-01-01 01:02:03.123456")).toDF("dateS", "tsS").select($"dateS".cast("date").as("d"), $"tsS".cast("timestamp").as("ts"))
df: org.apache.spark.sql.DataFrame = [d: date, ts: timestamp]
scala> df.write.parquet("/Users/maxim/tmp/before_1582/3_0_0_micros")
scala> spark.read.parquet("/Users/maxim/tmp/before_1582/3_0_0_micros").show(false)
+----------+--------------------------+
|d         |ts                        |
+----------+--------------------------+
|1001-01-01|1001-01-01 01:02:03.123456|
+----------+--------------------------+

Read the saved date/timestamp by Spark 2.4.5:
scala> spark.conf.set("spark.sql.session.timeZone", "America/Los_Angeles")
scala> spark.read.parquet("/Users/maxim/tmp/before_1582/3_0_0_micros").show(false)
+----------+--------------------------+
|d         |ts                        |
+----------+--------------------------+
|1001-01-01|1001-01-01 01:02:03.123456|
+----------+--------------------------+

@cloud-fan FYI

Test build #119805 has started for PR 27915 at commit 053861c.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24535/
Test PASSed.

Test build #119805 has finished for PR 27915 at commit 053861c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119805/
Test PASSed.

Test build #119819 has started for PR 27915 at commit d1e6d84.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24549/
Test PASSed.

Test build #119819 has finished for PR 27915 at commit d1e6d84.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119819/
Test PASSed.

Test build #119859 has started for PR 27915 at commit 63428ab.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24589/
Test PASSed.

Test build #119862 has started for PR 27915 at commit e590d36.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24592/
Test PASSed.

Test build #119866 has started for PR 27915 at commit 276d159.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24596/
Test PASSed.

Test build #119862 has finished for PR 27915 at commit e590d36.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119862/
Test FAILed.

Test build #119859 has finished for PR 27915 at commit 63428ab.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119859/
Test PASSed.

Test build #119874 has started for PR 27915 at commit 167b463.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24602/
Test PASSed.

Test build #119866 has finished for PR 27915 at commit 276d159.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119866/
Test PASSed.

Test build #119889 has started for PR 27915 at commit bbc4a1a.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24615/
Test PASSed.

Test build #119874 has finished for PR 27915 at commit 167b463.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119874/
Test PASSed.

Test build #119895 has started for PR 27915 at commit d7debb4.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24621/
Test PASSed.

Test build #119889 has finished for PR 27915 at commit bbc4a1a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119889/
Test PASSed.

Test build #119895 has finished for PR 27915 at commit d7debb4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119895/
Test PASSed.

@cloud-fan @HyukjinKwon Please, review the PR.

@mswit-databricks @adrian-ionescu FYI

@cloud-fan Regarding SQL config usage, WDYT of to do refactoring in a separate PR. I just don't want to introduce inconsistency by this PR. In the PR, I tried to follow existing style in fetching of SQL config values.

We already have SQLConf.get in ParquetRowConverter, so I'm not sure what is consistency now. Either is OK but SQLConf.get introduce fewer diffs. IMO fewer diffs is better.

Test build #119943 has started for PR 27915 at commit 6bebf3b.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24666/
Test PASSed.

Test build #119944 has started for PR 27915 at commit 8fa19a6.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24667/
Test PASSed.

Test build #119944 has finished for PR 27915 at commit 8fa19a6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119944/
Test FAILed.

jenkins, retest this, please

Test build #119952 has started for PR 27915 at commit 8fa19a6.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24676/
Test PASSed.

Test build #119943 has finished for PR 27915 at commit 6bebf3b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119943/
Test PASSed.

Test build #119952 has finished for PR 27915 at commit 8fa19a6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119952/
Test PASSed.

Test build #119970 has started for PR 27915 at commit a96392c.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24693/
Test PASSed.

Test build #119981 has started for PR 27915 at commit 5b52735.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24703/
Test PASSed.

Test build #119970 has finished for PR 27915 at commit a96392c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119970/
Test PASSed.

Test build #119981 has finished for PR 27915 at commit 5b52735.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119981/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24715/
Test PASSed.

Test build #119995 has started for PR 27915 at commit 184fcd8.

Test build #119995 has finished for PR 27915 at commit 184fcd8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119995/
Test PASSed.

thanks, merging to master/3.0!


Added action-setup-kubetools and action-setup-ecctl link to Utility part

action-setup-kube-tools:

https://github.com/yokawasa/action-setup-kube-tools

action-setup-ecctl:

https://github.com/yokawasa/action-setup-ecctl

These look great, thank you


Feature/add azure devops webhook

Add webhook support for Azure DevOps

Thanks a lot! Can you add at least one simple functional test?

@ramunasd sorry for the long break. The tests have now been added.


Requests with Large POST Body Immediately Goes into Delay Cancellation Right After the Server Received the Request

Environments

pekko-http:1.0.0
pekko:1.0.1
play-framework:3.0.0

Issue also observed with the following dependencies

akka-http:10.2.10
akka:2.6.21
play-framework:2.9.0

Issue description
I use pekko and pekko-http via play-framework and the following issue seems to only be reproducible when I receive the request via my nginx. If I hit my server through localhost:<server-port> with the same large request, the request does not suffers the same issue.
When server receives a request with a large POST body (~18400 bytes or more) from nginx, the request immediately goes into delayed cancellation after the request was received with the below debug log. Some debugging shows that this happens before the request is routed to the corresponding play Action and gets processed. If the server takes longer than the linger-timeout (default of 1 minute) to process the request, the connection is cancelled by the server.
[DEBUG] o.a.p.h.i.u.StreamUtils$DelayCancellationStage$$anon$11: Delaying cancellation for 1 minute

In the past, I've encountered these symptoms when using play-framework:2.8.x (akka:2.6.x and akka-http:10.1.x). At the time, I found that upgrading akka-http to 10.2.1 fixed the issue. However, after the upgrading to playframework:3.0.0 and adopting pekko, the same symptoms can be observed again. I've confirmed that the issue also happen on playframework:2.9.0 and its akka versions (listed above).
I'm not sure if this issue has been reintroduced in akka-http:10.2.10 and pekko has inheritted it and would like to ask for your assistance.
Debug Information
While I do not have a good minimum reproducible case to report, I've obtained some debug information I hope could help in figuring out the issue.
These logs contains the information that would have been printed by the following lines...
https://github.com/apache/incubator-pekko/blob/cfff9c53df859bb0f4407caf4821e7831dabeb19/stream/src/main/scala/org/apache/pekko/stream/impl/fusing/GraphInterpreter.scala#L551-L553
  println(
    s"$Name PUSH ${outOwnerName(connection)} -> ${inOwnerName(
        connection)}, ${connection.slot} (${connection.inHandler}) [${inLogicName(connection)}]")

https://github.com/apache/incubator-pekko/blob/cfff9c53df859bb0f4407caf4821e7831dabeb19/stream/src/main/scala/org/apache/pekko/stream/impl/fusing/GraphInterpreter.scala#L563-L564
  println(
    s"$Name PULL ${inOwnerName(connection)} -> ${outOwnerName(

https://github.com/apache/incubator-pekko/blob/cfff9c53df859bb0f4407caf4821e7831dabeb19/stream/src/main/scala/org/apache/pekko/stream/impl/fusing/GraphInterpreter.scala#L517-L519
    println(
      s"$Name CANCEL ${inOwnerName(connection)} -> ${outOwnerName(
          connection)} (${connection.outHandler}) [${outLogicName(connection)}]")

... and the cause for cancellation at
https://github.com/apache/incubator-pekko/blob/cfff9c53df859bb0f4407caf4821e7831dabeb19/stream/src/main/scala/org/apache/pekko/stream/impl/fusing/GraphInterpreter.scala#L522
The POST body sent is a text/plain body with the content data=0000000000[...] with the number of zero being specified by me to control the size of the body. The log contains all PUSH, PULL and CANCEL up to the time when the request gets to the play Controller.

18357 bytes request via nginx - connection goes into delay cancellation - connection-activities-18352-error.log
18356 bytes request via nginx - connection does not get cancelled - connection-activities-18351-success.log
18357 bytes direct request to localhost - connection does not get cancelled - connection-activities-18352-direct-success.log

Thank you very much for your time and if you require more information, please do not hesitate to ask.

Hi @raphaelNguyen, thanks for the report. Can you give a more high-level description of the problem? What are you trying to achieve and with which code and what happens in the error case?
The difference between the cases is that in the error case, the data comes in multiple chunks of data. There is no guarantee on how data is split up in the entity data stream. On local host and without TLS, it is far more likely that data is received in bigger chunks than with an actual connection with higher latency and maybe chunking caused by TLS and other networking components. In any case, as said before, there's no guarantee on how the data is split up. Making a guess, the problem here is that some component expects data to be split up in a certain way (or not at all). Any previous solution might have been accidental based on lucky timing.
Your logs show play.api.mvc.BodyParsers$TakeUpTo, fold and headOption stream components. Are these based on your code?

This seems to be an HTTP/1.0 request with Connection: close which is somewhat uncommon, so maybe something else is going on as well.

I can somewhat reproduce your issue, assuming that you cancel the request stream after having read all bytes (which is somewhat expected based on the TakeUpTo -> fold -> headOption flow but cancellation is somehow not shown in your logs?). If that's really the case, then the behavior is somewhat expected but probably under-documented. You must not cancel a request entity stream if you want to send out a response (but slurp the rest in, in this case, the rest would actually be empty but you still need to request it instead of cancelling the stream).
Here's the test, to be put into HttpServerSpec:
    "round-trip with HTTP/1.0 POST request with Default entity (entity received in bits) and Connection: close" in assertAllStagesStopped(new TestSetup {
      send(
        """POST / HTTP/1.0
          |Host: example.com
          |Connection: close
          |Content-Length: 12
          |
          |abcdef""")

      val entity =
      inside(expectRequest()) {
        case HttpRequest(POST, _, _, entity, _) =>
          entity.contentLengthOption shouldEqual Some(12)
          entity
      }
      requests.request(1) // emulate that the server with pull in more requests early

      val dataProbe = ByteStringSinkProbe()
      entity.dataBytes.runWith(dataProbe.sink)
      dataProbe.expectUtf8EncodedString("abcdef")

      send("ghijkl")
      dataProbe.expectUtf8EncodedString("ghijkl")
      dataProbe.cancel()
      // proper user behavior instead of cancel makes the test pass:
      // dataProbe.request(1)
      // dataProbe.expectComplete()

      // let the server some time to propagate messages / prepare the response
      Thread.sleep(2000)

      responses.sendNext(HttpResponse(protocol = HttpProtocols.`HTTP/1.0`))
      expectResponseWithWipedDate(
        """HTTP/1.0 200 OK
          |Server: pekko-http/test
          |Date: XXXX
          |Content-Length: 0
          |
          |""")

      shutdownBlueprint()
    })

Somewhat similar previous instance: https://github.com/akka/akka-http/issues/3458
(Above explanation about why localhost version work is still valid, request is received fully -> HttpEntity.Strict, data is now fully detached from the http connection, so whatever you do has no consequence on the actual connection).
I can remember a previous issue of head being used with Play leading to issues but cannot find the reference any more (years ago).

Hi @jrudolph. Thank you for spending the time diving into this issue. I'm learning a lot here.
I'll try to answer your questions to the best of my abilities but let me preface all this by stating that I'm using pekko and pekko-http stock configuration out of play-framework:3.0.0. The only configurations that I put on top is configuring some time-outs to be longer. In the application, I never touch pekko/pekko-http code directly. The logs I obtained was me putting down a few breakpoints at the specified lines and logging when the breakpoint is hit.

Can you give a more high-level description of the problem? What are you trying to achieve and with which code and what happens in the error case?

After upgrading to play-framework:3.0.0 from 2.8.20, I noticed that larger requests time out at around 1 minute.
Turning on all debug loggings shows me the debug message from StreamUtils stating that the connection is being put into delayed cancellation right after the request data is received by the server. In these error cases, the connection is killed by the server after linger-timeout even though the server was still working on processing the request normally. This was not what I expected as requests that are a few bytes less in size is being received and processed normally.
I can reproduce the issue with a bare-boned play-framework application, as long as the request is coming through nginx. A direct request does not seem to trigger this issue. Happens both on play-framework 3.0.0 and 2.9.0.
After posting this issue, I was also able to reproduce the problem in play-framework:2.9.0 but locking the akka-http version to 10.2.9 which I was on previously when using play-framework:2.8.20. This is making me think the problem might have been in how play 2.9.0/3.0.0 is using akka/pekko instead of a problem in akka/pekko itself.

The difference between the cases is that in the error case, the data comes in multiple chunks of data.

Monitoring the TCP/TLS comminucation, I can see that when directly hitting localhost, the data is being sent in in 1 chunk.
However, the case where I sent 18356 bytes request via nginx (1 byte less than error case), the TCP/TLS communication shows the same pattern of traffic as the error case, but my breakpoint loggings show that it's also being received in 1 chunk and did not error out, which I find interesting.

Your logs show play.api.mvc.BodyParsers$TakeUpTo, fold and headOption stream components. Are these based on your code?

To the best of my knowledge, these are not based on my code as I do not interact with pekko directly.

This seems to be an HTTP/1.0 request with Connection: close which is somewhat uncommon, so maybe something else is going on as well.

As far as I know, my nginx is performing the same as before I upgraded to play-framework:3.0.0 and using pekko. I will investigate this further. Maybe the way play-framework/pekko handling this has changed in the upgrade.

You must not cancel a request entity stream if you want to send out a response (but slurp the rest in, in this case, the rest would actually be empty but you still need to request it instead of cancelling the stream).

Potentially new play behaviour as I don't touch the stream directly. I will dig into play code a bit and maybe take this problem to them as well.
Thank you again for all your assistance thus far in diagnosing this issue.

After some tweaking, I was able to reproduce the issue on my minimal play application without going through nginx. KeepAlive needs to be disabled and the request needs to be sufficiently large.

I can reproduce this in Play, it's because we have HTTP pipelining enabled be default, see

https://github.com/playframework/playframework/issues/12351#issuecomment-1959694810

@mkurz does this work with Akka HTTP, ie have we broken this in Pekko HTTP?

@pjfanning Behaviour in Akka HTTP 10.2.x and Pekko HTTP v1 is the same, so you did not break anything.

Thank you @jrudolph and @mkurz and others for looking into this issue. This issue has been resolved for me in https://github.com/playframework/playframework/issues/12351#issuecomment-1959694810


Grep tests don't include flags

The tests for the grep exercise don't require the grep function to accept flags, and don't test for them, so it's not adequately testing the spec.

Would changing the tests cause any problem for already submitted programs? (I don't think so, but just checking). I can send a PR for this.

I don't think so either. Please do!


Allow ref callbacks on portals

I'm using the portal concept in one of my components. It works like this. render returns null and then the actual rendering is done onComponentDidMount and onComponentDidUpdate to an element which was previously appended to the body. This is very useful for tooltips and other overlay elements.
I found that this doesn't work very well with refs because Inferno expects a valid component instance, which is not available during onComponentDidUpdate. ref callbacks do not need instance, hence this pull request.

@mstijak thank you :)

That was quick :)

@mstijak when you have time, please can you check out Inferno 1.0-alpha2? If you want some info on setting it up, message me on Slack :)


Allow 100% with trailing fractional zeroes

Values like "100.00%" were not accepted, but values like "90.00%" were. So let's make it consistent. Verovio sometimes generates these values when converting from MusicXML.

I think if we don't want 100.00% then 90.00% should definitely not be valid. (However, I don't think deciding how numbers should be formatted is the role of the schema.)

I can see some value in values (pun intended) like 100.0%, 90.0% or tstamp="1.0, as they indicate precision, but beyond the first zero, there's little to gain with this argument. I'm happy to follow if others prefer more open values – it's what we have right now – and we can even allow trailinzg zeros on the 100, if that's what the community needs. I just wanted to bring up a different perspective, because that's what I typically see on that end ;-) Let's chat tomorrow about this – either solution is fine with me :-)

I agree there is no gain, but at the same time I don't see the need to 'clean' them since in term of data 1.0 and 1.00 are the same.

I think it would be an improvement in consistency if there were values above 100%  allowed.
Having 100.00% personally leads me to the assumption, that 100.56% is possible too.
So I see no improvement in encoding quality here (especially not for manual encoding).

Having 100.00% personally leads me to the assumption, that 100.56% is possible too.

Then you will see that 100.56% is actually an invalid value, which is what the schema stands for.
You can turn it the other way around. It leads you to the assumption that 99.99% is possible too, which is good. But reading guidelines is probably better than relying on assumptions ;-)

And, eventually, we're talking about a datatype called data.PERCENT.LIMITED, which has a <desc>ription as follows:

Positive decimal number between 0 and 100, followed by a percent sign "%".

We should not allow it to go over 100% ;-)


[SPARK-49357][CONNECT][PYTHON] Vertically truncate deeply nested protobuf message

What changes were proposed in this pull request?

Add a new message truncation strategy to limit the nesting level since the existing truncation strategies do not apply well to a deeply nested and large protobuf message.
Why are the changes needed?

There are instances where deeply nested protobuf messages cause performance problems on the client side when the logger is turned on.
Does this PR introduce any user-facing change?

No.
How was this patch tested?

Add a new test scenario to test_truncate_message in test_connect_basic.py.
Was this patch authored or co-authored using generative AI tooling?

No.

thanks, merged to master


Close laser emitter on Focal Length calibration

Close laser emitter during FL calibration and verify other calibration work according to table by algo team

How hard it is to add some kind of UT for this?
Is this phisibal?

I Don't think if it possible since viewer code is not reflected in the API

I Don't think if it possible since viewer code is not reflected in the API

No python API for it? python calibration example?


Steam password length/complexity causing error

I'm not sure exactly what the issue is - but my Steam password is over 60 chars length long, and includes essentially every possible scenario that KeePass allows.
When it asks for my password, I copy and paste it - and it comes back with "Incorrect Auth. Please check your username and password are correct"
I manually went to Steam, logged in with the exact credentials and it works.
I tried to re-do this a few times through the downgrader, viewed the password, everything looks the same as it does on the Steam website.
Thanks.

Do you per chance have characters such as " or \ in there?

Do you per chance have characters such as " or \ in there?

Yes

Alrighty, try adding \ before every instance of those

Im adding a little comment about it in next release


Still Get: Undefined mixin 'global-reset'.

I know this has been brought up on several occasions, and apparently there was a fix and you must move the Sass gem out of assets which I did (See below) but I still get the same error:
Hello. I have the following setup:
Rails version 3.1.0  Ruby v.1.9.2p290
Gemfile (pasted as is from the file)
gem 'sass-rails'
gem 'meta_search',    '>= 1.1.0.pre'```

Gem Versions:

```Using sass-rails (3.1.5)
Using activeadmin (0.3.4) ```

When trying to run:
```$> RAILS_ENV=production bundle exec rake assets:precompile```

Then I get:

rake aborted!
Undefined mixin 'global-reset'.
(in /Users/brad/.rvm/gems/ruby-1.9.2-p290/gems/activeadmin-0.3.4/app/assets/stylesheets/active_admin/_base.css.scss)

I forgot to mention that I do also have this in my production.rb
config.assets.precompile += %w[active_admin.css active_admin.js]

Based on this thread I bite the bullet and did it manually and it works. I was thinking it was already in the latest patch (ie. v 0.3.4) but its not.
To get it to work I added this:
@import "active_admin/mixins";
to the top of the scss files in the stylesheets files.

Relates to #810


Making an Arch User Repository (AUR) Package

Hey everyone,
I figured making a package for the Arch User Repository (AUR) might be very helpful to bring HIP to more people by making the installation straightforward for Arch/Manjaro users. It shouldn't be too hard since Arch packages are rather minimalist (here's the doc in the Arch wiki).
I'd gladly do the effort and start working on it as soon as I have a proper internet connection, but the issue is here in case someone more familiar with Arch packages wants to take the lead.
Regards,
Jules
PS: There already is an HIP package but it's only for the runtime, not the compiler.

Hey @JPenuchot The Arch community has been doing some work at https://github.com/rocm-arch/rocm-arch to get a bunch of ROCm packages working. Feel free to check it out. As of right now we have many ROCm 3.5 packages compiling however we only officially support Arch and Arch-repo compatible operating systems like EndeavourOS. There have been some reports with Manjaro users stating the packages do not work for them, since Manjaro is not Arch-repo compatible.
I believe this issue can be closed.

Hey, thanks a lot I didn't know about that project, I'll try it out as soon as I can


ARROW-6714: [R] Fix untested RecordBatchWriter case

I noticed this now-defunct call to table() while reviewing another PR. We clearly weren't testing this case because if you were to pass a data.frame in, you'd get a segfault. This patch adds tests and fixes the issue.

Codecov Report

Merging #5518 into master will decrease coverage by 11.41%.
The diff coverage is 100%.

@@             Coverage Diff             @@
##           master    #5518       +/-   ##
===========================================
- Coverage   88.69%   77.28%   -11.42%
===========================================
  Files         964       58      -906
  Lines      128215     4010   -124205
  Branches     1501        0     -1501
===========================================
- Hits       113724     3099   -110625
+ Misses      14126      911    -13215
+ Partials      365        0      -365

Impacted Files
Coverage Δ

r/R/record-batch-writer.R
88.88% <100%> (+28.88%)
:arrow_up:

python/pyarrow/ipc.pxi

cpp/src/parquet/column_page.h

cpp/src/plasma/test/external_store_tests.cc

cpp/src/arrow/array/builder_decimal.cc

cpp/src/plasma/client.cc

cpp/src/arrow/flight/internal.cc

cpp/src/arrow/compute/compute_test.cc

cpp/src/arrow/python/io.cc

python/pyarrow/hdfs.py

... and 898 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 46a14db...3c2eb81. Read the comment docs.


api: initial poc with grpc

Initial draft of how we might integrate an API that serves two purposes:

A remote API for multiple browser (and possible simple mobile) clients, aka, a client Hub
A local API for command-line, shell, and native framework clients

See forthcoming comments...

Looks cool so far. For sure still need to dig into the details, but looking good!

Wonder how authentication fits into this? Seems that some auth token would somehow scope a client's access to a set of Stores

Good question @asutula. I was thinking the same, leverage a JWT scope somehow. I like the per-store basis idea.

I pushed a bunch of changes here. The biggest one is an initial store Manager that is used by the grpc service. I'm not sure if we want to figure out how to "persist" the models so they can be re-hydrated by the manager as well.
I can make some inline comments in the morning. Gotta go hit the sack now.

Thanks for the review, @jsign! Here are some general notes below regarding this work and next steps:

I realize we're layering on a kind of "server" use-case to the stores, and exposing the full API to clients (other than local) may not be ideal. However, I'm finding it hard to determine where the line should be, which point to just needing to build some apps with it to get a better feel.
A good example of the above fuzziness is model registration: Should any client be able to register models? This is analogous to creating a SQL table, so probably not.
I believe we'll need some way to "reload" models into each store (and register them with the dispatcher). In the meantime, the client can just force registration whenever they start up. Maybe this is how it should always be? Hard to say.
No auth still, but there's an issue tracking the JWT work.

Next steps:

@sanderpick to add state change subscription in a separate PR
@asutula to take over flushing out the remaining APIs in a separate PR


No documentation for how to use ProgressiveMediaSource

Version
Media3 1.2.1
More version details
No response
Devices that reproduce the issue
N/A
Devices that do not reproduce the issue
No response
Reproducible in the demo app?
Yes
Reproduction steps
I'm trying to use ProgressiveMediaSource to play a local .mp4 video file using ExoPlayer. However, I've spent an hour browsing docs, and I can't find any documentation on how this thing is supposed to work. I would expect an example to be present in ProgressiveMediaSource or ProgressiveMediaSource.Factory, located here: https://developer.android.com/reference/androidx/media3/exoplayer/source/ProgressiveMediaSource. This is a common use case that with MediaPlayer was as simple as passing it a URI. I know ExoPlayer supports it too, but it's not obvious how to do it, so I think this is a docs bug that there isn't any documentation on how to do it.
Expected result
N/A
Actual result
N/A
Media
N/A
Bug Report

[X] You will email the zip file produced by adb bugreport to<EMAIL_ADDRESS>after filing this issue.

Update: I wasn't able to get ProgressiveMediaSource to work at all, it kept giving me "file not found" errors. But, I was able to use a media item directly to play my local file. Again, this took an entire afternoon of googling to find a working solution, I think the docs need to be updated to include this common use case. Here's my solution from this old stack overflow post:
val uri = RawResourceDataSource.buildRawResourceUri(R.raw.my_video)
val mediaItem = MediaItem.fromUri(uri)
player.setMediaItem(mediaItem)

https://stackoverflow.com/questions/40276012/how-to-get-local-video-uri-for-exoplayer-2-x
I had originally tried to use ProgressiveMediaSource since I thought you were supposed to create a media source for the player. Here's my non-working code, again, I was guessing here because I haven't found any usage docs:
// Throws "androidx.media3.datasource.FileDataSource$FileDataSourceException: java.io.FileNotFoundException: /2131886084: open failed: ENOENT (No such file or directory)"
val mediaSource = ProgressiveMediaSource.Factory(
    FileDataSource.Factory()
).createMediaSource(mediaItem)


nixos/minidlna: add user and group options

Description of changes
This allows the selection of a user/group to use for running the MiniDLNA daemon.  The minidlna user/group no longer gets created when a different user/group is used.
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

Tested on NixOS (x86_64-linux) via nixos-rebuild -I path/to/nixpkgs

[x] Using the default minidlna user/group
[x] Using a custom user/group

Rebased to make this change mergeable.  Tested on NixOS.

Please squash commits.

Squashed.

Hm, one of the CI checks is failing.


Elegant exclude certain files in the pages directory

we are developing a large react app, and we put a few non-page files in the folder.
├── src
|  ├── entry
|  ├── common
|  |  ├── components
|  |  ├── constants
|  |  ├── env
|  |  ├── hooks
|  |  ├── store.ts
|  |  ├── types
|  |  └── utils
|  └── pages
|     ├── choose-role
|     |  ├── api.ts
|     |  ├── components
|     |  ├── constants.ts
|     |  ├── utils.ts
|     |  └── index.tsx // the page file

the other files than index.tsx are not routes (e.g. /choose-role/utils
). So, we want to exclude these files when auto-generating the routes.
I can imagine three ways:

adding a _ before all these filenames. e.g. utils.ts => _utils.ts. awful tho. 😅
adding an excluded files list. (Thankfully, we have a good name convention for these files).
renaming the index.tsx => page.tsx(similar to the app dir, other files should be excluded by default.)

my personal preference is 3 ≈ 2 > 1.
is generouted adaptable for our case? would like to see your insights! ❤

Hey @aladdin-add, I see in your case many files at src/pages are .ts and actually those are ignored by default, as generouted looks only for pages with .tsx|.jsx extensions.
Currently in generouted, the ignore patterns for .tsx|.jsx pages are inspired by Docsuraus. You can ignore page(s) either by prefixing each page with _ or by prefixing a parent directory with _. So that should work as of now:
├── src
|  ├── entry
|  ├── common
|  |  ├── components
|  |  ├── constants
|  |  ├── env
|  |  ├── hooks
|  |  ├── store.ts
|  |  ├── types
|  |  └── utils
|  └── pages
|     ├── choose-role
|     |  ├── _components
|     |  |   ├── button.tsx   # ignored by parent _ prefix
|     |  |   └── card.tsx     # ignored by parent _ prefix
|     |  ├── _modal.tsx       # ignored _ prefix
|     |  ├── api.ts           # ignored .ts by default
|     |  ├── constants.ts     # ignored .ts by default
|     |  ├── utils.ts         # ignored .ts by default
|     |  └── index.tsx        # matches

Seems to be option 2 if I understand correctly.

Also I found out that the ignored paths were included in the generated types, but it was fixed few minutes ago. You can upgrade to v1.11.3 and try it out.
Lastly here's a related issue #9 for co-locating non-pages in src/pages.
Hope that helps!

Thanks for the quick reply! I mean option 2 is to opt-in something like:
export default defineConfig({
  plugins: [
    react(),
    generouted({excludedFiles: ["components/**/*"]})
  ]
});

so that components does not have to be prefixed("_components").  wdyt?

sure,I just like having an option; as how the option is designed, I'm totally open.

I've updated the plugins now to accept a source option in v1.11.4. I went with this option as it can be used for extending ignore patterns while having other potential use cases as well.
source can be string | string[], in your case it would make sense to use an array with the default pattern at the beginning of the array, then to include the ignore pattern(s) after in the array. I've tested with the following config:
export default defineConfig({
  plugins: [
    react(),
    generouted({ source: ['./src/pages/**/[\\w[-]*.{jsx,tsx}', '!./src/pages/**/components/**'] })
  ],
})

The second pattern start with ! to be ignored. You can check fast-glob docs for patterns reference.

For the <Routes> component to follow the same ignore pattern, you'll need to have a custom file, based on the current integrations at packages/generouted/src/* using generouted core. Should be something similar to #28

Big thanks! I'll give it a try.


Store encryption metadata outside the catalog

Will be tracked here https://perconadev.atlassian.net/browse/PG-832


Enable exported function/variable comment check in ci

Is your feature request related to a problem?
In branches less than release-5.1, comment check for exported function was enabled, such as
https://github.com/pingcap/tiflow/runs/4785508348?check_suite_focus=true
linting
  ✘  https://revive.run/r#exported  exported method Manager.UpdateChangeFeedCheckpointTs should have comment or be unexported
  ./cdc/sink/manager.go:165:1

✘ 1 problem (1 error, 0 warnings)

Errors:
  1  exported

However the latest golangci-lint disables this check.
Describe the feature you'd like
Enable two check and got tons of warnings
diff --git a/.golangci.yml b/.golangci.yml
index 8474f11ce..ffc5ab6f1 100644
--- a/.golangci.yml
+++ b/.golangci.yml
@@ -72,3 +72,6 @@ issues:
         - errcheck
         - gosec
         - makezero
+  include:
+    - EXC0012 # exported (.+) should have comment( \(or a comment on this block\))? or be unexported)
+    - EXC0014 # comment on exported (.+) should be of the form "(.+)..."

tools/bin/golangci-lint run --timeout 10m0s --skip-files kv_gen --skip-dirs dm,tests
cdc/sink/codec/craft/model.go:501:1: exported: comment on exported method RowChangedEventBuffer.RowsCount should be of the form "RowsCount ..." (revive)
// Number of rows batched in this buffer
^
cdc/sink/manager.go:130:1: exported: exported method Manager.UpdateChangeFeedCheckpointTs should have comment or be unexported (revive)
func (m *Manager) UpdateChangeFeedCheckpointTs(checkpointTs uint64) {
^
cdc/sink/mysql_params.go:35:2: exported: comment on exported const DefaultWorkerCount should be of the form "DefaultWorkerCount ..." (revive)
	// expose these two variables for redo log applier
	^
cdc/sink/mysql_params.go:37:2: exported: exported const DefaultMaxTxnRow should have comment (or a comment on this block) or be unexported (revive)
	DefaultMaxTxnRow   = 256
	^
cdc/sink/producer/kafka/kafka.go:262:2: exported: exported var NewAdminClientImpl should have comment or be unexported (revive)
	NewAdminClientImpl  kafka.ClusterAdminClientCreator = kafka.NewSaramaAdminClient
	^
cdc/capture/capture.go:110:1: exported: exported function NewCapture4Test should have comment or be unexported (revive)
func NewCapture4Test() *Capture {
^
pkg/pdtime/clock.go:109:1: exported: exported function NewClock4Test should have comment or be unexported (revive)
func NewClock4Test() Clock {
^
cdc/sink/codec/canal_flat.go:392:1: exported: exported method CanalFlatEventBatchEncoder.SetParams should have comment or be unexported (revive)
func (c *CanalFlatEventBatchEncoder) SetParams(params map[string]string) error {
^
cdc/sink/codec/canal_flat.go:410:1: exported: exported function NewCanalFlatEventBatchDecoder should have comment or be unexported (revive)
func NewCanalFlatEventBatchDecoder(data []byte, enableTiDBExtension bool) EventBatchDecoder {
^
cdc/sink/codec/interface.go:163:6: exported: exported type EncoderBuilder should have comment or be unexported (revive)
type EncoderBuilder interface {
     ^
cdc/sink/codec/json.go:54:1: exported: exported function NewColumn should have comment or be unexported (revive)
func NewColumn(value interface{}, tp byte) *column {
^
cdc/scheduler/util/table_set.go:55:2: exported: exported const AddingTable should have comment (or a comment on this block) or be unexported (revive)
	AddingTable = TableStatus(iota) + 1
	^
pkg/api/v1/changefeed.go:24:1: exported: comment on exported type ChangefeedsGetter should be of the form "ChangefeedsGetter ..."  (revive)
// NewBackendContext4Test returns a new pipeline context for test, and use the
^
cdc/processor/pipeline/system/system.go:57:1: exported: exported method System.Router should have comment or be unexported (revive)
func (s *System) Router() *actor.Router {
^
cdc/processor/pipeline/system/system.go:61:1: exported: exported method System.System should have comment or be unexported (revive)
func (s *System) System() *actor.System {
^
pkg/api/internal/rest/client.go:23:1: exported: comment on exported type HTTPMethod should be of the form "HTTPMethod ..." (with optional leading article) (revive)
// Enum types for HTTP methods.
^

Describe alternatives you've considered
No response
Teachability, Documentation, Adoption, Migration Strategy
No response

/assign


Tests for ARM, adds ManagedIdentityTrackedResource

Cadl Ranch Contribution Checklist:

https://github.com/Azure/cadl-ranch/issues/605

Supported scenarios

ManagedIdentityTrackedResource

createWithSystemAssignedOnly(put)
updateWithUserAssignedAndSystemAssigned(patch)
updateWithUserAssignedOnly(patch)
get

[x] I have written a scenario spec

[x] I have meaningful @scenario names. Someone can look at the list of scenarios and understand what I'm covering.

[x] I have written a mock API

[x] I have used @scenarioDocs for extra scenario description and to tell people how to pass my mock api check.

Did you follow https://github.com/Azure/cadl-ranch?tab=readme-ov-file#before-making-a-pull-request ?


Triton Backend Trains Slower Than Cuda?

More curious about what others are seeing....
I'm fine-tuning llama30b on a 24G titan RTX card and using same everything (model, dataset (alpaca), hyperparameters, gradient checkpointing on) and just trying out fine-tuning with different the two backends (cuda vs triton).
Cuda is a little more than 3s/it  and Triton is a little under 4s/it... I guess I was hoping/expecting triton to perform a bit better (but at least no worse...)
What are others seeing?
Thanks!

New cuda is slow AF too. I loaded the same exact GPTQv1 llama-30b 4bit model. Quadro P6000
Old implementation:
Output generated in 6.92 seconds (4.19 tokens/s, 29 tokens, context 382)
Output generated in 8.72 seconds (5.05 tokens/s, 44 tokens, context 382)

New Cuda
Output generated in 26.57 seconds (1.09 tokens/s, 29 tokens, context 376)
Output generated in 26.02 seconds (1.11 tokens/s, 29 tokens, context 376)

I can't get any model to load with Triton because all of what I downloaded had no group size, just the new format. I am looking for one to try or I'll encode one myself.
I'm assuming training will follow the same story.

Looks like there are some promising triton perf improvements in the upstream from the sterlind gptq-for-llama repo.  Any thoughts on getting sterlind bits merged into the qwopqwop repo?


Travis CI:  es6.object.to-string.js 'default' is not exported

Overview
START:
commonjs-proxy:/home/travis/build/carbon-design-system/carbon-components/node_modules/core-js/library/modules/es6.object.to-string.js (1:219) 'default' is not exported by 'node_modules/core-js/library/modules/es6.object.to-string.js'

☝ Is anyone aware of why our Travis CI is logging this out before running tests? Not sure how to resolve this myself. cc: @asudoh

@hellobrian Good catch, I suspect it's because rollup-plugin-commonjs is not applied to core-js for tests. The roll up config for tests can be found in karma.conf.js. Would you want to fix it? If not let me know and I can go ahead and fix it.

Hey @asudoh, sorry for the late reply. If it's okay, might be better if you fix it. Thanks for offering!


Merging Develop into Master 12.19.2019 - Email Testing for Dynamic templates, retroactive TNE release Notes, adding note on segmentation to MC how-to page

Description of the change:
Reason for the change:
Link to original source:

Closes #

Hello @Whatthefoxsays,Thanks again for the PR! We appreciate your contribution and look forward to                                              continued collaboration. Thanks!                                             Team SendGrid DX


postgres: Allow ignore_databases config option to be empty

What does this PR do?
This fixes a bug where setting ignore_databases: [] in the config would generate a broken query.
Motivation
A client is using databases that we typically ignore, and would like to monitor them.
Additional Notes
TODO: I'm not sure where the best place to add a test for this is.
Review checklist (to be filled by reviewers)

[ ] Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
[ ] PR title must be written as a CHANGELOG entry (see why)
[ ] Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
[ ] PR must have changelog/ and integration/ labels attached

Closing this PR since it's still a draft, feel free to re-open it if needed 🙂


[Extension type] Implement new nullaility rules in analyzer

Changes according to the https://github.com/dart-lang/language/pull/3434 has been implemented in CFE but not implemented in the analyzer yet. This causes failures of
co19/LanguageFeatures/Extension-types/upper_bound_A01_t05
co19/LanguageFeatures/Extension-types/upper_bound_A01_t06

This was implemented in e6b02b5895c616977ebf0a9f002b3e213629ca95, these tests don't fail anymore.


Migrate to the latest libp2p

Goal

Close #629

Updating this PR with new libp2p-ts

Ready for review


Fix places where I'm getting crashes.

I added checks for null in some places where I was getting crashes from freeing stuff that doesn't exist.
Keep in mind some of these crashes happen when switching scenes, so that might be why I'm encountering some of these errors.
I also changed to physics_process which runs at a constant rate (60 by default) rather than as fast as your CPU can tick if you have v sync off which for me costs around 0.3 ms. I guess you could also get an unfair advantage in multiplayer games :thinking:

Wait, does changing
if _field_background_grid:

to
if _field_background_grid != null:

really fix anything? As far as I know the two are equivalent.
I mean, I still prefer the x==null/x != null checks because they are more readable, but even the official tutorials use the x/not x checks from time to time. E.g: https://www.gdquest.com/tutorial/godot/gdscript/optimization-code/ (see the "leave loops and functions as soon as possible" section).
BTW, there are some git conflicts, you should probably rebase your branch onto the latest version of master.

really fix anything? As far as I know the two are equivalent.

Yeah it does, I think it's related to this : https://godotengine.org/article/dev-snapshot-godot-4-3-dev-1/
Read third bottom bullet point, or PR https://github.com/godotengine/godot/pull/73896
Freed objects are now treated differently from null in comparison operators ([GH-73896](https://github.com/godotengine/godot/pull/73896)), which ensures consistency for programmers when managing instances and memory.

I mean, I still prefer the x==null/x != null

Sure can do that.

BTW, there are some git conflicts. You should probably rebase your branch onto the latest version of master.`

I thought I had done this. Uhh,,, I messed up my git. I'm a open a separate PR


readme usage fix - set => put

set() method is not return anything

I'm so sorry @bshevchenko that this never got merged :( Somehow I missed the PR.
This got fixed since February and there's now both put and set that do the same thing: https://github.com/orbitdb/orbit-db-kvstore/blob/master/src/KeyValueStore.js#L18.
I'll merge this though as put is more common API for KeyValue stores.
Thank you for the contribution!


Update hybrid-modern-auth-overview.md

Added support note for hybrid agent.

@MSFTTracyP - please review and merge or reject changes

We've already updated https://docs.microsoft.com/en-us/exchange/hybrid-deployment/hybrid-agent and http://aka.ms/hmaom on the supportability statement. This is the last article that needs to be updated. Please merge the change.

@MSFTTracyP - can you please take a look and merge?


CIV-11541 - added new event.

JIRA link (if applicable)
NA
Change description
Added new event for Claim state change during LIP claimant intention.
Does this PR introduce a breaking change? (check one with "x")
[ ] Yes
[x] No

no changes


Lazy Loading

https://en.wikipedia.org/wiki/Lazy_loading

Thanks for the PR! In an effort to keep this repository clean and to focus on common patterns I will not merge this. Even the Wikipedia page is very shallow on this one.


Standardize usage of the term "test case" in the code and documentation

While thinking about the terminology we've been using to talk about tests for this framework, I realized that the term "test case" is being used to mean two distinct things:

A single test to be executed by the framework (i.e. an XCTestCase instance)
An XCTestCase subclass, including all of its test methods

After thinking about it more carefully and looking through the Apple XCTest header comments*, I've come to the conclusion that #1 is the correct usage. This PR is an attempt to improve our consistency around this.
Looking for feedback on both the underlying assumptions and the specific changes I'm proposing here!
* Note: Assuming folks agree with this usage suggestion, I'll be opening a Radar also, as there are a couple of points where the same confusion can be detected in the Apple XCTest headers.

Good idea! The changes also look good to me. @mike-ferris-apple, could you ask @swift-ci to please test?
Just spitballing, but maybe we could also one day add a "glossary" or something. It took me a while to understand what the following referred to:

Test case
Test run
Test suite
Test expectation (as opposed to assertion)

@swift-ci please test

Thanks, Daniel.  I would definitely be in support of pushing for clarity in the Xcode docs as well, so definitely don't shy away from filing radars on that!

One followup Radar posted: rdar://26002223


Get rid of redundant taxonomy_name from config

I'm pretty sure taxonomy_name is redundant from the config. I think its just used in extract_skills.py at the point of
if self.taxonomy_name == "toy":
            self.taxonomy_skills = load_toy_taxonomy()
        else:

but this could also be achieved with
if config_name=="extract_skills_toy":
   self.taxonomy_skills = load_toy_taxonomy()
else:

hmmm it's not redundant in so far as we use it for logging messages actually @lizgzil but that's its only use tbh

@india-kerle ok good to know - I guess we could use the name of the config file in the log? Just thinking of ways to slim down the configs

oh thats a great idea! lets do that


Show disclaimers at the bottom of charts

Where there are potential errors in the bumps chart (currently comments starting with #XXX) these should be printed as text at the bottom of the bumps chart from that year (underneath any crew names being added for the next year in multi-year charts, or underneath the bottom crew for single year charts).

The comments need rewording, and there are potentially quite a few missing, so would need to review sources.


fix(replica): error out replica if underlying file system doesn't support extent mapping

On branch fix-replica
Changes to be committed:
modified:   replica/replica.go
modified:   replica/extents.go
Not all file systems support file extent mapping, So replica
should error out if the underlying filesystem does not support
extent mapping.
This commit add check for the extent mapping support.
Create a temp file and send ioctl (FIEMAP and FIBMAP)
if it returns non zero value then exit, else proceed to
start replica.
This can be verified if the given path to the replica is formated
into the file-system which does not give file extent map. Few of
such file-system are NFS, ZFS etc.
Signed-off-by: Utkarsh Mani Tripathi<EMAIL_ADDRESS>

@yasker we should check for this in server.Start() function because if the replica get restarted on the same node server.Status() will return state: Closed and it will not reach to the construct() function. So replica will not be exited.

@yasker shouldn't we fatal the replica for this case?

@utkarshmani1997 I prefer not to panic deep in the logic. The error can be passed to the upper layer and eventually caused panic. But some errors may be gracefully handled but the caller won't even know there is an error if the function just decided to panic by itself. And catch all kinds of panics in the out layer is troublesome IMO. So as a general rule here, using error instead of panic deep in the logic.
And please squash your commits into one, which should make logic sense by itself.

It's a very good commit message, just needs a little tweak.
Can you remove following from the commit message? It's unnecessary.
On branch fix-replica
Changes to be committed:
    modified:   replica/replica.go

Also, change the title to:
replica: error out replica if the underlying file system doesn't support extent mapping

Thanks.

Thank you so much for the review as well as your guidance, learned a lot from this PR.


Check url for potential bots

Hello, this has passed as a non bot/crawler with this user agent:
Mozilla/5.1 (Windows NT 6.0; WOW64) AppleWebKit/537.37 (KHTML, like Gecko) Chrome/58.0.1145.75 Safari/537.37
However the url it visited is this:
/wp-content/plugins/bb-ultimate-addon/assets/css/uabb-admin-notice.css
It was a direct access to that url.
Same with this:
User agent:
Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36
Url visited:
/wp-login.php
I don't use WordPress so the result for them is a 404
The objective of some of those crawlers is to check for a given domain what kind of project they are using, in this case testing for WordPress

Hi,
Thanks for taking the time create an issue.
Unfortunately, in this instance, it is not possible for us to do anything as those user-agents you have listed have no unique signature that can be used to identify them. We have no way of knowing if they are indeed genuine, or not.
Thanks 👍


Game Updating

Is your feature request related to a problem? Please describe.
When trying to launch an outdated game, you currently just get an error. The 'Repair' option also doesn't seem to cover updating. Because of this, you have to either uninstall and reinstall entirely or go outside playnite to update.
Describe the solution you'd like
Since there doesn't seem to be a method for checking for updates, a manual context menu action calling install with --update-only or --repair-and-update would handle it fine.
Describe alternatives you've considered
Currently, I use the CLI to update unless it's a very small game.
Additional context
No response

Kinda bizarre why its not automatic, when legendary explicitly states when a game it not up to date on launch.
Skipping updates could be also added as a per game toggle if you are in offline mode (--skip-version-check on launch)


Propose: UI stuck partway through process

Describe the bug
I just tried to use the Propose that comes up when you click the three dots in the upper right of a post and I think it may be broken, at least the UI isn't indicating the process was completed successfully, etc.
To Reproduce
Steps to reproduce the behavior:

I made a Request post: https://near.social/mob.near/widget/MainPage.N.Post.Page?accountId=dawnkelly.near&blockHeight=116614755
Clicked the three dots in upper right corner of the post and selected Propose
The proposal modal popped up, I selected Text for my Proposal Type and the embed for my post was auto-populated (see screenshot below)
I clicked Next, the modal popped up showing the SocialDB data object to be saved and buttons to Confirm or Cancel, I selected Confirm and signed the wallet transaction
When the wallet transaction is finished, I'm taken back to the Propose modal with no indication of whether or not things succeeded. My only two options are to close the Propose modal or hit Next which repeats the saving of the same data.

Expected behavior
Everything is good through step 4 where you save data and sign the transaction in the wallet. Works great.
I would expect the wallet transaction to finish and then get some indication from the UI that the proposal was submitted successfully. This could look like a pop-up/toast with a message or a redirect to the Proposals feed where I would see my Proposal at/near the top of the list as one of the newest submissions.
Screenshots

Desktop (please complete the following information):

OS: iOS
Browser Chrome
Version

Additional context
The proposals were submitted successfully on-chain.
Transaction hashes:
https://nearblocks.io/txns/9LkhvMoWcTwnBvDD3smCaGyDsqmhFDNjLM9BRfSAXjrz
https://nearblocks.io/txns/B2EYdMArSu4JUr6LXPbRcXYmLqkxyhXSpFRZFKP5r4EC
I was not able to see them on the Proposal Feed earlier. I can't confirm if that is still the case as the Proposal widget seems to be down or being worked on right now.

hey @elliotBraem @dawnkelly09
Dawn your point is valid but I m not sure how we can solve this, because Near.call doesn't return promise so I can't know when the proposal is made, and since it's a deposit call we can't use the cache invalidation too.
My suggestion is:

Once the user clicks on "Create" to create the proposal, we can display a note letting them know where they can find their proposal on gateway, like a link, and then once the user confirm/copy it, we can close the modal and they can approve the txn.
I m open for feedback

Ah, that does make it tricky. As a user, a modal telling me where to check on the proposal would be nice. Then I can copy the url and click a "Dismiss" or similar button to close the modal and return to the Request feed.


Special characters + and # not supported for device ID

@kgremban

Can you review this PR?
IMPORTANT: When this content is ready to merge, you must add #sign-off in a comment or the approval may get overlooked.

#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team


cpanm is not executable by default

The documentation recommends installing the optional EV module here:
https://convos.by/doc/getting-started.html#optional-modules
using
$ ./script/cpanm --sudo EV

The ./script/cpanm file,  is not executable and we get a -bash: ./script/cpanm: Permission denied error.
Fixed trivially by chmod +x
./script/cpanm should be made executable by default.
I was not sure about git retaining file permissions - but the ./script/convos file, however, IS executable - which makes one think that this is possible to achieve.

Thanks! It can't be executable though, since I don't want that script to be installed, so instead I updated the documentation.


Clone not working

Hi,
I have tried cloning this Repo and running it on my local. To help you understand better, I am familiar with JS, nodeJS and git.
Steps: After clone when I type npm intsall it gives error in the installation part. Also the index.html inside app is giving error. I hope you consider to see the code once.

Have you successfully fixed the problem? Did you find any problem?

Need a lot of info. Would be helpful if I get your gmail id.

Yeah. Of course.
<EMAIL_ADDRESS>


Fix the algorithm in TextCodec::toUtf16

The algorithm was broken. Required to actually fix Try/OpenGothic#31.

[about original TextCodec]: It should just work :)
Thanks!


Add simple chroma example

Your checklist for this pull request
🚨Please review the guidelines for contributing to this repository.

[x] I have added necessary documentation (if appropriate)

Description
Please describe your pull request. Add Screenshots(if any).
❤Thank you!
Linked Issue
closes #15
Post merge checklist

[ ] Follow steps from the guidelines for contributing to this repository.

Looks good to me, thank you for contributing @selamanse


Fixed compilation error for Xcode 6.3 beta 4

Thanks for the PR @nduplessis. While this will fix the compilation issue, I don't think this is the appropriate way to address the root problem. I'll try to push up a PR today that fixes the root issue so that you can still use trailing closures as designed.

Hi @nduplessis, I just pushed up #408 which solves the root cause of this issue.


AVIF: "Error: source: bad seek to ..." with 0.28.1

What are the steps to reproduce?
Run the following script after unzipping the attached test image:
const fs = require('fs');
const sharp = require('sharp');

const inputStream = fs.createReadStream(
  'testimage.avif'
);

const sharpStream = sharp()
  .jpeg()
  .toFile('output.jpg', function (err) {
    console.log(err);
  });

inputStream.pipe(sharpStream);

What is the expected behaviour?
With sharp 0.27.2, an output image is correctly generated and null is logged to the console. With sharp 0.28.1, the bad seek error is logged. An output file is present but appears to be empty.
Are you able to provide a minimal, standalone code sample, without other dependencies, that demonstrates this problem?
See above.
Are you able to provide a sample image that helps explain the problem?
See attached.
What is the output of running npx envinfo --binaries --system?
  System:
    OS: macOS 10.15.7
    CPU: (8) x64 Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
    Memory: 39.01 MB / 16.00 GB
    Shell: 3.2.57 - /bin/bash
  Binaries:
    Node: 12.22.1 - ~/.nvm/versions/node/v12.22.1/bin/node
    npm: 6.14.12 - ~/.nvm/versions/node/v12.22.1/bin/npm

testimage.avif.zip

This looks like a 10-bit AVIF image.
$ heif-info -d testimage.avif | grep bit
| | | bits_per_channel: 10,10,10
| | | high_bitdepth: 1
| | | twelve_bit: 0

Please see the changelog for v0.28.0 at https://sharp.pixelplumbing.com/changelog#v0280---29th-march-2021

Prebuilt binaries limit AVIF support to the most common 8-bit depth.

We should probably try to improve the error messaging, which is likely to involve a libvips or libheif change.

@lovell ahhh ok, thanks for checking this. My confusion was having a previously working test in a package that depends on sharp fail when I tried to drop the newer version in. Just for my own understanding, does it blow up the size of the libraries to have the additional bit depths?
Definitely think a more specific error message wouold be really helpful, although I appreciate arranging something specific for this case could be a challenge. I was a little surprised that an empty file would be written, but I guess that is because the error actually comes out of the stream as it sees bytes that aren't understood (and as it happens the way it's used in the dependent module wouldn't result in this).

Yes, not supporting higher bit depths reduces the binary size of the aom dependency by ~30%. Output AVIF images from sharp/libvips were and still are always 8-bit per channel.
Please subscribe to https://github.com/lovell/sharp-libvips/issues/97 for updates about a possible switch to smaller/faster alternatives for AVIF that might allow reading higher bit depth images again.
It looks like libheif generates a "warning" for this image with the message Unspecified: Success (2.0) so that's probably the first thing to fix.

any hope of getting images with bit depth 10 supported?

@Zutatensuppe Please subscribe to https://github.com/lovell/sharp-libvips/issues/97 for updates about a possible switch to smaller/faster alternatives for AVIF that might allow reading higher bit depth images again.


Multithreading HowTo

Thank you for sharing this great resizing library! Results look so good!
Now I would like to get it a little faster using multithreading, but I don't know how. In your documentation I found class CImageResizerThreadPool... is this the right class for multithreading? Unfortunately I have no idea how to use it. Could you please share a small example, how to realize multithreaded resizing with your library? Thanks in advance for your help!

It is an extremely simple front-end. You should first gain knowledge how "worker thread pools" work in general, then you will be able to implement it.
My own implementation looks like this, but it needs custom-programmed thread pool and worker thread objects that call Workload -> process().
class CThreadPool : public avir :: CImageResizerThreadPool,
public CWorkerThreadPool
{
public:
int MaxThreadCount; // The number of threads to use.
virtual int getSuggestedWorkloadCount() const
{
	return( MaxThreadCount <= 0 ? CSystem :: getProcessorCount() :
		MaxThreadCount );
}

virtual void addWorkload( CWorkload* const Workload )
{
	VOXERRSKIP( add( new CResizeThread( Workload )));
}

virtual void startAllWorkloads()
{
	startAll();
}

virtual void waitAllWorkloadsToFinish()
{
	VOXERRSKIP( waitAllForFinish() );
}

virtual void removeAllWorkloads()
{
	removeAllThreads();
}

};

Thanks for your answer. In past I implemented a thread pool for another topic, so I will have a look if that helps here too. What I did not understand yet: does this strategy help a) to render a single picture faster, or does it b) help to render e.g. 4 pictures at nearly the same time like one (on a QuadCore CPU)?
If a) : I don't see howto divide the picture into parts, wouldn't be this necessary somehow?

On a 4-core processor the resizing speed increases by a factor of 3.2, so it does help to resize images faster. Algorithm divides the image automatically.

Hello Aleksey,
Unfortunately, I've been unable to re-scale an image with my thread pool.
The result is the striped picture:

But it's good when re-scaled with a single thread:

The code of the re-scaling thread pool is the following:
using thread_pool_base = thread_pool;
class avir_scale_thread_pool : public avir::CImageResizerThreadPool, public thread_pool_base
{
public:
    virtual int getSuggestedWorkloadCount() const override
    {
        return thread_pool_base::size();
    }

    virtual void addWorkload(CWorkload *const workload) override
    {
        _workloads.push(workload);
    }

    virtual void startAllWorkloads() override
    {
        while (!std::empty(_workloads))
        {
            _tasks.emplace_back(thread_pool_base::enqueue([](auto workload){ workload->process(); }, _workloads.front()));
            _workloads.pop();
        }
    }

    virtual void waitAllWorkloadsToFinish() override
    {
        for (auto &task : _tasks) task.wait();
    }

private:
    std::deque<task_future<void>> _tasks;
    std::queue<CWorkload*> _workloads;
};

Could please help me to investigate why the result is different from the expected?
Thanks in advance!

How do you initialize the thread pool?

Which thread library are you using?

It looks like not all threads are actually being executed, maybe some workload queue mistake.

You also probably need to remove items from _tasks if they are not autoremoved.

Here is the library that I use (attached to the message).
thread_pool.txt
Just rename it to *.hpp

The thread pool is utilized like this:
    nstd::avir_scale_thread_pool scaling_pool;
    nstd::avir::CImageResizerVars vars; vars.ThreadPool = &scaling_pool;
    nstd::avir::CImageResizerParamsUltra roptions;
    nstd::avir::CImageResizer<fpclass_dith> image_resizer { 8, 0, roptions};
    image_resizer.resizeImage(image, width, height, 0, new_image.get(), new_width, new_height, channels, 0, &vars);

Removing tasks didn't help:
    virtual void removeAllWorkloads()
    {
        _tasks.clear();
    }

Make sure thread_pool_base::size() returns correct value - should be the number of processors in the system. I have doubts that thread pool actually runs all workloads, make sure thread pool is functioning correctly. Test it by replacing workload->process(); with something like printf( "thread started\n" );. It should print this string thread_pool_base::size()*2 times.

It does. It returns the right number of cores.

the size of pool: 16                                                                                Workload...                                                                                Workload...                                                                                Workload...                                                                                Workload...                                                                                Workload...                                                                                Workload...                                                                                Workload...                                                                                Workload...                                                                                Workload...                                                                                Workload...                                                                                Workload...                                                                                Workload...                                                                                Workload...                                                                                Workload...                                                                                Workload...

It prints 15 times...

I changed the default value from:
std::max(std::thread::hardware_concurrency(), 2u) - 1u

to
std::max(std::thread::hardware_concurrency(), 2u)

It didn't change anything. The picture is still striped.

Okay. I've fixed it!
I just looked into your code and saw that you use the same workloads two times.
My mistake was that I removed workloads right after the first execution.
The proper thread pool looks like this:
using thread_pool_base = thread_pool;
class avir_scale_thread_pool : public avir::CImageResizerThreadPool, public thread_pool_base
{
public:
    virtual int getSuggestedWorkloadCount() const override
    {
        return thread_pool_base::size();
    }

    virtual void addWorkload(CWorkload *const workload) override
    {
        _workloads.emplace_back(workload);
    }

    **virtual void startAllWorkloads() override
    {
        for (auto &workload : _workloads) _tasks.emplace_back(thread_pool_base::enqueue([](auto workload){ workload->process(); }, workload));
    }**

    virtual void waitAllWorkloadsToFinish() override
    {
        for (auto &task : _tasks) task.wait();
    }

    virtual void removeAllWorkloads()
    {
        _tasks.clear();
        **_workloads.clear();**
    }

private:
    std::deque<std::future<void>> _tasks;
    std::deque<CWorkload*> _workloads;
};

I'm glad you've got it working, I'll leave this issue opened for others to learn.

Thank you!

Thanks at all! Got it working now too!

@masc4ii Hello! I've registered myself on the magiclantern.fm forum, and posted a couple of messages there, still waiting for moderation approval. Anyway, to get the information faster to you, here's what I've posted there:
Hi! Have the MLV App authors tried to apply non-linear "saturation" image transformations in a higher resolution, with a later downsizing step? This is not a common technique, but from the DSP standpoint it should look much better. "Aliasing" is not the whole story like in image resizing, there's also "harmonic distortion", which is not as apparent with images as it is with audio. Maybe worth a try.
A follow-up: the same actually applies to "linearization" or sRGB->linear conversion. It's a non-conventional approach and is resource-heavy, but probably it will fix the feel of all these gamma corrections being "not right".
I would like to add that making e.g. 3x upsampling followed by 3x downsampling is completely safe regarding the dynamic range. What is affected is frequency response, but this is "visible" when resizing smaller images mostly. Any high-resolution photo in most cases is already lacking in the highest frequencies due to limitations of the lenses mainly.

@masc4ii It's even a lot more safer with 2x upsample-downsample cycle (retains an unbelievable 120 dB range), but for best results I think 3x is needed.

@masc4ii To optimize the things in the pipeline it may be useful to just first upsample, then process the pipeline, then downsample. I do this in professional audio software, that's a very important feature for the users.

Hi @avaneev ,
thank you! We'll see when the mods enable your account... 😄 ...yes, here I can read already.
Until now all operations are done on the original resolution of the RAW image. For the final export after processing with ffmpeg we upscale with AVIR. I can imagine, that doing all those processing operations on a upscaled image could give better quality - but that would probably slow down the processing a lot, or am I wrong? One of the biggest downsides of our app currently is processing time for most users. In your example (3x upscaling): I would expect the application to be ~9x slower - am I correct, or didn't I understand correct?
What exactly do you mean with conversion sRGB->linear? As far as I know, we do the opposite: RAW sensor data is linear and we convert to "something good-looking or useful" like sRGB or those Log profiles.
@ilia3101 : most of the processing was your work - what do you say?! 👀
Do you own a ML capable camera? If you want, I could send you some MLV samples, so you could play with the app a little - if you like.
Thanks so much for your ideas!

Yes, that will slow-down all processing, maybe not by a factor of 9, but easily by a factor of 7. linear->sRGB conversion is the same situation, it's a non-linear sample mapping, so my proposal applies there, too.
I do not have an ML capable camera now, but I did a lot of "photography" when I had some Canon EOS middlerange camera and a couple of lenses in the past. Here you can see my "photoworks", on my audio samples product pages: https://www.voxengo.com/group/drum-samples/

@masc4ii This option can be made switchable, e.g. 1x, 2x or 3x oversampling. For short videos or nightly renders one could select 2x or 3x, 1x for other cases. This is a "transparent" option, it does not change things too much, but probably improves perceived dynamic range and "smoothness" of the footage.

@masc4ii One more note: my subjective feeling says that an image becomes apparently "more vivid" when non-linear sample mapping is applied at an increased resolution, and indeed it looks "smoother", but not in a "blurry" meaning. Dynamic range improvement is not too perceptible for me.

@masc4ii It looks more "cinematic" I would also say, closer to the "vintage" than "eye-popping" crispness of "modern".

@masc4ii Of course, it's important to apply any saturation/gamma transformation before the final resize, applying them afterwards won't make things look much better.


Doesn't work

I just installed this package and it doesn't work. No errors on the developer console, except deprecation warnings. Changing the only available setting does nothing. Enabling or disabling "Show Invisibles" also doesn't help. Restart Atom also does nothing.
Ubuntu 14.04, Atom 0.175.0.

Same here since I updated to the last Atom release (0.176) and updated all my packages.

Unfortunately, author of this package is not around... I'm not able fix issues. Due to deprecations, there some lines to fix. We need help :)

I think this is the reason : https://atom.io/docs/latest/upgrading/upgrading-your-package
But I don't have enough time to upgrade this package. I don't know the Atom API. Hope the author will be back ;)

I went ahead and hacked on this a bit and submitted a pull request. Probably not the cleanest but its working. https://github.com/gmoore-sjcorg/atom-color