[SPARK-37228][SQL][PYTHON] Implement DataFrame.mapInArrow in Python

What changes were proposed in this pull request?
This PR proposes to implement DataFrame.mapInArrow that allows users to apply a function with PyArrow record batches such as:
def do_something(iterator):
    for arrow_batch in iterator:
        # do something with `pyarrow.RecordBatch` and create new `pyarrow.RecordBatch`.
        # ...
        yield arrow_batch

df.mapInArrow(do_something, df.schema).show()

The general idea is simple. It shares the same codebase of DataFrame.mapInPandas except the pandas conversion logic.
Note that documentation will be done in another PR.
Why are the changes needed?
For usability and technical problems. Both are elabourated in more details at SPARK-37227.
Please also see the discussions at https://github.com/apache/spark/pull/26783.
Does this PR introduce any user-facing change?
Yes, this PR adds a new API:
import pyarrow as pa

df = spark.createDataFrame(
    [(1, "foo"), (2, None), (3, "bar"), (4, "bar")], "a int, b string")

def func(iterator):
    for batch in iterator:
        # `batch` is pyarrow.RecordBatch.
        yield batch

df.mapInArrow(func, df.schema).collect()

How was this patch tested?
Manually tested, and unit tests were added.

Test build #144960 has started for PR 34505 at commit 31f1ab3.

Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49431/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49431/

Test build #144960 has finished for PR 34505 at commit 31f1ab3.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ArrowStreamArrowUDFSerializer(ArrowStreamSerializer):
case class PythonMapInArrow(
trait MapInBatchExec extends UnaryExecNode
case class PythonMapInArrowExec(

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144960/

Test build #144961 has started for PR 34505 at commit ae75b34.

Test build #144963 has started for PR 34505 at commit eb6fdee.

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49432/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49432/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49432/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49434/

Test build #144961 has finished for PR 34505 at commit ae75b34.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ArrowStreamUDFSerializer(ArrowStreamSerializer):
case class PythonMapInArrow(
trait MapInBatchExec extends UnaryExecNode
case class PythonMapInArrowExec(

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49434/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49434/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144961/

Test build #144963 has finished for PR 34505 at commit eb6fdee.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144963/

Test build #144964 has started for PR 34505 at commit fcf2814.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49435/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49435/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49435/

Test build #144964 has finished for PR 34505 at commit fcf2814.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144964/

Test build #144972 has started for PR 34505 at commit 7d12ea6.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49443/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49443/

This PR is ready for a review.

cc @zero323, @ueshin, @BryanCutler, @viirya, @jpivarski, @revans2, @tgravescs, @LucaCanali, @Tagar

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49443/

Test build #144972 has finished for PR 34505 at commit 7d12ea6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144972/

cc @lgray

Thanks for the CC - I'll have to try this out soon, but will require updating our stuff to spark 3 (should be easy). It is, however, not the highest priority.

Thank you!!! This is exactly what we've been looking for; with this in place, we'll be able to process data like
>>> import awkward as ak
>>> df = spark.createDataFrame([([0, 1, 2],), ([],), ([3, 4],), ([5],), ([6, 7, 8, 9],)], ("x",))
>>> def func(iterator):
...     for batch in iterator:
...         akarray = ak.from_arrow(batch)
...         combinations = ak.combinations(akarray["x"], 2, axis=1)
...         yield from ak.to_arrow_table(ak.Array({"comb": combinations})).to_batches()
...
>>> df.mapInArrow(func, df.schema)

(Assuming the data with nested structure, such as the DataFrame[x: array<bigint>] above, are created in some scalable way, most likely by Laurelin. Our ROOT data are full of these nested lists. ak.combinations, etc. are Awkward Array functions for dealing with these things without Python iterations. Doing the same thing through Pandas turns all of our vectorized arrays into Python lists, but the Spark → Arrow → Awkward → Arrow → Spark transformation keeps them as contiguous arrays with minimal memory and CPU footprint...)

Thanks for checking this @jpivarski !!! Looks nice!

Hi @HyukjinKwon - thanks for this PR! I had a question about the python-facing API. Is it possible to have something similar to how current python/pandas UDFs' withColumn() can tell spark which columns are necessary for the inputs and pass the remainder of the columns through to the resulting DF without round-tripping them through python? For example (open-coding this, might be missing some syntax):

df = spark.createDataFrame(
    [(1, "foo"), (2, None), (3, "bar"), (4, "bar")], "a int, b string")

@udf(IntType())
def squared(x):
    return x * x

df = df.withColumn("squared", squared["a"]))

Would only ser/de the "a" column through the function, but the remaining column "b" would be in the resulting dataframe.

Sounds like that's close to the original proposal of  #26783. I was skeptical about that idea and it might not be worthwhile because:

We will have to take care of other restrictions, and variants like aggregation, window, etc all together, and thought that might not be worthwhile
I still think that Arrow is rather an internal format instead of user-facing.

So these made me propose one (developer) API that doesn’t require considering other restrictions
(e.g., the length of input should be the same as output's in case of scalar UDF in select) or variants.
In addition, I am pretty sure the current API can do what UDFs could do although it might be less optimizing.
However, I am open to other suggestions.

Test build #145056 has started for PR 34505 at commit c09b702.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49526/

Test build #145060 has started for PR 34505 at commit e6d7730.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49530/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49526/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49526/

Test build #145056 has finished for PR 34505 at commit c09b702.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49530/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145056/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49530/

Test build #145060 has finished for PR 34505 at commit e6d7730.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145060/

retest this please

Test build #145075 has started for PR 34505 at commit e6d7730.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49544/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49544/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49544/

Test build #145075 has finished for PR 34505 at commit e6d7730.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145075/

thanks @HyukjinKwon, I think this API makes sense for our use cases at the moment

There's something this PR has made me begin to ponder. The "stream" object used by is created by the socket library's makefile() API: https://docs.python.org/3/library/socket.html#socket.socket.makefile
This means it is not a traditional file (i.e., the BSD socket API does not support posix read and write, so this is just a convenience provided by python). If a pipe were to be used instead of a socket, it seems conceivable that arrow data structures could be written to the pipe via the vmsplice() syscall, which would effectively do zero-copy movement of data from python to the JVM executor (I believe the virtual memory pages simply get assigned to the pipe file descriptor inside the kernel). My understanding was that the python worker.py process is always on the same machine as the JVM executor, so this seems like a reasonable speedup to consider.

As an option if it makes sense - Arrow also comes with Plasma https://arrow.apache.org/blog/2017/08/08/plasma-in-memory-object-store/

Plasma, an in-memory object store that is being developed as part of Apache Arrow

Ray uses Plasma heavily.
Would something like Plasma add scalability here for zero-copy shared-memory exchanges between jvm and python?

The "stream" object used by is created by the socket library's makefile() API: https://docs.python.org/3/library/socket.html#socket.socket.makefile

Sounds like an idea but I would like to avoid this approach for now because:

I am actually investigating a way of true zero copy by shared memory between JVM and Python side. Using socket isn't actually true zero copy. It does copy from JVM to Python side although they are in a streaming approach.
Using socket is I think too low-level API, and current API should be able to allow users to use byte stream way within the UDF itself - the cost of wrapping Arrow instance on the top of the actual Arrow bytes would be trivial.
3.I think using higher level is more common than the socket approach.

@Tagar it is conceivable to use something like plasma for zero-copy sharing of data between a spark JVM executor and the python worker. However, there are many hurdles that I see:

You would need to implement a brand new allocator (in addition to the "Unsafe"/off-heap allocator and the "JVM native memory" allocator), which creates a Memory-mapped file, and the does all allocations within that file.
Shared memory is not a resource which is scheduled by most schedulers, AFAIK. I know yarn doesn't handle it. I also know that the spark standalone scheduler doesn't handle it. Maybe k8s does, but I don't know. As an alternative to using shared memory, the JVM executor can create a memory-mapped file (as above), but instead of making it in shared memory, it can make it in its own private memory space, and then share that file descriptor with its child worker.py process via memfd_create(): https://man7.org/linux/man-pages/man2/memfd_create.2.html . However, memfd_create() is a relatively now syscall which is only in linux! (That is probably okay if this is behind a configuration option.)
Spark itself would have to use Arrow's format as its execution format to do zero-copying sharing between python and java. Unfortunately, almost all spark SQL operations are done on its InternalRow format (except for some special cases involving parquet and other columnar file stores I think), which is not a columnar format. I am almost certain that @tgravescs and others have thought about this before and made JIRA issues about supporting a columnar format (possibly arrow, possibly not) in Spark, but I don't know what happened to it.

All that is meant to say that true zero-copy seemingly requires touching a lot of parts of Spark. My proposal to move from using a BSD socket to using a pipe and using vmsplice just (potentially) fixes one thing: the copy done from python to JVM. It cannot even speed up the copying from the pipe to python's memory because of limitations of the vmsplice() syscall.

I am actually investigating a way of true zero copy by shared memory between JVM and Python side. Using socket isn't actually true zero copy. It does copy from JVM to Python side although they are in a streaming approach.

@HyukjinKwon I'm afraid there has been a misunderstanding. I agree that using a socket isn't zero copy at all. I am just saying that it is what it is done today. Anyway, I would be very interested in learning more about your thoughts on zero copy between JVM and python.

Let me address the rest of comments on codes by EOD today.

Test build #145133 has started for PR 34505 at commit 857fc77.

Test build #145135 has started for PR 34505 at commit af30c17.

Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49606/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49606/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49604/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49604/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49604/

Test build #145133 has finished for PR 34505 at commit 857fc77.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145133/

Test build #145135 has finished for PR 34505 at commit af30c17.

This patch fails from timeout after a configured wait of 500m.
This patch merges cleanly.
This patch adds no public classes.

retest this please

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145135/

Test build #145160 has started for PR 34505 at commit af30c17.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49631/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49631/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49631/

Test build #145160 has finished for PR 34505 at commit af30c17.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145160/

Will merge this one in few more days if there aren't more comments.

Merged to master.

Thanks guys!


Add wrapper for chmod

As the subject says, add a trivial wrapper for the chmod system call.

Welcome to Rust, Julio!  One question about this PR: can you give an example of when this function would be necessary as opposed to using std::os::unix::PermissionsExt::set_mode ?

Hey! I know your name :)
Uh... well, never? Hadn't seen that yet. So I guess this is useless :-P

Hold on... the set_mode you quote modifies a Permissions in-memory object... but how do you later apply those permissions to the actual file?

@jmmv see std::fs::set_permissions

Oh nice; thanks a lot. See https://github.com/jmmv/sandboxfs/commit/2dc36620fb2e90a84eade4deeb14bdc0c80e2898 for context.


Testing/installation test

This adds a basic test fixing #35 and also attempts to fix #16 using delta's
I mimicked the way phoenix is testing its installer task. The only problem is that we don't generate a test directory. We don't provide any way for testing jobs (currently).
I didn't test that every file got written, rather I asserted that we could load the sample dashboard

Hello, @JanStevens! This is your first Pull Request that will be reviewed by Ebert, an automatic Code Review service. It will leave comments on this diff with potential issues and style violations found in the code as you push new commits. You can also see all the issues found on this Pull Request on its review page. Please check our documentation for more information.

Coverage remained the same at 87.162% when pulling bcf21ad576d3ad0b3d9b8d40ab93ff9adbb73106 on playpasshq:testing/installation-test into c927a709af51183ea652a9c726fc34f0497a1deb on kittoframework:master.

Please split this in 2 PRs one for #16 and another for the installer tests.

Nevermind about https://github.com/kittoframework/kitto/pull/71#issuecomment-268378965
I commited (attributed to you) as 7dc8af7a0

Coverage remained the same at 87.162% when pulling 725682c45692ac28750b888d3904da343e50dad0 on playpasshq:testing/installation-test into c927a709af51183ea652a9c726fc34f0497a1deb on kittoframework:master.

Coverage remained the same at 87.162% when pulling 2ea09e380c2473998c6a78831fe736e6deb76f0c on playpasshq:testing/installation-test into 7dc8af7a032a234f5597747c16d4a5b56e172291 on kittoframework:master.

Cherry-picked this as 46d3f1744e1e
Note: Code for the test is adapted from https://github.com/phoenixframework/phoenix/blob/v1.2.1/test/mix/tasks/phoenix.new_test.exs
Thanks @JanStevens :+1:


update the speakers.

Fixes #35

[ ] Please always align the last row of speakers to the center
I have centred the last child on devices with a width greater than 1145px. do it across all devices is a bit tricky and would require a lot of effort. Is that ok with you like that if not then please open an issue and I'll look into addressing it.


Captions not detected in QuickTime even with full parsing

I have a 37 second Quicktime file with a c708 track wtih 608 and 708 streams where the first caption byte at 00:00:30:00.  MediaInfo (22.09) doesn't display any text streams in the file even with the --full flag.
I created a 128k dump to send for testing and weirdly, in that case a text stream is detected (but only the 708 stream).
full_mediainfo.txt
SBON2202100H.mov.head.txt
I can send the full file for troubleshooting if necessary.

I created a 128k dump to send for testing and weirdly, in that case a text stream is detected (but only the 708 stream).

It creates a MediaInfo track if no data, hum... It is a bug :).

I can send the full file for troubleshooting if necessary.

Yes, please.

first caption byte at 00:00:30:00 [...] even with the --full flag

 --full flag is for a full report, not a full parsing.
Test with  --ParseSpeed=1

Thanks, adding --ParseSpeed=1 does in fact display both caption streams.
I sent you a link to the file via email.
Aside from the potential bug from the partial file, is there any way to determine if a text track exists, even if there's no captions?  Right now our validation is flagging an error if upstream asserts that there are captions when there are not, but this is a valid use case.  For this 30 second file adding full parsing doesn't add much processing time, but I am reluctant to universally enable it for larger content.  The way I see it, we'd want to use MediaInfo/MediaConch to see if there is a text track at all, and reject if not - if there is, then pass it on to a tool with full essence decoding that would flag the lack of any captions.
Any thoughts on how we could accomplish that?

is there any way to determine if a text track exists, even if there's no caption data within the parsing interval?

--File_Eia608_DisplayEmptyStream=1 --File_Eia708_DisplayEmptyStream=1.
It is disabled by default because people were complaining that empty content was displayed as existing, not easy to find a good balance between speed, accuracy, and relevance of default result.

Thanks, that's very helpful. Is this exposed anywhere in MediaConch? Right now it's actually MediaConch that is failing for lack of a text track in our workflow.

MediaInfo CLI options are usable with MediaConch CLI (and also the lib part, I think).
For GUI (both MediaInfo and MediaConch), we need to add them, but IIUC you need only CLI so not urgent.

@plondino while working on the GUI I remarked that there is an incompatibility between this request and https://github.com/MediaArea/MediaInfoLib/issues/1882, so I added an option without incompatibilities at https://github.com/MediaArea/MediaInfoLib/pull/2085.
"Old" options are still there but the last one replaces the previous one, please use the new option instead when you can.
I also added extra/InternalDetectionKind field when no content is detected, with values Command (indicating that no 608/708 content was found, only commands) or Stream (indicating that no 608/708 content or command was found, only an empty 680/708 stream), in case it helps e.g. if you use self.mediainfo.Option('File_File_DisplayCaptions ', 'Stream') and then want to know if a content was actually detected .


Exception when special characters in shared text

Hi, and thank you for the effort you've put into this package.
When sharing text with special characters, for example line breaks, the app will crash when trying to parse the share intent.
Steps to reproduce: Copy text with new lines e.g.

App will crash with the following debug log:
DEBUG  useShareIntent[onChange] {"value": "{ \"text\": \"Accelerate innovation
Our\" }"}
ERROR  SyntaxError: JSON Parse error: U+0000 thru U+001F is not allowed in string, js engine: hermes

need two fixes :

Correctly handle parse error exception
Fix the json encoding workflow from ios (swift) to JS.

fixed in 1.1.0


Add support of Symbol action

Hi, I added simple patch to support symbol action in reduxMiddleware

Thanks.  I auto-generate that dist file... (I should really just remove it from git!)

Thanks for the patch!


Update build-release.yml to solve Node12 and artifact deprecated warnings

Questions
Answers

Description?
as recommended in recent build-release annotations

Type?
improvement

BC breaks?
no

Deprecations?
no

Fixed ticket?
Issue, please write "Fixes #[issue number]" here.

How to test?
This PR and next PR pass automatic checks without warnings of deprecation.

Right! These annotations are not visible on GitHub Mobile so it's better to add all relevant information inside issue/pr description.

I'm guessing the plan is to use organization github actions in all modules, like in https://github.com/PrestaShop/psgdpr/pull/176
Isn't it @atomiix @kpodemski ?

this discuss

It's probably different, having a shared ESLint configuration leads to bugs in the CI, bug with nodejs compatibility, it was a real pain to maintain because of this (Or I might not be qualified enough to maintain it properly myself)

Close as SharakPL and atomix have a plan to use organization github actions in all modules :)


Popover Overlay can set non response events

i need popover overlay do not respond to any events, but subcomponents  can，like View pointerEvents: 'box-none' .

@BaLaLaLs Thanks for reporting this. We'll add this feature in the coming release.


Button bar for Results tab taking half screen after long SQL error message

I am using DBeaver to query Hive. Hive can throw a very long error messages that include stack traces etc.. This causes DBeaver UI to brake.

It is like that since forever and it is quite a problem since sometimes you can wait long a time of a result and you can't see it in the end.

What is your OS?
What is your DBeaver version (if it is not 4.0.2 - please update it)?


--disable-openmp still builds with openmp

Building master on Ubuntu:
> ./configure --disable-openmp
...
>  make | grep -i openmp
libtool: link: g++  ...   ***-fopenmp*** -Wl,-soname -Wl,libtesseract.so.4 -o .libs/libtesseract.so.4.0.1
...
libtool: link: g++ -g -O2 -std=c++14 -o .libs/tesseract tesseract-tesseractmain.o  ./.libs/libtesseract.so -llept -lrt -lpthread ***-fopenmp***

I'm a total noob regarding C++/Makefiles, so I don't really know what could be wrong or how to really disable it.

Please provide more details (at lease configure output and config.log)

Here is the output of ./configure --disable-openmp
configure.txt
And here is config.log
config.log

There is nothing in log that instruct tesseract to use -fopenmp.
Can you provide also make log (make 2>&1 | tee make.log)?

In my case tesseract-4.0.0 was built on CentOS 6 as $ ./configure --disable-openmp as well.
However when I run lstmtraining I get:
lstmtraining: symbol lookup error: /usr/local/lib/libtesseract.so.4: undefined symbol: GOMP_parallel
ldd /usr/local/lib/libtesseract.so.4
linux-vdso.so.1 =>  (0x00007ffff6cbc000)
liblept.so.5 => /usr/local/lib/liblept.so.5 (0x00007fee72364000)
libpng12.so.0 => /usr/lib64/libpng12.so.0 (0x00007fee72114000)
libjpeg.so.62 => /usr/lib64/libjpeg.so.62 (0x00007fee71ebc000)
libtiff.so.3 => /usr/lib64/libtiff.so.3 (0x00007fee71c54000)
libz.so.1 => /lib64/libz.so.1 (0x00007fee71a34000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fee71814000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007fee71504000)
libm.so.6 => /lib64/libm.so.6 (0x00007fee7127c000)
libc.so.6 => /lib64/libc.so.6 (0x00007fee70ee4000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fee70cc4000)
/lib64/ld-linux-x86-64.so.2 (0x000055c53e44f000)

Please use recent git code when reporting issue.

@iragerasimov, maybe /usr/local/lib/libtesseract.so.4 is not the result of your build, but another version which was built with OpenMP?

Thanks! Looked more closely at /usr/local/lib/libtesseract.so.4 and after re-build got training tools working

Thank you!
On Tue, Mar 26, 2019 at 7:28 PM Irina<EMAIL_ADDRESS>wrote:

Thanks! Looked more closely at /usr/local/lib/libtesseract.so.4 and after
re-build got training tools working
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/tesseract-ocr/tesseract/issues/2323#issuecomment-476940233,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AYwCDTWJFmMiewtBDIdmrumzK8nqBBFZks5vatdOgaJpZM4b1Qnj
.

In my case, it appears something was not "clean" during the build.
After I deleted everything, checked out again and re-run everything, it built fine, without open-mp.


Provide tools for directly downloading the datasets from timeseriesclassification.com

Is your feature request related to a problem? Please describe.
it would be useful for users to be able to download the classification/regression problems directly from timeseriesclassification.com in code. Apparently tslearn does this, and it seems like a good idea
Describe the solution you'd like
I'm not sure where it should be located, perhaps with the load data tools
Describe alternatives you've considered
You can manually download the data, but this is labour intensive and means you need to confront the data format, when perhaps you do not need to.

Hey can I have this?

Yep 👍

I think we want some function to download data sets  and save them in a default location (i.e. sktime/datasets/data) so that we can simply load them the next time we need them.
This may help:

Please check out https://tslearn.readthedocs.io/en/stable/gen_modules/datasets/tslearn.datasets.UCR_UEA_datasets.html#tslearn.datasets.UCR_UEA_datasets
This may also be helpful https://github.com/fastai/fastai/blob/e34fee816f5dcbf05a9bc9b44e617902a52549fc/fastai/data/external.py#L243

Thanks, I'll have a look at them

Closed by #430


The Infinite Mixture Models tutorial error

It is not possible to use:
k = map(t -> length(unique(chain[:z].value[t,:,:])), 1:iterations);

it throws the error:
ERROR: ArgumentError: index z not found

This example uses an old version of MCMCChains. With newer versions you have to use
zvars = namesingroup(chain, :z)
k = map(t -> length(unique(chain[t, zvars, :])), 1:iterations)

(or something similar based on namesingroup, the code is untested). The issue will be fixed by the reorganization and updates in https://github.com/TuringLang/TuringTutorials/pull/113.

No it doesn't work. The following works, but it just produces a vector of ones:
k = map(t -> length(unique(get_params(chain[t,zvars,:]))), 1:iterations);

I assume the problem is that you use get_params: it returns a NamedTuple of parameters (see https://turinglang.github.io/MCMCChains.jl/dev/chains/#MCMCChains.get_params-Tuple{Chains}). I guess
k = map(t -> length(unique(Array(chain[t, zvars, :]))), 1:iterations)

is better (I assumed, apparently incorrectly since it failed (?), that chain[t, zvars, :] returns already an array).

That works better, but what is the logic behind this array construction?

Array(chain) returns an array of the values in the chain, where as default the different chains are appended (i.e., the output has #iterations * #chains rows and #parameters columns): https://turinglang.github.io/MCMCChains.jl/dev/getting-started/#Exporting-Chains

OK, fine. I have run this example with some different settings and samplers. One strange thing occurs when I change from the SMC() (which mixes badly) to the PG() sampler. When I use SMC() the output in zvars and consequently the k variable corresponds well with the output in the locations variable  muvars = namesingroup(chain, :μ), i.e. there is one location for each cluster. However, when the PG() sampler is used, this is no longer the case, i.e. there are more locations than clusters. However, the number of clusters seems to be correct.


⚠ Bofrost.com has degraded performance

In 3baa16a, Bofrost.com (https://www.bofrost.com) experienced degraded performance:

HTTP code: 200
Response time: 1257 ms

Resolved: Bofrost.com performance has improved in 3bfed37.


cssFile[file.path] = {styles: file.contents.toString()};

Hello,
After upgrading to version 2.1.0 we are starting receiving the following error
/Users/gvnn/Projects/thick/make-wally-thick/node_modules/gulp-clean-css/index.js:31
    cssFile[file.path] = {styles: file.contents.toString()};
                                               ^

TypeError: Cannot read property 'toString' of null
    at DestroyableTransform.transform [as _transform] (/Users/gvnn/Projects/thick/make-wally-thick/node_modules/gulp-clean-css/index.js:31:48)
    at DestroyableTransform.Transform._read (/Users/gvnn/Projects/thick/make-wally-thick/node_modules/readable-stream/lib/_stream_transform.js:159:10)
    at DestroyableTransform.Transform._write (/Users/gvnn/Projects/thick/make-wally-thick/node_modules/readable-stream/lib/_stream_transform.js:147:83)
    at doWrite (/Users/gvnn/Projects/thick/make-wally-thick/node_modules/readable-stream/lib/_stream_writable.js:347:64)
    at writeOrBuffer (/Users/gvnn/Projects/thick/make-wally-thick/node_modules/readable-stream/lib/_stream_writable.js:336:5)
    at DestroyableTransform.Writable.write (/Users/gvnn/Projects/thick/make-wally-thick/node_modules/readable-stream/lib/_stream_writable.js:274:11)
    at Transform.ondata (_stream_readable.js:555:20)
    at emitOne (events.js:96:13)
    at Transform.emit (events.js:188:7)
    at readableAddChunk (_stream_readable.js:176:18)
    at Transform.Readable.push (_stream_readable.js:134:10)
    at Transform.push (_stream_transform.js:128:32)
    at afterTransform (_stream_transform.js:77:12)
    at TransformState.afterTransform (_stream_transform.js:54:12)
    at Transform.stream._transform (/Users/gvnn/Projects/thick/make-wally-thick/node_modules/gulp-postcss/index.js:19:14)
    at Transform._read (_stream_transform.js:167:10)

We pipe to clean-css right after autoprefixer, like this
var b = gulp.src(sources)

    b = b.pipe(sasspipe)
        .pipe(postcss([
            autoprefixer({ browsers: ['last 3 versions', '> 5%', 'IE >= 0']})
        ]));

    if(!debug) {
        b = b.pipe(cleanCSS({ compatibility: 'ie8', processImport: false }));
    }

    return b
        .pipe(gulp.dest(dest))
        .pipe(module.size());

Just for a little bit of context, we have developed a series of tasks for our internal development pipeline, the code that I posted is used in our repo https://github.com/StudioThick/opine-styles it might be useful for testing.

@gvnn yikes! Yes I pushed some changes yesterday in attempts for fix some other stuff. Hang tight as I'll get it up today for you

@gvnn also I'm striving to get this done quickly for you but my testing doesn't catch quite everything. Could you do me a favor? Crack open (in your project) ~/node_modules/gulp-clean-css/index.js:31
and change
cssFile[file.path] = {styles: file.contents.toString()};

to
cssFile[file.path] = {styles: file.contents ? file.contents.toString() : file.toString()};
Let me know if that resolves your error?

I am having the same error, the fix you provided works.

cool I'll get it up soon, are you guys passing files with no contents? couple minutes...

2.1.1 is up. Can you please pull and confirm if fixed or terribly busted still? 🤓

Seems to be working again 👍

Thanks guys 👍  Apologies for the late reply, it was bed time in Australia 🛌

Not entirely sure how but I got this issue using v4.0.0?
I had to update node_modules/gulp-clean-css/index.js line 24 from:
[file.path]: {styles: file.contents.toString()}

To
[file.path] : {styles: file.contents ? file.contents.toString() : file.toString()}

This fixed the issue for me


misc: add 6220 Maestro bin range support, drop pattern regex

This adds the range that is requested in https://github.com/stripe/jquery.payment/issues/157, the exhaustive BIN ranges can be found here (http://www.barclaycard.co.uk/business/files/bin_rules.pdf)—aren't quite close to implementing those in current jquery.payment so addressed #157 directly.
r? @SlexAxton

This felt more like a combinatorics and regex quiz, but it feels fine. I agree with @jamesreggio in getting rid of an unnecessary regex for startsWith.


[SR-15683] Program crashes when adding many tasks to a TaskGroup on Windows

Previous ID
SR-15683

Radar
None

Original Reporter
@MarSe32m

Type
Bug

Environment
Windows, snapshot: swift-DEVELOPMENT-SNAPSHOT-2021-12-23-a.

Additional Detail from JIRA

Votes
0

Component/s

Labels
Bug, Concurrency, Windows

Assignee
None

Priority
Medium

md5: 33cb31c018b26c413b20c4f4c1356a81

Issue Description:
When adding many tasks to a TaskGroup, the program crashes when waiting for the tasks to finish. This doesn't happen on "relatively low" values of upperBound. For example when upperBound = 2500, the code executes fine to the end and prints "Done: 2500". But when for example upperBound > 10000, the program crashes. I cannot reproduce this behavior on macOS or Linux.
@main
struct App {
    static func main() async {
        let upperBound = 10000
        await withTaskGroup(of: Void.self) { group in
            for i in 0..<upperBound {
                group.addTask {
                    print(i)
                }
            }
        }
        print("Done:", upperBound)
    }
}

I cannot reproduce this anymore with nightly toolchain. I think the fix that resolved https://github.com/apple/swift/issues/57976 also resolved this.

Great to hear, thanks for looping back to let us know :+1:


 PCRE failure during the build step?:
https://travis-ci.com/github/JuliaPerf/PProf.jl/jobs/389738246
└ @ Pkg.Types /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Types.jl:949
    Cloning registry from "https://github.com/JuliaRegistries/General.git"
      Added registry `General` to `~/.julia/registries/General`
   Updating registry at `~/.julia/registries/General`
   Updating git-repo `https://github.com/JuliaRegistries/General.git`
  Installed OrderedCollections ─ v1.3.1
  Installed pprof_jll ────────── v0.0.0-20191205061153+0
  Installed ProtoBuf ─────────── v0.8.0
ERROR: PCRE JIT error: no more memory
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:33
  [2] jit_compile
    @ ./pcre.jl:138 [inlined]
  [3] compile(regex::Regex)
    @ Base ./regex.jl:73
  [4] Regex(pattern::String, compile_options::UInt32, match_options::UInt32)
    @ Base ./regex.jl:37
  [5] Regex
    @ ./regex.jl:60 [inlined]
  [6] parse(::Type{Base.BinaryPlatforms.Platform}, triplet::String; validate_strict::Bool)
    @ Base.BinaryPlatforms ./binaryplatforms.jl:716
  [7] parse
    @ ./binaryplatforms.jl:713 [inlined]
  [8] HostPlatform

https://github.com/JuliaLang/julia/pull/37688#issuecomment-697214481

UndefVarError: pprof not defined

That's due to https://github.com/JuliaPackaging/BinaryBuilder.jl/issues/913, you can safely ignore it for the time being

Awesome. Thanks, makes sense! I'll check back in on this in a few days then :)


Fixes arrow 0.17 compatiblity.

What do these changes do?
arrow::compute::CanCast is available since v0.17.
Related issue number
N/A.

Codecov Report

Merging #214 (daf2e21) into main (804ca53) will increase coverage by 0.12%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main     #214      +/-   ##
==========================================
+ Coverage   77.19%   77.31%   +0.12%
==========================================
  Files          55       55
  Lines        4262     4263       +1
==========================================
+ Hits         3290     3296       +6
+ Misses        972      967       -5

Impacted Files
Coverage Δ

src/server/async/socket_server.cc
83.77% <0.00%> (+1.95%)
:arrow_up:


Dont enable default features for host dependencies

Fixes https://github.com/microsoft/vcpkg/discussions/19781, depends on #173

I think this is not the correct way to do this; the best way to do this is just ask for default-features: false.

the best way to do this is just ask for default-features: false.

So I should run vcpkg install --dry-run, take a list of all ports that are a host dependency and add them with default-features: false to my vcpkg.json?

Can this be moved forward?
This would be a use case with effect on CI: https://github.com/microsoft/vcpkg/pull/23668

This PR is currently wrong according to #292 iiuc. If I remember correctly A[x] -> B[x] -> C[x] -> D[x], root has a host dependency on A and normal on C and x is always a default feature it currently installs A, B, C[x], D[x]

Hm, the focus of #292 was on a port being once requested at least twice, with different feature options. This shouldn't drop the default features if at least one port doesn't exclude them.
But what if there is only one request, with default-features disabled?

But what if there is only one request (or: if there are only requests) with default-features disabled?

Then they should be not enabled if it is a host dependency. This does work.

This PR is currently wrong according to #292 iiuc. If I remember correctly A[x] -> B[x] -> C[x] -> D[x], root has a host dependency on A and normal on C and x is always a default feature it currently installs A, B, C[x], D[x] instead of A[x], B[x], C[x], D[x]

If now everything is a [core] dependency (except from root to C), it now installs A, B, C[x], D[x] as expected

I am in doubt that this approach fixes the root issue. I consider that a library maintainer knows exactly the dependencies of its own library, that can be another library, a header file or just a single feature of a library. It does not make a difference if this is a host or a target dependency.
For me the purpose of the default feature set is that a library maintainer can propose a set of feature, that allows the majority of user to use the library it without dealing with features at all. There is no reason the the package maintainer is empowered to override users decision and enable unwanted feature via a default feature set by default.
@ras0219-msft

It is reasonable for this to be some sort of opt in, such as a setting in the manifest file that says "pretend I depended on [core] of every package in my tree".

That does not work, because that are normally two persons. The package maintainer can state "default-features": false but the application developer gets unnecessary dependency in its build, many of them will never notice that.

Consider that the consequence of using this with opencv as a transitive dependency would mean that user applications wouldn't actually be able to load any image files.

I think you have such tree in mind:
<user-app> - <magic-for-png> - <opencv[core,png]>
Is this correct?
For me it is still obvious that this does not rectify to add something like tiff support to opencv. If the user-app likes to use tiff files directly with openvc it has to state that:
<user-app> - <magic-for-png> - <opencv[core,png]>
<user-app> - <opencv[core,tiff]>
vcpkg will install <opencv[core,tiff,png]> to satisfy both.
or he may just use the default  if this suits best, and he don't want to deal with features.
If later the maintainer of <magic-for-png> decides that it does no longer depend on <opencv[core,png]> he can remove the dependency, and the <user-app> developer can use the updated vcpkg tree without regressions.

I think a separate setting that specifically affects host dependencies actually makes a lot of sense because those host dependencies can't affect the runtime of the application and so the reasoning behind default features isn't applicable.

I don't see a different in reasoning, we want only required dependencies that is true in the same way for host and target. There are different reasonable defaults though.

@daschuer It seems your comment here is not related to the host dependencies aspect. Please move it to one of the other places.

It is an answer to @ras0219-msft post above.

Ping for re-consideration. It is a significant difference for host utilities from qt5-tools which don't need GUI, cf. https://github.com/microsoft/vcpkg/pull/23668. And gui really needs to remain a default feature, no matter if native or cross build.

It is a significant difference for host utilities from qt5-tools which don't need GUI

Data from x86-windows CI, host x64-windows:
Building qt5-base[core,gui]:x64-windows...
Elapsed time to handle qt5-base:x64-windows: 6.659 min
Building qt5-activeqt[core]:x64-windows...
Elapsed time to handle qt5-activeqt:x64-windows: 39.88 s
Building qt5-imageformats[core]:x64-windows...
Elapsed time to handle qt5-imageformats:x64-windows: 6.465 s
Building qt5-svg[core]:x64-windows...
Elapsed time to handle qt5-svg:x64-windows: 17.41 s
Building qt5-declarative[core,quick]:x64-windows...
Elapsed time to handle qt5-declarative:x64-windows: 8.544 min
Building qt5-tools[core,gui]:x64-windows...
Elapsed time to handle qt5-tools:x64-windows: 2.173 min

What we want:
Building qt5-base[core]:x64-windows...
Elapsed time to handle qt5-base:x64-windows: < 1 min
Building qt5-tools[core]:x64-windows...
Elapsed time to handle qt5-tools:x64-windows: < 1 min

@BillyONeal Any news? :)

I believe @ras0219-msft still had backcompat concerns about this but not positive...

Would be nice if he could write them down so that they can be discussed. I would say that the benefits (greatly reduced build times) overwhelms the disadvantages (maybe backcompat(?)).
Disabling default features for the host triplet is currently impossible if you want default features for the target triplet and sometimes default = target triplet.

@ras0219-msft @JavierMatosD @valeriaconde @vicroms @AugP and I discussed this today and have the following observations. (Most of these were originally pointed out by @ras0219-msft )

We do not think we can accept this in this form because it breaks our "no manifest changes means no changes" contract with customers.
One option would be to add a vcpkg-minimum-required or similar which would let a manifest opt in to this. The problem is this carries legacy forward forever.
Another option would be to add a separate host-default-features field which would let new versions of ports make the right choice. The unfortunate thing here is that it should almost always be the empty set. Or it could be a bool that just says it's the empty set.
Combine (2) and (3) (the vcpkg-minimum-required goes in the port rather than the consuming manifest)
Another option would be to deprecate most uses of default features. (Maybe the cure is worse than the disease) @vicroms specifically points out that this will likely not be popular among less experienced users of vcpkg.

@vicroms (and others) do not want (5)
@ras0219-msft @vicroms and I really like (4)
Since most of us preferred that that's as a dev team what we would ask for, but we are now concerned that we need to sell this stuff internally since it's a meaningful product feature change.
(I'm not leaving you with action items because we didn't agree on any that we could conclusively tell you would be a path forward)

We do not think we can accept this in this form because it breaks our "no manifest changes means no changes" contract with customers.

You could declare it as a bug fix like https://github.com/microsoft/vcpkg-tool/pull/292. Joke aside I guess this is only relevant if you use a builtin-baseline, otherwise there are already changes without changes to the manifest when you change for example the vcpkg-repo commit.
So what about comparing the builtin-baseline against a commit from which the new policy applies instead of the current one. This would be the same behavior as doing (3) as an bulk action in one commit but without (2) so that we don't carries legacy forward forever.

Or you explicitly do a major step forward, bundling several major changes. Such a defaulting to x64  on x64 windows hosts.
(I'm really tired of all these qt*webengine:host rebuilds when we only need non-gui host qt*tools.)

(I'm really tired of all these qtwebengine:host rebuilds when we only need non-gui host qttools.)

I mean we could only enable this change for the ci command


Code Quality: Updated license year

Updates the copyright year to 2025. I've also made it more concise, by removing the "See the LICENSE" sentence and included the original year of publishing as discussed in #16085.
- // Copyright (c) 2024 Files Community
- // Licensed under the MIT License. See the LICENSE.
+ // Copyright (c) 2018-2025 Files Community
+ // Licensed under the MIT License.

Thank you for your work on this. I pushed a commit to main to resolve this change.


Add tracing context in Storage Files

Added tracing context to sync and async storage file clients.
Added T with Response "overload in sync and async storage file clients.

NOTE
Flux return types to be updated in later PR's
Closes: #4474

Did you mean to add Jianghao, Josh, and I as reviewers?

Did you mean to add Jianghao, Josh, and I as reviewers?

@alzimmermsft yeah :| ..added now


tt-rss-theme-feedly: Init at 1.4.0

Motivation for this change
I use the theme every day because the default one looks crappy.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[x] Fits CONTRIBUTING.md.

@GrahamcOfBorg build tt-rss-theme-feedly

Success on x86_64-darwin (full log)
Attempted: tt-rss-theme-feedly
Partial log (click to expand)
no Makefile, doing nothing
installing
post-installation fixup
strip is /nix/store/qfxcr8c4fg7lkybrny9n2sb77bmippdx-cctools-binutils-darwin/bin/strip
patching script interpreter paths in /nix/store/6ib5w6g3rd89q5bs56l4g9v8g2p977ys-tt-rss-theme-feedly-1.4.0
/nix/store/gpy3r9ss5ngfkib8ylx7jzgahq7m0x5b-patch-shebangs.sh: line 22: warning: command substitution: ignored null byte in input
/nix/store/gpy3r9ss5ngfkib8ylx7jzgahq7m0x5b-patch-shebangs.sh: line 22: warning: command substitution: ignored null byte in input
/nix/store/gpy3r9ss5ngfkib8ylx7jzgahq7m0x5b-patch-shebangs.sh: line 22: warning: command substitution: ignored null byte in input
/nix/store/gpy3r9ss5ngfkib8ylx7jzgahq7m0x5b-patch-shebangs.sh: line 22: warning: command substitution: ignored null byte in input
/nix/store/6ib5w6g3rd89q5bs56l4g9v8g2p977ys-tt-rss-theme-feedly-1.4.0

Success on aarch64-linux (full log)
Attempted: tt-rss-theme-feedly
Partial log (click to expand)
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/ni93gb99k2yqzqldrdq4ryzz3cwwlnfk-tt-rss-theme-feedly-1.4.0
strip is /nix/store/zrs21zqcchgyabjf4xfimncdq16njizc-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/ni93gb99k2yqzqldrdq4ryzz3cwwlnfk-tt-rss-theme-feedly-1.4.0
/nix/store/gpy3r9ss5ngfkib8ylx7jzgahq7m0x5b-patch-shebangs.sh: line 22: warning: command substitution: ignored null byte in input
/nix/store/gpy3r9ss5ngfkib8ylx7jzgahq7m0x5b-patch-shebangs.sh: line 22: warning: command substitution: ignored null byte in input
/nix/store/gpy3r9ss5ngfkib8ylx7jzgahq7m0x5b-patch-shebangs.sh: line 22: warning: command substitution: ignored null byte in input
/nix/store/gpy3r9ss5ngfkib8ylx7jzgahq7m0x5b-patch-shebangs.sh: line 22: warning: command substitution: ignored null byte in input
checking for references to /build in /nix/store/ni93gb99k2yqzqldrdq4ryzz3cwwlnfk-tt-rss-theme-feedly-1.4.0...
/nix/store/ni93gb99k2yqzqldrdq4ryzz3cwwlnfk-tt-rss-theme-feedly-1.4.0

@pSub Fixed

@GrahamcOfBorg build tt-rss-theme-feedly

Success on x86_64-darwin (full log)
Attempted: tt-rss-theme-feedly
Partial log (click to expand)
no configure script, doing nothing
installing
post-installation fixup
strip is /nix/store/qfxcr8c4fg7lkybrny9n2sb77bmippdx-cctools-binutils-darwin/bin/strip
patching script interpreter paths in /nix/store/0lly2i1zjwc5582369cm707cc3qqh9cz-tt-rss-theme-feedly-1.4.0
/nix/store/gpy3r9ss5ngfkib8ylx7jzgahq7m0x5b-patch-shebangs.sh: line 22: warning: command substitution: ignored null byte in input
/nix/store/gpy3r9ss5ngfkib8ylx7jzgahq7m0x5b-patch-shebangs.sh: line 22: warning: command substitution: ignored null byte in input
/nix/store/gpy3r9ss5ngfkib8ylx7jzgahq7m0x5b-patch-shebangs.sh: line 22: warning: command substitution: ignored null byte in input
/nix/store/gpy3r9ss5ngfkib8ylx7jzgahq7m0x5b-patch-shebangs.sh: line 22: warning: command substitution: ignored null byte in input
/nix/store/0lly2i1zjwc5582369cm707cc3qqh9cz-tt-rss-theme-feedly-1.4.0

Success on x86_64-linux (full log)
Attempted: tt-rss-theme-feedly
Partial log (click to expand)
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/cj9zzj90pgv3p9gjcvxahj7s4xz9jbrp-tt-rss-theme-feedly-1.4.0
strip is /nix/store/1hi76hr87bd1y1q1qjk0lv8nmcjip1c8-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/cj9zzj90pgv3p9gjcvxahj7s4xz9jbrp-tt-rss-theme-feedly-1.4.0
/nix/store/gpy3r9ss5ngfkib8ylx7jzgahq7m0x5b-patch-shebangs.sh: line 22: warning: command substitution: ignored null byte in input
/nix/store/gpy3r9ss5ngfkib8ylx7jzgahq7m0x5b-patch-shebangs.sh: line 22: warning: command substitution: ignored null byte in input
/nix/store/gpy3r9ss5ngfkib8ylx7jzgahq7m0x5b-patch-shebangs.sh: line 22: warning: command substitution: ignored null byte in input
/nix/store/gpy3r9ss5ngfkib8ylx7jzgahq7m0x5b-patch-shebangs.sh: line 22: warning: command substitution: ignored null byte in input
checking for references to /build in /nix/store/cj9zzj90pgv3p9gjcvxahj7s4xz9jbrp-tt-rss-theme-feedly-1.4.0...
/nix/store/cj9zzj90pgv3p9gjcvxahj7s4xz9jbrp-tt-rss-theme-feedly-1.4.0

Success on aarch64-linux (full log)
Attempted: tt-rss-theme-feedly
Partial log (click to expand)
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/ki3iid143x3jwwm3q2wcjrk5g8ni7mgp-tt-rss-theme-feedly-1.4.0
strip is /nix/store/zrs21zqcchgyabjf4xfimncdq16njizc-binutils-2.30/bin/strip
patching script interpreter paths in /nix/store/ki3iid143x3jwwm3q2wcjrk5g8ni7mgp-tt-rss-theme-feedly-1.4.0
/nix/store/gpy3r9ss5ngfkib8ylx7jzgahq7m0x5b-patch-shebangs.sh: line 22: warning: command substitution: ignored null byte in input
/nix/store/gpy3r9ss5ngfkib8ylx7jzgahq7m0x5b-patch-shebangs.sh: line 22: warning: command substitution: ignored null byte in input
/nix/store/gpy3r9ss5ngfkib8ylx7jzgahq7m0x5b-patch-shebangs.sh: line 22: warning: command substitution: ignored null byte in input
/nix/store/gpy3r9ss5ngfkib8ylx7jzgahq7m0x5b-patch-shebangs.sh: line 22: warning: command substitution: ignored null byte in input
checking for references to /build in /nix/store/ki3iid143x3jwwm3q2wcjrk5g8ni7mgp-tt-rss-theme-feedly-1.4.0...
/nix/store/ki3iid143x3jwwm3q2wcjrk5g8ni7mgp-tt-rss-theme-feedly-1.4.0


added file stream endpoint

reference to https://app.clubhouse.io/terminalsystems/story/18073
Test cases not made

Hi @jsonsivar  i have made the test case which is working fine on my system but tests are failing here on these automated tests on github

Hi @jsonsivar i have made the test case which is working fine on my system but tests are failing here on these automated tests on github

I think it's this line https://github.com/FleekHQ/space-daemon/pull/232/files#diff-88f68c1abe4820637faea91d3156be54e20ff233e23f361c78d9cb7d94d12ac8R474.  Can you try path.join iirc so it handles OS path delimter.


upd20241107

Update gRPC, protobuf and protovalidate dependencies
Update buf generator settings and versions to match gRPC versions
Re-generate using pb-directory v0.31.7

The changes have already been merged in other PRs


Add missing migration to getting-started example

This migration is the effect of https://github.com/Electron100/butane/pull/158

It would be good to have https://github.com/Electron100/butane/pull/175 considered first, then I'll refresh this PR if that one is approved.  Thus avoiding revising these files multiple times.


SymmetricalComponents

Reported by ahaumer on 16 May 2016 21:10 UTC
The documentation of the functions in Electrical.MultiPhase dealing with the symmetric orientation as well as the block SymmetricalComponents in Electrical.QuasiStationary.MultiPhase needs to be enhanced, i.e. an explanation of how the phase angles of m-phase systems are distributed as well as the order of symmetrical components (positive, negative and zero sequence). Furthermore 2 components are missing:
function symmetricBackTransformationMatrix (counterpart of symmetricTransformationMatrix) and block FromSymmetricalComponents (counterpart of SymmetricalComponents).

Migrated-From: https://trac.modelica.org/Modelica/ticket/1997

Comment by ahaumer on 17 May 2016 18:50 UTC
Partly resolved by 4d12a040a9ed2b13aff757ae4764d99a9a30b7f4 (documentation, function symmetricBackTransformationMatrix, block FromSymmetricalComponents). Correctness of function symmetricBackTransformationMatrix has to be checked once more.

Comment by ahaumer on 21 May 2016 15:59 UTC
Even worse: functions symmetricOrientationMatrix and symmetricTransformationMatrix give non-zero elements where the matrix elements should be zero. Bug-fixing (MSL 3.2.2 maintenance and trunk) recommended.

Comment by ahaumer on 21 May 2016 16:44 UTC
resolved by bf41d47b4c170d45cf542a83a585167fbe00c03d, a7f7cffe2dec9d3bacb0a9c8da8afa19819d4fc7

Modified by ahaumer on 21 May 2016 16:44 UTC

Comment by ahaumer on 22 May 2016 12:59 UTC
Following the argumentation in this ticket, the zero inductor in component Modelica.Magnetic.FundamentalWave.BasicMachines.Components.SymmetricMultiPhaseWinding should be present only in case if mBase>2.
Additionally, shouldn't a multiStar be used (I'm not 100% sure)? in Modelica.Magnetic.FundamentalWave.BasicMachines.Components.SymmetricMultiPhaseCageWinding

Comment by ahaumer on 22 May 2016 13:01 UTC
fixed by 60ea8119ac031e7604db642cd409ecee92e97b74, f7c31ff9ac44422789481c777f017ff7d32c0c53

Modified by beutlich on 16 Aug 2016 09:17 UTC

Comment by beutlich on 16 Aug 2016 09:17 UTC
Toni, please fill the release notes field. Thanks.


Fix old registry test

The web console issue isn't actually a problem with the console itself, more that the test of wether the old registry file exists is failing (indicating it can't find the file)
Originally posted by @stewmcl in https://github.com/ibm-messaging/mq-ansible/issues/8#issuecomment-1429365209

closed by  Removed redundant test #21


feat: add endpoint to export saved queries using new format

SUMMARY

In line with https://github.com/apache/incubator-superset/issues/11167, this PR adds a new endpoint to export saved queries in the zipped YAML format. This will allow users to easily migrate saved queries between Superset instances.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

N/A
TEST PLAN

Added unit tests for the API and for the new ExportSavedQueriesCommand.
ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Changes UI
[ ] Requires DB Migration.
[ ] Confirm DB Migration upgrade and downgrade tested.
[X] Introduces new feature or API
[ ] Removes existing feature or API

A rebase should fix the broken E2E test (see #11453)

Codecov Report

Merging #11447 into master will decrease coverage by 1.27%.
The diff coverage is 97.14%.

@@            Coverage Diff             @@
##           master   #11447      +/-   ##
==========================================
- Coverage   62.26%   60.99%   -1.28%
==========================================
  Files         863      418     -445
  Lines       40978    26018   -14960
  Branches     3693        0    -3693
==========================================
- Hits        25516    15869    -9647
+ Misses      15283    10149    -5134
+ Partials      179        0     -179

Flag
Coverage Δ

#javascript
?

#python
60.99% <97.14%> (-0.93%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

superset/charts/api.py
76.98% <ø> (ø)

superset/databases/api.py
83.49% <ø> (-7.08%)
:arrow_down:

superset/models/sql_lab.py
90.97% <ø> (ø)

superset/queries/saved_queries/commands/export.py
95.34% <95.34%> (ø)

superset/datasets/api.py
92.77% <100.00%> (+0.04%)
:arrow_up:

superset/queries/saved_queries/api.py
97.43% <100.00%> (+1.13%)
:arrow_up:

superset/queries/saved_queries/schemas.py
100.00% <100.00%> (ø)

superset/db_engines/hive.py
0.00% <0.00%> (-85.72%)
:arrow_down:

superset/databases/commands/create.py
31.91% <0.00%> (-59.58%)
:arrow_down:

superset/db_engine_specs/hive.py
53.90% <0.00%> (-30.08%)
:arrow_down:

... and 471 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b2636f0...7f8174a. Read the comment docs.


Parse aggr functions from tremor-script/docs

This removes the dependency on the old tremor-www-docs repo.
It depends on a new release of tremor-script containing the docs folder, which will be 0.12 until then this shouldnt be merged.

I think this changed with troy and is no longer relevant (in this form)?

Honestly, I don't know. Need to check.

aggregate functions are now done via a "fake" librarary module


Torrent filename got truncated for some titles? [BUG]

Bug/Issue
I ran v1.20 today which worked but some of the downloaded torrents seemed to have there filenames truncated causing the file to lose the file extension too? I'm assuming a string limit must be at play somewhere?
This title was fine:
[Yameii] DEAD DEAD DEMONS DEDEDEDE DESTRUCTION - S01E09 [English Dub] [CR WEB-DL 1080p] [4CD3F265]
But this got truncated:
[Yameii] BARTENDER Glass of God - S01E01 [English Dub] [CR WEB-DL 1080p] [8B740E5E] (Bartender: Kami no Glass)
Debug Log Message
The logs didn't show any problems,
Additional Information
No response
Code Changes
As mentioned I'm assuming the variable used for the writing of the fetched torrent may have a limit wherever it's set, also I did wonder if the torrents actual filename could be better to use as it seems the codes using the torrents title for the filename rather than the  .torrents actual download filename which from what I could see was a lower character count than the title? (to see what I mean manually download the torrent file from the url above to see the default filename, it drops the bracketed text from the downloads filename so isn't as long as the title text)

The character limit would definitely interfere with the longer named series, I had not thought of that.
I could implement something that uses a simple format such as, '[Uploader] Series Name - S01E01'?
The video quality could also be included into the file name, but that may interfere with the long titles.

If you look at non rss nyaa torrent pages the download torrent filename seems shorter than the title on the page itself, could that filename variable be used? I've yet to experiment myself yet to see if it reduces the character count enough, but I'm assuming it's shorter for a reason, I'm thinking it could work?

If you look at non rss nyaa torrent pages the download torrent filename seems shorter than the title on the page itself, could that filename variable be used? I've yet to experiment myself yet to see if it reduces the character count enough, but I'm assuming it's shorter for a reason, I'm thinking it could work?

It looks like the Nyaa website removes the encoding info from the title when you download a torrent file directly. I could implement the same thing where the files are named as [Username] Series - S01E01.torrent.
Is that what you're thinking?


xls data cell format

I have the impression that for xls import the data cell format must be a string.  I get import failures when the cell input type is for instance numeric.

@bmihelac I'm having an issue with the format as well. Here is my model with two charfields:
class Coupon(models.Model):
    office_code = models.CharField(
        max_length=10,
        unique=True
        )
    online_code = models.CharField(
        max_length=10,
        unique=True
        )

    class Meta:
        verbose_name = 'Coupon Pair'
        verbose_name_plural = 'Coupon Pairs'

    def __unicode__(self):
        return unicode(self.id)

And, the admin:
class CouponResource(resources.ModelResource):

    class Meta:
        model = Coupon
        exclude = ('id', )
        import_id_fields = ['office_code', 'online_code', ]
        skip_unchanged = True

class CouponAdmin(ImportExportModelAdmin):

    resource_class = CouponResource

    class Meta:
        model = Coupon
        list_per_page = 20

On a test import, if a column, say office_code, only contains numbers, the importer will add a '.0' on the end of each. In the excel file, I've tried using the format tool to format as text, but this didn't help. How do I go about saving without adding the periods and zeros?
Thanks a lot for the helpful plugin!

@bmihelac any updates on this? I'm getting ValueError: invalid literal for int() with base 10: '' as well:
Traceback (most recent call last):
File "/Users/sassanharadji/.virtualenvs/mhelli/lib/python3.5/site-packages/import_export/resources.py", line 408, in import_row
instance, new = self.get_or_init_instance(instance_loader, row)
File "/Users/sassanharadji/.virtualenvs/mhelli/lib/python3.5/site-packages/import_export/resources.py", line 232, in get_or_init_instance
instance = self.get_instance(instance_loader, row)
File "/Users/sassanharadji/.virtualenvs/mhelli/lib/python3.5/site-packages/import_export/resources.py", line 226, in get_instance
return instance_loader.get_instance(row)
File "/Users/sassanharadji/.virtualenvs/mhelli/lib/python3.5/site-packages/import_export/instance_loaders.py", line 33, in get_instance
return self.get_queryset().get(**params)
File "/Users/sassanharadji/.virtualenvs/mhelli/lib/python3.5/site-packages/django/db/models/query.py", line 378, in get
clone = self.filter(*args, **kwargs)
File "/Users/sassanharadji/.virtualenvs/mhelli/lib/python3.5/site-packages/django/db/models/query.py", line 790, in filter
return self._filter_or_exclude(False, *args, **kwargs)
File "/Users/sassanharadji/.virtualenvs/mhelli/lib/python3.5/site-packages/django/db/models/query.py", line 808, in _filter_or_exclude
clone.query.add_q(Q(*args, **kwargs))
File "/Users/sassanharadji/.virtualenvs/mhelli/lib/python3.5/site-packages/django/db/models/sql/query.py", line 1243, in add_q
clause, _ = self._add_q(q_object, self.used_aliases)
File "/Users/sassanharadji/.virtualenvs/mhelli/lib/python3.5/site-packages/django/db/models/sql/query.py", line 1269, in _add_q
allow_joins=allow_joins, split_subq=split_subq,
File "/Users/sassanharadji/.virtualenvs/mhelli/lib/python3.5/site-packages/django/db/models/sql/query.py", line 1203, in build_filter
condition = self.build_lookup(lookups, col, value)
File "/Users/sassanharadji/.virtualenvs/mhelli/lib/python3.5/site-packages/django/db/models/sql/query.py", line 1099, in build_lookup
return final_lookup(lhs, rhs)
File "/Users/sassanharadji/.virtualenvs/mhelli/lib/python3.5/site-packages/django/db/models/lookups.py", line 19, in __init__
self.rhs = self.get_prep_lookup()
File "/Users/sassanharadji/.virtualenvs/mhelli/lib/python3.5/site-packages/django/db/models/lookups.py", line 57, in get_prep_lookup
return self.lhs.output_field.get_prep_lookup(self.lookup_name, self.rhs)
File "/Users/sassanharadji/.virtualenvs/mhelli/lib/python3.5/site-packages/django/db/models/fields/__init__.py", line 744, in get_prep_lookup
return self.get_prep_value(value)
File "/Users/sassanharadji/.virtualenvs/mhelli/lib/python3.5/site-packages/django/db/models/fields/__init__.py", line 976, in get_prep_value
return int(value)
ValueError: invalid literal for int() with base 10: ''

It doesn't happen with xlsx, just happens with xls.

I fixed it with overriding clean method of CharWidget:
class CharWidget(widgets.CharWidget):

    def clean(self, value, row=None, *args, **kwargs):
        if isinstance(value, float):
            value = int(value)
            return "{}".format(value)
        return value

@walterliang Did you try the fix that was submitted by @bcanyelles ?


Issue with SagePay Forms

Sorry I am unable to understand your documentation.  I have setup a function with the following code:
$gateway = OmniPay::create('SagePay\Form')->initialize([
            'vendor' => 'OurVendorID',
            'testMode' => true,
            'encryptionKey' => 'OurEncryptionKey',
        ]);
        $response = $gateway->purchase([
            'amount' => '99.99',
            'currency' => 'GBP',
            'transactionId' => 'FC-TEST01',
            'description' => 'Pizzas for everyone at PHPNE',
            'returnUrl' => 'https://example.com/success',
            'failureUrl' => 'https://example.com/failure'
        ]);

        $result = $gateway->completePurchase()->send();

This just results in an error about crypt:
Missing or invalid "crypt" parameter
Instead I then use the following from your documentation:
$crypt = $_GET['crypt']; // or supplied by your framework
$result = $gateway->completeAuthorize(['crypt' => $crypt])->send();

Again this gives an error relating to Crypt:
Undefined index: crypt
So then I try switching in code found in the test functions:
$crypt = '@BE1508740B97BEC235E9E8474168E3D4DEAD71C1395CDAC1A24F4784A71135AE797F5451F48EB2EAF5669EB3F6953F466FC3A03E5815E607C4DD18C03D7BDD61D176DB4DF131DC0F02DFC5145FFBA841651641BDAA405E72F4EF8849C2B2FD1F08763E3E66EAF3C479429A92014C7B8316F3B446BA09D28A821EE81C243E3DDD6F4C1F41D6ADEAC74D42B221645ADCC69E2F22ECDCAA010F63CCD02EAF0CE20F98439DAC7C31E528A7574656191150C1CC6CCD9FBEFCBFC9B34029FAB8F62DBC1C1618628FF0529B4E66B8AB857A79CC2FD0F14299A3505D22F964322755E6190EDD5BD42066D88154F950585236B6A2951D28BDA474E3FB17638DAA2F6304EAEA3AB9513DD0604D447000208D55DA4FDF544BEE00B5744170C1FC1E6DC6AADA07BEF6EB1FA46C14B99C3371491816A5C2CF7E03EDF6D58142767F7550DFFFE634FA56532605768FFEC1B20BEC32177816DDAB804149E9A301E495F11568F58896E90B1AC5776C2F12CE578955292F640A1E81213AAAA7856CD622FC241C65AC08215B94F933FD47050E0CFB2BFA85D7570A9D401B2366001AA377C50825B6D3893BDEC46D87F9121EC39DBF948F69399B8E842635556C7BC08E2E0C85CFA3EA2F9E4582ABDB3EB9668C83EDC34A5862F3CBC1A10935A189493D42D5C3FE4AF2BFF29B8464934B221B5A56B407F4E638B6766E6C706996A0252F2DDB30AFC8C1DA65F89987F4A9AA317BB104ED42F749DE3A43C39C515B67B3EA619D4092843DA551EEFC22CF672F53F753DEB93F1E8487BFE89C0100AFB9799238B24E523B0D6D53F824DEC0F6897F97FA507648D13276841B3B2E98030CBCD8E04F5E2C06C57BCC6F6C346440E620718F8877FFE969';

$result = $gateway->completeAuthorize(['crypt' => $crypt])->send();

This then results in a blank page.
I fee the documentation is missing some vital parts to get this working.  Could you provide me with a basic example function that will redirect to SagePay forms?

On that last section, what does $result contain? Is it the result of the transaction? You may be getting an exception thrown if the $crypt value is not valid for your encryption key.
Okay, the process is:

That first section of code you posted will create a $result that is a redirect object. Use it to redirect to Sage Pay. The redirect will be a POST, and a very quick and dirty way of doing that will be to use $result->redirect(); This will output a self-POSTing form and send the user off to Sage Pay to enter their card details.
The next part of the code is to handle the result when the user returns from Sage Pay. It will be in a different endpoint on your site. Set up the $gateway object in the same way as before then execute the code to check the result. When the user returns from Sage Pay the URL you supplied as the return URL will have the crypt parameter. Just issue $result = $gateway->completeAuthorize()->send(); as a minimum and the gateway driver will attempt to find the crypt parameter for you. You can pass it in as a parameter just in case you have a framework that gets in the way, but normally it should work.

send() in step 1 does not actually send anything remotely, but it gives you can object with all the details of the redirect that you need to execute to send the user off.
send() in step 2 does not actually send anything either, but reads the payment result from the current URL and decodes it to give you the result of the payment.
The user flow is not explained fully, I agree, so it should be read in conjunction with the description of the flow in the Sage Pay documentation. I hope the above helps.

Another way to do the redirect could be a bit of PHP code like this, which will give the user a button to press to go to the Sage Pay site:
echo '<form method="'.$result->getRedirectMethod().'" action="'.$result->getRedirectUrl().'" accept-charset="UTF-8">';
foreach ($result->getRedirectData() as $name => $value) {
    echo '<input type="hidden" name="'.$name.'" value="'.htmlspecialchars($value).'" />';
}
echo '
<button type="submit">Go To Sage Pay</button>
';
echo '</form>';

The above form could POST to an iframe if you like. It's completely your choice, and Omnipay does not try to limit how you can set up your front end.

Thanks for your response, I believe I might be having an issue with the creation of the 'crypt'.
I have tried to add:
$response->redirect();
But this gives me the error:
'This response does not support redirection.'
How do you go about generating the crypt value?  I did find the following code:
$crypt = openssl_encrypt($response, 'aes-128-cbc', $key, OPENSSL_RAW_DATA, $key);
$key being my encryption key but I the error is along the lines of the $response must be a string.
Sorry about this, payment methods confuse me to no end!

I have the out put of my result here, any thoughts on where I am going wrong? Thanks:
Omnipay\SagePay\Message\Response Object ( [request:protected] => Omnipay\SagePay\Message\Form\CompletePurchaseRequest Object ( [service:protected] => [VPSProtocol:protected] => 3.00 [liveEndpoint:protected] => https://live.sagepay.com/gateway/service [testEndpoint:protected] => https://test.sagepay.com/gateway/service [parameters:protected] => Symfony\Component\HttpFoundation\ParameterBag Object ( [parameters:protected] => Array ( [vendor] => REDACTED [testMode] => [referrerId] => [language] => [useOldBasketFormat] => [exitOnResponse] => [apply3DSecure] => [useAuthenticate] => [accountType] => [encryptionKey] => REDACTED [cryptx] => @BE1508740B97BEC235E9E8474168E3D4DEAD71C1395CDAC1A24F4784A71135AE797F5451F48EB2EAF5669EB3F6953F466FC3A03E5815E607C4DD18C03D7BDD61D176DB4DF131DC0F02DFC5145FFBA841651641BDAA405E72F4EF8849C2B2FD1F08763E3E66EAF3C479429A92014C7B8316F3B446BA09D28A821EE81C243E3DDD6F4C1F41D6ADEAC74D42B221645ADCC69E2F22ECDCAA010F63CCD02EAF0CE20F98439DAC7C31E528A7574656191150C1CC6CCD9FBEFCBFC9B34029FAB8F62DBC1C1618628FF0529B4E66B8AB857A79CC2FD0F14299A3505D22F964322755E6190EDD5BD42066D88154F950585236B6A2951D28BDA474E3FB17638DAA2F6304EAEA3AB9513DD0604D447000208D55DA4FDF544BEE00B5744170C1FC1E6DC6AADA07BEF6EB1FA46C14B99C3371491816A5C2CF7E03EDF6D58142767F7550DFFFE634FA56532605768FFEC1B20BEC32177816DDAB804149E9A301E495F11568F58896E90B1AC5776C2F12CE578955292F640A1E81213AAAA7856CD622FC241C65AC08215B94F933FD47050E0CFB2BFA85D7570A9D401B2366001AA377C50825B6D3893BDEC46D87F9121EC39DBF948F69399B8E842635556C7BC08E2E0C85CFA3EA2F9E4582ABDB3EB9668C83EDC34A5862F3CBC1A10935A189493D42D5C3FE4AF2BFF29B8464934B221B5A56B407F4E638B6766E6C706996A0252F2DDB30AFC8C1DA65F89987F4A9AA317BB104ED42F749DE3A43C39C515B67B3EA619D4092843DA551EEFC22CF672F53F753DEB93F1E8487BFE89C0100AFB9799238B24E523B0D6D53F824DEC0F6897F97FA507648D13276841B3B2E98030CBCD8E04F5E2C06C57BCC6F6C346440E620718F8877FFE969 ) ) [httpClient:protected] => Omnipay\Common\Http\Client Object ( [httpClient:Omnipay\Common\Http\Client:private] => Http\Adapter\Guzzle6\Client Object ( [client:Http\Adapter\Guzzle6\Client:private] => GuzzleHttp\Client Object ( [config:GuzzleHttp\Client:private] => Array ( [handler] => GuzzleHttp\HandlerStack Object ( [handler:GuzzleHttp\HandlerStack:private] => Closure Object ( [static] => Array ( [default] => Closure Object ( [static] => Array ( [default] => GuzzleHttp\Handler\CurlMultiHandler Object ( [factory:GuzzleHttp\Handler\CurlMultiHandler:private] => GuzzleHttp\Handler\CurlFactory Object ( [handles:GuzzleHttp\Handler\CurlFactory:private] => Array ( ) [maxHandles:GuzzleHttp\Handler\CurlFactory:private] => 50 ) [selectTimeout:GuzzleHttp\Handler\CurlMultiHandler:private] => 1 [active:GuzzleHttp\Handler\CurlMultiHandler:private] => [handles:GuzzleHttp\Handler\CurlMultiHandler:private] => Array ( ) [delays:GuzzleHttp\Handler\CurlMultiHandler:private] => Array ( ) ) [sync] => GuzzleHttp\Handler\CurlHandler Object ( [factory:GuzzleHttp\Handler\CurlHandler:private] => GuzzleHttp\Handler\CurlFactory Object ( [handles:GuzzleHttp\Handler\CurlFactory:private] => Array ( ) [maxHandles:GuzzleHttp\Handler\CurlFactory:private] => 3 ) ) ) [parameter] => Array ( [$request] => [$options] => ) ) [streaming] => GuzzleHttp\Handler\StreamHandler Object ( [lastHeaders:GuzzleHttp\Handler\StreamHandler:private] => Array ( ) ) ) [parameter] => Array ( [$request] => [$options] => ) ) [stack:GuzzleHttp\HandlerStack:private] => Array ( [0] => Array ( [0] => Closure Object ( [parameter] => Array ( [$handler] => ) ) [1] => prepare_body ) ) [cached:GuzzleHttp\HandlerStack:private] => ) [allow_redirects] => Array ( [max] => 5 [protocols] => Array ( [0] => http [1] => https ) [strict] => [referer] => [track_redirects] => ) [http_errors] => 1 [decode_content] => 1 [verify] => 1 [cookies] => [headers] => Array ( [User-Agent] => GuzzleHttp/6.3.3 curl/7.60.0 PHP/7.1.20 ) ) ) ) [requestFactory:Omnipay\Common\Http\Client:private] => Http\Message\MessageFactory\GuzzleMessageFactory Object ( ) ) [httpRequest:protected] => Symfony\Component\HttpFoundation\Request Object ( [attributes] => Symfony\Component\HttpFoundation\ParameterBag Object ( [parameters:protected] => Array ( ) ) [request] => Symfony\Component\HttpFoundation\ParameterBag Object ( [parameters:protected] => Array ( ) ) [query] => Symfony\Component\HttpFoundation\ParameterBag Object ( [parameters:protected] => Array ( ) ) [server] => Symfony\Component\HttpFoundation\ServerBag Object ( [parameters:protected] => Array ( [REDIRECT_STATUS] => 200 [HTTP_HOST] => dev.1stchoice.lan [HTTP_CONNECTION] => keep-alive [HTTP_CACHE_CONTROL] => max-age=0 [HTTP_UPGRADE_INSECURE_REQUESTS] => 1 [HTTP_USER_AGENT] => Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36 [HTTP_DNT] => 1 [HTTP_ACCEPT] => text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8 [HTTP_REFERER] => http://dev.1stchoice.lan/sagecheckout/confirm [HTTP_ACCEPT_ENCODING] => gzip, deflate [HTTP_ACCEPT_LANGUAGE] => en-US,en;q=0.9 [HTTP_COOKIE] => _ga=GA1.2.985968222.1545147037; cookieconsent_status=dismiss; remember_web_59ba36addc2b2f9401580f014c7f58ea4e30989d=eyJpdiI6Iit4Y0JyVjVYNGpHQUxIb01ZbzR6Smc9PSIsInZhbHVlIjoibjFEeEZQSmZveHVwXC9DdGNXaTVaNHI2eFJscGNXdkxDdlJUaVVXYXRYYUw4ditKQ3d0VUxtUWo5ZmdmRzg0V1JYMU0rc3BLYVRzaUFqbHVvRit5ZVV4blQxa09oM2RZdXRpdXpcL1ZwalVFOD0iLCJtYWMiOiJlMGJkNTI2ODlmN2IzYzcxNGNlN2U2MWZjMGU1OTNhNDczODIyM2U3ODY0MDA3YzkzMWQxYTQ1NTNkYTMwOTkwIn0%3D; _gid=GA1.2.1755130210.1553515611; _gat=1; 1stchoice_session=eyJpdiI6InJFS3U5NUNIUFMwbm8wTnkzM1ZKZnc9PSIsInZhbHVlIjoiSWhOOGFCS3R3NVwvVTlMSUFSWTVaMUthc05UUXVJYTVvdnR1dGc3V2xcL2tzcWJaVWN6ekFMb3BmSzBcL2x0TzQzVyIsIm1hYyI6ImVhNTA1ZDlmY2ZlMzIyMDRkMjdhYTRlZjhjN2JhMzE1NjIwOGQyOGFmY2I4NDYxYzExOTk2ZTQ0ZjIzOTQ1OTQifQ%3D%3D [PATH] => /usr/bin:/bin:/usr/sbin:/sbin [SERVER_SIGNATURE] => [SERVER_SOFTWARE] => Apache/2.2.34 (Unix) mod_wsgi/3.5 Python/2.7.13 PHP/7.1.20 mod_ssl/2.2.34 OpenSSL/1.0.2o DAV/2 mod_fastcgi/mod_fastcgi-SNAP-0910052141 mod_perl/2.0.9 Perl/v5.24.0 [SERVER_NAME] => dev.1stchoice.lan [SERVER_ADDR] => <IP_ADDRESS> [SERVER_PORT] => 80 [REMOTE_ADDR] => <IP_ADDRESS> [DOCUMENT_ROOT] => /Applications/MAMP/htdocs/1stchoice/staging/public_html [SERVER_ADMIN] =><EMAIL_ADDRESS>[SCRIPT_FILENAME] => /Applications/MAMP/htdocs/1stchoice/staging/public_html/index.php [REMOTE_PORT] => 64998 [REDIRECT_URL] => /sagepay/create/FC-569882E [GATEWAY_INTERFACE] => CGI/1.1 [SERVER_PROTOCOL] => HTTP/1.1 [REQUEST_METHOD] => GET [QUERY_STRING] => [REQUEST_URI] => /sagepay/create/FC-569882E [SCRIPT_NAME] => /index.php [PHP_SELF] => /index.php [REQUEST_TIME_FLOAT] =><PHONE_NUMBER>.31 [REQUEST_TIME] =><PHONE_NUMBER> [argv] => Array ( ) [argc] => 0 ) ) [files] => Symfony\Component\HttpFoundation\FileBag Object ( [parameters:protected] => Array ( ) ) [cookies] => Symfony\Component\HttpFoundation\ParameterBag Object ( [parameters:protected] => Array ( [_ga] => GA1.2.985968222.1545147037 [cookieconsent_status] => dismiss [remember_web_59ba36addc2b2f9401580f014c7f58ea4e30989d] => eyJpdiI6Iit4Y0JyVjVYNGpHQUxIb01ZbzR6Smc9PSIsInZhbHVlIjoibjFEeEZQSmZveHVwXC9DdGNXaTVaNHI2eFJscGNXdkxDdlJUaVVXYXRYYUw4ditKQ3d0VUxtUWo5ZmdmRzg0V1JYMU0rc3BLYVRzaUFqbHVvRit5ZVV4blQxa09oM2RZdXRpdXpcL1ZwalVFOD0iLCJtYWMiOiJlMGJkNTI2ODlmN2IzYzcxNGNlN2U2MWZjMGU1OTNhNDczODIyM2U3ODY0MDA3YzkzMWQxYTQ1NTNkYTMwOTkwIn0= [_gid] => GA1.2.1755130210.1553515611 [_gat] => 1 [1stchoice_session] => eyJpdiI6InJFS3U5NUNIUFMwbm8wTnkzM1ZKZnc9PSIsInZhbHVlIjoiSWhOOGFCS3R3NVwvVTlMSUFSWTVaMUthc05UUXVJYTVvdnR1dGc3V2xcL2tzcWJaVWN6ekFMb3BmSzBcL2x0TzQzVyIsIm1hYyI6ImVhNTA1ZDlmY2ZlMzIyMDRkMjdhYTRlZjhjN2JhMzE1NjIwOGQyOGFmY2I4NDYxYzExOTk2ZTQ0ZjIzOTQ1OTQifQ== ) ) [headers] => Symfony\Component\HttpFoundation\HeaderBag Object ( [headers:protected] => Array ( [host] => Array ( [0] => dev.1stchoice.lan ) [connection] => Array ( [0] => keep-alive ) [cache-control] => Array ( [0] => max-age=0 ) [upgrade-insecure-requests] => Array ( [0] => 1 ) [user-agent] => Array ( [0] => Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36 ) [dnt] => Array ( [0] => 1 ) [accept] => Array ( [0] => text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8 ) [referer] => Array ( [0] => http://dev.1stchoice.lan/sagecheckout/confirm ) [accept-encoding] => Array ( [0] => gzip, deflate ) [accept-language] => Array ( [0] => en-US,en;q=0.9 ) [cookie] => Array ( [0] => _ga=GA1.2.985968222.1545147037; cookieconsent_status=dismiss; remember_web_59ba36addc2b2f9401580f014c7f58ea4e30989d=eyJpdiI6Iit4Y0JyVjVYNGpHQUxIb01ZbzR6Smc9PSIsInZhbHVlIjoibjFEeEZQSmZveHVwXC9DdGNXaTVaNHI2eFJscGNXdkxDdlJUaVVXYXRYYUw4ditKQ3d0VUxtUWo5ZmdmRzg0V1JYMU0rc3BLYVRzaUFqbHVvRit5ZVV4blQxa09oM2RZdXRpdXpcL1ZwalVFOD0iLCJtYWMiOiJlMGJkNTI2ODlmN2IzYzcxNGNlN2U2MWZjMGU1OTNhNDczODIyM2U3ODY0MDA3YzkzMWQxYTQ1NTNkYTMwOTkwIn0%3D; _gid=GA1.2.1755130210.1553515611; _gat=1; 1stchoice_session=eyJpdiI6InJFS3U5NUNIUFMwbm8wTnkzM1ZKZnc9PSIsInZhbHVlIjoiSWhOOGFCS3R3NVwvVTlMSUFSWTVaMUthc05UUXVJYTVvdnR1dGc3V2xcL2tzcWJaVWN6ekFMb3BmSzBcL2x0TzQzVyIsIm1hYyI6ImVhNTA1ZDlmY2ZlMzIyMDRkMjdhYTRlZjhjN2JhMzE1NjIwOGQyOGFmY2I4NDYxYzExOTk2ZTQ0ZjIzOTQ1OTQifQ%3D%3D ) ) [cacheControl:protected] => Array ( [max-age] => 0 ) ) [content:protected] => [languages:protected] => [charsets:protected] => [encodings:protected] => [acceptableContentTypes:protected] => [pathInfo:protected] => [requestUri:protected] => [baseUrl:protected] => [basePath:protected] => [method:protected] => [format:protected] => [session:protected] => [locale:protected] => [defaultLocale:protected] => en [isHostValid:Symfony\Component\HttpFoundation\Request:private] => 1 [isForwardedValid:Symfony\Component\HttpFoundation\Request:private] => 1 ) [response:protected] => Omnipay\SagePay\Message\Response Object *RECURSION* [currencies:protected] => [zeroAmountAllowed:protected] => 1 [negativeAmountAllowed:protected] => ) [data:protected] => Array ( ) )

Have you got the purpose of steps 1 and 2 that I posted earlier? What I'm seeing here looks like they are mushed together. Only step 1 creates a redirect object. You need to focus on that first - setting up the data, then redirecting the user to Sage Pay. The completeAuthorize() is something you do when the user returns to your success URL.
Are you getting the initial redirect?

Here, play with this, it runs the code in the documentation:
https://omnipay.acadweb.co.uk/sagepay3x/test3.php

Ah! I see what you mean.  I think I was misunderstanding there.  I will have a play and get back to you, cheers, had a moment of clarity now.

Needs a diagram really. This may help just to set the picture: https://www.sagepay.co.uk/support/12/36/sage-pay-form In that diagram, $gateway->purchase() sets up an object to handle flow "2", and $getway->completeAuthorize() handles what comes back in flow "3".

Thanks very much :) I'll have another crack at it tonight and see what I can get working.

Thanks I think I have the process clear in my head now.  I am coming up with this error though:
LOG.error: Call to undefined method Omnipay\SagePay\Message\Form\PurchaseRequest::redirect()
I must not have included something correctly although I do have:
use Omnipay\Omnipay;
At the top of my controller.  Any idea what I have missed out?

redirect() is in the purchase response, not the request.
Remember $response = $request->send() where $request = $gateway->purchase().

Thanks! I have it redirecting to sagepay as expected.  Now just got to get my success function working.
Thanks very much for all your help :)

No problem. Please close this issue when you are ready, and feel free to contribute to the documentation, even if it is just one line - everything helps.

Closing as answered.


Error When Enabling Nomadnet Node on Mx Linux

Describe the Bug
On MXLinux, when enabling the node feature in the config of nomadnet (enabling nomadnet pages), nomadnet closes with the following error:
Traceback (most recent call last):
  File "/home/frk/.local/bin/nomadnet", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/frk/.local/lib/python3.11/site-packages/nomadnet/nomadnet.py", line 51, in main
    program_setup(configarg, rnsconfigarg, daemon, console)
  File "/home/frk/.local/lib/python3.11/site-packages/nomadnet/nomadnet.py", line 11, in program_setup
    app = nomadnet.NomadNetworkApp(
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/frk/.local/lib/python3.11/site-packages/nomadnet/NomadNetworkApp.py", line 336, in __init__
    self.message_router.enable_propagation()
  File "/home/frk/.local/lib/python3.11/site-packages/LXMF/LXMRouter.py", line 466, in enable_propagation
    raise e
  File "/home/frk/.local/lib/python3.11/site-packages/LXMF/LXMRouter.py", line 437, in enable_propagation
    peer = LXMPeer.from_bytes(serialised_peer, self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/frk/.local/lib/python3.11/site-packages/LXMF/LXMPeer.py", line 42, in from_bytes
    peer = LXMPeer(router, dictionary["destination_hash"])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/frk/.local/lib/python3.11/site-packages/LXMF/LXMPeer.py", line 111, in __init__
    self.destination = RNS.Destination(self.identity, RNS.Destination.OUT, RNS.Destination.SINGLE, APP_NAME, "propagation")
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/frk/.local/lib/python3.11/site-packages/RNS/Destination.py", line 171, in __init__
    raise ValueError("Can't create outbound SINGLE destination without an identity")
ValueError: Can't create outbound SINGLE destination without an identity

It seems to be an identity error but it's not! (or maybe an error for the identity of the nomad node!)
When disabling the "enable node" in the config, nomadnet starts working again without problems, but i have only the nomad client, i want to enable node to serve pages. There is a way to fix om my machine, or is a nomadnet problem? i tested also on debian, mint, windows, but never got this error!!
On windows cannot also get pages working due to a "stty" error...windows doesn't have stty, so i think the problem is there.

OS: MX Linux 23
Python version: 3.11.2
Program version: Nomadnet 0.5.3

Thank you! Transferred to LXMF repository, as this is where the bug is located.

thank you for your great and hard work mark!!!
i just managed to resolve the problem disabling the propagation node in the config file!!!
Verbose explanation:
in the config file:
disable_propagation = False
changed into:
disable_propagation = true
the nomadnet client return to open up normally without errors,
and now with keeping node hosting active:
enable_node = yes
it works perfectly and i'm able to host pages, but losing the ability to host a propagation node, that i'm currently running on a meshchat client on the same machine.
another user found the same issue on his raspberry nomadnet installation and pointed out to me, resolving my the issue too (don't know his OS version, sorry)


Single JVM Example is broken

HDFS resource service can't find the path id for a file in a JAR

merged into develop


Running Gradle task 'assembleDebug'... Forever!

I've downloaded android studio yesterday to start studying flutter, but i'm facing a very frustrating issue.
It just doesn't start at all, and every time I create a new flutter project and try to run, it shows an error that the task "Clean" already exists, once I fix that, I see another error in the kotlin version, once I fix that, my app just don't finish the "assembleDebug" at all.
Steps to Reproduce

Create a sample flutter project
Run the app

Logs

I got confused and closed the wrong issue, actually this one is solved, and the other is active.

Hi @K-Schaeffer
could you provide your flutter run --verbose log?

I fixed it with this:

Then, I removed my Flutter install files, and downloaded the git version.
Now I'm facing another issue: https://github.com/flutter/flutter/issues/61688


Immediately evaluate reactor.observe

reactor.observe only triggers on changes.  It seems like it should trigger immediately using the current result of the getter.  Otherwise, if you don't know the present state and don't want to leak memory, you have to do something like this:
var stopObserving;

var onValue = (value) => {
  var completed = doStuffWith(value);

  if (completed)
    stopObserving();
};

onValue(reactor.evaluate(getter));
var stopObserving = reactor.observe(getter, onValue);

Instead of simply reactor.observe(getter, onValue).
This could either be corrected by default, corrected with a flag or new method (like observeImmediately), or left as is.  What do you think?

I ran into this same issue when implementing a data-binding layer  between Nuclear and VueJS.
Your solution of immediately evaluating and then observing is what I ended up doing.
In terms of leaking memory, i've found it best to not use reactor.observe many places but instead build it into a UI framework or conventionalize around some lifecycle of obsevation, ie a React Component.
By relying on observe as a way to read data from your system you lose a predictability and understandability of NuclearJS.  In fact we have built an entire frontend and only use observe outside of a component binding maybe 3 times.

The pattern you mention above @appsforartists seems like a pretty easy way to do this.
Having to evaluate before observe isn't a huge cost to incur and if we coupled evaluate with observe you could never get the behavior of "observing without first evaluating" without a flag passed to observe.
For now I am going to keep the API as is, since we are in 1.0.x