[SPARK-8000][SQL] Support for auto-detecting data sources.

What changes were proposed in this pull request?
https://issues.apache.org/jira/browse/SPARK-8000
This PR adds the support for detecting data source by extension and "magic number" in Parquet for read().load().
As I described in comments, detection follows the steps below:

Detection steps follows:

Firstly, this tries to check if one of the given paths has the extension. If it has,
it detects data source based on that.
If the path does not have an extension, then it tries to find a single leaf file from
the gien path and then it tries to find an extension. If it has, it detects data source
based on that.
If this even fails, then it tries to read Parquet "magic number" to decide.

Each test has been added for each datasource.
How was the this patch tested?
This was tested with unittests and with dev/run_tests for coding style

Test build #51552 has started for PR 11270 at commit 23ba726.

Test build #51552 has finished for PR 11270 at commit 23ba726.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51552/
Test FAILed.

retest this please

Test build #51558 has started for PR 11270 at commit 23ba726.

Test build #51558 has finished for PR 11270 at commit 23ba726.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51558/
Test FAILed.

Retest this please

retest this please

Test build #51560 has started for PR 11270 at commit 23ba726.

Test build #51560 has finished for PR 11270 at commit 23ba726.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51560/
Test FAILed.

Merged build finished. Test FAILed.

In order to avoid breaking changes (e.g. we can always read Parquet with load), maybe we want to special case handle for Parquet beyond looking at file names.
I looked at the binary protocol (see https://github.com/Parquet/parquet-format), and it looks like Parquet always start with "PAR1" in the beginning of the file. That is to say, if the first four bytes are: 0x50, 0x41, 0x52, 0x31, then it is a Parquet file.

@rxin Actually, as you know, spark.sql.sources.default can be different datasource, so I think we might have to add some logics to validate all datasources from files in this way or add nothing to avoid breaking changes.
If we go for the validation, there are several concerns.

For Parquet we might be able to use "magic number" you just said but as far as I remember there is no such thing for ORC but it just starts with index data. (Maybe for CSV and JSON we might be able to do this by reading few data from the first of files).

For reading few bytes can be simply done by reading them directly but if we need to read other stuff (for example, reading footer from ORC to validate) this will bring complexity just like this in Parquet.

Driver-side overhead would be pretty much increased because basically we need to touch each file to make sure it has a datasource of all the datasources.

Could we maybe handle this issue in different PR?

retest rhis please

retest this please

Test build #51584 has started for PR 11270 at commit 23ba726.

I'd be ok dropping spark.sql.sources.default in 2.0.
I think that is rarely used.

Also this is some best effort thing. I'd just pick one file to read and test that, rather than testing all the files.
If a folder has mixed Parquet and other files, then unfortunately an error will be thrown at runtime. I don't know how users will be able to read that anyway.

@rxin thanks! I will update soon.

Test build #51584 has finished for PR 11270 at commit 23ba726.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51584/
Test FAILed.

Merged build finished. Test FAILed.

I submitted some more commits. In summary,

Added a DataSourceDetect class separatly.
Now, it only picks a single file. If the given path is directory, it goes deep and picks a single file if the directory does not have the extension.
I did not remove sqlContext.conf.defaultDataSourceName here as I see it is referred from relatively a lot of classes (so I thought I could do this in another PR).

I submitted some more commits. In summary,

Added a DataSourceDetect class separatly.
Now, it only picks a single file. If the given path is directory, it goes deep and picks a single file if the directory does not have the extension.
I did not remove sqlContext.conf.defaultDataSourceName here as I see it is referred from relatively a lot of classes (so I thought I could do this in another PR).

I submitted some more commits. In summary,

Added a DataSourceDetect class separatly.
Now, it only picks a single file. If the given path is directory, it goes deep and picks a single file if the directory does not have the extension.
I did not remove sqlContext.conf.defaultDataSourceName here as I see it is referred from relatively a lot of classes (so I thought I could do this in another PR).

I submitted some more commits. In summary,

Added a DataSourceDetect class separatly.
Now, it only picks a single file. If the given path is directory, it goes deep and picks a single file if the directory does not have the extension.
I did not remove sqlContext.conf.defaultDataSourceName here as I see it is referred from relatively a lot of classes (so I thought I could do this in another PR).

Test build #51671 has started for PR 11270 at commit 856062a.

Test build #51671 has finished for PR 11270 at commit 856062a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51671/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51703/
Test FAILed.

retest this please

Test build #51706 has started for PR 11270 at commit dc2c454.

Test build #51706 has finished for PR 11270 at commit dc2c454.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51706/
Test FAILed.

Test build #51718 has started for PR 11270 at commit 28101a1.

Test build #51718 has finished for PR 11270 at commit 28101a1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51718/
Test FAILed.

Merged build finished. Test FAILed.

Test build #51740 has started for PR 11270 at commit cb3d4e8.

Test build #51740 has finished for PR 11270 at commit cb3d4e8.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51740/
Test FAILed.

I made some tests ignored for testing default data source and also commented some pyspark tests instead of setting unittest.skip() as they include some primitive save and load tests.
I think I should make another PR (or follow-up) to remove the default data source option if this is merged.

retest this please

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51752/
Test FAILed.

Test build #51755 has started for PR 11270 at commit 7a4c049.

Test build #51755 has finished for PR 11270 at commit 7a4c049.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51755/
Test PASSed.

Test build #51955 has started for PR 11270 at commit d00a1b5.

@rxin I found an API listFiles() which returns an iterator. So, this will not list up in any case but just trying to find a single file.
Also, I added some more tests for all the cases.

Test build #51958 has started for PR 11270 at commit 8b06436.

@rxin I found an API listFiles() which returns an iterator. So, this will not list up all files in any case but just try to find a single file.
Also, I added some more tests for all the cases.

Test build #51955 has finished for PR 11270 at commit d00a1b5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51955/
Test FAILed.

Test build #51961 has started for PR 11270 at commit d2a1ecf.

Sorry, listFiles() calls listStatus() internally. It looks there is no way to fetch a file deep without listing files.

Test build #51961 has finished for PR 11270 at commit d2a1ecf.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51961/
Test FAILed.

retest this please

Test build #51964 has started for PR 11270 at commit d2a1ecf.

Test build #51964 has finished for PR 11270 at commit d2a1ecf.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51964/
Test FAILed.

Merged build finished. Test FAILed.

Test build #51958 has finished for PR 11270 at commit 8b06436.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51958/
Test PASSed.

@rxin If you think we should not list up not even once then, should we maybe then just detect the source only by given paths without listing up and then just leave the sqlContext.conf.defaultDataSourceName option?
So, in other words,
├── iamjson.json                 # Detect success by the extension of `iamjson.json`
│   ├── part-001
│   └── part-002
├── iamjson                      # Try use `sqlContext.conf.defaultDataSourceName` and then
│   │                            # throw an exception in Parquet-side.
│   ├── part-001
│   └── part-002
├── iamparquet.parquet           # Detect success by the extension of `iamparquet.parquet`
│   ├── part-001.parquet
│   └── part-002.parquet
└── iamparquet                   # Just use `sqlContext.conf.defaultDataSourceName`
    ├── part-001.parquet
    └── part-002.parquet

I will take an action as soon as I could have some feedback for this conflict.

Let me close this for now because I could not come up with a good idea.

Sorry I didn't have much time. I will think about it too.

@HyukjinKwon, sorry to bother you about such an old issue, do you know if any such magic file detection functionality now exists in any main branch spark versions?

@HyukjinKwon does this functionality still exist in spark anywhere?

Answering my own question, I think its already located at:  DataSource.lookupDataSource

No, this isn't fixed yet. We can't auto-detect the source when we read due to the concern about the overhead of listing files.


[BUG] Java process does not get killed when data prepper itself terminates

Describe the bug
The running java process does not get killed when data prepper itself terminates. This is problematic when running data prepper inside of a docker container using kubernetes HorizontalPodAutoscaler to control data prepper. Right now data prepper will terminate for some reason however the container/pod does not terminate and will stick around as a zombie pod- holding onto resources but unable to function. I believe this is because PID 1 (the java cmd running data prepper) does not get terminated when the data-prepper process stops.
To Reproduce
Really you just need data prepper to try to use more resources than is available in a container/pod. Here is one way

Run data prepper in kubernetes
Set the pod resource limit to something low like cpu 100m to force it to run into resource issues
Configure data prepper to accept data
Fire a lot of data at data prepper really fast (One way is to build up a buffer in your source and then try to send it to data prepper all at once) What I did was force restart data prepper a few times while simultaneously generated a bunch of source data trying to send to data prepper. Then data prepper gets overwhelmed, tries to get more resources, throws an error and terminates

Sample error
2023-04-03T17:18:30,799 [pool-7-thread-83] ERROR com.amazon.dataprepper.plugins.source.otellogs.OTelLogsGrpcService - Failed to write the request of size 2380052 due to:
java.util.concurrent.TimeoutException: Pipeline [log-pipeline] - Buffer does not have enough capacity left for the size of records: 2611, timed out waiting for slots.
	at org.opensearch.dataprepper.plugins.buffer.blockingbuffer.BlockingBuffer.doWriteAll(BlockingBuffer.java:123) ~[blocking-buffer-2.1.0.jar:?]
	at org.opensearch.dataprepper.model.buffer.AbstractBuffer.writeAll(AbstractBuffer.java:100) ~[data-prepper-api-2.1.0.jar:?]
	at org.opensearch.dataprepper.plugins.MultiBufferDecorator.writeAll(MultiBufferDecorator.java:39) ~[data-prepper-core-2.1.0.jar:?]
	at com.amazon.dataprepper.plugins.source.otellogs.OTelLogsGrpcService.processRequest(OTelLogsGrpcService.java:106) ~[otel-logs-source-2.1.0.jar:?]
	at com.amazon.dataprepper.plugins.source.otellogs.OTelLogsGrpcService.lambda$export$0(OTelLogsGrpcService.java:80) ~[otel-logs-source-2.1.0.jar:?]
	at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:141) ~[micrometer-core-1.10.3.jar:1.10.3]
	at com.amazon.dataprepper.plugins.source.otellogs.OTelLogsGrpcService.export(OTelLogsGrpcService.java:80) ~[otel-logs-source-2.1.0.jar:?]
	at io.opentelemetry.proto.collector.logs.v1.LogsServiceGrpc$MethodHandlers.invoke(LogsServiceGrpc.java:246) ~[opentelemetry-proto-0.16.0-alpha.jar:0.16.0]
	at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182) ~[grpc-stub-1.49.1.jar:1.49.1]
	at com.linecorp.armeria.server.grpc.AbstractServerCall.invokeOnMessage(AbstractServerCall.java:384) ~[armeria-grpc-1.22.1.jar:?]
	at com.linecorp.armeria.server.grpc.AbstractServerCall.lambda$onRequestMessage$2(AbstractServerCall.java:348) ~[armeria-grpc-1.22.1.jar:?]
	at com.linecorp.armeria.internal.shaded.guava.util.concurrent.SequentialExecutor$1.run(SequentialExecutor.java:123) ~[armeria-1.22.1.jar:?]
	at com.linecorp.armeria.internal.shaded.guava.util.concurrent.SequentialExecutor$QueueWorker.workOnQueue(SequentialExecutor.java:235) ~[armeria-1.22.1.jar:?]
	at com.linecorp.armeria.internal.shaded.guava.util.concurrent.SequentialExecutor$QueueWorker.run(SequentialExecutor.java:180) ~[armeria-1.22.1.jar:?]
	at com.linecorp.armeria.common.RequestContext.lambda$makeContextAware$3(RequestContext.java:566) ~[armeria-1.22.1.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
2023-04-03T17:18:30,800 [log-pipeline-sink-worker-4-thread-1] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [log-pipeline] - Shutting down process workers
2023-04-03T17:19:01,806 [log-pipeline-sink-worker-4-thread-1] WARN  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [log-pipeline] - Workers did not terminate in time, forcing termination
2023-04-03T17:19:05,278 [log-pipeline-processor-worker-3-thread-1] ERROR org.opensearch.dataprepper.pipeline.common.FutureHelper - FutureTask is interrupted or timed out
2023-04-03T17:19:06,287 [log-pipeline-processor-worker-3-thread-1] INFO  org.opensearch.dataprepper.pipeline.ProcessWorker - Processor shutdown phase 1 complete.
2023-04-03T17:19:07,300 [log-pipeline-processor-worker-3-thread-1] INFO  org.opensearch.dataprepper.pipeline.ProcessWorker - Beginning processor shutdown phase 2, iterating until buffers empty.
2023-04-03T17:19:07,300 [log-pipeline-sink-worker-4-thread-1] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [log-pipeline] - Shutting down process workers
Exception in thread "HTTP-Dispatcher" Exception in thread "I/O dispatcher 1" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
2023-04-03T17:20:00,693 [log-pipeline-sink-worker-4-thread-1] WARN  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [log-pipeline] - Workers did not terminate in time, forcing termination
2023-04-03T17:20:00,693 [metrics-pipeline-processor-worker-5-thread-1] ERROR org.opensearch.dataprepper.pipeline.common.PipelineThreadPoolExecutor - Pipeline [metrics-pipeline] process worker encountered a fatal exception, cannot proceed further
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space: failed reallocation of scalar replaced objects
	at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:?]
	at java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:?]
	at org.opensearch.dataprepper.pipeline.common.PipelineThreadPoolExecutor.afterExecute(PipelineThreadPoolExecutor.java:70) ~[data-prepper-core-2.1.0.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1137) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: java.lang.OutOfMemoryError: Java heap space: failed reallocation of scalar replaced objects
2023-04-03T17:20:00,693 [metrics-pipeline-processor-worker-5-thread-1] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [metrics-pipeline] - Received shutdown signal with processor shutdown timeout PT30S and sink shutdown timeout PT30S. Initiating the shutdown process
2023-04-03T17:20:00,693 [log-pipeline-processor-worker-3-thread-1] ERROR org.opensearch.dataprepper.pipeline.ProcessWorker - Encountered exception during pipeline log-pipeline processing
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@416a9cb0[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@706b5ab7[Wrapped task = org.opensearch.dataprepper.pipeline.Pipeline$$Lambda$1099/0x00000008014068d8@6dccc3d5]] rejected from org.opensearch.dataprepper.pipeline.common.PipelineThreadPoolExecutor@6f8299f9[Shutting down, pool size = 1, active threads = 1, queued tasks = 1, completed tasks = 109]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365) ~[?:?]
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) ~[?:?]
	at org.opensearch.dataprepper.pipeline.Pipeline.lambda$publishToSinks$4(Pipeline.java:262) ~[data-prepper-core-2.1.0.jar:?]
	at org.opensearch.dataprepper.pipeline.router.DataFlowComponentRouter.route(DataFlowComponentRouter.java:45) ~[data-prepper-core-2.1.0.jar:?]
	at org.opensearch.dataprepper.pipeline.router.Router.route(Router.java:42) ~[data-prepper-core-2.1.0.jar:?]
	at org.opensearch.dataprepper.pipeline.Pipeline.publishToSinks(Pipeline.java:261) ~[data-prepper-core-2.1.0.jar:?]
	at org.opensearch.dataprepper.pipeline.ProcessWorker.postToSink(ProcessWorker.java:117) ~[data-prepper-core-2.1.0.jar:?]
	at org.opensearch.dataprepper.pipeline.ProcessWorker.doRun(ProcessWorker.java:98) ~[data-prepper-core-2.1.0.jar:?]
	at org.opensearch.dataprepper.pipeline.ProcessWorker.run(ProcessWorker.java:52) ~[data-prepper-core-2.1.0.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
2023-04-03T17:20:00,694 [metrics-pipeline-processor-worker-5-thread-1] INFO  org.opensearch.dataprepper.plugins.source.otelmetrics.OTelMetricsSource - Stopped otel_metrics_source.
2023-04-03T17:20:00,694 [metrics-pipeline-processor-worker-5-thread-1] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [metrics-pipeline] - Shutting down process workers
2023-04-03T17:20:00,694 [metrics-pipeline-processor-worker-5-thread-1] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [metrics-pipeline] - Encountered interruption terminating the pipeline execution, Attempting to force the termination
2023-04-03T17:20:00,694 [metrics-pipeline-processor-worker-5-thread-1] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [metrics-pipeline] - Shutting down process workers
2023-04-03T17:20:00,694 [pool-4-thread-1] ERROR org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1 - I/O reactor terminated abnormally
org.apache.http.nio.reactor.IOReactorException: I/O dispatch worker terminated abnormally
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:359) ~[httpcore-nio-4.4.15.jar:4.4.15]
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221) ~[httpasyncclient-4.1.5.jar:4.1.5]
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ~[httpasyncclient-4.1.5.jar:4.1.5]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: java.lang.OutOfMemoryError: Java heap space

Expected behavior
PID 1 to be terminated, and then the container/pod to terminate because PID 1 (the init cmd for the container) was terminated.
I ran this after it crashed & you can see PID 1 was not terminated
data-prepper-775db9b7b6-msg2v:/usr/share/data-prepper# ps -A
PID   USER     TIME  COMMAND
    1 root      5:23 java -Ddata-prepper.dir=/usr/share/data-prepper -Dlog4j.configurationFile=/usr/share/data-prepper/config/log4j2-rolling.properties -cp /u
  482 root      0:00 sh -c clear; (bash || ash || sh)
  489 root      0:00 bash
  490 root      0:00 ps -A

Environment (please complete the following information):

Docker
Kubernetes
Data prepper v2.1.0

Additional context
Note: data prepper does not crash when the resource cpu/mem limit is gone and I can view the resources used spike way over my preset limit if I run the same test.

It appears that we have seen a similar issue when exceptions are thrown in the OpenSearch sink.
2023-04-05T14:29:51.247 [log-pipeline-sink-worker-2-thread-2] ERROR org.opensearch.dataprepper.pipeline.common.PipelineThreadPoolExecutor - Pipeline [log-pipeline] process worker encountered a fatal exception, cannot proceed further
java.util.concurrent.ExecutionException: java.lang.NullPointerException
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:?]
at java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:?]
at org.opensearch.dataprepper.pipeline.common.PipelineThreadPoolExecutor.afterExecute(PipelineThreadPoolExecutor.java:70) ~[data-prepper-core-2.2.0-SNAPSHOT.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1129) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.lang.NullPointerException
at org.opensearch.dataprepper.plugins.sink.opensearch.BulkRetryStrategy.handleRetry(BulkRetryStrategy.java:195) ~[opensearch-2.2.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.plugins.sink.opensearch.BulkRetryStrategy.execute(BulkRetryStrategy.java:145) ~[opensearch-2.2.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink.lambda$flushBatch$6(OpenSearchSink.java:243) ~[opensearch-2.2.0-SNAPSHOT.jar:?]
at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:141) ~[micrometer-core-1.10.3.jar:1.10.3]
at org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink.flushBatch(OpenSearchSink.java:240) ~[opensearch-2.2.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink.doOutput(OpenSearchSink.java:228) ~[opensearch-2.2.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.model.sink.AbstractSink.lambda$output$0(AbstractSink.java:54) ~[data-prepper-api-2.2.0-SNAPSHOT.jar:?]
at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:141) ~[micrometer-core-1.10.3.jar:1.10.3]
at org.opensearch.dataprepper.model.sink.AbstractSink.output(AbstractSink.java:54) ~[data-prepper-api-2.2.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.pipeline.Pipeline.lambda$publishToSinks$3(Pipeline.java:286) ~[data-prepper-core-2.2.0-SNAPSHOT.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
... 2 more
2023-04-05T14:29:51.253 [log-pipeline-processor-worker-1-thread-1] ERROR org.opensearch.dataprepper.pipeline.common.FutureHelper - FutureTask failed due to:
java.util.concurrent.ExecutionException: java.lang.NullPointerException
at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:?]
at java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:?]
at org.opensearch.dataprepper.pipeline.common.FutureHelper.awaitFuturesIndefinitely(FutureHelper.java:29) ~[data-prepper-core-2.2.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.pipeline.ProcessWorker.postToSink(ProcessWorker.java:118) ~[data-prepper-core-2.2.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.pipeline.ProcessWorker.doRun(ProcessWorker.java:98) ~[data-prepper-core-2.2.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.pipeline.ProcessWorker.run(ProcessWorker.java:45) ~[data-prepper-core-2.2.0-SNAPSHOT.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.lang.NullPointerException
at org.opensearch.dataprepper.plugins.sink.opensearch.BulkRetryStrategy.handleRetry(BulkRetryStrategy.java:195) ~[opensearch-2.2.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.plugins.sink.opensearch.BulkRetryStrategy.execute(BulkRetryStrategy.java:145) ~[opensearch-2.2.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink.lambda$flushBatch$6(OpenSearchSink.java:243) ~[opensearch-2.2.0-SNAPSHOT.jar:?]
at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:141) ~[micrometer-core-1.10.3.jar:1.10.3]
at org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink.flushBatch(OpenSearchSink.java:240) ~[opensearch-2.2.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink.doOutput(OpenSearchSink.java:228) ~[opensearch-2.2.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.model.sink.AbstractSink.lambda$output$0(AbstractSink.java:54) ~[data-prepper-api-2.2.0-SNAPSHOT.jar:?]
at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:141) ~[micrometer-core-1.10.3.jar:1.10.3]
at org.opensearch.dataprepper.model.sink.AbstractSink.output(AbstractSink.java:54) ~[data-prepper-api-2.2.0-SNAPSHOT.jar:?]
at org.opensearch.dataprepper.pipeline.Pipeline.lambda$publishToSinks$3(Pipeline.java:286) ~[data-prepper-core-2.2.0-SNAPSHOT.jar:?]
... 5 more

@danweller18 , I believe this should be resolved in Data Prepper 2.2. Please take a look and let us know if this addresses your issue.
Thank you for reporting it!

@dlvenable unfortunately this solution did not resolve my issue
However I did watch as my pod got restarted 3 times before it got into a stuck state again. Before this PR, data prepper would not get restarted even 1 time. So the PR did improve the previous state.
In the picture you can see the cpu limit was set at 400mb and the pod was requesting closer to 1000mb at times.
I got a shell into the container and it is still has process pid 1 running but data prepper is currently in a vegetable state
last bit of the logs
2023-04-25T15:44:53,138 [pool-8-thread-26] ERROR org.opensearch.dataprepper.plugins.source.otellogs.OTelLogsGrpcService - Failed to write the request of size 72959 due to:
java.util.concurrent.TimeoutException: Pipeline [log-pipeline] - Buffer does not have enough capacity left for the number of records: 80, timed out waiting for slots.
	at org.opensearch.dataprepper.plugins.buffer.blockingbuffer.BlockingBuffer.doWriteAll(BlockingBuffer.java:123) ~[blocking-buffer-2.2.0.jar:?]
	at org.opensearch.dataprepper.model.buffer.AbstractBuffer.writeAll(AbstractBuffer.java:100) ~[data-prepper-api-2.2.0.jar:?]
	at org.opensearch.dataprepper.plugins.MultiBufferDecorator.writeAll(MultiBufferDecorator.java:39) ~[data-prepper-core-2.2.0.jar:?]
	at org.opensearch.dataprepper.plugins.source.otellogs.OTelLogsGrpcService.processRequest(OTelLogsGrpcService.java:106) ~[otel-logs-source-2.2.0.jar:?]
	at org.opensearch.dataprepper.plugins.source.otellogs.OTelLogsGrpcService.lambda$export$0(OTelLogsGrpcService.java:80) ~[otel-logs-source-2.2.0.jar:?]
	at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:141) ~[micrometer-core-1.10.5.jar:1.10.5]
	at org.opensearch.dataprepper.plugins.source.otellogs.OTelLogsGrpcService.export(OTelLogsGrpcService.java:80) ~[otel-logs-source-2.2.0.jar:?]
	at io.opentelemetry.proto.collector.logs.v1.LogsServiceGrpc$MethodHandlers.invoke(LogsServiceGrpc.java:246) ~[opentelemetry-proto-0.16.0-alpha.jar:0.16.0]
	at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182) ~[grpc-stub-1.49.1.jar:1.49.1]
	at com.linecorp.armeria.server.grpc.AbstractServerCall.invokeOnMessage(AbstractServerCall.java:384) ~[armeria-grpc-1.22.1.jar:?]
	at com.linecorp.armeria.server.grpc.AbstractServerCall.lambda$onRequestMessage$2(AbstractServerCall.java:348) ~[armeria-grpc-1.22.1.jar:?]
	at com.linecorp.armeria.internal.shaded.guava.util.concurrent.SequentialExecutor$1.run(SequentialExecutor.java:123) ~[armeria-1.22.1.jar:?]
	at com.linecorp.armeria.internal.shaded.guava.util.concurrent.SequentialExecutor$QueueWorker.workOnQueue(SequentialExecutor.java:235) ~[armeria-1.22.1.jar:?]
	at com.linecorp.armeria.internal.shaded.guava.util.concurrent.SequentialExecutor$QueueWorker.run(SequentialExecutor.java:180) ~[armeria-1.22.1.jar:?]
	at com.linecorp.armeria.common.RequestContext.lambda$makeContextAware$3(RequestContext.java:566) ~[armeria-1.22.1.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
2023-04-25T15:44:53,139 [log-pipeline-sink-worker-4-thread-1] INFO  org.opensearch.dataprepper.plugins.source.otellogs.OTelLogsSource - Stopped otel_logs_source.
2023-04-25T15:44:53,139 [log-pipeline-sink-worker-4-thread-1] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [log-pipeline] - Shutting down processor process workers.
2023-04-25T15:45:14,978 [metrics-pipeline-processor-worker-7-thread-1] WARN  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [metrics-pipeline] - Workers did not terminate in time, forcing termination of processor workers.
2023-04-25T15:45:14,979 [metrics-pipeline-processor-worker-7-thread-1] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [metrics-pipeline] - Shutting down sink process workers.
2023-04-25T15:45:14,979 [metrics-pipeline-processor-worker-7-thread-1] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [metrics-pipeline] - Encountered interruption terminating the pipeline execution, Attempting to force the termination of sink workers.
2023-04-25T15:45:14,979 [metrics-pipeline-processor-worker-7-thread-1] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [metrics-pipeline] - Pipeline fully shutdown.
2023-04-25T15:45:23,140 [log-pipeline-sink-worker-4-thread-1] WARN  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [log-pipeline] - Workers did not terminate in time, forcing termination of processor workers.
2023-04-25T15:45:23,140 [log-pipeline-sink-worker-4-thread-1] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [log-pipeline] - Shutting down sink process workers.
2023-04-25T15:45:23,140 [log-pipeline-processor-worker-3-thread-1] ERROR org.opensearch.dataprepper.pipeline.common.FutureHelper - FutureTask is interrupted or timed out
2023-04-25T15:45:23,140 [log-pipeline-processor-worker-3-thread-1] INFO  org.opensearch.dataprepper.pipeline.ProcessWorker - Processor shutdown phase 1 complete.
2023-04-25T15:45:23,140 [log-pipeline-processor-worker-3-thread-1] INFO  org.opensearch.dataprepper.pipeline.ProcessWorker - Beginning processor shutdown phase 2, iterating until buffers empty.
2023-04-25T15:45:23,141 [log-pipeline-processor-worker-3-thread-1] ERROR org.opensearch.dataprepper.pipeline.ProcessWorker - Encountered exception during pipeline log-pipeline processing
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4f6ccdf2[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@65933952[Wrapped task = org.opensearch.dataprepper.pipeline.Pipeline$$Lambda$1111/0x0000000801401eb0@dc15bf1]] rejected from org.opensearch.dataprepper.pipeline.common.PipelineThreadPoolExecutor@80bd459[Shutting down, pool size = 1, active threads = 1, queued tasks = 1, completed tasks = 435]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365) ~[?:?]
	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) ~[?:?]
	at org.opensearch.dataprepper.pipeline.Pipeline.lambda$publishToSinks$5(Pipeline.java:312) ~[data-prepper-core-2.2.0.jar:?]
	at org.opensearch.dataprepper.pipeline.router.DataFlowComponentRouter.route(DataFlowComponentRouter.java:45) ~[data-prepper-core-2.2.0.jar:?]
	at org.opensearch.dataprepper.pipeline.router.Router.route(Router.java:42) ~[data-prepper-core-2.2.0.jar:?]
	at org.opensearch.dataprepper.pipeline.Pipeline.publishToSinks(Pipeline.java:311) ~[data-prepper-core-2.2.0.jar:?]
	at org.opensearch.dataprepper.pipeline.ProcessWorker.postToSink(ProcessWorker.java:140) ~[data-prepper-core-2.2.0.jar:?]
	at org.opensearch.dataprepper.pipeline.ProcessWorker.doRun(ProcessWorker.java:121) ~[data-prepper-core-2.2.0.jar:?]
	at org.opensearch.dataprepper.pipeline.ProcessWorker.run(ProcessWorker.java:57) [data-prepper-core-2.2.0.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
2023-04-25T15:45:53,140 [log-pipeline-sink-worker-4-thread-1] WARN  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [log-pipeline] - Workers did not terminate in time, forcing termination of sink workers.
2023-04-25T15:45:53,141 [log-pipeline-sink-worker-4-thread-1] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [log-pipeline] - Pipeline fully shutdown.


Enchantments not showing up

Enchants are not showing in the enchanting table as an option(set to .5 in the config for a 50% chance). They dont show as an option and seem to just overwrite enchants instead. For example a protection 4 enchant has a chance to become any one of your enchants from the plugin, overwhelmingly seeming to prefer soulbound 1. If I could get them to show in the game on the enchanting table that would be great!

I'm not sure what you mean by "show in the game on the enchanting table" since the enchanting table is obfuscated. The plugin does replace X% of vanilla enchants (defined by config) from the enchanting table to attempt to keep balance (instead of adding them).
I've gone ahead and implemented a small fix for the randomness/level issue, however. Give the newest build a try and see if that fixes things.
https://jenkins.egg82.me/job/egg82/job/AdvancedEnchantments/job/master/

This appears to have fixed the randomness, as the enchants are being replaced as they should be but with fair returns, Protection 4 gives repairing 3 or similar instead of soulbound 1


New translations and default subject

Added Polish and French translations.
Added simple feature to set default e-mail subject for contact form when it's used for product availabilty requests.

Rationale:
Within lfs-theme/catalog/products/product_inline.html we have a simple 'Send request' button that appears when product is not available. Often clients redirected from specific product to contact form are asking just 'when it will be available?' so that shop owner doesn't even know what the question is about.
New feature implemented here lets us simply add product_id to the request, eg by changing product_lnline.html snippet to:
<button class="emphasized"                 onclick="window.location='{% url 'lfs_contact_form' %}?product_id={{ product.pk }}'; return false;"                 name="">{% trans 'Send request' %}</button>
to get nice e-mail subject like: "[MyShop contact form] Availability of 'Product 1' (728)"

Nice!


[loc][Query][vscode-cmake-tools] Localizability of 0; "package.c_cpp.debuggers.symbolLoadInfo.loadAll.description"

Hello developers,
please help localization team to answer question from translator.
In case translator is not sure about context/meaning/localizability of string we raise a question.
Source:
If true, symbols for all libs will be loaded, otherwise no solib symbols will be loaded. Default value is true.
File Name:
Src\vscode-extensions\vscode-cmake-tools.xlf
Query:
what is a solib?

This looks like a cpptools string.  Moving to cpptools.


update_all for models / scopes

@cheerfulstoic I'm trying to implement a temporary replacement. What could be a good implementation for this?
I thought about something like:
def update_all(params = {})
  query_as(:c).set(c: params).exec
end

or maybe:
def update_all(updates, bindings = {})
  if updates.is_a? Hash
    query_as(:n).set(n: values).exec
  elsif updates.is_a? String
    query_as(:n).set(values).params(bindings).exec
  else
    raise ArgumentError, 'Invalid parameters.'
  end
end

I think you're on a good path.  If you come up with a good final solution I'm happy to throw it in the gem (or you can make a PR and get credit ;)  I have a few thoughts:

First, I would call bindings parameters or params to be consistent.
I would probably use a case statement for the second implementation
These should all start with the all method so that you could run Model.where('something').update_all() and it should only update the objects that you have scoped to.

Maybe something like:
def update_all(updates, params = {})
  query = all.query_as(:n)

  case updates
  when Hash
    query.set(n: values).exec
  when String
    query.set(values).params(params).exec
  else
    raise ArgumentError, 'Invalid parameters.'
  end
end

There's also the argument to be made that Cypher makes this sort of thing really easy with SET, so that maybe users should just choose whichever of these paths that they want manually.  In the end, though, it's good to have compatibility with ActiveRecord so that plugins can work more easily between the two.
Let me know if you have any other questions!


Add CLI and fix edge case with gzipped text files

This PR aims to address #15 (Unicode issue with try/except workaround)  and #5 (adds command line interface with ability to specify ignore paths and enables dodgy --help)
There is also a version bump.
Note this PR also makes use of GitHub actions (see .github/workflows/pythonpackage.yml) and could be combined with GitHub's secrets to add $TWINE_USERNAME and $TWINE_PASSWORD to enable automatic uploading of dodgy to PyPI.

Coverage decreased (-1.3%) to 28.395% when pulling dd84b073b7037d514f3e24de515713ddc7ca3286 on Lightslayer:master into bc0ab1dd876e7aae9a6372e637687a0cd7793931 on landscapeio:master.

Coverage decreased (-1.3%) to 28.395% when pulling dd84b073b7037d514f3e24de515713ddc7ca3286 on Lightslayer:master into bc0ab1dd876e7aae9a6372e637687a0cd7793931 on landscapeio:master.


[DRAFT] Shorter Quantum Circuit paper notebook dataset

This folder contains a notebook that explores a dataset related to Shorter Quantum Circuits research paper.
The dataset contains results of running the algorithms described in the paper
and reproduces numerical results described in the paper.
Note that URL is the notebook need updating with its final version before completing this PR

Could you please add some information to the description as described in the following guide: https://github.com/microsoft/Quantum/blob/main/samples/azure-quantum/CONTRIBUTING.md#creating-the-prs


chore: add used sidecar images to vhd images

What type of PR is this?
/kind bug
What this PR does / why we need it:
add used sidecar images to vhd images
livenessprobe: v2.12.0 and csi-node-driver-registrar:v2.10.1 (v2.11.0 is unused now) are still used by aks 1.27, 1.28, add those sidecar images back to vhd images,
Which issue(s) this PR fixes:

Fixes #
Requirements:

[ ] uses conventional commit messages

[ ] includes documentation
[ ] adds unit tests
[ ] tested upgrade from previous version

Special notes for your reviewer:
Release note:
none

Pull Request Test Coverage Report for Build<PHONE_NUMBER>8
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 71.263%

Totals

Change from base Build<PHONE_NUMBER>6:
0.0%

Covered Lines:
2589

Relevant Lines:
3633

💛  - Coveralls


Update dependencies and Travis

Signed-off-by: Waldemar Quevedo<EMAIL_ADDRESS>

Coverage increased (+0.1%) to 91.173% when pulling 1e1c65ae724b314e1f7952e255d7e33ef81ce159 on updates into 348798afea05238066697d9f7f6b9a6cc2a8758c on master.


Accept .. as a search directory

Just allows directory-prefixed arguments, such as foo/bar, which indicate that just should start its search for the justfile not in the invocation directory, but in foo/.
However, due to the way that arguments are parsed, ../ is accepted, but .. is not. However, since .. cannot be the name of a recipe, it would be safe to give .. the same meaning as ../.

It is done!


Create a Reader for .doc, .docx, and .pdf files

There are currently two unimplemented template files (for reading .doc and .docx, and .pdf files, respectively)

[x] .odt files
[ ] .doc files
[ ] .docx files
[ ] .pdf files

Reading Word documents and PDF files are a bit less straight-forward than plaintext files. A library that is compatible with the license for this project may be advisible.
https://github.com/HaveIBeenPwned/EmailAddressExtractor/blob/9d9efa8cd4d74600db1aa3d63006bad760be81f0/src/Objects/Readers/PdfReader.cs#L2-L11
https://github.com/HaveIBeenPwned/EmailAddressExtractor/blob/9d9efa8cd4d74600db1aa3d63006bad760be81f0/src/Objects/Readers/DocumentReader.cs#L2-L11

Docx is xml so e-mails should be readable using plaintext search.

@jfbourke Do you want to split your OpenDocumentTextReader.cs to support .docx files?
The only difference I can see at first glance is .odt reads from content.xml and .docx reads from word/document.xml.
If you don't want to I can, but I want everyone to have a chance to contribute and your implementation works well :smiley:

@GStefanowich I've split the implementation to support the two file formats, added .pptx as well.

@jfbourke Looks great!


How to show default button after spinning?

I'm making a login button with EMSpinnerButton.
When login is failed, i want to make button normal.
how can i do so?

@masuhara
// Expand animation
button.animate(animation: .expand)

@aminaura you are cool guy


KAFKA-4733: Improve Streams Reset Tool console output

Added general explanation of the tool and what it does. Also added few details to the arguments.

@guozhangwang Call for final review and merging.

I feel like I'll need to start paying the review debt for @guozhangwang soon...


Short solution needed: "How to add proxy" (proxychains)

Please help us write most modern and shortest code solution for this issue:
How to add proxy (technology: proxychains)
Fast way
Just write the code solution in the comments.
Prefered way

Create pull request with a new code file inside inbox folder.
Don't forget to use comments explain solution.
Link to this issue in comments of pull request.

How to add new proxy to proxychains


urql-introspection plugin query does not match the urql devtools query causing error

Describe the bug
The introspection query that the urql-introspection plugin uses to generate the schema does not match the one that urql's devtoolsExchange uses. Specifically, it is missing __typename in several places.
This would be fine, however when urql's cacheExchange (@urql/exchange-graphcache) uses this generated schema, an uncaught error occurs when the devtoolsExchange (@urql/devtools) attempts to request its own introspection query.
The problem is that the devtoolsExchange's query includes some fields that aren't known to cacheExchange due to the differences in the introspection query.
To Reproduce
Steps to reproduce the behavior:

use graphql-codegen with the urql-inspection plugin to generate a schema
import the schema into urql cacheExchange
add the devtoolsExchange in urql client
load the app in a browser
Result: app will load, then once the urql devtools requests its own introspection query, cacheExchange will throw an error when checking the response.

My GraphQL schema:

Is quite large and not the issue here.

My GraphQL operations:

########################################
# graphql-codegen introspection query:
########################################
query IntrospectionQuery {
  __schema {
    queryType {
      name
    }
    mutationType {
      name
    }
    subscriptionType {
      name
    }
    types {
      ...FullType
    }
    directives {
      name
      description
      locations
      args {
        ...InputValue
      }
    }
  }
}

fragment FullType on __Type {
  kind
  name
  description
  fields(includeDeprecated: true) {
    name
    description
    args {
      ...InputValue
    }
    type {
      ...TypeRef
    }
    isDeprecated
    deprecationReason
  }
  inputFields {
    ...InputValue
  }
  interfaces {
    ...TypeRef
  }
  enumValues(includeDeprecated: true) {
    name
    description
    isDeprecated
    deprecationReason
  }
  possibleTypes {
    ...TypeRef
  }
}

fragment InputValue on __InputValue {
  name
  description
  type {
    ...TypeRef
  }
  defaultValue
}

fragment TypeRef on __Type {
  kind
  name
  ofType {
    kind
    name
    ofType {
      kind
      name
      ofType {
        kind
        name
        ofType {
          kind
          name
          ofType {
            kind
            name
            ofType {
              kind
              name
              ofType {
                kind
                name
              }
            }
          }
        }
      }
    }
  }
}

########################################
# urql-devtools introspection query:
########################################
query IntrospectionQuery {
  __schema {
    queryType {
      name
      __typename
    }
    mutationType {
      name
      __typename
    }
    subscriptionType {
      name
      __typename
    }
    types {
      ...FullType
      __typename
    }
    directives {
      name
      description
      locations
      args {
        ...InputValue
        __typename
      }
      __typename
    }
    __typename
  }
}

fragment FullType on __Type {
  kind
  name
  description
  fields(includeDeprecated: true) {
    name
    description
    args {
      ...InputValue
      __typename
    }
    type {
      ...TypeRef
      __typename
    }
    isDeprecated
    deprecationReason
    __typename
  }
  inputFields {
    ...InputValue
    __typename
  }
  interfaces {
    ...TypeRef
    __typename
  }
  enumValues(includeDeprecated: true) {
    name
    description
    isDeprecated
    deprecationReason
    __typename
  }
  possibleTypes {
    ...TypeRef
    __typename
  }
}

fragment InputValue on __InputValue {
  name
  description
  type {
    ...TypeRef
    __typename
  }
  defaultValue
}

fragment TypeRef on __Type {
  kind
  name
  ofType {
    kind
    name
    ofType {
      kind
      name
      ofType {
        kind
        name
        ofType {
          kind
          name
          ofType {
            kind
            name
            ofType {
              kind
              name
              ofType {
                kind
                name
                __typename
              }
              __typename
            }
            __typename
          }
          __typename
        }
        __typename
      }
      __typename
    }
    __typename
  }
}

Here is the diff between the two queries:
5d4
<       __typename
9d7
<       __typename
13d10
<       __typename
17d13
<       __typename
25d20
<         __typename
27d21
<       __typename
29d22
<     __typename
42d34
<       __typename
46d37
<       __typename
50d40
<     __typename
54d43
<     __typename
58d46
<     __typename
65d52
<     __typename
69d55
<     __typename
78d63
<     __typename
107d91
<                 __typename
109d92
<               __typename
111d93
<             __typename
113d94
<           __typename
115d95
<         __typename
117d96
<       __typename
119d97
<     __typename

My codegen.yml config file:

overwrite: true
schema: 'http://localhost:4000/graphql'
generates:
  src/graphql/schema.ts:
    plugins:
      - 'urql-introspection'
    config:
      useTypeImports: true
      includeEnums: true
      includeInputs: true
      includeDirectives: true
      includeScalars: true

Expected behavior
Ideally the two schemas should match or at least not conflict so much that it causes page-breaking errors.
One possibility could be adding an includeTypenames: boolean option to the urql-introspection plugin to adjust the query.
Environment:

OS: macOS 11.3
@graphql-codegen/cli: 1.21.4
@graphql-codegen/urql-introspection: 1.2.0
NodeJS: v14.16.0

Additional context
Apologies but I don't see how I could put this into a codesandbox in any meaningful way. For reference here is how the client is instantiated in my application:
import { createClient, dedupExchange, fetchExchange } from 'urql';
import { cacheExchange } from '@urql/exchange-graphcache';
import { devtoolsExchange } from '@urql/devtools';
import schema from '@src/graphql/schema';

const client = createClient({
  url: 'http://localhost:4000/graphql',
  requestPolicy: 'cache-first',
  exchanges: [
    devtoolsExchange,
    dedupExchange,
    cacheExchange({ schema }),
    fetchExchange,
  ],
});

The specific error is the following:
Unhandled Rejection (Graphcache Error): Invalid Object type: The type `__Schema` is not an object in the defined schema, but the GraphQL document is traversing it.
(Caused At: "IntrospectionQuery" query)
https://bit.ly/2XbVrpR#3

This error originates here: https://github.com/FormidableLabs/urql/blob/4c1a9d9bd3df55207aeff1ffc3e2b3262f27cbc9/exchanges/graphcache/src/ast/schemaPredicates.ts#L70
Removing the devtoolsExchange from the client removes the error, but at the expense of having no devtools 😢

Of course, happy to provide a stack trace. Apologies, timezone differences delayed my reply, though it sounds like you've already solved the issue (many thanks for that).
Here is a screenshot of the source-mapped stacktrace for the record, since copy-paste from the browser wouldn't preserve the formatting.

If the cause of the problem was a difference between the result of devtools' query and what graphecache expects, then I'm not sure how my initial report made an incorrect assumption. Perhaps my terminology is simply lacking. At any rate I was simply trying to provide as much information as I had gleaned from debugging myself.

I can confirm that upgrading to<EMAIL_ADDRESS>has resolved the issue for my installation. Will leave the ticket open in case you have a specific policy around confirming/labelling/closing issues.

@disbelief Awesome! No worriers though, I merely mean that more details around errors without speculation often helps us to resolve an issue quicker 🙌 Then again, since we have time to address these issues during normal work hours we often are able to fix them on the same day either way


Add before-hook-creation to upstream hooks

This has been an issue in some places because Helm doesn't remove the hook before trying to create it again.
It has been fixed upstream here: https://github.com/kyverno/kyverno/commit/2f4b8230304ff3de2001735791eb44ee1e69bedd , but this change will only make it on Kyverno v1.12.

oh wow, I messed this up real good

Okay now it's fixed.

Fine with me but what is the need to get this in before v1.12?

I feel this will be a problem that will start happening to more and more customers when people start moving to v20+.
I'm unsure if v1.12 will land before that.


Add TreeSelect-component

I like this project, can add TreeSelect-component like ant-design?
i think it is very necessary

And Tree-component, please

you can use https://github.com/cipchk/ng-tree-antd with @cipchk as an temp solution.


Wrap hljs

[x] Others (Update, fix, translation, etc...)

Introduced in https://github.com/hexojs/hexo/pull/3827 https://github.com/hexojs/hexo/pull/2901

Pending release of https://github.com/hexojs/hexo/pull/3827 https://github.com/hexojs/hexo/pull/3848


Quote suggestion

There should be a form which everyone can use to suggest quotes. Nothing special. Just a text field, a submit button and probably CAPTCHA (I think of reCAPTCHA or other similar services).
The suggested quotes should be put in a pending queue. Admin can watch this queue and approve or disapprove the quotes from it. So, it requires to add a new table to our current database scheme for pending quotes and implement an admin panel for managing them.
Currently, we don't have any decent authorization mechanism, so let's use basic auth for any admin actions. You can find examples of using basic auth in LogList here.

This can be closed


U1: Create a login screen

Referencing the UI mocks and using Android Studio, assemble a component containing a form with a username field, password field, a submit button, a link to the registration screen, and all relevant visuals and titles

"LOGIN
Create email feild: email field must only accept the @pride.hofstra.edu domain name
Create password field: password field must contain at least 6 characters
Create form submission button: form must contain a button labeled ""Login""
Create link to Registration Form: form must contain a button labeled ""Register"", next to a text prompt ""Don't have an account?"" that will convert the component to the Registration form
Successful Login: (funtionality IP)
Insuccessful Login: An error message must appear below the submission button upon a failed login that says, ""Please check your login information and try again!""
REGISTRATION
Create first name field: first name field
Create last name field: last name field
Create email feild: email field must only accept the @pride.hofstra.edu domain name (must display a message under the field if email does not have this domain name ""Email must be a valid @pride.hofstra.edu account"" that will prevent the form from being submitted)
Create password field:
password field must contain at least 6 characters (must display a message under field if feild contains less than 6 characters ""Password must conatin 6 characters"" that will prevent the form from being submitted)
Create form submission button: form must contain a button labeled ""Register"" that will render the Login component upon sucessful registration
Create link to Login Form: form must contain a button labaled ""Login"", nect to a text prompt ""Have an account?"" that will convert the component to the Login Form"


Using Makefile to source with --install or --no-use returns "Error 3"

Operating system and version: Ubuntu 22.04.3 LTS in Docker
nvm debug output:

nvm --version: v0.39.5
$SHELL: /bin/bash
$SHLVL: 1
whoami: 'root'
${HOME}: /root
${NVM_DIR}: '${HOME}/.nvm'
${PATH}: ${HOME}/.pulumi/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
$PREFIX: ''
${NPM_CONFIG_PREFIX}: ''
$NVM_NODEJS_ORG_MIRROR: ''
$NVM_IOJS_ORG_MIRROR: ''
shell version: 'GNU bash, version 5.1.16(1)-release (aarch64-unknown-linux-gnu)'
uname -a: 'Linux 5.15.49-linuxkit #1 SMP PREEMPT Tue Sep 13 07:51:32 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux'
checksum binary: 'sha256sum'
OS version: Ubuntu 22.04.3 LTS
random-funcs:       srandom/random
regex-funcs:        internal
compiled limits:
sprintf buffer      8192
maximum-integer<PHONE_NUMBER>
awk: /usr/bin/awk, mawk 1.3.4 20200120
curl: /usr/bin/curl, curl 7.81.0 (aarch64-unknown-linux-gnu) libcurl/7.81.0 OpenSSL/3.0.2 zlib/1.2.11 brotli/1.0.9 zstd/1.4.8 libidn2/2.3.2 libpsl/0.21.0 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.43.0 librtmp/2.3 OpenLDAP/2.5.16
wget: not found
git: not found
grep: /usr/bin/grep (grep --color=auto), grep (GNU grep) 3.7
sed: /usr/bin/sed, sed (GNU sed) 4.8
cut: /usr/bin/cut, cut (GNU coreutils) 8.32
basename: /usr/bin/basename, basename (GNU coreutils) 8.32
rm: /usr/bin/rm, rm (GNU coreutils) 8.32
mkdir: /usr/bin/mkdir, mkdir (GNU coreutils) 8.32
xargs: /usr/bin/xargs, xargs (GNU findutils) 4.8.0
nvm current: none
which node:
which iojs:
which npm:
npm config get prefix: bash: npm: command not found
npm root -g: bash: npm: command not found

nvm ls output:

            N/A
iojs -> N/A (default)
node -> stable (-> N/A) (default)
unstable -> N/A (default)
lts/* -> lts/hydrogen (-> N/A)
lts/argon -> v4.9.1 (-> N/A)
lts/boron -> v6.17.1 (-> N/A)
lts/carbon -> v8.17.0 (-> N/A)
lts/dubnium -> v10.24.1 (-> N/A)
lts/erbium -> v12.22.12 (-> N/A)
lts/fermium -> v14.21.3 (-> N/A)
lts/gallium -> v16.20.2 (-> N/A)
lts/hydrogen -> v18.17.1 (-> N/A)

How did you install nvm?

RUN curl https://raw.githubusercontent.com/creationix/nvm/master/install.sh | bash
What steps did you perform?

Create an .nvmrc file with the content v20.4.0
Create a Makefile:

nvm-install:
	. ${NVM_DIR}/nvm.sh --install

nvm-no-use:
	. ${NVM_DIR}/nvm.sh --no-use

Create an ubuntu docker image with nvm installed:

FROM ubuntu:latest

RUN apt-get -y update
RUN apt-get -y install curl
RUN apt-get -y install make

RUN curl https://raw.githubusercontent.com/creationix/nvm/master/install.sh | bash

Run a docker container with the current directory mounted as a volume: docker run -it -v $(PWD):/tmp
In the container run, cd to the mounted directory /tmp then run make nvm-install and make nvm-no-use

What happened?
Got Error 3 for both command outputs:
root@d235e50c2915:/tmp# make nvm-install
. /root/.nvm/nvm.sh --install
make: *** [Makefile:22: nvm-install] Error 3
root@d235e50c2915:/tmp# make nvm-no-use
. /root/.nvm/nvm.sh --no-use
make: *** [Makefile:25: nvm-no-use] Error 3

What did you expect to happen?
nvm installs the version of node in the .nvmrc file in the directory.
Is there anything in any of your profile files that modifies the PATH?

No. Nothing in the default .bashrc or .profile files in ubuntu:latest has anything that modifies the path.

On my Mac, when I have no default alias and have .nvmrc set to a non-installed version, . $NVM_DIR/nvm.sh has a zero exit code, . $NVM_DIR/nvm.sh --install attempts to install the version, and . $NVM_DIR/nvm.sh --no-use  is the same as without --no-use.
I certainly see a lot of your base utilities are the GNU version, so maybe one of those is the issue. Presumably nvm_auto none, nvm_auto install, and nvm_auto use all have the same behavior?
What happens if you set -x and run each of those three commands?

Are you doing this with a Makefile that you're calling in an interactive shell within the Docker container? I think Makefile is messing with something.
make --version output:
GNU Make 4.3
Built for aarch64-unknown-linux-gnu
Copyright (C) 1988-2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

No, certainly not; it's certainly possible make is causing the issue - can you run those commands with set -x in the makefile?

In docker, each "RUN" needs to re-source nvm.sh due to the way docker works (that's my understanding, at least).
Either way, closing since there's been no followup. Happy to reopen if that changes.


SignMessage succeed, but verify signedMessage failed when I use 'Sui Wallet' and 'Suiet' to sign. But it was all right before update.

SignMessage succeed, but verify signedMessage failed when I use 'Sui Wallet' and 'Suiet' to sign. But it was all right before update.

Duplicated #203
Please follow the instructions there. Feel free to give feedback if there is still any error.


When using submit button ! Streamlit chat bot is not showing success message

I am using streamlit_feedback in my chat app , but i m not able to print any success messages, when a feedback is submitted , I also want to store the in a Mongo db , can someone help me in that, it would be great
As soon as I am clicking the submit button , it is re running the app without any toast alert!
This is my code snippet
def handle_feedback():
            st.write(st.session_state.fb_k)
            st.toast("✔ Feedback received!")

if prompt := st.chat_input(" Ask your document?"):
        st.chat_message("user").markdown(prompt)
        st.session_state.messages.append({"role": "user", "content": prompt})
        with st_lottie_spinner(loading_animation, quality='high', height='70px', width='70px'):
            ans,unique_links = get_response_from_django(prompt)
            with st.chat_message("assistant"):
                st.markdown(ans)

        streamlit_feedback(
                        feedback_type="thumbs",
                        optional_text_label="[Optional] Please provide an explanation",
                        align="flex-start",
                        key=f"fb_k",
                        on_submit=handle_feedback,
            )

The streamlit_feedback component triggers a page reload when submitted, this is how streamlit components work. The on_submit function is only then run when your app reaches the streamlit-feedback() call on the rerun.
To fix this you can remove the streamlit_feedback component from the if statement


Improvements/316

https://github.com/OAGr/foretold/issues/316
Please, consider this PR after all previous.

After fixes.


⚠ ClamAV has degraded performance

In 69389e2, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 171 ms

Resolved: ClamAV performance has improved in 5c5e85c.


target parameter of post_transition signal should be the resulting state

If the target of the transition is GET_STATE or RETURN_VALUE, post_transition gets the State object instead of the resulting state.

Thanks for the fix.
Btw, could you show for what purpose do you use signals? For all 6 years of django-fsm development and usage I newer have to use signals in mine projects.

We sometimes trigger notifications or activity stream object creation on state transitions. Examples:
• when an e-signature document object changes state to signed, trigger an email notification
• when an application changes state to declined, send a Slack message to someone


location of physical packages storage

Where are packages physically stored?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 078a8c96-ec59-c10b-e48e-8c5586c8cb09
Version Independent ID: 200f5723-2404-298d-e6bb-533592a28ff3
Content: Package Management in VSTS and TFS
Content Source: docs/package/overview.md
Product: devops
GitHub Login: @elbatk
Microsoft Alias: elbatk

Adding @alexmullans to answer this for ya, @verbaloid

@verbaloid in TFS or VSTS? Assuming the former, they're stored in the database along with your other TFS data.

Thanks @alexmullans , closing this issue for now but feel free to reach out if you have more questions @verbaloid .

thank you


barrier = cattle_grid

Hello - Cruiser user here (both Android and Desktop) with Mapsforge maps.
Graphhopper will not route me over a barrier = cattle_grid, eg:
https://www.openstreetmap.org/node/2923770193
This route, with Cruiser Desktop using offline Graphhopper takes me on a very interesting reroute when I (a motorcycle) could easily ride over it.  This is with default "car", "fast" options and no avoidances - stock routing options.  Same results on the Android app.

I also use BRouter, and had to modify the profile to take into account level-crossings, which are tagged differently on OSM despite being the same "electronic barrier" type.  Of course there is lack of consistency with people's edits of public data to consider, but what is Graphhopper's "problem" with a drive-over cattle-grid?!
I am fairly active on the Cruiser board, and have written about this situation here.

Seems to work just fine?
https://graphhopper.com/maps/?point=55.365403%2C-3.851056&point=55.365755%2C-3.849066&profile=car&layer=OpenStreetMap
https://graphhopper.com/maps/?point=55.347399%2C-3.872085&point=55.371413%2C-3.829582&profile=car&layer=OpenStreetMap


[Bug] Temporary table does not use partition expiry from incremental model configuration

Is this a new bug in dbt-bigquery?

[X] I believe this is a new bug in dbt-bigquery
[X] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior
In our GCP project, we set a default partition expiry of 3 years. However, we have one model that we want to go back to 5 years. This model has the following config:
{{
    config(
        alias = 'tom_demo',
        materialized = 'incremental',
        incremental_strategy = 'insert_overwrite',
        partition_by = {'field': 'date', 'data_type': 'date'},
        on_schema_change = 'append_new_columns',
        partition_expiration_days = 366 * 5
    )
}}

When attempting to backfill this model for 2021-01-01 (more than 3 years ago), the model is running but the resulting partition is empty. The query returns data but the temp table uses the default project partition expiry of 3 years (instead of the 5 years specified by the model).
Expected Behavior
I expect the temp table to have the same expiry of the model it wishes to merge into.
Steps To Reproduce
In your gcp project, attempt to run the model for a date within your models config partition expiry (but greater than your projects default expiry). For example this project has a default expiry of 3 years, which means this partition for 2021-01-01 will  is outside of the temp table expiry. (but within the tom_demo model's expiry).
    create or replace table `project`.`intermediate`.`tom_demo__dbt_tmp`

    partition by snapshot_date

    OPTIONS(
      description="""creating temp table to show""",

      expiration_timestamp=TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 12 hour)
    )
    as (

      select
      DATE("2021-01-01") as snapshot_date,
      1 as col
    )

Relevant log output
No response
Environment
- OS: linux
- Python: 3.11
- dbt-core: 1.6.6
- dbt-bigquery: 1.6.6

Additional Context
No response

@dbeatty10 Would you accept a PR for this? Just got hit by this again. We had a dataset that had a default partition expiration of 7 days. Couldn't understand why incremental model's with a partition expiration of 3 years were failing to insert any data.


with postgres.Backend job.RunAfter is not always honored

Steps to reproduce:

Create a queue and handler
Add a job that has RunAfter: time.Now().Add(time.Hour)

Expected:

Job doesn't run for an hour

Actual:

Job runs immediately

In a fork, I added logging to confirm:
https://github.com/pconstantinou/neoq/blob/main/backends/postgres/postgres_backend.go#L928

Are you able to reproduce with a shorter time frame? E.g. 1 minute?

Yes. I could give you a test if you’d like.
On Thu, Feb 29, 2024 at 7:13 AM Adriano Caloiaro @.***>
wrote:

Are you able to reproduce with a shorter time frame? E.g. 1 minute?
—
Reply to this email directly, view it on GitHub
https://github.com/acaloiaro/neoq/issues/120#issuecomment-1971353866,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAKR3GURQW2DW3VPXHKBKGDYV5CR7AVCNFSM6AAAAABD2UVPG2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZRGM2TGOBWGY
.
You are receiving this because you authored the thread.Message ID:
@.***>

@pconstantinou I think we're going to need that reproduction test. I just added TestFutureJobProcessing to cover this case, and it's currently passing. In my manual testing, I tested a few jobs with RunAfter ranging between 30 seconds and 10 minutes, and all jobs ran promptly at the specified RunAfter.
There must be something unique about your situation that this test doesn't cover.
The only thing I can think of is that you set the JobCheckInterval on neoq.Config to a value greater than the time between time.Now() and RunAfter. It's default is 1 second. I've considered not exposing JobCheckInterval for this reason. If this does turn out to be the case, it's probably best addressed with documentation. Or perhaps not exposing JobCheckInterval at all.


Logs leaking credentials

Same issue as before but this is for the mail catcher.
The logs contain the username and password in plain text as it logs all messages coming into the container. This isn't just an issue with the daspanel password being shown but also anything being sent through the container.
It's also listing the MYSQL root password, daspanel api key, etc.

@OmgImAlexis  Your are right, mail-catcher (mailhog) is loging in stdout every message received by it. I will try to solve this issue is next 3 days.

Mailhog has no option to suppress log messages. To solve the problem I had to redirect all error messages and logs  to /dev/null.
You will need to update your images by running:
docker pull daspanel/mailhog-mail-catcher

Looks good.


[SPARK-18751][Core]Fix deadlock when SparkContext.stop is called in Utils.tryOrStopSparkContext

What changes were proposed in this pull request?
When SparkContext.stop is called in Utils.tryOrStopSparkContext (the following three places), it will cause deadlock because the stop method needs to wait for the thread running stop to exit.

ContextCleaner.keepCleaning
LiveListenerBus.listenerThread.run
TaskSchedulerImpl.start

This PR adds SparkContext.stopInNewThread and uses it to eliminate the potential deadlock. I also removed my changes in #15775 since they are not necessary now.
How was this patch tested?
Jenkins

cc @rxin

Test build #69750 has started for PR 16178 at commit 7131a96.

Test build #69751 has started for PR 16178 at commit 2cd4282.

Test build #69750 has finished for PR 16178 at commit 7131a96.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69750/
Test PASSed.

Test build #69751 has finished for PR 16178 at commit 2cd4282.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69751/
Test PASSed.

Test build #69821 has started for PR 16178 at commit ae3d42c.

Test build #69821 has finished for PR 16178 at commit ae3d42c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69821/
Test PASSed.

LGTM

Thanks! Merging to master and 2.1.


[1pt] Add tool to push data to S3 (for Hydroviz)

A few things are covered in this PR.

FIM Outputs to HydroViz AWS (Amazon Web Service) S3. While the issue asked for FIM3 to auto push it to S3, a decision was made to make it for FIM4 as a stand alone tool.  It can still be integrated easily into the FIM 3 API tool, but the level of effort, considering that the tool is about to be shelved, made it smarter to make this as a stand alone tool for FIM4.  This tool can be run as a command line tool or integrated against other code based. It can also be run on any output including FIM3 output folder.
This tool has a whitelist system with wildcards, to select which files are sent up to S3. A requirement exists to control while files are copied up to S3. While designed and tested for HydroViz, it is configurable to any S3 bucket address and/or folders. While not tested, it is expected to be able to push files and folders to ESIP if required. See issue 625.

Adding AWS to the FIM4 required an upgrade to the docker images for AWS Boto3. Piggy-backing on this, we took the opportunity to add three additional packages, Jupyter, JupyterLab and Ipympl. Jupyter is being used by a number of developers and this makes it easier for them to use Jupyter tools with FIM code.

A minor correction to gms_run_branch.sh to move a code block down the page after variable setup and validation.

During output processing while overwriting past run (gms_pipeline.sh, gms_run_unit.sh, gms_run_branch.sh), the src_optimization log directory was not be cleaned. Old logs from previous runs could be carried forward. This is now corrected. See Issue 661.

Additions

src

data

README.md: This system was designed with the expectation that more AWS interaction will be coming in the near future. As a result, a class model using inheritance was created to make future enhancements easier and more standardized. This document talks about the tool and process. Note: The new data folder can be used for AWS external communication, but could also be used for other external data communication modules.
aws

aws_base.py:  With implementing a new class and inheritance system, this file has properties and a method that all child class will be expected to use and share. This makes it quicker and easier to added new AWS tools and helps keep consistant patterns and standards.
aws_creds_template.env: There are a number of ways to validate credentials to send data up to S3. We have chosen to use an .env file that can be passed into the tool from any location. This is the template for that .env file. Later versions may be changed to use AWS profile security system.
s3.py: This file pushes file and folders up to a defined S3 bucket and root folder. Note: while it is designed only for puts (pushing to S3), hooks were added in case functional is added later for gets (pull from S3).

Changes

utils

shared_functions.py:  A couple of new features

Added a method which accepts a path to a .lst or .txt file with a collection of data and load it into a  python list object. It can be used for a list of HUCS, file paths, or almost anything.
A new method for quick addition of current date/time in output.
A new method for quick calculation and formatting of time duration in hours, min and seconds.
A new method for search for a string in a given python list. It was designed with the following in mind, we already have a python list loaded with whitelist of files to be included in an S3 push. As we iterate through files from the file system, we can use this tool to see if the file should be pushed to S3. This tool can easily be used contexts and there is similar functionality in other FIM4 code that might be able to this method.

Dockerfile : Removed a line for reloading Shapely in recent PRs, which for some reason is no longer needed after adding the new BOTO3 python package. Must be related to python packages dependencies. This removed Shapely warning seen as a result of another recent PR.

Pipfile and Pipfile.lock:  Updates for the four new python packages, boto3 (for AWS), jupyter, jupyterlab and ipympl. We have some staff that use Jupyter in their dev actitivies. Adding this package into the base Docker image will make it easier for them.

Testing

gms_pipeline.sh, gms_run_unit.sh and gms_run_branch.sh were tested for a couple HUCs to make sure all python dependencies are still fine.
Alpha testing was done to ensure all is still well.
Docker image creation tests were completed
Tests were done to ensure the new Docker images they auto load Jupyter tools.
Considerable testing against the new S3 tool including:

different files / folder combinations
validating that it fails if invalid credentials are passed in.
testing of various combinations of files / folder paths and wildcard character usage for the new whitelist.lst file.
testing of various combinations (pass and fail) for the CLI input for the s3.py file.

Ready for merge. It is pushing a full BED run of <IP_ADDRESS> from previous_fim folder up to Hydroviz (with whitelist filtering) as we speak and is doing just fine. I will check in the morning to see how it finished up.

And I updated the code to move the code up one level as we discussed earlier today.

Now updated to use bash / AWS CLI instead of Boto3. Boto3 was taking 24 hours plus to push all of the files up to S3, but aws cli can do it in 3.5 hours. With AWS CLI, it is not as easy to track and catch errors, but Boto3 does not yet have a bulk (recursive folders) tool and even with adding its performance upgrades, it is not good to be reasonable.
This is again ready for merging.  New Docker images will need to be created after merge.


Aztec Connect Style Uniswap Bridge

Goal: Support something similar to Aztec connect in function that uses L2 to aggregate
Extension of #843 with a prior issue at #879, updated here to elaborate on the tasks for clarity.
### Tasks
- [x] Write a new  Uniswap portal contract with unambigious messages
- [x] Public flow for Uniswap with current token bridge impl
- [x] Private flow for Uniswap with current token bridge impl
- [ ] https://github.com/AztecProtocol/aztec-packages/issues/2493
- [ ] Add aggregation. Current impl #1796.
- [ ] https://github.com/AztecProtocol/aztec-packages/issues/2173
- [ ] https://github.com/AztecProtocol/aztec-packages/issues/2291

### Dependencies
- [ ] https://github.com/AztecProtocol/aztec-packages/issues/1743
- [ ] https://github.com/AztecProtocol/aztec-packages/issues/2170

Closing this for now as it have no priority at the moment.


RFC: Env flag for npm packages to conditionally load WASM binaries instead of native binaries

The problem
Web development toolchains are increasingly including binaries for compilers/bundlers as they provide substantially faster performance than JS equivalents. The ecosystem is also increasingly adopting WebAssembly for distributing & executing binaries as it provides far greater security guarantees, are directly inspectable, and "build once and run everywhere".
Today esbuild and swc are the most popular of these new tools, and both release a WASM version in a separate package in lockstep with their native binaries (esbuild-wasm and @swc/wasm). As such, consumers of these packages need to determine at runtime whether they should use the native binaries or the WASM binary.
Parcel v2 currently uses a custom flag process.env.PARCEL_SWC_WASM that indicates it should require the WASM binary and not the platform specific variant. Other toolchains that have adopted swc/esbuild (Vite, Next.js, Snowpack, etc) need a similar way of determining this as well, and custom flags for each package is not ideal.
Proposed Solution: process.env.REQUIRES_WASM
I propose we standardize on a new flag, process.env.REQUIRES_WASM, that instructs npm packages to use WASM binaries instead of unsandboxed platform specific binaries.
This is important for environments that require high security guarantees (i.e. targets of supply chain attacks like enterprises and OSS devs), WebAssembly based containers like WebContainer or kubelet, in-browser playgrounds, and other cases where native binaries cannot/should not be executed.
Indicates WASM is required, not just preferred.
Importantly, this flag does not indicate that the host environment prefers WASM. Instead it is an instruction that the host requires WASM binaries and cannot (or does not) allow execution of unsandboxed native binaries.
Intended functionality
If process.env.REQUIRES_WASM is true at runtime, npm packages should import WASM binaries (or a JS file generated via emscripten, etc) instead of corresponding platform specific binaries.
Should a package not have a WASM binary available (and/or can't polyfill the functionality otherwise), the program should throw an error if process.env.REQUIRES_WASM is present.

I would propose to use an exports/imports field condition instead of a (runtime) environment variable.
An exports field condition has the advantage that it's statically known instead of at runtime. This allows to avoid using import() and instead import ... from "..." can be used.
see also

https://webpack.js.org/guides/package-exports/
https://nodejs.org/api/cli.html#cli_c_condition_conditions_condition

Great idea!
Another angle to this problem is: many of these tools rely on the postinstall hook if npm. One common problem they encounter is that if you run npm install --ignore-scripts, they postinstall hook won't be called and the native binaries won't be installed. Some times users have this on without knowing it (such as in an .npmrc) and file issues about it.
If we could convince npm to make this option the default it would push these types of tools to prioritize wasm and make it the primary way to use their tools.

(I'm commenting here because I was pinged on Twitter as the author of esbuild, but I don't have any stake in this PR)

If process.env.REQUIRES_WASM is true at runtime, npm packages should import WASM binaries (or a JS file generated via emscripten, etc) instead of corresponding platform specific binaries.

Each binary executable is many megabytes. Solutions which require shipping binary executables for multiple platforms in the same package are very undesirable because then the weight of the package (installation time and disk size) would be huge. The reason esbuild's WebAssembly package is separate from the native package is to reduce the weight of the package. Having that be a run-time switch instead of an install-time switch would increase the weight of the package by a lot, and is likely not something I would ever consider doing for esbuild.

If we could convince npm to make this option the default it would push these types of tools to prioritize wasm and make it the primary way to use their tools.

This isn't ever going to happen for esbuild. WASM has a number of major drawbacks and I would never want it to be the primary way to interact with esbuild. If there were no drawbacks, there wouldn't be a reason to ship native executables. Some of the drawbacks:

AFAIK there's still no way to get node to cache the translated WASM assembly code so every time you run it, it's recompiled from scratch. This can be a huge performance penalty over native.

Node has a pretty bad bug where the WASM compilation must finish before node is allowed to exit: https://github.com/nodejs/node/issues/36616. This causes short runs of large WASM programs (e.g. CLI scenarios) to take way, way longer than they should (up to 40x longer).

Depending on the scenario, WASM's 2gb address size limit can be prohibitive and lead to crashes or large slowdowns as available memory gets low. Native binaries do not have this restriction and generally have a better chance of working correctly.

In esbuild's case WASM can easily be 10x slower than native. WASM will never be the default implementation.

Another angle to this problem is: many of these tools rely on the postinstall hook of npm. One common problem they encounter is that if you run npm install --ignore-scripts, they postinstall hook won't be called and the native binaries won't be installed. Some times users have this on without knowing it (such as in an .npmrc) and file issues about it.

I'm currently experimenting with an alternative installation strategy for esbuild that uses optionalDependencies instead of a postinstall script. If it works, that would solve this particular problem without needing any new environment flags and without relying on WASM. See this post for more information: https://github.com/evanw/esbuild/issues/789#issuecomment-890596192. The only drawback is that optionalDependencies doesn't provide a way to have a default "fallback" dependency if none of the optional dependencies apply. There's another proposal from Yarn that attempts to address that need: https://github.com/yarnpkg/berry/issues/2751.

Thanks @evanw, the optionalDependencies trick sounds interesting. Am I reading this right that it would install the wasm version for webcontainer (which needs to run in the browser)?

Another idea, probably for the Node.js security team is, if there was a way to disable 3rd party packages from launching subprocesses that would help discourage the install-binary patterns. Deno has this with --allow-run (this applies to all code, not just 3rd party), and the result is that you see mostly wasm used there.

@EricSimons Given the discussion here and elsewhere, I think standardizing on an export map condition could be the best immediate-impact solution here. stackblitz could run install with --condition=requires_wasm (bikeshed on name) and everything should just work.

@sokra @matthewp I think export maps definitely could be a good solution here. It does diverge from how folks are currently handling this issue atm, as both Next and Parcel load the appropriate binary using runtime conditions. I'd be curious to see if these teams would have any reservations on this approach (I imagine Next would be on board as Tobias works on it, but also cc'ing @devongovett for the Parcel team's take).

Thanks @evanw, the optionalDependencies trick sounds interesting. Am I reading this right that it would install the wasm version for webcontainer (which needs to run in the browser)?

No, it has nothing to do with WASM. I just mentioned it since it's a way of avoiding problems with --ignore-scripts and postinstall scripts. If you need to run esbuild in the browser, then you'll need to explicitly install the WASM version of esbuild instead of the native version.

@evanw thanks for taking the time to respond here- some extra perspective/info that might be relevant:

Solutions which require shipping binary executables for multiple platforms in the same package are very undesirable because then the weight of the package (installation time and disk size) would be huge

AFAIK many npm packages are starting to ship included binaries because it removes the CPU time (and often errors) that happen on install and during CI jobs. It also eliminates the need for postinstall, which is ideal from a security POV. I do understand why you don't want to go this route for esbuild though.

Depending on the scenario, WASM's 2gb address size limit can be prohibitive and lead to crashes or large slowdowns as available memory gets low. Native binaries do not have this restriction and generally have a better chance of working correctly.

FYI that this is now 4gb as of May 2020, and wasm64 just hit stage 3 in June which will remove this limit. v8 has already begun work on implementation.

In esbuild's case WASM can easily be 10x slower than native. WASM will never be the default implementation.

As you pointed out in a previous issue, this is in part because esbuild-wasm can currently only run single threaded whereas the native variant has full multithreading. WebAssembly Threads are now available in Chrome/Edge/Firefox and enables high performance multithreading. We use this for WebContainers and the performance has been able to exceed local execution speeds. This might be worth exploring for esbuild-wasm.
Big fan of esbuild and would love to see/help it adopt the latest improvements for the webassembly version!

Isn't this what the "browser" field in package.json for? It's widely supported by pretty much every bundler, and allows substituting the whole module or specific files for the browser. We already use it for some of our native packages, and I think that was the plan for the rest as well. The environment variable we currently use for other packages is just a temporary hack I believe.

@devongovett for bundling use cases definitely, but for environments that provide full Node runtimes requiring strong security guarantees (i.e. targets of supply chain attacks like enterprises and OSS devs, WebAssembly based containers like WebContainer or kubelet, etc) having the browser field affect Node's real resolution algorithm could be problematic for a few reasons:

Since the browser field is mostly used for bundling use cases, they often specify overrides that patch Node.js code in a way where it would run in a browser but not in Node
Ignoring the above, the host env would have to modify Node's resolution algorithm to account for this, which isn't ideal

So IMO given the advent and growth of WASM adoption in npm packages for local envs, It seems there's a bit of a middle ground we need to accomodate that the browser doesn't seem well suited for.

Hmm interesting. Unfortunately, I think WASM has a couple drawbacks over native packages, which is why we distribute native binaries by default.

Like @evanw said, WASM is significantly slower in our experience. For our case, this is not due to multi-threading. In Parcel, threading is handled in JS using Node's worker_threads, and the native module runs within those workers (so, single threaded). We've seen WASM run up to 30x slower than native in some cases. In particular, the inability to pass buffers between JS and WASM without copying is a huge problem for us.
Some features currently don't work in WASM, e.g. anything involving the filesystem. For our browser REPL, we've reimplemented some of these in JS using a Memory FS layer, but that wouldn't really be the expected way to use Parcel in Node.

For those reasons, I don't think WASM would be the default way to use Parcel, but perhaps could be opt-in. The first issue is fine if you accept the perf hit, but the second I'm not sure how to solve.

I think opt-in would definitely be the right way to handle this. It seems like there's growing consensus that @sokra's idea of standardizing on an export map condition is the right path for a standardized opt-in, so if that sounds sensible I think I'm going to modify the original issue to reflect an export map condition proposal. Also happy to continue discussing alternative ideas otherwise though! 👍

Where do export conditions get standardized? In Node core? Looks like node only supports "import", "require", "node", and "default". One of the reasons Parcel has not yet implemented package exports support is that it's unclear to us how additional conditions get standardized so that they actually work the same way across tools.

+1 to using standardized conditional exports, but that is just a part of the solution.
It would also be great to address the package payload size issue that @evanw brought up. We've struggled with this a bit and ended up using optionalDependencies as the workaround.
Ideally, we could codify/standardize this "non-js extension" part of the package system as well (and update clients not to waste bandwidth, and create noise in the terminal) by honoring the os value  specified in package.json (or consider moving the os specifier info to be part of parent's package.json's optionalDependencies, or specified via a new package.json field like platformDependencies (and there are likely other solutions to consider).
The last missing bit is to standardize a wasm-specific os specifier in package.json, for example "os": "wasm".
This combo would ensure wide-applicability of the design (including for wasm environments), while solve the current hurdles that npm packages with native/non-js extensions face.
So to recap, an ideal solution would not only ensure that the (preferably static) module resolution occurs in a platform-aware way but that also the package installation is performed in platform-aware way to improve the efficiency and DX of the package delivery system.

@devongovett Node.js wouldn't be involved in standardizing an export condition unless it planned on using it itself. I don't think this is one they would use. I think this would be a community standard, which like browser is just an informal agreement among tool makers.

Actually I said that and then @bmeck said they might be interested: https://twitter.com/bradleymeck/status/1423375532503207942. Bradley, is nodejs/node the right place to propose a recognized condition?

yes nodejs/node is the right place, just make an issue and/or PR. This seems like it wouldn't be too controversial except getting the right names for condition and how to enable the mode (flag vs config file vs API)

I personally agree that using exports and imports can solve this in an elegant yet flexible way. Ideally there's a standard condition baked into Node.js that we can use for this which would also disable dlopen as @bmeck mentioned already on Twitter.
More specifically we could try to standardize a --no-addons or --no-native-addons (similar to --no-warnings or --no-force-async-hooks-checks) which disables process.dlopen and enables a no-addons or no-native-addons condition.
Open source maintainers could then provide a different entry point given the no-addons condition:
{
  "exports": {
    ".": {
      "no-addons": "./wasm-entry-point.js",
      "default": "./native-entry-point.js"
    }
  }
}

This would also address the performance concerns in a neat way because by default it uses the "native entry point" and only falls back to the WASM version if the condition is provided, either directly -C=no-addons or through a CLI option --no-addons which does a bit more than just providing a condition.

It thought a bit more about this and wanted to propose another option which is a combination of an imports mapping and optionalDependencies.
If a package depends for example on esbuild then a maintainer could conditionally use the WASM versions with an imports mapping:
{
  "imports": {
    "#esbuild": {
      "no-addons": "esbuild-wasm",
      "default": "esbuild"
    }
  }
}

The library could then import #esbuild and it conditionally imports either the WASM or native package. This requires esbuild-wasm to be specified in optionalDependencies. The downside here is that the initial install gets a little slower because all optional dependencies will be installed.

You might be interested in the following RFC for Yarn: https://github.com/yarnpkg/berry/issues/2751. Each approach attempts to solve different goals, but some of them seem shared.

I like the idea of this issue being solved at the package manager level to avoid downloading unnecessary binaries. I'm still digesting the RFC that @arcanis shared, but will comment there if I have any thoughts.

I agree that if this is handled on the package manager level that'd be ideal so we don't have to download packages which won't be used. I think this with a condition makes up a good solution which should cover most use cases.