RUMM-2385 Intercept window touch events as Session Replay TouchData

What does this PR do?
A brief description of the change being made with this pull request.
Motivation
What inspired you to submit this pull request?
Additional Notes
Anything else we should know when reviewing?
Review checklist (to be filled by reviewers)

[ ] Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
[ ] Make sure you discussed the feature or bugfix with the maintaining team in an Issue
[ ] Make sure each commit and the PR mention the Issue number (cf the CONTRIBUTING doc)

Codecov Report

Merging #1009 (af0db01) into feature/session-replay-vo (922fd29) will increase coverage by 0.10%.
The diff coverage is 87.25%.

@@                      Coverage Diff                      @@
##           feature/session-replay-vo    #1009      +/-   ##
=============================================================
+ Coverage                      82.99%   83.09%   +0.10%
=============================================================
  Files                            307      316       +9
  Lines                           9980    10181     +201
  Branches                        1628     1675      +47
=============================================================
+ Hits                            8282     8459     +177
- Misses                          1193     1196       +3
- Partials                         505      526      +21

Impacted Files
Coverage Δ

...nal/tracking/AndroidXFragmentLifecycleCallbacks.kt
93.48% <ø> (ø)

...droid/sessionreplay/processor/SnapshotProcessor.kt
100.00% <ø> (ø)

...m/datadog/android/sessionreplay/recorder/IntExt.kt
100.00% <ø> (ø)

...com/datadog/android/sessionreplay/recorder/Node.kt
66.67% <66.67%> (+66.67%)
:arrow_up:

...ionreplay/recorder/mapper/ButtonWireframeMapper.kt
66.67% <66.67%> (ø)

...android/sessionreplay/recorder/SnapshotProducer.kt
77.14% <77.14%> (+27.14%)
:arrow_up:

...y/recorder/mapper/ViewScreenshotWireframeMapper.kt
80.00% <80.00%> (ø)

...onreplay/recorder/mapper/GenericWireframeMapper.kt
85.71% <85.71%> (ø)

...ssionreplay/recorder/mapper/ViewWireframeMapper.kt
88.46% <88.46%> (ø)

...ssionreplay/recorder/mapper/TextWireframeMapper.kt
89.66% <89.66%> (ø)

... and 12 more


Enable OSSF scorecard workflow

Start using the OpenSSF scorecard to track overall health and security of this repository.  This PR also adds a badge to the main page to see the results, which can be clicked on to see details (which will only show results once the PR is merged causing the workflow to run for the first time).

Merging since approved and no other comments for 2 days, hoping this is the right process.


convert all loops to higher order functions

Description
remove iterators for higher-order functions
Benchmarks 🤷
develop:

higher_order:

posting for comments - not sure what I am even trying.

Probably should go in this direction. Bit o a waste of a night but good to know at least.

Probably should go in this direction. Bit o a waste of a night but good to know at least.

Definitely an interesting test. I wonder what the performance looks like after having gone through the babel -> js conversion as part of us supporting node js. Would it transpile the iterators to for loops to improve performance?

Definitely an interesting test. I wonder what the performance looks like after having gone through the babel -> js conversion as part of us supporting node js. Would it transpile the iterators to for loops to improve performance?

closing this out, I'll bee adding a benchmark PR from another branch

closing this out, I'll bee adding a benchmark PR from another branch

^

^


[Enhancement][Opensearch] Readiness Probes

Hi :). Hopefully this is not a duplicated issue.
Is your feature request related to a problem? Please describe.
During rolling restarts of data nodes, Readiness probes (https://github.com/opensearch-project/helm-charts/blob/main/charts/opensearch/values.yaml#L351) are not waiting the cluster to be green again before selecting the data node as Ready. This causes nodes to be restarted really fast causing the cluster to go into RED state quite easily (depending on the number of nodes, shards, replicas...).
It can also break that migration of .kibana indices showing the following message:
[resource_already_exists_exception]: index [.kibana_Y/XXXXXXXXX_XXXXXXXX] already exists
Describe the solution you'd like
It would be great to have a similar solution ElasticSearch is using on their charts. They are waiting the cluster to be green again before marking the node as Ready.
Describe alternatives you've considered
As a temporary fix. I increased the initialDelaySeconds to wait the cluster to load the node and find the shards again. But with a large amount of data, it's required several minutes. it's hard to find a value for this.

@prudhvigodithi Could please look into the issue. Thanks

Hey @ic-ruben-burgue thanks for bringing this up, looks similar to https://github.com/opensearch-project/helm-charts/issues/307, the ideal way is to use the cluster health endpoint  which is unauthenticated for the probe.
 curl -XGET "http://localhost:9200/_cluster/health?pretty"
{
  "cluster_name" : "opensearch",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "discovered_master" : true,
  "discovered_cluster_manager" : true,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

@TheAlgo @bbarani @peterzhuamazon WDYT?

Hey I just found one more huddle, this is not true as soon the the security plugin pitches in, example as follows
curl -XGET "https://localhost:9200/_cluster/health?pretty"  -k
Unauthorized

I have tested the command before the security plugin is installed which works fine. This should be ideally open without authentication.
@nknize @dblock I dont see we have any unauthenticated endpoints once the security is installed to check the cluster health, we should have one to test the health of the cluster without authentication before routing the traffic, this will help in ensuring the cluster is green and identify it without any user password, WDYT?
Adding @bbarani  @CEHENKLE
Thank you


Documentation at docs.rs does not specify required crate features

I just looked up https://docs.rs/uefi/0.16.0/uefi/table/boot/struct.BootServices.html#method.find_handles and was surprised to find out that my imported uefi crate didn't have a find_handles function. I had to look into the source code to realize that find_handles is only available when the "exts" feature is enabled.
docs.rs supports a syntax to specify the required features and make them part of the public documentation. An example is https://docs.rs/ntfs/0.2.0/ntfs/struct.NtfsTime.html#impl-TryFrom<OffsetDateTime>
Steps required:

Add a package.metadata.docs.rs section to Cargo.toml like https://github.com/ColinFinck/ntfs/blob/b0058b9b9a19bd6727a8be7ef603d1e4421255ed/Cargo.toml#L40-L42
Enable the doc_cfg feature for the newly added docsrs argument: https://github.com/ColinFinck/ntfs/blob/b0058b9b9a19bd6727a8be7ef603d1e4421255ed/src/lib.rs#L37
Add a corresponding cfg_attr for every cfg(feature(...)) attribute: https://github.com/ColinFinck/ntfs/blob/b0058b9b9a19bd6727a8be7ef603d1e4421255ed/src/time.rs#L36

This works at least for impls and fns.
Would be great to see that in the next version of uefi-rs on docs.rs :)

Thanks for the suggestion, I agree this would be a nice improvement to the docs. I put up a PR for it, with one deviation from what you suggested: using doc_auto_cfg so that we don't have to annotate each item behind a feature config.

I've merged that change and pushed a minor 0.16.1 release. Confirmed that docs.rs now shows the feature badges, e.g.: https://docs.rs/uefi/0.16.1/uefi/table/boot/struct.BootServices.html#method.find_handles


Add Ontario to the explicit key list

Ontario released it's vaccine certificate today.
Keys available at https://prd.pkey.dhdp.ontariohealth.ca/.well-known/jwks.json

I just merged #16 which add the same key, but thank you for your contribution !


Feature Request: kubectl cost prices

While kubectl cost currently shows costs for various assets, it doesn't show the source of pricing data.
/allNodePricing from https://github.com/opencost/opencost/blob/develop/pkg/costmodel/router.go#L1757 returns JSON for on-prem similar to
{
    "code": 200,
    "status": "success",
    "data": {
        "default": {
            "CPU": "10.25",
            "RAM": "3.50",
            "GPU": ""
        },
        "default,gpu": {
            "CPU": "10.25",
            "RAM": "3.50",
            "GPU": "2.95"
        },
        "default,spot": {
            "CPU": "0.6655",
            "RAM": "0.892",
            "GPU": ""
        }
    }
}

Public clouds have more complicated responses, but this would make it easier to see prices applied (and potentially customized). Even only showing On Demand pricing would be useful.

Please assign this issue to me

Hi @saif-xgrid we welcome contributions, but I'd like to understand where you're heading with the implementation before you get too deep. Would you post a brief outline?
@mattray would you elaborate on the ask a little bit? What exactly does a user want to accomplish? Do they want to see the price-per-resource of nodes in their cluster?

@michaelmdresser I think the data returned from /allNodePricing makes it easier to understand node costs, especially when you're overriding costs. Presentation may be a bit difficult, but I think this is an API call that would be useful from the CLI as well. Maybe this is too ambitious, given there are 16 SKUs within the JSON returned from my 4 node cluster on AWS.
@saif-xgrid if you want to do it, maybe take a look at the /addNodePricing from an AWS or other cloud deployment and provide a mock-up of output from the returned JSON.

Thanks for the extra detail @mattray. If using default pricing or custom pricing, I think this makes sense as a feature to get a nicely-formatted view of configured node pricing.
However, with cloud providers I'm concerned that we won't give correct data. Kubecost does best-effort guesses for node pricing for cloud providers, but the ultimate answer is always provided by the bill, which is why we have have cloud provider integrations and reconciliation. I'm not intimately familiar with the /allNodePricing endpoint, but I'm worried that we'll give a misleading response for users running on cloud providers who are not being billed default rates due to discounts, regional variation, etc.
Again, though, I'm not super familiar with the endpoint. It may account for some of these details already. Some research is required to make sure we play well with Kubecost.
@kubecost/product how can we best coordinate with an open source volunteer here? We're in a tricky spot in which closed-source Kubecost code may influence a contribution.


Trim should trim all whitespace characters

The trim/ltrim/rtrim functions say in their docstrings that they trim whitespace. However, they were by default only trimming space characters. I've replaced the space with the regex whitespace metacharacter, so they should behave like clojure.string/trim/triml/trimr

Thanks


How can I select how many times an alternative appears?

I want to be able to weight the alternatives, but I cant find any param that allows me to do that.
I really don't understand how the alternatives are "assigned", is it totally random?

Alternatives are assigned using a deterministic algorithm  based on the client_id. This allows for clients to "predict" which alternative is going to be assigned. Relevant code is here.

I was checking this http://facebook.github.io/planout/docs/random-operators.html
WeightedChoice

Not sure if I should post on this thread but I can't figure out how to set Confidence as well. No param for that

This project doesn't implement the planout algorithm.


Problems running with version from master branch

Hi,
I like to use the extending palettes feature so I installed the master version...
Unfortunately I'm getting this error:
>material-tools.cmd -c angular-material-config.json

module.js:471
    throw err;
    ^

Error: Cannot find module '../lib/MaterialCLI'
    at Function.Module._resolveFilename (module.js:469:15)
    at Function.Module._load (module.js:417:25)
    at Module.require (module.js:497:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (C:\...\npm\node_modules\angular-material-tools\bin\material-tools:8:1)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)

Thanks in advance!

I use
npm.cmd install -g git+https://github.com/angular/material-tools.git#master
for installation (that succeeds) but still the error :-(

Why are you installing it from GitHub? Try using npm install -g angular-material-tools.

I like to use the extending palettes feature...

Released a new version that should have it.

Great - Thanks a lot!

ok so cli tools are available again but the now I get this error (with palette extending config):
[08:52:54]: Error: Missing colors 50, 100, 200, 300, 400, 500, 600, 700, 800, 900, A100, A200, A400, A700 in palette _grey!
    at checkPaletteValid (C:\...\.material-cache\1.1.1\module\angular-material.js:5761:13)
    at Object.definePalette (C:...\.material-cache\1.1.1\module\angular-material.js:5744:22)
    at ThemeBuilder._registerPalette (C:\...\npm\node_modules\angular-material-tools\lib\builders\ThemeBuilder.js:74:34)
    at C:\...\npm\node_modules\angular-material-tools\lib\builders\ThemeBuilder.js:55:81
    at Function.Utils.forEach (C:\...\npm\node_modules\angular-material-tools\lib\common\Utils.js:42:17)
    at ThemeBuilder._buildThemingService (C:\...\npm\node_modules\angular-material-tools\lib\builders\ThemeBuilder.js:55:23)
    at new ThemeBuilder (C:\...\npm\node_modules\angular-material-tools\lib\builders\ThemeBuilder.js:22:14)
    at MaterialTools.MaterialBuilder._buildTheme (C:\...\npm\node_modules\angular-material-tools\lib\builders\MaterialBuilder.js:63:34)
    at C:\...\npm\node_modules\angular-material-tools\lib\MaterialTools.js:121:50

It seems the new code is not in the release... can you check again?
Thanks

Sorry, I hadn't pushed all of the commits properly last time. I've released v1.0.0-beta.5 that should sort it out. Can you give it another try?

It works now! Thanks again.
One last question: If I include the generated angular-material.js & angular-material.css all the style-tags in the head are still generated. How can I prevent that?

ok got it - $mdThemingProvider.disableTheming();


Tested on Stretch

I changed code to meet latest version of Raspbian Stretch and R Pi 3.

I apologize for not seeing this PR sooner. Other than a few code format OCD items, all looks good to me. Thank you for testing it on the Pi3. Thank you for comprehending my madness enough to actually get this working w/ dnsmasq :)
I will probably give this a whirl on my Pi2 as well just in case (maybe pointless since Pi4s are here).
If you don't mind, could you rebase the 3 commits and squash them into one for the merge?  Once again, much thanks!!
👍

One other small note - if you dont mind, can we revert the AP name and pwd in config.json?

Sure, I will do that right away.
On Thu, 27 Jun 2019 at 04:26, Shaba Abhiram<EMAIL_ADDRESS>wrote:

One other small note - if you dont mind, can we revert the AP name and pwd
in config.json?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/sabhiram/raspberry-wifi-conf/pull/33?email_source=notifications&email_token=AH3QZ7MIM4QJPSSNQNXU3YDP4QQMTA5CNFSM4HAWMXAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYVPDCY#issuecomment-506130827,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AH3QZ7MHBHWCPXHGYBKLDFDP4QQMTANCNFSM4HAWMXAA
.

Reverted. Please test if you got chance. I noticed that in some cases
password zzzzzzz won't work as it is considered too simple...
On Thu, 27 Jun 2019 at 08:59, Zoran Stankovic<EMAIL_ADDRESS>wrote:

Sure, I will do that right away.
On Thu, 27 Jun 2019 at 04:26, Shaba Abhiram<EMAIL_ADDRESS>wrote:

One other small note - if you dont mind, can we revert the AP name and
pwd in config.json?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/sabhiram/raspberry-wifi-conf/pull/33?email_source=notifications&email_token=AH3QZ7MIM4QJPSSNQNXU3YDP4QQMTA5CNFSM4HAWMXAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYVPDCY#issuecomment-506130827,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AH3QZ7MHBHWCPXHGYBKLDFDP4QQMTANCNFSM4HAWMXAA
.

Perfect, thanks for that note - I might just make the password change on master after the merge. Thanks for all the hard work!


fix(sbb-form-field): remove label property and attribute

In order to support SSR we have to remove the label property. As replacement the <label> tag should be used.
BREAKING CHANGE: label property and attribute of <sbb-form-field> was removed. Use <label> tag inside <sbb-form-field> to provide the label information. E.g. <sbb-form-field label="Example">...</sbb-form-field> becomes <sbb-form-field><label>Example</label>...</sbb-form-field>

Codecov Report
Attention: Patch coverage is 95.23810% with 1 lines in your changes are missing coverage. Please review.

:exclamation: No coverage uploaded for pull request base (main@e374eb9). Click here to learn what that means.

Files
Patch %
Lines

src/components/form-field/form-field/form-field.ts
95.23%
1 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2523   +/-   ##
=======================================
  Coverage        ?   93.18%
=======================================
  Files           ?      301
  Lines           ?    24918
  Branches        ?     2032
=======================================
  Hits            ?    23220
  Misses          ?     1669
  Partials        ?       29

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Ignore lastCopied if not paste.

Fixes #5613.
An alternative fix here would be to ensure that lastCopied’s text always includes a trailing newline when it’s line-wise.

Thanks for the fix! Merged.


Multiple Cookiebot instances for an app with multiple domains.

Hi,
I have a Nuxt 3 app with multiple domains. For every domain you should have a different Cookiebot instance. Is it possible to add multiple tokens? One for every domain?
Any ideas?

Hi @pieterjanmaes, thank you for your issue :-)
Currently this is not possible, but it is a good enhancement. I'll add it to the roadmap to pick this up.

Hi @bnachtweh,
For now w're gonna add Cookiebot with our GTM.
Thanks for putting it on the roadmap!


[SPARK-18922][SQL][CORE][STREAMING][TESTS] Fix all identified tests failed due to path and resource-not-closed problems on Windows

What changes were proposed in this pull request?
This PR proposes to fix all the test failures identified by testing with AppVeyor.
Scala - aborted tests
WindowQuerySuite:
  Exception encountered when attempting to run a suite with class name: org.apache.spark.sql.hive.execution.WindowQuerySuite *** ABORTED *** (156 milliseconds)
   org.apache.spark.sql.AnalysisException: LOAD DATA input path does not exist: C:projectssparksqlhive   argetscala-2.11   est-classesdatafilespart_tiny.txt;

OrcSourceSuite:
 Exception encountered when attempting to run a suite with class name: org.apache.spark.sql.hive.orc.OrcSourceSuite *** ABORTED *** (62 milliseconds)
   org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.lang.IllegalArgumentException: Can not create a Path from an empty string);

ParquetMetastoreSuite:
 Exception encountered when attempting to run a suite with class name: org.apache.spark.sql.hive.ParquetMetastoreSuite *** ABORTED *** (4 seconds, 703 milliseconds)
   org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.lang.IllegalArgumentException: Can not create a Path from an empty string);

ParquetSourceSuite:
 Exception encountered when attempting to run a suite with class name: org.apache.spark.sql.hive.ParquetSourceSuite *** ABORTED *** (3 seconds, 907 milliseconds)
   org.apache.spark.sql.AnalysisException: Path does not exist: file:/C:projectsspark  arget mpspark-581a6575-454f-4f21-a516-a07f95266143;

KafkaRDDSuite:
 Exception encountered when attempting to run a suite with class name: org.apache.spark.streaming.kafka.KafkaRDDSuite *** ABORTED *** (5 seconds, 212 milliseconds)
   java.io.IOException: Failed to delete: C:\projects\spark\target\tmp\spark-4722304d-213e-4296-b556-951df1a46807

DirectKafkaStreamSuite:
 Exception encountered when attempting to run a suite with class name: org.apache.spark.streaming.kafka.DirectKafkaStreamSuite *** ABORTED *** (7 seconds, 127 milliseconds)
   java.io.IOException: Failed to delete: C:\projects\spark\target\tmp\spark-d0d3eba7-4215-4e10-b40e-bb797e89338e
   at org.apache.spark.util.Utils$.deleteRecursively(Utils.scala:1010)

ReliableKafkaStreamSuite
 Exception encountered when attempting to run a suite with class name: org.apache.spark.streaming.kafka.ReliableKafkaStreamSuite *** ABORTED *** (5 seconds, 498 milliseconds)
   java.io.IOException: Failed to delete: C:\projects\spark\target\tmp\spark-d33e45a0-287e-4bed-acae-ca809a89d888

KafkaStreamSuite:
 Exception encountered when attempting to run a suite with class name: org.apache.spark.streaming.kafka.KafkaStreamSuite *** ABORTED *** (2 seconds, 892 milliseconds)
   java.io.IOException: Failed to delete: C:\projects\spark\target\tmp\spark-59c9d169-5a56-4519-9ef0-cefdbd3f2e6c

KafkaClusterSuite:
 Exception encountered when attempting to run a suite with class name: org.apache.spark.streaming.kafka.KafkaClusterSuite *** ABORTED *** (1 second, 690 milliseconds)
   java.io.IOException: Failed to delete: C:\projects\spark\target\tmp\spark-3ef402b0-8689-4a60-85ae-e41e274f179d

DirectKafkaStreamSuite:
 Exception encountered when attempting to run a suite with class name: org.apache.spark.streaming.kafka010.DirectKafkaStreamSuite *** ABORTED *** (59 seconds, 626 milliseconds)
   java.io.IOException: Failed to delete: C:\projects\spark\target\tmp\spark-426107da-68cf-4d94-b0d6-1f428f1c53f6

KafkaRDDSuite:
Exception encountered when attempting to run a suite with class name: org.apache.spark.streaming.kafka010.KafkaRDDSuite *** ABORTED *** (2 minutes, 6 seconds)
   java.io.IOException: Failed to delete: C:\projects\spark\target\tmp\spark-b9ce7929-5dae-46ab-a0c4-9ef6f58fbc2

Java - failed tests
Test org.apache.spark.streaming.kafka.JavaKafkaRDDSuite.testKafkaRDD failed: java.io.IOException: Failed to delete: C:\projects\spark\target\tmp\spark-1cee32f4-4390-4321-82c9-e8616b3f0fb0, took 9.61 sec

Test org.apache.spark.streaming.kafka.JavaKafkaStreamSuite.testKafkaStream failed: java.io.IOException: Failed to delete: C:\projects\spark\target\tmp\spark-f42695dd-242e-4b07-847c-f299b8e4676e, took 11.797 sec

Test org.apache.spark.streaming.kafka.JavaDirectKafkaStreamSuite.testKafkaStream failed: java.io.IOException: Failed to delete: C:\projects\spark\target\tmp\spark-85c0d062-78cf-459c-a2dd-7973572101ce, took 1.581 sec

Test org.apache.spark.streaming.kafka010.JavaKafkaRDDSuite.testKafkaRDD failed: java.io.IOException: Failed to delete: C:\projects\spark\target\tmp\spark-49eb6b5c-8366-47a6-83f2-80c443c48280, took 17.895 sec

org.apache.spark.streaming.kafka010.JavaDirectKafkaStreamSuite.testKafkaStream failed: java.io.IOException: Failed to delete: C:\projects\spark\target\tmp\spark-898cf826-d636-4b1c-a61a-c12a364c02e7, took 8.858 sec

Scala - failed tests
PartitionProviderCompatibilitySuite:
 - insert overwrite partition of new datasource table overwrites just partition *** FAILED *** (828 milliseconds)
   java.io.IOException: Failed to delete: C:\projects\spark\target\tmp\spark-bb6337b9-4f99-45ab-ad2c-a787ab965c09

 - SPARK-18635 special chars in partition values - partition management true *** FAILED *** (5 seconds, 360 milliseconds)
   org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.lang.IllegalArgumentException: Can not create a Path from an empty string);

 - SPARK-18635 special chars in partition values - partition management false *** FAILED *** (141 milliseconds)
   org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.lang.IllegalArgumentException: Can not create a Path from an empty string);

UtilsSuite:
 - reading offset bytes of a file (compressed) *** FAILED *** (0 milliseconds)
   java.io.IOException: Failed to delete: C:\projects\spark\target\tmp\spark-ecb2b7d5-db8b-43a7-b268-1bf242b5a491

 - reading offset bytes across multiple files (compressed) *** FAILED *** (0 milliseconds)
   java.io.IOException: Failed to delete: C:\projects\spark\target\tmp\spark-25cc47a8-1faa-4da5-8862-cf174df63ce0

StatisticsSuite:
 - MetastoreRelations fallback to HDFS for size estimation *** FAILED *** (110 milliseconds)
   org.apache.spark.sql.catalyst.analysis.NoSuchTableException: Table or view 'csv_table' not found in database 'default';

SQLQuerySuite:
 - permanent UDTF *** FAILED *** (125 milliseconds)
   org.apache.spark.sql.AnalysisException: Undefined function: 'udtf_count_temp'. This function is neither a registered temporary function nor a permanent function registered in the database 'default'.; line 1 pos 24

 - describe functions - user defined functions *** FAILED *** (125 milliseconds)
   org.apache.spark.sql.AnalysisException: Undefined function: 'udtf_count'. This function is neither a registered temporary function nor a permanent function registered in the database 'default'.; line 1 pos 7

 - CTAS without serde with location *** FAILED *** (16 milliseconds)
   java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: file:C:projectsspark%09arget%09mpspark-ed673d73-edfc-404e-829e-2e2b9725d94e/c1

 - derived from Hive query file: drop_database_removes_partition_dirs.q *** FAILED *** (47 milliseconds)
   java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: file:C:projectsspark%09arget%09mpspark-d2ddf08e-699e-45be-9ebd-3dfe619680fe/drop_database_removes_partition_dirs_table

 - derived from Hive query file: drop_table_removes_partition_dirs.q *** FAILED *** (0 milliseconds)
   java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: file:C:projectsspark%09arget%09mpspark-d2ddf08e-699e-45be-9ebd-3dfe619680fe/drop_table_removes_partition_dirs_table2

 - SPARK-17796 Support wildcard character in filename for LOAD DATA LOCAL INPATH *** FAILED *** (109 milliseconds)
   java.nio.file.InvalidPathException: Illegal char <:> at index 2: /C:/projects/spark/sql/hive/projectsspark	arget	mpspark-1a122f8c-dfb3-46c4-bab1-f30764baee0e/*part-r*

HiveDDLSuite:
 - drop external tables in default database *** FAILED *** (16 milliseconds)
   org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.lang.IllegalArgumentException: Can not create a Path from an empty string);

 - add/drop partitions - external table *** FAILED *** (16 milliseconds)
   org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.lang.IllegalArgumentException: Can not create a Path from an empty string);

 - create/drop database - location without pre-created directory *** FAILED *** (16 milliseconds)
   org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.lang.IllegalArgumentException: Can not create a Path from an empty string);

 - create/drop database - location with pre-created directory *** FAILED *** (32 milliseconds)
   org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.lang.IllegalArgumentException: Can not create a Path from an empty string);

 - drop database containing tables - CASCADE *** FAILED *** (94 milliseconds)
   CatalogDatabase(db1,,file:/C:/projects/spark/target/tmp/warehouse-d0665ee0-1e39-4805-b471-0b764f7838be/db1.db,Map()) did not equal CatalogDatabase(db1,,file:C:/projects/spark/target/tmp/warehouse-d0665ee0-1e39-4805-b471-0b764f7838be\db1.db,Map()) (HiveDDLSuite.scala:675)

 - drop an empty database - CASCADE *** FAILED *** (63 milliseconds)
   CatalogDatabase(db1,,file:/C:/projects/spark/target/tmp/warehouse-d0665ee0-1e39-4805-b471-0b764f7838be/db1.db,Map()) did not equal CatalogDatabase(db1,,file:C:/projects/spark/target/tmp/warehouse-d0665ee0-1e39-4805-b471-0b764f7838be\db1.db,Map()) (HiveDDLSuite.scala:675)

 - drop database containing tables - RESTRICT *** FAILED *** (47 milliseconds)
   CatalogDatabase(db1,,file:/C:/projects/spark/target/tmp/warehouse-d0665ee0-1e39-4805-b471-0b764f7838be/db1.db,Map()) did not equal CatalogDatabase(db1,,file:C:/projects/spark/target/tmp/warehouse-d0665ee0-1e39-4805-b471-0b764f7838be\db1.db,Map()) (HiveDDLSuite.scala:675)

 - drop an empty database - RESTRICT *** FAILED *** (47 milliseconds)
   CatalogDatabase(db1,,file:/C:/projects/spark/target/tmp/warehouse-d0665ee0-1e39-4805-b471-0b764f7838be/db1.db,Map()) did not equal CatalogDatabase(db1,,file:C:/projects/spark/target/tmp/warehouse-d0665ee0-1e39-4805-b471-0b764f7838be\db1.db,Map()) (HiveDDLSuite.scala:675)

 - CREATE TABLE LIKE an external data source table *** FAILED *** (140 milliseconds)
   org.apache.spark.sql.AnalysisException: Path does not exist: file:/C:projectsspark	arget	mpspark-c5eba16d-07ae-4186-95bb-21c5811cf888;

 - CREATE TABLE LIKE an external Hive serde table *** FAILED *** (16 milliseconds)
   org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.lang.IllegalArgumentException: Can not create a Path from an empty string);

 - desc table for data source table - no user-defined schema *** FAILED *** (125 milliseconds)
   org.apache.spark.sql.AnalysisException: Path does not exist: file:/C:projectsspark	arget	mpspark-e8bf5bf5-721a-4cbe-9d6	at scala.collection.immutable.List.foreach(List.scala:381)d-5543a8301c1d;

MetastoreDataSourcesSuite
 - CTAS: persisted bucketed data source table *** FAILED *** (16 milliseconds)
   java.lang.IllegalArgumentException: Can not create a Path from an empty string

ShowCreateTableSuite:
 - simple external hive table *** FAILED *** (0 milliseconds)
   org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.lang.IllegalArgumentException: Can not create a Path from an empty string);

PartitionedTablePerfStatsSuite:
 - hive table: partitioned pruned table reports only selected files *** FAILED *** (313 milliseconds)
   org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.lang.IllegalArgumentException: Can not create a Path from an empty string);

 - datasource table: partitioned pruned table reports only selected files *** FAILED *** (219 milliseconds)
   org.apache.spark.sql.AnalysisException: Path does not exist: file:/C:projectsspark	arget	mpspark-311f45f8-d064-4023-a4bb-e28235bff64d;

 - hive table: lazy partition pruning reads only necessary partition data *** FAILED *** (203 milliseconds)
   org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.lang.IllegalArgumentException: Can not create a Path from an empty string);

 - datasource table: lazy partition pruning reads only necessary partition data *** FAILED *** (187 milliseconds)
   org.apache.spark.sql.AnalysisException: Path does not exist: file:/C:projectsspark	arget	mpspark-fde874ca-66bd-4d0b-a40f-a043b65bf957;

 - hive table: lazy partition pruning with file status caching enabled *** FAILED *** (188 milliseconds)
   org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.lang.IllegalArgumentException: Can not create a Path from an empty string);

 - datasource table: lazy partition pruning with file status caching enabled *** FAILED *** (187 milliseconds)
   org.apache.spark.sql.AnalysisException: Path does not exist: file:/C:projectsspark	arget	mpspark-e6d20183-dd68-4145-acbe-4a509849accd;

 - hive table: file status caching respects refresh table and refreshByPath *** FAILED *** (172 milliseconds)
   org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.lang.IllegalArgumentException: Can not create a Path from an empty string);

 - datasource table: file status caching respects refresh table and refreshByPath *** FAILED *** (203 milliseconds)
   org.apache.spark.sql.AnalysisException: Path does not exist: file:/C:projectsspark	arget	mpspark-8b2c9651-2adf-4d58-874f-659007e21463;

 - hive table: file status cache respects size limit *** FAILED *** (219 milliseconds)
   org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.lang.IllegalArgumentException: Can not create a Path from an empty string);

 - datasource table: file status cache respects size limit *** FAILED *** (171 milliseconds)
   org.apache.spark.sql.AnalysisException: Path does not exist: file:/C:projectsspark	arget	mpspark-7835ab57-cb48-4d2c-bb1d-b46d5a4c47e4;

 - datasource table: table setup does not scan filesystem *** FAILED *** (266 milliseconds)
   org.apache.spark.sql.AnalysisException: Path does not exist: file:/C:projectsspark	arget	mpspark-20598d76-c004-42a7-8061-6c56f0eda5e2;

 - hive table: table setup does not scan filesystem *** FAILED *** (266 milliseconds)
   org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.lang.IllegalArgumentException: Can not create a Path from an empty string);

 - hive table: num hive client calls does not scale with partition count *** FAILED *** (2 seconds, 281 milliseconds)
   org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.lang.IllegalArgumentException: Can not create a Path from an empty string);

 - datasource table: num hive client calls does not scale with partition count *** FAILED *** (2 seconds, 422 milliseconds)
   org.apache.spark.sql.AnalysisException: Path does not exist: file:/C:projectsspark	arget	mpspark-4cfed321-4d1d-4b48-8d34-5c169afff383;

 - hive table: files read and cached when filesource partition management is off *** FAILED *** (234 milliseconds)
   org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.lang.IllegalArgumentException: Can not create a Path from an empty string);

 - datasource table: all partition data cached in memory when partition management is off *** FAILED *** (203 milliseconds)
   org.apache.spark.sql.AnalysisException: Path does not exist: file:/C:projectsspark	arget	mpspark-4bcc0398-15c9-4f6a-811e-12d40f3eec12;

 - SPARK-18700: table loaded only once even when resolved concurrently *** FAILED *** (1 second, 266 milliseconds)
   org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.lang.IllegalArgumentException: Can not create a Path from an empty string);

HiveSparkSubmitSuite:
 - temporary Hive UDF: define a UDF and use it *** FAILED *** (2 seconds, 94 milliseconds)
   java.io.IOException: Cannot run program "./bin/spark-submit" (in directory "C:\projects\spark"): CreateProcess error=2, The system cannot find the file specified

 - permanent Hive UDF: define a UDF and use it *** FAILED *** (281 milliseconds)
   java.io.IOException: Cannot run program "./bin/spark-submit" (in directory "C:\projects\spark"): CreateProcess error=2, The system cannot find the file specified

 - permanent Hive UDF: use a already defined permanent function *** FAILED *** (718 milliseconds)
   java.io.IOException: Cannot run program "./bin/spark-submit" (in directory "C:\projects\spark"): CreateProcess error=2, The system cannot find the file specified

 - SPARK-8368: includes jars passed in through --jars *** FAILED *** (3 seconds, 521 milliseconds)
   java.io.IOException: Cannot run program "./bin/spark-submit" (in directory "C:\projects\spark"): CreateProcess error=2, The system cannot find the file specified

 - SPARK-8020: set sql conf in spark conf *** FAILED *** (0 milliseconds)
   java.io.IOException: Cannot run program "./bin/spark-submit" (in directory "C:\projects\spark"): CreateProcess error=2, The system cannot find the file specified

 - SPARK-8489: MissingRequirementError during reflection *** FAILED *** (94 milliseconds)
   java.io.IOException: Cannot run program "./bin/spark-submit" (in directory "C:\projects\spark"): CreateProcess error=2, The system cannot find the file specified

 - SPARK-9757 Persist Parquet relation with decimal column *** FAILED *** (16 milliseconds)
   java.io.IOException: Cannot run program "./bin/spark-submit" (in directory "C:\projects\spark"): CreateProcess error=2, The system cannot find the file specified

 - SPARK-11009 fix wrong result of Window function in cluster mode *** FAILED *** (16 milliseconds)
   java.io.IOException: Cannot run program "./bin/spark-submit" (in directory "C:\projects\spark"): CreateProcess error=2, The system cannot find the file specified

 - SPARK-14244 fix window partition size attribute binding failure *** FAILED *** (78 milliseconds)
   java.io.IOException: Cannot run program "./bin/spark-submit" (in directory "C:\projects\spark"): CreateProcess error=2, The system cannot find the file specified

 - set spark.sql.warehouse.dir *** FAILED *** (16 milliseconds)
   java.io.IOException: Cannot run program "./bin/spark-submit" (in directory "C:\projects\spark"): CreateProcess error=2, The system cannot find the file specified

 - set hive.metastore.warehouse.dir *** FAILED *** (15 milliseconds)
   java.io.IOException: Cannot run program "./bin/spark-submit" (in directory "C:\projects\spark"): CreateProcess error=2, The system cannot find the file specified

 - SPARK-16901: set javax.jdo.option.ConnectionURL *** FAILED *** (16 milliseconds)
   java.io.IOException: Cannot run program "./bin/spark-submit" (in directory "C:\projects\spark"): CreateProcess error=2, The system cannot find the file specified

 - SPARK-18360: default table path of tables in default database should depend on the location of default database *** FAILED *** (15 milliseconds)
   java.io.IOException: Cannot run program "./bin/spark-submit" (in directory "C:\projects\spark"): CreateProcess error=2, The system cannot find the file specified

UtilsSuite:
 - resolveURIs with multiple paths *** FAILED *** (0 milliseconds)
   ".../jar3,file:/C:/pi.py[%23]py.pi,file:/C:/path%..." did not equal ".../jar3,file:/C:/pi.py[#]py.pi,file:/C:/path%..." (UtilsSuite.scala:468)

How was this patch tested?
Manually via AppVeyor as below:
Scala - aborted tests
WindowQuerySuite - all passed
OrcSourceSuite:
- SPARK-18220: read Hive orc table with varchar column *** FAILED *** (4 seconds, 417 milliseconds)
  org.apache.spark.sql.execution.QueryExecutionException: FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
  at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:625)
ParquetMetastoreSuite - all passed
ParquetSourceSuite - all passed
KafkaRDDSuite - all passed
DirectKafkaStreamSuite - all passed
ReliableKafkaStreamSuite - all passed
KafkaStreamSuite - all passed
KafkaClusterSuite - all passed
DirectKafkaStreamSuite - all passed
KafkaRDDSuite - all passed

Java - failed tests
org.apache.spark.streaming.kafka.JavaKafkaRDDSuite - all passed
org.apache.spark.streaming.kafka.JavaDirectKafkaStreamSuite - all passed
org.apache.spark.streaming.kafka.JavaKafkaStreamSuite - all passed
org.apache.spark.streaming.kafka010.JavaDirectKafkaStreamSuite - all passed
org.apache.spark.streaming.kafka010.JavaKafkaRDDSuite - all passed

Scala - failed tests
PartitionProviderCompatibilitySuite:
- insert overwrite partition of new datasource table overwrites just partition (1 second, 953 milliseconds)
- SPARK-18635 special chars in partition values - partition management true (6 seconds, 31 milliseconds)
- SPARK-18635 special chars in partition values - partition management false (4 seconds, 578 milliseconds)

UtilsSuite:
- reading offset bytes of a file (compressed) (203 milliseconds)
- reading offset bytes across multiple files (compressed) (0 milliseconds)

StatisticsSuite:
- MetastoreRelations fallback to HDFS for size estimation (94 milliseconds)

SQLQuerySuite:
 - permanent UDTF (407 milliseconds)
 - describe functions - user defined functions (441 milliseconds)
 - CTAS without serde with location (2 seconds, 831 milliseconds)
   java.lang.IllegalArgumentException: Can not create a Path from an empty string
 - derived from Hive query file: drop_database_removes_partition_dirs.q (734 milliseconds)
 - derived from Hive query file: drop_table_removes_partition_dirs.q (563 milliseconds)
 - SPARK-17796 Support wildcard character in filename for LOAD DATA LOCAL INPATH (453 milliseconds)

HiveDDLSuite:
 - drop external tables in default database (3 seconds, 5 milliseconds)
 - add/drop partitions - external table (2 seconds, 750 milliseconds)
 - create/drop database - location without pre-created directory (500 milliseconds)
 - create/drop database - location with pre-created directory (407 milliseconds)
 - drop database containing tables - CASCADE (453 milliseconds)
 - drop an empty database - CASCADE (375 milliseconds)
 - drop database containing tables - RESTRICT (328 milliseconds)
 - drop an empty database - RESTRICT (391 milliseconds)
 - CREATE TABLE LIKE an external data source table (953 milliseconds)
 - CREATE TABLE LIKE an external Hive serde table (3 seconds, 782 milliseconds)
 - desc table for data source table - no user-defined schema (1 second, 150 milliseconds)

MetastoreDataSourcesSuite
 - CTAS: persisted bucketed data source table (875 milliseconds)

ShowCreateTableSuite:
 - simple external hive table (78 milliseconds)

PartitionedTablePerfStatsSuite:
 - hive table: partitioned pruned table reports only selected files (1 second, 109 milliseconds)
- datasource table: partitioned pruned table reports only selected files (860 milliseconds)
 - hive table: lazy partition pruning reads only necessary partition data (859 milliseconds)
 - datasource table: lazy partition pruning reads only necessary partition data (1 second, 219 milliseconds)
 - hive table: lazy partition pruning with file status caching enabled (875 milliseconds)
 - datasource table: lazy partition pruning with file status caching enabled (890 milliseconds)
 - hive table: file status caching respects refresh table and refreshByPath (922 milliseconds)
 - datasource table: file status caching respects refresh table and refreshByPath (640 milliseconds)
 - hive table: file status cache respects size limit (469 milliseconds)
 - datasource table: file status cache respects size limit (453 milliseconds)
 - datasource table: table setup does not scan filesystem (328 milliseconds)
 - hive table: table setup does not scan filesystem (313 milliseconds)
 - hive table: num hive client calls does not scale with partition count (5 seconds, 431 milliseconds)
 - datasource table: num hive client calls does not scale with partition count (4 seconds, 79 milliseconds)
 - hive table: files read and cached when filesource partition management is off (656 milliseconds)
 - datasource table: all partition data cached in memory when partition management is off (484 milliseconds)
 - SPARK-18700: table loaded only once even when resolved concurrently (2 seconds, 578 milliseconds)

HiveSparkSubmitSuite:
 - temporary Hive UDF: define a UDF and use it (1 second, 745 milliseconds)
 - permanent Hive UDF: define a UDF and use it (406 milliseconds)
 - permanent Hive UDF: use a already defined permanent function (375 milliseconds)
 - SPARK-8368: includes jars passed in through --jars (391 milliseconds)
 - SPARK-8020: set sql conf in spark conf (156 milliseconds)
 - SPARK-8489: MissingRequirementError during reflection (187 milliseconds)
 - SPARK-9757 Persist Parquet relation with decimal column (157 milliseconds)
 - SPARK-11009 fix wrong result of Window function in cluster mode (156 milliseconds)
 - SPARK-14244 fix window partition size attribute binding failure (156 milliseconds)
 - set spark.sql.warehouse.dir (172 milliseconds)
 - set hive.metastore.warehouse.dir (156 milliseconds)
 - SPARK-16901: set javax.jdo.option.ConnectionURL (157 milliseconds)
 - SPARK-18360: default table path of tables in default database should depend on the location of default database (172 milliseconds)

UtilsSuite:
 - resolveURIs with multiple paths (0 milliseconds)

After resolving the aborted tests, there is a test failure identified as below:
OrcSourceSuite:
- SPARK-18220: read Hive orc table with varchar column *** FAILED *** (4 seconds, 417 milliseconds)
  org.apache.spark.sql.execution.QueryExecutionException: FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
  at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$runHive$1.apply(HiveClientImpl.scala:625)

This does not look due to this problem so this PR does not fix it here.

Build started: [TESTS] ALL

Test build #70783 has started for PR 16451 at commit e58f0bd.

Test build #70783 has finished for PR 16451 at commit e58f0bd.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70783/
Test FAILed.

Merged build finished. Test FAILed.

retest this please

Test build #70787 has started for PR 16451 at commit e58f0bd.

Test build #70787 has finished for PR 16451 at commit e58f0bd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70787/
Test PASSed.

Merged build finished. Test PASSed.

Build started: [TESTS] org.apache.spark.streaming.kafka.ReliableKafkaStreamSuite

Test build #70822 has started for PR 16451 at commit 40f964d.

Build started: [TESTS] org.apache.spark.streaming.kafka.DirectKafkaStreamSuite

Test build #70823 has started for PR 16451 at commit 991ca7e.

Test build #70822 has finished for PR 16451 at commit 40f964d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70822/
Test FAILed.

Test build #70823 has finished for PR 16451 at commit 991ca7e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70823/
Test FAILed.

retest this please

Test build #70837 has started for PR 16451 at commit 991ca7e.

Build started: [TESTS] org.apache.spark.streaming.kafka010.DirectKafkaStreamSuite

Test build #70837 has finished for PR 16451 at commit 991ca7e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70837/
Test FAILed.

retest this please

Test build #70843 has started for PR 16451 at commit 991ca7e.

Test build #70843 has finished for PR 16451 at commit 991ca7e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70843/
Test PASSed.

Merged build finished. Test PASSed.

Build started: [TESTS] org.apache.spark.streaming.kafka010.DirectKafkaStreamSuite

Test build #70858 has started for PR 16451 at commit d50d10c.

Test build #70858 has finished for PR 16451 at commit d50d10c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70858/
Test PASSed.

Merged build finished. Test PASSed.

Test build #70864 has started for PR 16451 at commit 1ecf9d7.

Test build #70866 has started for PR 16451 at commit 6ff6cdd.

Build started: [TESTS] org.apache.spark.streaming.kafka010.DirectKafkaStreamSuite
Build started: [TESTS] org.apache.spark.sql.hive.execution.SQLQuerySuite
Build started: [TESTS] org.apache.spark.sql.hive.MetastoreDataSourcesSuite
Build started: [TESTS] org.apache.spark.streaming.kafka.ReliableKafkaStreamSuite

Now, there are 30ish tests failed on Windows which I could identify via AppVeyor tests here which I am currently working on. Their causes do not look resource or path related problems

Test build #70864 has finished for PR 16451 at commit 1ecf9d7.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70864/
Test FAILed.

retest this please

Build started: [TESTS] org.apache.spark.sql.hive.PartitionedTablePerfStatsSuite

Test build #70869 has started for PR 16451 at commit 2faa531.

Hi @srowen, do you mind if I ask to check whether the changes look reasonable? (I will double check if the tests are really passed after the tests above are finished.)

Now, there are 30ish tests failed on Windows which I could identify via AppVeyor tests here which I am currently working on. Their causes do not look resource or path related problems

Test build #70869 has finished for PR 16451 at commit 2faa531.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70869/
Test FAILed.

retest this please

Test build #70870 has started for PR 16451 at commit 2faa531.

Test build #70866 has finished for PR 16451 at commit 6ff6cdd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70866/
Test PASSed.

Test build #70871 has started for PR 16451 at commit 162cdf1.

Test build #70871 has finished for PR 16451 at commit 162cdf1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70871/
Test FAILed.

Let me just push a small commit fixing the "not" condition just mainly to retrigger the test.

Test build #70875 has started for PR 16451 at commit 79a17f4.

Test build #70870 has finished for PR 16451 at commit 2faa531.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70870/
Test FAILed.

Test build #70875 has finished for PR 16451 at commit 79a17f4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70875/
Test FAILed.

retest this please

Test build #70881 has started for PR 16451 at commit 79a17f4.

I really don't think using the full path makes HiveSparkSubmitSuite.scala flaky but if it seems frequently failed, let me use the original code path for OSs except Windows.

Test build #70881 has finished for PR 16451 at commit 79a17f4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70881/
Test PASSed.

retest this please

Test build #70891 has started for PR 16451 at commit 79a17f4.

Test build #70891 has finished for PR 16451 at commit 79a17f4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #70892 has started for PR 16451 at commit 79a17f4.

retest this please

Test build #70893 has started for PR 16451 at commit 79a17f4.

Test build #70892 has finished for PR 16451 at commit 79a17f4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70892/
Test FAILed.

Test build #70893 has finished for PR 16451 at commit 79a17f4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70893/
Test PASSed.

Merged build finished. Test PASSed.

retest this please

I just checked each except for the one below is passed in Windows via concatenated full-log.
It seems DirectKafkaStreamSuite is flaky due to the occasional failure of removing created temp directory (the same problem with https://github.com/apache/spark/pull/16451#discussion_r94562756 but that one is constantly failed)
It is sometimes passed  but sometimes failed as below:
DirectKafkaStreamSuite:
 Exception encountered when attempting to run a suite with class name: org.apache.spark.streaming.kafka010.DirectKafkaStreamSuite *** ABORTED *** (59 seconds, 626 milliseconds)
   java.io.IOException: Failed to delete: C:\projects\spark\target\tmp\spark-426107da-68cf-4d94-b0d6-1f428f1c53f6

Test build #70899 has finished for PR 16451 at commit 79a17f4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70899/
Test PASSed.

Merged build finished. Test PASSed.

test this please

Test build #70953 has started for PR 16451 at commit 78c5ba0.

test this please

retest this please

Test build #70954 has started for PR 16451 at commit 78c5ba0.

retest this please

Test build #70955 has started for PR 16451 at commit 78c5ba0.

Test build #70955 has finished for PR 16451 at commit 78c5ba0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70955/
Test FAILed.

Merged build finished. Test FAILed.

Test build #70953 has finished for PR 16451 at commit 78c5ba0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70953/
Test PASSed.

Build started: [TESTS] org.apache.spark.sql.hive.HiveSparkSubmitSuite

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70954/
Test PASSed.

Merged build finished. Test PASSed.

@srowen, regarding https://github.com/apache/spark/pull/16451#discussion_r94572449, would it be okay to leave as is or should I give a shot for Utils.deleteRecursively?

Utils.deleteRecursively does seem to delete the contents first, so I don't know why it fails on Windows.

Maybe, I misunderstood this comment and also I might be wrong but let me try to explain at my best.
On Windows, if we have a folder below and call Utils.deleteRecursively(".\\tmp"),
tmp
└── resource1.txt

It will try to remove resource1.txt first. If anything opened resource1.txt without closing, Windows does not seem allowing to delete resource1.txt and also the folder tmp unless the object referencing resource1.txt closes it or is garbage-collected.
For example,

Windows

scala> new java.io.FileInputStream("resource1.txt")
res2: java.io.FileInputStream = java.io.FileInputStream@4980d3

scala> new java.io.File("resource1.txt").delete()
res3: Boolean = false

scala> new java.io.File("resource1.txt").exists()
res4: Boolean = true

Mac

scala> new java.io.FileInputStream("resource1.txt")
res6: java.io.FileInputStream = java.io.FileInputStream@13805618

scala> new java.io.File("resource1.txt").delete()
res7: Boolean = true

scala> new java.io.File("resource1.txt").exists()
res8: Boolean = false

So, if deleting resource1.txt throws an exception in finally block by true from !file.delete() and true from file.exists(). Afterwords, another exception is thrown by the failure of removing tmp because the directory is not empty in finally block.
This suppresses the original exception message from the failure of removing the contents but throws a new exception that says the failure of removing the parent directory.

Test build #71009 has started for PR 16451 at commit 4e0b166.

Build started: [TESTS] org.apache.spark.streaming.kafka010.DirectKafkaStreamSuite
Build started: [TESTS] org.apache.spark.streaming.kafka.ReliableKafkaStreamSuite

Test build #71009 has finished for PR 16451 at commit 4e0b166.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71009/
Test FAILed.

Merged build finished. Test FAILed.

@HyukjinKwon are you still working on this or believe this is a good point to commit this?

Ah, I just found anothet one. Let me check all other errors again. They are now pretty few (30-ish).

 - recovery with file input stream *** FAILED *** (10 seconds, 205 milliseconds)
   The code passed to eventually never returned normally. Attempted 660 times over 10.014272499999999 seconds. Last failure message: Unexpected internal error near index 1
   \
    ^. (CheckpointSuite.scala:680)

 - SPARK-18220: read Hive orc table with varchar column *** FAILED *** (2 seconds, 563 milliseconds)
   org.apache.spark.sql.execution.QueryExecutionException: FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z

 - rolling file appender - size-based rolling (compressed) *** FAILED *** (15 milliseconds)
   1000 was not less than 1000 (FileAppenderSuite.scala:128)

 - recover from node failures with replication *** FAILED *** (34 seconds, 613 milliseconds)
   org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 6.0 failed 4 times, most recent failure: Lost task 1.3 in stage 6.0 (TID 33, localhost, executor 28): java.io.IOException: org.apache.spark.SparkException: Failed to get broadcast_6_piece0 of broadcast_6
   ...
   Caused by: org.apache.spark.SparkException: Failed to get broadcast_6_piece0 of broadcast_6
   ...

recovery with file input stream - seems the same problem with this.
SPARK-18220: read Hive orc table with varchar column - I could not see the cause at the first look but it seems not related with this problem because apparently the test does not use any path.
rolling file appender - size-based rolling (compressed) - this test seems possible flaky. It was passed in https://ci.appveyor.com/project/spark-test/spark/build/503-6DEDA384-4A91-45CD-AD26-EE0757D3D2AC/job/etb359vk0fwbrqgo
recover from node failures with replication - this test seems possibly flaky too. It was passed in https://ci.appveyor.com/project/spark-test/spark/build/503-6DEDA384-4A91-45CD-AD26-EE0757D3D2AC/job/etb359vk0fwbrqgo
Other test failures are deal with in https://github.com/apache/spark/pull/16501.
Let me add a fix for the first one in CheckpointSuite after verifying it.

Test build #71038 has started for PR 16451 at commit 54ff4cb.

Build started: [TESTS] org.apache.spark.streaming.CheckpointSuite

(FWIW, I am pretty sure of this PR for now)

Test build #71040 has started for PR 16451 at commit f3c08fd.

Test build #71038 has finished for PR 16451 at commit 54ff4cb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71038/
Test PASSed.

Merged build finished. Test PASSed.

Test build #71040 has finished for PR 16451 at commit f3c08fd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71040/
Test PASSed.

Merged build finished. Test PASSed.

Merged to master


The builds should test upgrades from the last major release

It would be great if builds could (perhaps optionally) run e2e on an upgraded cluster: deploy a cluster using the previous version of subctl, re-deploy it using the just-built version of subctl, and run e2e.

This is planned just after 0.7 is released, using https://github.com/submariner-io/submariner-operator/pull/711.

FTR, this was implemented in https://github.com/submariner-io/submariner-operator/pull/778.


link against memcpy and memset properly

when linking without libc, build memcpy.o and memset.o. we can cache these objects with a hash of (release/debug, conditional compilation values, target triple/data layout).
generate a compile unit without memcpy/memset exported. when linking the final output, use ar rcs foo.a foo.o memcpy.o memset.o. Then ld -o foo foo.a (or equivalent).
This accomplishes several things:

omit memcpy/memset when the functions are unused
avoid compiling memcpy/memset with every compiler invocation
allow linking zig-created .o files with libc .o files, so that a single project can have both c code and zig code and everything can be linked together. in this case zig generated code will depend on the libc memset/memcpy implementations.

Why have separate memcpy.o and memset.o instead of prebuilding a mem.a and linking with ld -o foo foo.o mem.a? Doesn't that still provide the selective linking feature we want?

That should work too.


[BUG]row_offset repeat last batch cause wrong result

There is a bug in data_collector.hpp.
When pre_nnz_[local_id * num_params + j] == nnz, the copy of row_offset is skipped, so row_offset remain the same as last batch. But the same nnz does not mean the same value. It will cause embedding vector sit in wrong row.
https://github.com/NVIDIA/HugeCTR/blob/36612f9ecca93248379d5fac6eb405a736a23547/HugeCTR/include/data_readers/data_collector.hpp#L364-L380

Thank you for your feedback! If one_hot_ == true, I think "the same nnz does mean the same value in row_offset buffer. Does it make sense to you?

The fowering two row_offset have the same nnz(5), but they are different.
row_offset1: 0,1,2,2,3,3,3,4,4,4,4,4,4,4,4,4,4,5
row_offset2: 0,1,2,2,3,3,3,4,5,5,5,5,5,5,5,5,5,5

If it's one hot input, row_offset can only be 0,1,2,3,4,5. In your example: '0,1,2,2,3,3,3,4,4,4,4,4,4,4,4,4,4,5' slot 2 has no feature values (nnz == 0) which is not one hot.

It's true that one-hot can only be 0,1,2,3,4,5 in one GPU trainning, and it fine not updating that.
But in multi-GPU trainning, things become different. Here is a full example:
Assuming there are 4 GPU in system.
row_offset A:
GPU0: 0,0,0,1,1,1,1,1,2,3,4,4,4,4,4,
GPU1: 0,0,0,0,0,1,1,1,1,1,1,1,1,2,3,
GPU2: 0,1,2,2,3,3,3,3,3,3,3,4,4,4,4,
GPU3: 0,0,0,0,0,0,1,2,2,2,2,2,3,3,3,
Explain: one key only appear in one GPU. It's one-hot. You can add print in your code to see the real row_offset.
If last GPU0 row_offset is 0,1,1,1,1,1,1,1,2,3,4,4,4,4,4, which has the same nnz, if not update, the row_offset will become:
row_offset B:
GPU0: 0,1,1,1,1,1,1,1,2,3,4,4,4,4,4,
GPU1: 0,0,0,0,0,1,1,1,1,1,1,1,1,2,3,
GPU2: 0,1,2,2,3,3,3,3,3,3,3,4,4,4,4,
GPU3: 0,0,0,0,0,0,1,2,2,2,2,2,3,3,3,
It's wrong. And it's not one-hot anymore. Key0 appears in GPU0 and GPU2.

Closing due to age.


[AMORO-1960] Refactor data expiration to TableMaintainer

Why are the changes needed?

Enhance #1960.
Brief change log

Convenient to expand and manage

How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[x] Run test locally before making a pull request

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not documented)

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

I understand that the only difference between partition expiration in change+base and Iceberg is the calculation logic for determining which partitions need to expire. Change+base requires consideration of two tables. So, can we calculate them separately and pass the results to change and base using MixedTableMaintainer?

LGTM


Virtual Time Support

Introduction
This very primitive patch is the result of our paper [“VT-Mininet: Virtual-time-enabled Mininet for Scalable and Accurate Software-Define Network Emulation”], to appear on ACM Sigcomm Symposium on SDN Research (SOSR) 2015.
Mininet, as an examplary and ordinary network emulator, uses the system clock across all the containers, even if a container is not being scheduled to run. This leads to the issue of temporal fidelity, especially with high workloads.
Virtual time sheds the light on the issue of preserving temporal fidelity for large-scale emulation. The key insight is to trade time with system resources via precisely scaling the time of interactions between containers and physical devices by a factor of N (also called time dilation factor), hence, making an emulated network appear to be N times faster from the viewpoints of applications in the container than it actually is.
How to Use Virtual Time
When creating hosts, feed another parameter time dilation factor, "tdf". For example:
Host = custom( CPULimitedHost, sched = 'cfs', period_us = 100000, cpu = 0.8, tdf = 4 )
......# create Class for topo, switches etc
net = Mininet( topo = Topo, host = Host, switch = Switch, controller = Controller, waitConnected = True, link = Link )

TDF's default value is 1, e.g. no virtual time. So this patch should NOT cause compatible issues.
Test
I test this patch on my Dell XPS 8700 Desktop running on Ubuntu 14.04 LTS. We integrated virtual time with Mininet and conducted many experimental evaluations. Details can be found in our SOSR2015 paper. There is another similar paper on PADS2015 named A Virtual Time System for Linux-container-based Emulation of Software-defined Networks, though it focuses on simulation & emulation system and gives more comprehensive experimental results.
The integrated Mininet code, together with kernel patches, is also separately published at my Github website. In other words, to run Mininet with virtual time, one must first rebuild one's kernel. How to patch kernel is described in that repository's README file.
TODO List

How to provide kernel patch? Option 1: provide patch file with respect to particualr kernel version. Option 2: provide already rebuilt virtual machine  image.
Summary some tests into one or two demos and place them under example diretory.
If the application inside container is at a very deep position of the process tree, setTDF will need to do cascading operations so that TDF of all processes in the tee, including host and all other applications inside the container, is properly changed. This is not implemented yet.
There are many other code regarding adaptive virtual time system. It is mentioned and discussed in both PADS and SOSR papers. However, it is still too immature.

Hello,
I'm trying to use your patch to make a simple test between two hosts connected through a switch. I need UDP traffic and I'm configuring links to 5 Mbps, TDF=2 and iperf udpBW=8Mbps but I receive next error with iperf 2:
[  3] WARNING: did not receive ack of last datagram after 10 tries.
If I use iperf3 test hangs on client command after launching iperf server.
If I run tests under 5Mbps, i.e. 4 Mbps, everything runs properly on iperf 2 and you can see the Time Dilation is working since a 10 seconds test takes around a little bit more than 20 seconds and data are transmitted properly.
Do you know if I missing something important? Due to time dilation I think that I should be able to run tests between 5Mbps and 10Mbps with TDF=2.
Thank you very much in Advance

Have you patched linux kernel? As you can see from the TODO list, I haven't
decided how to post kernel patch for virtual time into
mininet's repository. See this repo
https://github.com/littlepretty/VirtualTimeForMininet, which has patch with
regard to kernel version 3.16.3 and some test examples.
On Mon, Jul 20, 2015 at 3:26 PM, imyelmo<EMAIL_ADDRESS>wrote:

Hello,
I'm trying to use your patch to make a simple test between two hosts
connected through a switch. I need UDP traffic and I'm configuring links to
5 Mbps, TDF=2 and iperf udpBW=8Mbps but I receive next error with iperf 2:
[ 3] WARNING: did not receive ack of last datagram after 10 tries.
If I use iperf3 test hangs on client command after launching iperf server.
If I run tests under 5Mbps, i.e. 4 Mbps, everything runs properly on iperf
2 and you can see the Time Dilation is working since a 10 seconds test
takes around a little bit more than 20 seconds and data are transmitted
properly.
Do you know if I missing something important? Due to time dilation I think
that I should be able to run tests between 5Mbps and 10Mbps with TDF=2.
Thank you very much in Advance
—
Reply to this email directly or view it on GitHub
https://github.com/mininet/mininet/pull/521#issuecomment-123022356.

--
Best Wishes
Yours sincerely
Yan Jiaqi
PhD Student @ CS Department, IIT, Chicago, IL
E-mail<EMAIL_ADDRESS>

Hello,
yes of course, the kernel 3.6.13 is patched with your install script and mininet show the TDF properly. The question is why with TDF=2 and BW=5MB it cannot be transmitted 8Mbps since due to TDF the real bandwidth through the switch should be only 4Mbps. Actually, it is quite strange.
Dr. Isaías Martínez Yelmo
Universidad de Alcalá de Henares
Escuela Politécnica - Dpto. Automática
Campus Universitario N-II, km. 32
Sector Verde. Planta 2. Despacho E-243
28871 Alcalá de Henares Madrid
Spain
E-mail: isaias.martinezy<EMAIL_ADDRESS>Web: http://www.uah.es/pdi/isaias_martinezy
On 2015-07-20, at 23:17, Yan Jiaqi <notifications<EMAIL_ADDRESS>wrote:
Have you patched linux kernel? As you can see from the TODO list, I haven't
decided how to post kernel patch for virtual time into
mininet's repository. See this repo
https://github.com/littlepretty/VirtualTimeForMininet, which has patch with
regard to kernel version 3.16.3 and some test examples.
On Mon, Jul 20, 2015 at 3:26 PM, imyelmo <notifications<EMAIL_ADDRESS>wrote:

Hello,
I'm trying to use your patch to make a simple test between two hosts
connected through a switch. I need UDP traffic and I'm configuring links to
5 Mbps, TDF=2 and iperf udpBW=8Mbps but I receive next error with iperf 2:
[ 3] WARNING: did not receive ack of last datagram after 10 tries.
If I use iperf3 test hangs on client command after launching iperf server.
If I run tests under 5Mbps, i.e. 4 Mbps, everything runs properly on iperf
2 and you can see the Time Dilation is working since a 10 seconds test
takes around a little bit more than 20 seconds and data are transmitted
properly.
Do you know if I missing something important? Due to time dilation I think
that I should be able to run tests between 5Mbps and 10Mbps with TDF=2.
Thank you very much in Advance
—
Reply to this email directly or view it on GitHub
https://github.com/mininet/mininet/pull/521#issuecomment-123022356.

--
Best Wishes
Yours sincerely
Yan Jiaqi
PhD Student @ CS Department, IIT, Chicago, IL
E-mail: littlepretty881203<EMAIL_ADDRESS>—
Reply to this email directly or view it on GitHubhttps://github.com/mininet/mininet/pull/521#issuecomment-123042622.

Honestly, I have not yet tested how virtual time would effect UDP
connections.
Did you run with TCP connections?
On Mon, Jul 20, 2015 at 4:38 PM, imyelmo<EMAIL_ADDRESS>wrote:

Hello,
yes of course, the kernel 3.6.13 is patched with your install script and
mininet show the TDF properly. The question is why with TDF=2 and BW=5MB it
cannot be transmitted 8Mbps since due to TDF the real bandwidth through the
switch should be only 4Mbps. Actually, it is quite strange.
Dr. Isaías Martínez Yelmo
Universidad de Alcalá de Henares
Escuela Politécnica - Dpto. Automática
Campus Universitario N-II, km. 32
Sector Verde. Planta 2. Despacho E-243
28871 Alcalá de Henares Madrid
Spain
E-mail: isaias.martinezy<EMAIL_ADDRESS>Web: http://www.uah.es/pdi/isaias_martinezy
On 2015-07-20, at 23:17, Yan Jiaqi<EMAIL_ADDRESS><EMAIL_ADDRESS>wrote:
Have you patched linux kernel? As you can see from the TODO list, I
haven't
decided how to post kernel patch for virtual time into
mininet's repository. See this repo
https://github.com/littlepretty/VirtualTimeForMininet, which has patch
with
regard to kernel version 3.16.3 and some test examples.
On Mon, Jul 20, 2015 at 3:26 PM, imyelmo<EMAIL_ADDRESS><EMAIL_ADDRESS>wrote:

Hello,
I'm trying to use your patch to make a simple test between two hosts
connected through a switch. I need UDP traffic and I'm configuring links
to
5 Mbps, TDF=2 and iperf udpBW=8Mbps but I receive next error with iperf
2:
[ 3] WARNING: did not receive ack of last datagram after 10 tries.
If I use iperf3 test hangs on client command after launching iperf
server.
If I run tests under 5Mbps, i.e. 4 Mbps, everything runs properly on
iperf
2 and you can see the Time Dilation is working since a 10 seconds test
takes around a little bit more than 20 seconds and data are transmitted
properly.
Do you know if I missing something important? Due to time dilation I
think
that I should be able to run tests between 5Mbps and 10Mbps with TDF=2.
Thank you very much in Advance
—
Reply to this email directly or view it on GitHub
https://github.com/mininet/mininet/pull/521#issuecomment-123022356.

--
Best Wishes
Yours sincerely
Yan Jiaqi
PhD Student @ CS Department, IIT, Chicago, IL
E-mail: littlepretty881203<EMAIL_ADDRESS>—
Reply to this email directly or view it on GitHub<
https://github.com/mininet/mininet/pull/521#issuecomment-123042622>.
—
Reply to this email directly or view it on GitHub
https://github.com/mininet/mininet/pull/521#issuecomment-123051548.

--
Best Wishes
Yours sincerely
Yan Jiaqi
PhD Student @ CS Department, IIT, Chicago, IL
E-mail<EMAIL_ADDRESS>

TCP connection are running properly. Thinking about the ongoing problem, could it be that iperf UDP implementation uses a library to get time references that is not being considered by your contribution?

Then let's dug into iperf's source code. Which version you use for experiment, 2 or 3?
Sent from my iPhone

On Jul 22, 2015, at 08:11, imyelmo<EMAIL_ADDRESS>wrote:
TCP connection are running properly. Thinking about the ongoing problem, could it be that iperf UDP implementation uses a library to get time references that is not being considered by your contribution?
—
Reply to this email directly or view it on GitHub.

I have tested both of them with the same result. You can choose the most suitable for you. They may address the timing in a similar way; although differences exit since iperf3 tries to solve performance issues of UDP at high rates.

Do you have any news about this issue? I've come back from my vacations. Let me know if you need some  help

As I've mentioned before, the kernel API for this isn't quite right.
The kernel feature is clock namespaces and should be implemented as such.
The right way to do this is to use the existing unshare and setns system calls, and to use sysfs to configure it as needed.
Once it's implemented in this fashion, we might have a chance of getting it accepted upstream. In particular, it will be important to demonstrate why clock namespaces is/are a generally useful feature, which I think it should be for:

testing drivers under different timing conditions
testing applications under different timing conditions
testing the kernel and scheduler under different timing conditions

Basically clock namespaces offers a way to decouple real time from reported wall clock time, usually for the purposes of virtually accelerating real hardware.

My apologies. I am busy with something else since August and still haven't
got time to lookup your issue. Would you mind let me see your test script?
On Thu, Sep 17, 2015 at 1:46 AM, imyelmo<EMAIL_ADDRESS>wrote:

Do you have any news about this issue? I've come back from my vacations.
Let me know if you need some help
—
Reply to this email directly or view it on GitHub
https://github.com/mininet/mininet/pull/521#issuecomment-140984591.

--
Best Wishes
Yours sincerely
Yan Jiaqi
PhD Student @ CS Department, IIT, Chicago, IL
E-mail<EMAIL_ADDRESS>

Totally agree with you about the kernel API. This old, raw pull request is
made before SOSR. I am merging these additional system call into the
existing ones.
Also, we think it may be useful if kernel can enable mininet to
freeze/unfreeze process. This will be helpful when one integrate emulation
with simulators and thus need to pause mininet when doing simulation
calculations.
On Fri, Sep 18, 2015 at 5:50 PM, lantz<EMAIL_ADDRESS>wrote:

As I've mentioned before, the kernel API for this isn't quite right.
The kernel feature is clock namespaces and should be implemented as
such.
The right way to do this is to use the existing unshare and setns system
calls, and to use sysfs to configure it as needed.
Once it's implemented in this fashion, we might have a chance of getting
it accepted upstream. In particular, it will be important to demonstrate
why clock namespaces is/are a generally useful feature, which I think it
should be for:

testing drivers under different timing conditions
testing applications under different timing conditions
testing the kernel and scheduler under different timing conditions

Basically clock namespaces offers a way to decouple real time from
reported wall clock time, usually for the purposes of virtually
accelerating real hardware.
—
Reply to this email directly or view it on GitHub
https://github.com/mininet/mininet/pull/521#issuecomment-141588050.

--
Best Wishes
Yours sincerely
Yan Jiaqi
PhD Student @ CS Department, IIT, Chicago, IL
E-mail<EMAIL_ADDRESS>

The freeze/unfreeze idea is an interesting one (literally stopping wall clock time.)

BTW, why you suggest use /sysfs, not /proc, to configure dilation? I think
dilation is a per process/namespace/container thing. So maybe
/proc/$pid/ns/dilation is a more appropriate place?
On Tue, Sep 22, 2015 at 3:42 PM, lantz<EMAIL_ADDRESS>wrote:

The freeze/unfreeze idea is an interesting one (literally stopping wall
clock time.)
—
Reply to this email directly or view it on GitHub
https://github.com/mininet/mininet/pull/521#issuecomment-142413756.

--
Best Wishes
Yours sincerely
Yan Jiaqi
PhD Student @ CS Department, IIT, Chicago, IL
E-mail<EMAIL_ADDRESS>

By procfs I meant /proc.
Yes that's where the clock namespace anchor  (which you attach to using setns for example) should be for a given process, but a few questions remain:

can you make a persistent clock namespace without an anchor process?
how do you set the time dilation factor or other parameters for the clock namespace?
does it make sense to use sysfs to allow you to configure the parameters usig sysctl?

I also think that it should be /proc/$pid/ns/clock - although I guess you could call them "time dilation namespaces" but I really think the name is/should be clock namespaces. I guess it could be clk if you wanted it to be three letters. ;-p
Alternately:

would it make more sense to use process control groups instead of namespaces?

I tend to think that the feature is most correctly thought of as clock namespaces, but it could also be a process control group feature.

BTW for most up-to-date version of comments, please check page - usually I make a few edits/tweaks after my initial draft.

Hello,
I'm trying to get a fresh install of your virtual time implementation in order to provide you a script to see the issue. I did several tests and I'm not sure about all the changes I did. However, no I'm finding problems to compile the kernel:
log info is in build.log
err info is in build.err
step 1. make src code for kernel and module
CHK     include/config/kernel.release
CHK     include/generated/uapi/linux/version.h
CHK     include/generated/utsrelease.h
CALL    scripts/checksyscalls.sh
CHK     include/generated/compile.h
PASYMS  arch/x86/realmode/rm/pasyms.h
LDS     arch/x86/realmode/rm/realmode.lds
LD      arch/x86/realmode/rm/realmode.elf
RELOCS  arch/x86/realmode/rm/realmode.relocs
OBJCOPY arch/x86/realmode/rm/realmode.bin
AS      arch/x86/realmode/rmpiggy.o
LD      arch/x86/realmode/built-in.o
LD      arch/x86/built-in.o
BC      kernel/timeconst.h
/bin/sh: 1: bc: not found
make[1]: *** [kernel/timeconst.h] Error 127
make: *** [kernel] Error 2
step 1.2. make modules
CHK     include/config/kernel.release
CHK     include/generated/uapi/linux/version.h
CHK     include/generated/utsrelease.h
CALL    scripts/checksyscalls.sh
Building modules, stage 2.
MODPOST 3881 modules
step 2. install modules
INSTALL arch/x86/crypto/aes-x86_64.ko
Can't read private key
make[1]: *** [arch/x86/crypto/aes-x86_64.ko] Error 2
make: *** [modinst] Error 2
step 3. install kernel
sh ./arch/x86/boot/install.sh 3.16.3 arch/x86/boot/bzImage
System.map "/boot"
*** Missing file: arch/x86/boot/bzImage
*** You need to run "make" before "make install".
make[1]: *** [install] Error 1
make: *** [install] Error 2
make/install complete for kernel 3.16.3
step 4. update GRUB
grub-mkconfig: You must run this as root
please reboot OS and press ESC on GRUB interface
following software should be rebuilt: iperf-3, basic tests and mininet
Build.err does not give any further information and there is no build.log file.
Have you done some changes recently?

No, you see, there is no commit since 3 month ago.
BTW, since I am re-implementing the "virtual time kernel", it may take a
while for me to push a new set of working code.
On Mon, Sep 28, 2015 at 5:16 AM, imyelmo<EMAIL_ADDRESS>wrote:

Hello,
I'm trying to get a fresh install of your virtual time implementation in
order to provide you a script to see the issue. I did several tests and I'm
not sure about all the changes I did. However, no I'm finding problems to
compile the kernel:
log info is in build.log
err info is in build.err
step 1. make src code for kernel and module
CHK include/config/kernel.release
CHK include/generated/uapi/linux/version.h
CHK include/generated/utsrelease.h
CALL scripts/checksyscalls.sh
CHK include/generated/compile.h
PASYMS arch/x86/realmode/rm/pasyms.h
LDS arch/x86/realmode/rm/realmode.lds
LD arch/x86/realmode/rm/realmode.elf
RELOCS arch/x86/realmode/rm/realmode.relocs
OBJCOPY arch/x86/realmode/rm/realmode.bin
AS arch/x86/realmode/rmpiggy.o
LD arch/x86/realmode/built-in.o
LD arch/x86/built-in.o
BC kernel/timeconst.h
/bin/sh: 1: bc: not found
make[1]: *** [kernel/timeconst.h] Error 127
make: *** [kernel] Error 2
step 1.2. make modules
CHK include/config/kernel.release
CHK include/generated/uapi/linux/version.h
CHK include/generated/utsrelease.h
CALL scripts/checksyscalls.sh
Building modules, stage 2.
MODPOST 3881 modules
step 2. install modules
INSTALL arch/x86/crypto/aes-x86_64.ko
Can't read private key
make[1]: *** [arch/x86/crypto/aes-x86_64.ko] Error 2
make: *** [modinst] Error 2
step 3. install kernel
sh ./arch/x86/boot/install.sh 3.16.3 arch/x86/boot/bzImage
System.map "/boot"
*** Missing file: arch/x86/boot/bzImage
*** You need to run "make" before "make install".
make[1]: *** [install] Error 1
make: *** [install] Error 2
make/install complete for kernel 3.16.3
step 4. update GRUB
grub-mkconfig: You must run this as root
please reboot OS and press ESC on GRUB interface
following software should be rebuilt: iperf-3, basic tests and mininet
Build.err does not give any further information and there is no build.log
file.
Have you done some changes recently?
—
Reply to this email directly or view it on GitHub
https://github.com/mininet/mininet/pull/521#issuecomment-143700928.

--
Best Wishes
Yours sincerely
Yan Jiaqi
PhD Student @ CS Department, IIT, Chicago, IL
E-mail<EMAIL_ADDRESS>

I think that  I have found the problem. I will let you know of my progress. I will try to get a simple test to show the issue. I hope it will useful for the current release an the next one. Thanks for your help.

I am not sure what persistent means for a clock namespace? In my
implementation, each host is an anchor and all its applications(its
children) only inherit TDF from it.
As for set TDF, my first thought was to use /proc/$pid/ns/clk. But the
problems is, all /proc/$pid/ns/ files are actually special symbolic files.
When you read this file, it just give you its inode number, which you can
use as an ID to distinguish whether two processes belong to the same ns. I
am really not sure about what will happen if you write sth to this symbolic
link. When I follow this link, it just give its inode number back.
So /proc/$pid/ns/clk is the place to show if some processes are in the same
clock namespace, and this is the only usage of clock namespace?
Since “set TDF” involves userspace IO operations, a more simple way is to
put TDF at /proc/$pid/tdf and handle writes/reads to it. However, changing
TDF during host running is more complicated than I thought.

On Thu, Sep 24, 2015 at 3:20 PM, lantz<EMAIL_ADDRESS>wrote:

By procfs I meant /proc.
Yes that's where the clock namespace anchor (which you attach to using
setns for example) should be for a given process, but a few questions
remain:

can you make a persistent clock namespace without an anchor process?
how do you set the time dilation factor or other parameters for the
clock namespace?
does it make sense to use sysfs to allow you to configure the
parameters usig sysctl?

I also think that it should be /proc/$pid/ns/clock - although I guess you
could call them "time dilation namespaces" but I really think the name
is/should be clock namespaces. I guess it could be clk if you wanted it
to be three letters. ;-p
Alternately:

would it make more sense to use process control groups instead of
namespaces?

I tend to think that the feature is most correctly thought of as clock
namespaces, but it could also be a process control group feature.
—
Reply to this email directly or view it on GitHub
https://github.com/mininet/mininet/pull/521#issuecomment-143040341.

--
Best Wishes
Yours sincerely
Yan Jiaqi
PhD Student @ CS Department, IIT, Chicago, IL
E-mail<EMAIL_ADDRESS>

I am not sure what persistent means for a clock namespace? In my
implementation, each host is an anchor and all its applications(its
children) only inherit TDF from it.
As for set TDF, my first thought was to use /proc/$pid/ns/clk. But the
problems is, all /proc/$pid/ns/ files are actually special symbolic files.
When you read this file, it just give you its inode number, which you can
use as an ID to distinguish whether two processes belong to the same ns. I
am really not sure about what will happen if you write sth to this symbolic
link. When I follow this link, it just give its inode number back.
So /proc/$pid/ns/clk is the place to show if some processes are in the same
clock namespace, and this is the only usage of clock namespace?
Since “set TDF” involves userspace IO operations, a more simple way is to
put TDF at /proc/$pid/tdf and handle writes/reads to it. However, changing
TDF during host running is more complicated than I thought.

I'm still looking forward to clock namespaces - it's obviously a useful feature for Mininet, and the right API is almost certainly to follow the namespace approach. It also seems that it would be useful for Linux in general to test timing-dependent behavior and to expose race conditions and other bugs.

Sorry for not being active for several month. Other projects caught me. Recently we applied virtual time to synchronize simulation and emulation, by entirely freeze/stop the emulated hosts. So right now the system is a little bit messy. I will first test and post the dilation portion of the system hopefully before next week. Then maybe the dilation+freeze system later.

No problem, we are also working on several things . I can try to perform some tests during July if possible.

I have been testing dilating link throughput in this separate repo VirtualTimeKernel. It contains kernel changes and a patched mininet based on an old version. There are two main issues, mentioned in the README, that I am still working on. If the virtual time interface (also mentioned in README) now looks OK for mininet, I will begin to patch on the current source code of mininet.


Configure Azure Load Balancer To configure Azure Load Balancer, we recommend that you use the Azure Standard Load Balancer SKU and then

When you are directing traffic from cluster nodes to virtual host name of LB, got a connection time out error.  the same works if you reach the backend VMs directly(bypassing the LB) or if they disable Floating IP on the LB rules
Since the setup was using Standadrd internal LB, which by design doesn’t have a internet connectivity.
The corosync was using udp protocol for communication.
Resolution
For the first issue,  please enable Public IP for the VM’s
For the 2nd Issue ,please allow UDP protocol in NSG rules.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 44665e2e-a3ea-15c6-3a18-b40d34eb7e0f
Version Independent ID: bdad828b-17bb-8db9-c15d-784430aafd07
Content: Set up IBM Db2 HADR on Azure virtual machines (VMs) on RHEL
Content Source: articles/virtual-machines/workloads/sap/high-availability-guide-rhel-ibm-db2-luw.md
Service: virtual-machines-linux
GitHub Login: @msjuergent
Microsoft Alias: juergent

We are currently cleaning up old issues and closing out items that are greater than 90 days old. If an issue is still present, please open a new feedback item on the document so we can prioritize correctly.
#please-close


LUCI infra failure: "build.summary_markdown: too big to accept"

https://chromium-swarm.appspot.com/task?id=63218ab260e2c010&o=true&w=true
[W2023-06-29T13:34:13.388307-07:00 19826 0 client.go:357] RPC failed permanently: rpc error: code = InvalidArgument desc = build.summary_markdown: too big to accept (4426 > 4000 bytes) {"host":"cr-buildbucket.appspot.com", "method":"UpdateBuild", "service":"buildbucket.v2.Builds"}
[I2023-06-29T13:34:13.388348-07:00 19826 0 options.go:227] dropped 4 items over 3.1829302s
[E2023-06-29T13:34:13.388433-07:00 19826 0 annotate.go:273] original error: rpc error: code = InvalidArgument desc = build.summary_markdown: too big to accept (4426 > 4000 bytes)

goroutine 1:
From frame 0 to 0, the following wrappers were found:
  internal reason: MultiError 1/1: following first non-nil error.

#0 bbagent/main.go:510 - main.readyToFinalize()
  reason: fatal error in buildbucket.UpdateBuild

The real failure is

But the markdown summary should be fixed as well.
@christopherfujino do we have a separate bug to track the test failure?

The real failure is
Waiting for application to render first frame...
[2023-06-29 13:01:20.017507] [STDOUT] stdout: [+10013 ms] First frame is taking longer than expected...
[2023-06-29 13:01:20.017637] [STDOUT] stdout: [        ] View ID: _flutterView/0x10483b420
[2023-06-29 13:01:20.032007] [STDOUT] stdout: [  +14 ms] Isolate isolates/3315763852458143 state:
[2023-06-29 13:01:20.037333] [STDOUT] stdout: [   +5 ms] {"type":"Isolate","id":"isolates/3315763852458143","number":"3315763852458143","name":"main","isSystemIsolate":false,"isolateGroupId":"isolateGroups/2187587266607717932","isolateFlags":[{"name":"copy_parent_code","valueAsString":"false"},{"name":"is_system_isolate","valueAsString":"false"}],"startTime":1688068868723,"runnable":true,"livePorts":0,"pauseOnExit":false,"pauseEvent":{"type":"Event","kind":"Resume","timestamp":1688068868787,"isolateGroup":{"type":"@IsolateGroup","id":"isolateGroups/2187587266607717932","number":"2187587266607717932","name":"main.dart","isSystemIsolateGroup":false},"isolate":{"type":"@Isolate","id":"isolates/3315763852458143","number":"3315763852458143","name":"main","isSystemIsolate":false,"isolateGroupId":"isolateGroups/2187587266607717932"}},"breakpoints":[],"exceptionPauseMode":"None","rootLib":{"type":"@Library","id":"libraries/@342330433","fixedId":true,"name":"","uri":"package:flutter_view/main.dart"},"extensionRPCs":["ext.ui.window.impellerEnabled","ext.dart.io.getHttpEnableTimelineLogging","ext.dart.io.setHttpEnableTimelineLogging","ext.dart.io.httpEnableTimelineLogging","ext.dart.io.getSocketProfile","ext.dart.io.startSocketProfiling","ext.dart.io.pauseSocketProfiling","ext.dart.io.socketProfilingEnabled","ext.dart.io.clearSocketProfile","ext.dart.io.getVersion","ext.dart.io.getHttpProfile","ext.dart.io.getHttpProfileRequest","ext.dart.io.clearHttpProfile","ext.flutter.exit","ext.flutter.connectedVmServiceUri","ext.flutter.activeDevToolsServerAddress","ext.flutter.timeDilation","ext.flutter.debugDumpRenderTree","ext.flutter.debugDumpSemanticsTreeInTraversalOrder","ext.flutter.debugDumpSemanticsTreeInInverseHitTestOrder","ext.flutter.profileRenderObjectPaints","ext.flutter.profileRenderObjectLayouts","ext.flutter.debugDumpApp","ext.flutter.debugDumpFocusTree","ext.flutter.showPerformanceOverlay","ext.flutter.didSendFirstFrameEvent","ext.flutter.didSendFirstFrameRasterizedEvent","ext.flutter.fastReassemble","ext.flutter.profileWidgetBuilds","ext.flutter.profileUserWidgetBuilds"]}
[2023-06-29 13:01:20.037395] [STDOUT] stdout: [        ] Received VM events:
[2023-06-29 13:01:20.037463] [STDOUT] stdout: [        ] Flutter.ServiceExtensionStateChanged: [ExtensionData {extension: ext.flutter.activeDevToolsServerAddress, value: http://<IP_ADDRESS>:9104/}]
[2023-06-29 13:01:20.037493] [STDOUT] stdout:            Flutter.ServiceExtensionStateChanged: [ExtensionData {extension: ext.flutter.connectedVmServiceUri, value: http://<IP_ADDRESS>:51102/n18v17LclxU=/}]

@vashworth Do we have an ios bug tracking this issue already?

@vashworth Do we have an ios bug tracking this issue already?

I just made a issue to track this error: https://github.com/flutter/flutter/issues/129836

@vashworth Do we have an ios bug tracking this issue already?

I just made a issue to track this error: #129836

Thank you @vashworth !

It might be possible that some functionality was removed recently which caused this. This seems to not be happening very frequently, leaving as P2.

We haven't seen this issue in a long time. Closing as not replicable.