Data quality issue with CVE-2024-38356 & CVE-2024-38357

The CVE ID
Two CVEs originating from GHSAs are affected by the same underlying issue:

CVE-2024-38356 / GHSA-9hcv-j9pv-qmph
CVE-2024-38357 / GHSA-w9jx-4g6g-rp7x

Describe the data quality issue observed
Some vulnerable versions (including the most recent vulnerable versions of the library) are omitted from the affected versions list of the CVEs while being correct in the GHSAs.
Additional context
Both GHSAs cover vulnerabilities that were found across three supported versions of TinyMCE, with two open-source fixed versions available:
Affected versions: <5.11.0, >=6.0.0 <6.8.4, >=7.0.0 <7.2.0
Patched versions: 5.11.0, 6.8.4, 7.2.0
There are also additional downstream software packages mentioned in the GHSA where TinyMCE is directly embedded. The GHSA definitions on OSV.dev are fine, as they rely directly on GHSA-9hcv-j9pv-qmph.json & GHSA-w9jx-4g6g-rp7x.json.
The CVE affected versions on OSV.dev appear to use the Git commit URLs provided by GitHub on the GHSA. If this worked correctly, they would omit the fix in 5.11.0 from the definition (as 5.x is now patched only under a commercial license for long-term support customers) and additional downstream software packages, but would probably work for 6.x & 7.x.
Unfortunately, for some reason while two commit URLs are present in the GHSA data, only one commit URL appears in the references section on NVD:

https://nvd.nist.gov/vuln/detail/CVE-2024-38356
https://nvd.nist.gov/vuln/detail/CVE-2024-38357

In both cases the commit that appears is the one for 6.x, which results in 7.0.0 onwards not appearing as vulnerable versions even though they are explicitly mentioned in the CVE text and the associated GHSA.
Suggested changes to record
Ideally the CVE would mirror the version ranges specified in the GHSA, as the GHSA is the canonical source of the affected & fixed versions. The NVD record leaves little doubt this is the case by referring to "GitHub, Inc." as the source.

Ideally the CVE would mirror the version ranges specified in the GHSA, as the GHSA is the canonical source of the affected & fixed versions. The NVD record leaves little doubt this is the case by referring to "GitHub, Inc." as the source.

The reason why this isn't the case is because the CVE to OSV conversion focuses on deriving commit ranges, not versions, as the objective of converting these records is to facilitate vulnerability scanning by commit hash.
This FAQ entry discusses what OSV.dev with OSV records it imports:

Both version and commit enumeration populate the affected.versions[] field, which assists with precise version matching.

Any versions added to the affected[].versions array are Git tags that fall within the affected commit range and nothing is knowable about the versioning scheme employed.
The OSV records generated from the conversion do not contain any affected[].package information because:
a) there is no broadly applicable way to derive an ecosystem name for the subject of the CVEs converted
b) as there is no ecosystem name, the package name is nonsensical

The CVE affected versions on OSV.dev appear to use the Git commit URLs provided by GitHub on the GHSA.

Close. Any commits in the CVE's references are assumed to be a more accurate fix commit that what may be derivable from any versions supplied in the CVE record.
Additionally, because https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2024-38356 and https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2024-38357 are yet to be analysed by the NVD, they are lacking any machine-readable details about the affected versions. Even if that was available, the commit in the references would be preferred.

If this worked correctly, they would omit the fix in 5.11.0 from the definition (as 5.x is now patched only under a commercial license for long-term support customers) and additional downstream software packages, but would probably work for 6.x & 7.x.

I think this is something of an edge case of an edge case, resulting in the combination of necessary assumptions we make for the scale we're operating at. I'm not sure that there's anything actionable that can be done here that would work at scale, and I'm also struggling to think of any enhancements to the underlying CVE that would have resulted in a more accurate conversion outcome.
/cc @oliverchang for his thoughts...

@tjdett before I close this out as unactionable, I wanted to make sure we capture what use cases do and/or do not work with the data as it is today, as that may help review the situation from the perspective of the ultimate objective, which is vulnerability detection...
So, from the perspective of calling OSV.dev's API with either a package or a commit, can you give some examples of false negatives that occur with the data in its current form?
(It's best to work backwards from what is being scanned, rather than looking at specific records and trying to work forwards to what you have)


fix test cases

This PR includes:

all test cases under test/verify are fixed and passed, except check-image, because this file is large and seems image page are not ready yet.
added back rhel-86.json in test/files, and added one more step to copy this file in vm.install, this is to run test in rhel-8-6 os.

Thanks for the contribution! We are no longer testing/releasing into RHEL 8.6 so no need to readd the repo override. Also, I am confused why you have commented out so much of the test cases. I would prefer if you kept the failing cases failing instead of commenting them out to make them pass. Or, we can remove unnecessary tests. If a test is failing and it isn't important then I think we should remove it instead of commenting it out.

I keep rhel86 here because there is no rhel87 or rhel91 vm available when I run test. Do you know how to run test cases on rhel87 or rhel91?
⨯ https://cockpit-images.eu-central-1.linodeobjects.com/
⨯ https://cockpit-images.us-east-1.linodeobjects.com/
⨯ https://cockpit-11.apps.cnfdb2.e2e.bos.redhat.com/images/
image-download: image-download: couldn't find file anywhere: rhel-9-1-5064e97ed9d10dea1ad85e719ee8bfb85ebb9378360edcd94f60b1d0e1a4268c.qcow2

⨯ https://cockpit-images.eu-central-1.linodeobjects.com/
⨯ https://cockpit-images.us-east-1.linodeobjects.com/
⨯ https://cockpit-11.apps.cnfdb2.e2e.bos.redhat.com/images/
image-download: image-download: couldn't find file anywhere: rhel-8-7-6e65aa1a33596efbb6a5fadec040fac2a19fc8d29b8d47fcf71fc829ee43fb1d.qcow2

I commented out test cases/code blocks that are failed due to bugs, and I also put the bug link in the first line of commented codes, to indicate why they are commented out. Do you prefer to keep them there before the bugs get fixed?

For unnecessary tests, I didn't see any of them till now, will remove them if I found.

@martinpitt

image-download: image-download: couldn't find file anywhere: rhel-9-1-5064e97ed9d10dea1ad85e719ee8bfb85ebb9378360edcd94f60b1d0e1a4268c.qcow2

Any thoughts on how to fix this?

@yih-redhat , @jkozol : The cockpit-11 image store is on its way out, as these machines are unfortunately going away. I uploaded the current rhel-8-7 and rhel-9-1 images there for short-term experimentation.
In order to download Red Hat images in the long term, you need an S3 token. Please contact me on Red Hat IRC (pitti), Signal, or GPG email if you want to work on this regularly, then I can create a token for you.

@yih-redhat , @jkozol : The cockpit-11 image store is on its way out, as these machines are unfortunately going away. I uploaded the current rhel-8-7 and rhel-9-1 images there for short-term experimentation.
In order to download Red Hat images in the long term, you need an S3 token. Please contact me on Red Hat IRC (pitti), Signal, or GPG email if you want to work on this regularly, then I can create a token for you.

Thank you, I will send you an email about this.


Add basic auth capability for private Docker registries

Provide a detailed description of the proposed feature
Today, thanks to https://github.com/Homebrew/brew/pull/11766, only bearer tokens are supported to authenticate against a private Docker registry. Configuration example:
export HOMEBREW_ARTIFACT_DOMAIN=https://artifactory.your-company.com/artifactory/brew-remote
export HOMEBREW_DOCKER_REGISTRY_TOKEN=<artifactory-access-token>

I propose to implement the support of basic authentication. Configuration example of my proposal:
export HOMEBREW_ARTIFACT_DOMAIN=https://artifactory.your-company.com/artifactory/brew-remote
export HOMEBREW_DOCKER_REGISTRY_BASIC_USER=<username>
export HOMEBREW_DOCKER_REGISTRY_BASIC_PWD=<password>

... which would send an Authorization: Basic ...  header instead of a Bearer one:
https://github.com/Homebrew/brew/blob/371122da557d770aa4555f258a904aa008f9ce8b/Library/Homebrew/download_strategy.rb#L599
What is the motivation for the feature?
In our case we use Artifactory as a mirror. It supports both authentication methods:

access token (Bearer)
username / API key (Basic)

However, only super admins can generate access tokens that don't expire.
Regular users can only generate access tokens with a 3600s duration maximum.
On the other hand, regular users can generate a random API key that can be used as a password in a basic auth that doesn't expire.
Also, many other Docker registries only support basic auth.
How will the feature be relevant to at least 90% of Homebrew users?
The support of this feature will really help using homebrew in large companies where you are behind a proxy and you are forced to use a mirror like Artifactory with restricted permissions.
What alternatives to the feature have been considered?
Today we developed this wrapper around homebrew that automatically refreshes Artifactory's access token when it's expired.
The dev experience becomes quite decent, but it's still hacky.
Snippet to include in your ~/.zshrc or equivalent:
export HOMEBREW_ARTIFACT_DOMAIN=https://artifactory.your-company.com/artifactory/brew-remote

brewtoken() {
  if [[ -f ~/.brewtoken.sh ]]; then
    source ~/.brewtoken.sh
  fi

  if [[ ! -z "${BREWTOKEN_TS:-}" && "${1:-}" != "--force" ]]; then
    if [[ $(( $(date +%s) - $BREWTOKEN_TS )) -le 3300 ]]; then
      # Token not expired > we silently exit
      return
    fi
  fi

  >&2 echo "Requesting access token from Artifactory..."
  local rtRes=$(jfrog rt access-token-create)
  local rtAccessToken=$(echo "${rtRes}" | tr -d ' ' | grep -o '"access_token":"[^"]*"' | tr -d \" | sed 's/^access_token://') # we cannot use jq because we need homebrew to install it :-)
  if [[ -z "${rtAccessToken}" ]]; then
    >&2 echo "Failed to fetch access token:"
    >&2 echo "${rtRes}"
    return 1
  fi

  echo "export BREWTOKEN_TS=$(date +%s)" > ~/.brewtoken.sh
  echo "export HOMEBREW_DOCKER_REGISTRY_TOKEN=\"${rtAccessToken}\"" >> ~/.brewtoken.sh
  source ~/.brewtoken.sh
}

brew() {
  if [[ "${1:-}" =~ 'install|upgrade|reinstall|fetch|formulae|search|update' ]]; then
    brewtoken
  fi
  /usr/local/bin/brew $@
}

... and you first have to install the JFrog CLI and execute jfrog config add to set your credentials up
(if you don't want to, you can rewrite the script above with simple curl calls)

Will review a pull request for this.

Unfortunately I don't have any Ruby skills, so if someone's willing to open a PR, go on :)
Otherwise, I can give it a try in my free time later

Unfortunately I don't have any Ruby skills, so if someone's willing to open a PR, go on :) Otherwise, I can give it a try in my free time later

I would like to work on this issue


Update v1.6.0 broke Laravel's builds

What is this feature about (expected vs actual behaviour)?
Seems like release v1.6.0 broke Laravel's build and subsequently all PR's that are sent in since are failing as well: https://github.com/laravel/framework/runs/4920401730?check_suite_focus=true#step:8:138
It seems most likely that this commit is the culprit: https://github.com/voku/portable-ascii/commit/acaf86813ba78f0787bcf6294fc1b55c044ae7a5
Changing the behaviour here seems like a breaking change to me and should probably be done in a major release?
How can I reproduce it?
voku\helper\ASCII::to_ascii('пиздюк.txt');

Expected output: pizdyuk.txt (pre 1.6.0)
Given output: pizdiuk.txt (as of 1.6.0)
Does it take minutes, hours or days to fix?
Think this is an easy fix by just rolling back the commit and tagging a new patch release.

I am currently in the hospital (my little girl had her heart operation last week, everything went well 🙏) and I will take a look this evening.
"It's more a fix for the current behaviour for Russia ascii mappings, but I can re-tag with a major version anyway. Should I remove the old tag? 🤔" - https://twitter.com/suckup_de/status/1485641365233049605?t=ncRafqzpgbWbFXIiL2HFUw

Ow man, don't worry about it at all. We already fixed it on our end by limiting the constraint to <1.6.0. This is in no way urgent 👍
Hope your girl recovers soon! ❤

@driesvints I reverted the "ASCII"-mappings and tagged 1.6.1 and 2.0.0 https://github.com/voku/portable-ascii/compare/1.6.1...2.0.0

@voku thanks a lot! 👏


AOT Compatibility

I'd like to suggest some improvements to make wasmtime-dotnet more AOT friendly. I'm thinking along the lines of enabling static linking against wasmtime, which could really improve scenarios where the C# assembly is AOT compiled.. More details about .NET's native AOT and how it interops with native code can be found here: Native AOT with .NET.
Why This Matters:
AOT offers significant performance improvements and having wasmtime-dotnet support this more seamlessly would be a huge win. It could enhance the library's usability in high-performance scenarios.
Suggested Changes:

Enable static linking in wasmtime-dotnet.
Ensure compatibility with .NET AOT binaries.

I'm happy to implement the necessary changes myself if it is something that will be accepted.

Hi @andrewmd5, thanks very much for the informational request!
I definitely think this would be something we would accept, but I'm not entirely sure what the work entails. Is it mostly just a different way of building the project?


Lana 3

Timed the block out with player movement 10. Added a canvas to the block out so the player can move.

did it manually


[Android] Memory leak when a process is run in a isolate

I tried to create my own isolate and keep it alive to execute some heavy work.
But I got memory issue. App memory continue increasing when using app.
Example code

code sample

import 'dart:async';
import 'dart:isolate';

import 'package:flutter/foundation.dart';

class _CrossIsolatesMessage<Q, R> {
  final SendPort sender;
  final SendPort? errorPort;
  ComputeCallback<Q, R>? handler;
  Q? message;

  _CrossIsolatesMessage({
    required this.sender,
    this.errorPort,
    this.handler,
    this.message
  });
}

class FSIsolate {
  SendPort? _isolateSendPort;
  // Isolate _newIsolate;
  bool _isInitialized = false;
  final ReceivePort resultPort = ReceivePort();
  Isolate? _isolate;
  FSIsolate();

  void kill() {
    resultPort.close();
    _isolate?.kill();
  }

  Future<void> initialize() async {
    if (_isInitialized) {
      return Future<void>.value(null);
    } else {
      _isolate = await Isolate.spawn<_IsolateConfiguration>(
        _spawn,
        _IsolateConfiguration(
          resultPort.sendPort,
        ),
        errorsAreFatal: false,
      );
      _isolateSendPort = await resultPort.first;
      _isInitialized = true;
    }
  }

  Future<R> parseData<Q, R>(ComputeCallback<Q, R> handler, Q data) {
    var completer = Completer<R>();
    ReceivePort port = ReceivePort();
    ReceivePort errorPort = ReceivePort();
    try {
      _isolateSendPort!.send(_CrossIsolatesMessage<Q, R>(
          message: data,
          errorPort: errorPort.sendPort,
          handler: handler,
          sender: port.sendPort));
      port.listen((message) {
        completer.complete(message);
      });
      errorPort.listen((error) {
        completer.completeError(error);
      });
    } catch (e) {
      rethrow;
    }
    return completer.future;
  }

}

Future<void> _spawn<Q, R>(_IsolateConfiguration configuration) async {
  var ourReceivePort = ReceivePort();

  configuration.resultPort.send(ourReceivePort.sendPort);
  ourReceivePort.listen((message) async {
    var data = message as _CrossIsolatesMessage;
    SendPort replyToPort = data.sender;
    try {
      var result = await data.handler!(data.message);
      replyToPort.send(result);
    } catch (e) {
      data.errorPort!.send(e);
    }
  });
}

@immutable
class _IsolateConfiguration {
  const _IsolateConfiguration(
      this.resultPort,
      );
  final SendPort resultPort;
}
typedef ComputeCallback<Q, R> = FutureOr<R>? Function(Q message);

final _fsisolate = FSIsolate();

  void _incrementCounter() {
    setState(() {
      for (var i = 0; i < 1000; i++) {
        _fsisolate.initialize().then((value) => _fsisolate.parseData(_parseAndDecode, '{"glossary": {    "title": "example glossary",		"GlossDiv": {        "title": "S",			"GlossList": {            "GlossEntry": {                "ID": "SGML",					"SortAs": "SGML",					"GlossTerm": "Standard Generalized Markup Language",					"Acronym": "SGML",					"Abbrev": "ISO 8879:1986",					"GlossDef": {                    "para": "A meta-markup language, used to create markup languages such as DocBook.",						"GlossSeeAlso": ["GML", "XML"]                },					"GlossSee": "markup"            }        }    }}}'));
      }
      // This call to setState tells the Flutter framework that something has
      // changed in this State, which causes it to rerun the build method below
      // so that the display can reflect the updated values. If we changed
      // _counter without calling setState(), then the build method would not be
      // called again, and so nothing would appear to happen.
      _counter++;
    });
  }

I also attach demo project for this issue.
Flutter doctor:
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 2.10.2, on macOS 11.6.2 20G314 darwin-x64, locale
    en-VN)
[!] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
    ✗ cmdline-tools component is missing
      Run `path/to/sdkmanager --install "cmdline-tools;latest"`
      See https://developer.android.com/studio/command-line for more details.
    ✗ Android license status unknown.
      Run `flutter doctor --android-licenses` to accept the SDK licenses.
      See https://flutter.dev/docs/get-started/install/macos#android-setup for
      more details.
[✓] Xcode - develop for iOS and macOS (Xcode 13.0)
[✓] Chrome - develop for the web
[✓] Android Studio (version 4.2)
[✓] VS Code (version 1.63.2)
[✓] Connected device (3 available)
    ! Error: Unable
to prepare Duong's iphone for development.  (code -1)
[✓] HTTP Host Availability

! Doctor found issues in 1 category.

isolate_leak_2.zip (Updated)

Hi @ngminhduong,
I ran your sample project on Android and pressed the increment button once and the app is crashing. Could you please what target platform are you compiling for and how you are checking the app memory usage?

logs
Launching lib/main.dart on Redmi K20 Pro in debug mode...
Warning: Mapping new ns http://schemas.android.com/repository/android/common/02 to old ns http://schemas.android.com/repository/android/common/01
Warning: Mapping new ns http://schemas.android.com/repository/android/generic/02 to old ns http://schemas.android.com/repository/android/generic/01
Warning: Mapping new ns http://schemas.android.com/sdk/android/repo/addon2/02 to old ns http://schemas.android.com/sdk/android/repo/addon2/01
Warning: Mapping new ns http://schemas.android.com/sdk/android/repo/repository2/02 to old ns http://schemas.android.com/sdk/android/repo/repository2/01
Warning: Mapping new ns http://schemas.android.com/sdk/android/repo/sys-img2/02 to old ns http://schemas.android.com/sdk/android/repo/sys-img2/01
Warning: Mapping new ns http://schemas.android.com/repository/android/common/02 to old ns http://schemas.android.com/repository/android/common/01
Warning: Mapping new ns http://schemas.android.com/repository/android/generic/02 to old ns http://schemas.android.com/repository/android/generic/01
Warning: Mapping new ns http://schemas.android.com/sdk/android/repo/addon2/02 to old ns http://schemas.android.com/sdk/android/repo/addon2/01
Warning: Mapping new ns http://schemas.android.com/sdk/android/repo/repository2/02 to old ns http://schemas.android.com/sdk/android/repo/repository2/01
Warning: Mapping new ns http://schemas.android.com/sdk/android/repo/sys-img2/02 to old ns http://schemas.android.com/sdk/android/repo/sys-img2/01
✓  Built build/app/outputs/flutter-apk/app-debug.apk.

Exited (sigterm)

flutter doctor -v
[✓] Flutter (Channel stable, 2.10.2, on macOS 12.1 21C52 darwin-arm, locale en-GB)
    • Flutter version 2.10.2 at /Users/mahesh/Documents/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 097d3313d8 (2 days ago), 2022-02-18 19:33:08 -0600
    • Engine revision a83ed0e5e3
    • Dart version 2.16.1
    • DevTools version 2.9.2

[✓] Android toolchain - develop for Android devices (Android SDK version 31.0.0)
    • Android SDK at /Users/mahesh/Library/Android/sdk
    • Platform android-31, build-tools 31.0.0
    • ANDROID_HOME = /Users/mahesh/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.8+10-b944.6916264)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 13.2.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • CocoaPods version 1.10.2

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 4.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.8+10-b944.6916264)

[✓] IntelliJ IDEA Community Edition (version 2021.2.1)
    • IntelliJ at /Applications/IntelliJ IDEA CE.app
    • Flutter plugin version 61.2.4
    • Dart plugin version 212.5080.8

[✓] VS Code (version 1.63.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.34.0

[✓] Connected device (3 available)
    • Redmi K20 Pro (mobile) • <IP_ADDRESS>:5555 • android-arm64  • Android 11 (API 30)
    • macOS (desktop)        • macos            • darwin-arm64   • macOS 12.1 21C52 darwin-arm
    • Chrome (web)           • chrome           • web-javascript • Google Chrome 98.0.4758.102

[✓] HTTP Host Availability
    • All required HTTP hosts are available

• No issues found!

Hi @maheshmnj
I update demo project. Can you help check again.
I check memory using Flutter dev tool
https://drive.google.com/file/d/1WeY50MWZ9bwDVMdjHUDHLjtqoZqQFe4f/view?usp=sharing
I also include a video how i perform a test.

Hi @ngminhduong, Thanks for filing the issue. I can reproduce the same behavior there is a increase in memory usage when the process is run in a different isolate and the memory doesn't seem to release.
Below is the memory usage before/after the process was run in isolate.

Before
After

flutter doctor -v
[✓] Flutter (Channel stable, 2.10.2, on macOS 12.1 21C52 darwin-arm, locale en-GB)
    • Flutter version 2.10.2 at /Users/mahesh/Documents/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 097d3313d8 (2 days ago), 2022-02-18 19:33:08 -0600
    • Engine revision a83ed0e5e3
    • Dart version 2.16.1
    • DevTools version 2.9.2

[✓] Android toolchain - develop for Android devices (Android SDK version 31.0.0)
    • Android SDK at /Users/mahesh/Library/Android/sdk
    • Platform android-31, build-tools 31.0.0
    • ANDROID_HOME = /Users/mahesh/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.8+10-b944.6916264)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 13.2.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • CocoaPods version 1.10.2

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 4.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.8+10-b944.6916264)

[✓] IntelliJ IDEA Community Edition (version 2021.2.1)
    • IntelliJ at /Applications/IntelliJ IDEA CE.app
    • Flutter plugin version 61.2.4
    • Dart plugin version 212.5080.8

[✓] VS Code (version 1.63.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.34.0

[✓] Connected device (3 available)
    • Redmi K20 Pro (mobile) • <IP_ADDRESS>:5555 • android-arm64  • Android 11 (API 30)
    • macOS (desktop)        • macos            • darwin-arm64   • macOS 12.1 21C52 darwin-arm
    • Chrome (web)           • chrome           • web-javascript • Google Chrome 98.0.4758.102

[✓] HTTP Host Availability
    • All required HTTP hosts are available

• No issues found!

[✓] Flutter (Channel master, 2.11.0-0.0.pre.582, on macOS 12.1 21C52 darwin-arm, locale en-GB)
    • Flutter version 2.11.0-0.0.pre.582 at /Users/mahesh/Documents/flutter_master
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision<PHONE_NUMBER> (5 days ago), 2022-02-16 00:00:11 -0500
    • Engine revision b09cf53269
    • Dart version 2.17.0 (build 2.17.0-105.0.dev)
    • DevTools version 2.10.0

[✓] Android toolchain - develop for Android devices (Android SDK version 31.0.0)
    • Android SDK at /Users/mahesh/Library/Android/sdk
    • Platform android-31, build-tools 31.0.0
    • ANDROID_HOME = /Users/mahesh/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.8+10-b944.6916264)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 13.2.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • CocoaPods version 1.10.2

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 4.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.8+10-b944.6916264)

[✓] IntelliJ IDEA Community Edition (version 2021.2.1)
    • IntelliJ at /Applications/IntelliJ IDEA CE.app
    • Flutter plugin version 61.2.4
    • Dart plugin version 212.5080.8

[✓] VS Code (version 1.63.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.34.0

[✓] Connected device (3 available)
    • Redmi K20 Pro (mobile) • <IP_ADDRESS>:5555 • android-arm64  • Android 11 (API 30)
    • macOS (desktop)        • macos            • darwin-arm64   • macOS 12.1 21C52 darwin-arm
    • Chrome (web)           • chrome           • web-javascript • Google Chrome 98.0.4758.102

[✓] HTTP Host Availability
    • All required HTTP hosts are available

• No issues found!

Try watching your app's resource usage in the Dart Observatory.  The flutter run tool will print a URL that can be used to access the Observatory.
The Observatory home page will show all the active isolates in the app process.  Make sure that all isolates spawned by the app are exiting as expected and that no isolates are being leaked.

Hi @jason-simmons
Why app memory always increase when I execute job from isolate.
What's wrong with my example.

I also found that when I kill the current isolate, memory was decreased but it was not released all memory.

cc @dnfield

Per my reading of this thread, no actionable leak in the engine has been identified. Per @jason-simmons comment, checking the profile in observatory would be the way to go.

The reproduction that would be interesting to look at here is a case where an isolate is spawned and cannot be killed/exited, or that even after it exits it retains significant memory. Please feel free to add a case that demonstrates that if you find one, or file a new bug - but please make sure it contains a reproducible example. Your example here doesn't actually attempt to kill the isolate, and it's also not clear to me that you're properly closing all your ports (which can keep an isolate alive).


Content of All Genres and Ui

Description
I would like to contribute in writing proper and eye catching content for all the genres, as a content writer I would write the content with better Ui like highlight content and hovering effects.
Screenshots

Checklist

[X] I have checked the existing issues
[X] I have read the [Contributing Guidelines]
[X] I have read the discussion tab thoroughly and got the project idea
[X] I am willing to work on this issue (optional)
[X] I am a GSSoC'23 contributor

@SiddhantPawar03 I have created a pull request regarding this task, please verify and merge it.


Verify that "Mihoești" is listed in the dropdown menu when searched

Description:
"Mihoești" should appear in the dropdown menu with correct spelling and locality information.
Preconditions:
The website is up and running.

Step 2
Observe the dropdown menu.
Expected results:
"Mihoești" is displayed and selectable from the dropdown menu.


tabby: 0.10.0 -> 0.11.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for tabby is: Self-hosted AI coding assistant
meta.homepage for tabby is: https://github.com/TabbyML/tabby
meta.changelog for tabby is: https://github.com/TabbyML/tabby/releases/tag/v0.11.0
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 0.11.0 with grep in /nix/store/zclyl8fy202mp981kvzqzd9cvxhm0dfg-tabby-0.11.0
found 0.11.0 in filename of file in /nix/store/zclyl8fy202mp981kvzqzd9cvxhm0dfg-tabby-0.11.0

Rebuild report (if merged into master) (click to expand)

2 total rebuild path(s)

2 package rebuild(s)

First fifty rebuilds by attrpath

tabby

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/zclyl8fy202mp981kvzqzd9cvxhm0dfg-tabby-0.11.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A tabby https://github.com/r-ryantm/nixpkgs/archive/3e1464aff56e5c26996e974a0a5702357a01a127.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/3e1464aff56e5c26996e974a0a5702357a01a127#tabby

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/zclyl8fy202mp981kvzqzd9cvxhm0dfg-tabby-0.11.0
ls -la /nix/store/zclyl8fy202mp981kvzqzd9cvxhm0dfg-tabby-0.11.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review --extra-nixpkgs-config '{ allowInsecurePredicate = x: true; }' run on x86_64-linux 1

1 package built:

tabby

Maintainer pings
cc @ghthor for testing.

This segfaults by itself because of tabby's strict version requirements of llama-cpp, but seems to work absolutely perfectly in combination with https://github.com/NixOS/nixpkgs/pull/310743.


DFS Travesal in Binary Tree

Issue #248 (dfs traversal) is solved in this PR

@rudrakshi99 @raksha009 finish this soon

@rudrakshi99 ,I had resolve the merge conflicts.


DEdit for Jupiter - crashes when easily overloaded.

I'd like to preface this submission by saying this is definitely an outlandish scenario, and may not actually indicate a problem with DDrawCompat itself. I'm submitting this because I want to ensure all of my bases are covered.
The program I'm using is not a game in and of itself; rather, it is a world editor program for the LithTech Jupiter engine named "DEdit". I have checked the list of unsupported cases and confirmed that this program does not fall under any of those categories. (The game for which the program was made does not suffer from these problems.) For reasons I've not yet been able to figure out, this program appears to have become wildly unstable running on any modern Intel hardware (post 2020), constantly crashing with unspecified access violation errors. DDrawCompat was the only thing I've been able to use thus far to stabilize the program (for the most part) with a significant performance reduction; however, attempting to close larger world files will still cause crashes for the same access violations.
Limited testing was done on two other sets of hardware. Neither had problems remotely like this.
The log file for this run is here. Although I did use a lot of process-specific configuration settings, because the crash in question is easily replicatable, I can say with relative confidence that very few, if any, of these settings have had any performance impact or have been able to make the crash avoidable following testing.
The mini-crash dump is here. A full one can be provided if needed.
I am already (painfully) aware that the 0xc0000005 error could apply to literally anything under the sun, but if nothing else, it would at least give me peace of mind if I knew the culprit was elsewhere beyond this wrapper. Any potential insight would be appreciated.

The crash you posted should be fixed by v0.5.2, though I'm not sure because I wasn't able to reproduce it with earlier versions either.
I don't know about the performance issues, I'd need more info on how to reproduce them exactly. Since your GPU was using D3D9On12 by default, v0.5.2 should now automatically switch it to igd9trinity32.dll instead, which hopefully will have better performance as well.

 The crashes have indeed mostly dissipated, as have the performance issues. I'm not sure if you were able to actually go out of your way to install this software yourself, but if that's what you did, I can't thank you enough. This was the only major barrier to making the software work with my laptop, and although it's still not perfect (viewports don't tend to show anything until I've switched to software rendering and then back to hardware manually), it's far better than it was before.
If there's anything else posing an issue I'll let you know. Thanks again!

viewports don't tend to show anything until I've switched to software rendering and then back to hardware manually

This may have been caused by the windowed mode presentation issue in v0.5.2. Is it by chance fixed in v0.5.3? If not, I'll need some more help on how to reproduce the problem, as I haven't even figured out how to load anything in the editor yet.

No, unfortunately the issue is not fixed in v0.5.3. Reproducing the problem should be as simple as attempting to create a new World file, but I've not seen DEdit without preset functionality before, so we may just not be using the same program, so to speak.
The version of DEdit I'm using is bundled alongside the editing tools for the game TRON 2.0. For me, the build number is 109.21. I'll assemble a video showing current vs. intended behavior when I get the chance - however unfortunately I cannot send you a working world file to open for yourself, as this version is (to my knowledge) quite closely tied to the unique version of LithTech that runs the game (LithTech Triton). Getting that specific version may be the only way we can actually compare results.
 (The installer bundle also comes with its own uninstaller that can clean up all of the other software once you're done.)

I didn't have much time to put this video together, so I apologize for the imperfections. A caption denotes what segment of the video is currently playing.
https://youtu.be/JT_uyzKaoWI


Incorrect Java signature for value classes

Minimized code
final class Foo(val value: String) extends AnyVal

final class Bar {
  def boxedStringy = Option(new Foo("hi"))
}

Output
$ javap Bar
Compiled from "A.scala"
public final class Bar {
  public Bar();
  public scala.Option<java.lang.String> boxedStringy();
}

Expectation
Scala 2 (2.13.4) is right:
$ javap Bar
Compiled from "A.scala"
public final class Bar {
  public scala.Option<Foo> boxedStringy();
  public Bar();
}

AKA forward-port https://github.com/scala/scala/pull/8127 for Scala 3.

My team has just hit this. Kindly requesting some love for this ticket 😅
Our understanding is that any signatures that mention value classes should currently be avoided in calls from Java.

I think I was also hit by this today in https://github.com/FasterXML/jackson-module-scala/pull/675

On that jackson-module-scala ticket, @jtjeferreira suggests that the bug was fixed in Scala 2 in https://github.com/scala/scala/pull/8127. Maybe it can be forward ported, I don't know if java signature building is similar in Scala 3.

@lrytz there's a WIP forward-port at https://github.com/scala/scala3/pull/20463 (where the contributor is stuck and could use help)

@lrytz there's a WIP forward-port at #20463 (where the contributor is stuck and could use help)

Yes, I tried to fix it but this was my first time looking into the compiler code so I got stuck. Would be helpful if someone pointed me in the right direction and point in which phase (or phases) I would have to do the fix.
OTOH I was also wondering if the same fix is applied like in scala2, this will be a binary incompatible change. So it can only be done in scala 3.6?

OTOH I was also wondering if the same fix is applied like in scala2, this will be a binary incompatible change. So it can only be done in scala 3.6?

fwiw, the Scala 2 fix was made in 2.12.9, it wasn't held for 2.13.0
as @raboof wrote at https://github.com/lightbend-labs/mima/pull/299#issuecomment-494744322,

when a method's erased signature has remained the same, but its full signature has changed [...] Such a change is not necessarily binary incompatible, but typically worthwhile to manually investigate and explicitly confirm

offhand, it seems to me like it would be fair game for an LTS backport as long as it was carefully release-noted, but I'm not fully confident in my own opinion here


Opentelemetry option added for tracing

Supports noop, stdouttracer and otlphttp tracer
noop is the default, so no tracing will take place by default

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Documentation change: Explaining the behavior of focus stylebox overlapping the pressed stylebox

Fixes #42322
Issue: No explanation as to why a focus stylebox is displayed over pressed stylebox, when a node is pressed.
After discussion, a doc addition is made in Stylebox class reference to explain why focus stylebox is displayed over pressed stylebox, even when a node is pressed.

@Calinou Would it be possible to add hacktoberfest-accepted label to this PR, as I'm participating in it?

@Calinou Could you review this PR as I'm participating in Hacktoberfest, so it can be counted?

Thanks!

Cherry-picked for 3.2.4.


Binding value to a YYYY-MM-DD value fails

Setting a YYYY-MM-DD value manually works, but if using Polymer data binding or programmatically setting the same string value, it does not get converted to a Date which causes and exception in date comparison (_dateEquals in vaadin-month-calendar.html)

The current value is of type Date. You can bind a string value after this is implemented: https://github.com/vaadin/vaadin-date-picker/issues/51#issuecomment-187733692

Hi,
Before all thanks for this great date-picker.
Then I'm still gettin trouble with this. In my UT I try to set the value like this :
myDatePicker.value = '2015-01-01';
but I get the following error :
Error: date1.getFullYear is not a function
  vaadin-month-calendar.Polymer._dateEquals at /vaadin-date-picker/vaadin-month-calendar.html:133
  vaadin-month-calendar.Polymer._isSelected at /vaadin-date-picker/vaadin-month-calendar.html:154
  TemplateInstance.Polymer.Base.extend._annotatedComputationEffect at /polymer/polymer.html:1591
  TemplateInstance.Polymer.Bind._modelApi._effectEffects at /polymer/polymer.html:1394
  TemplateInstance.Polymer.Bind._modelApi._propertySetter at /polymer/polymer.html:1378
  TemplateInstance.Polymer.Bind._modelApi.__setProperty [as __setPropertyOrig] at /polymer/polymer.html:1387
  TemplateInstance.Polymer.Templatizer.__setPropertyImpl [as __setProperty] at /polymer/polymer.html:3696
     dom-repeat.Polymer._forwardParentProp at /polymer/polymer.html:4470
                    dom-repeat.<anonymous> at /polymer/polymer.html:3776
  dom-repeat.Polymer.Base.extend._functionEffect at /polymer/polymer.html:1550

Hi
Are you sure you’re using alpha4? If you set the value like you describe above, you’d get that exact error message with alpha2 (wrong value type), but with alpha4 this should work just fine.
In fact, in alpha4 line 133 in vaadin-month-calendar is empty.

Yep you're right I was still in alpha2. My mistake, I was pretty sure to bump to alpha 4 😓
Thank you.


Interoperability with dpctx

Fix of the Data parallel control module integration (formerly known as DPPY, DPPL). All the code that depends on dpctx is moved to a separate module to avoid unwanted dependency in the big daal4py.

Overall looks ok - and we might merge in this configuration.  The only question that is not clear - how we will work in case of dpctl use from conda


Problem installing Telepathy

Hello
I am using KALI linux virtual machine
I installed Telepathy according to the tutorial
I moved to the Telepathy folder
And when I try to run the command
telepathy -t durov
I am encountering an error
telepathy: command not found
I'd love to know why he doesn't recognize Telepathy
And how to solve the problem
And another question I wanted to ask where do I enter the Telegram API details in Telepathy
Thank you

Hi there!
From the directory where Telepathy is installed, instead try:
$ python3 telepathy.py -t durov
When it runs the first time you’ll be able to enter API details.
Let me know how you get on
Jordan Wildon he/him
Prose Intelligence, Ltd. | Founder & CEO
Email: @.***
CONFIDENTIALITY NOTICE: This email and any attachments may contain confidential information intended only for the use of the individual or entity named above. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or action taken in reliance on the contents of this email is strictly prohibited. If you have received this email in error, please notify the sender immediately and delete the email and any attachments from your system. Please be advised that email communications may not be secure and could be subject to interception, data corruption, or unauthorized access. Therefore, we cannot guarantee the security or confidentiality of information transmitted by email. By opening this email, you agree to assume all risks associated with the use of email for communication purposes.

From: phpll-cyber @.>
Sent: Wednesday, May 31, 2023 12:26:00 AM
To: proseltd/Telepathy @.>
Cc: Subscribed @.***>
Subject: [proseltd/Telepathy] Problem installing Telepathy (Issue #53)
Hello
I am using KALI linux virtual machine
I installed Telepathy according to the tutorial
I moved to the Telepathy folder
And when I try to run the command
telepathy -t durov
I am encountering an error
telepathy: command not found
I'd love to know why he doesn't recognize Telepathy
And how to solve the problem
And another question I wanted to ask where do I enter the Telegram API details in Telepathy
Thank you
—
Reply to this email directly, view it on GitHubhttps://github.com/proseltd/Telepathy/issues/53, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AVGBB5YT5TVWFOOL72PXQ4LXIZ6YRANCNFSM6AAAAAAYUVQT3A.
You are receiving this because you are subscribed to this thread.Message ID: @.***>

Hello!
Im not him but i have the same issue. I tried the way you described and get this:
python3: can't open file '/home/amr/telepathy.py': [Errno 2] No such file or directory
Should i also install python3?


What is the foreground mode?

  The atom editor has a foreground option -f.  What does it mean to run a browser process in the foreground?  I don't see how a browser could run in a background process.

The atom command will exit immediately after opening the file, and atom -f will wait until the editor quits and print Chromium debug information in the meanwhile. There is no concept of foreground mode in atom-shell.


Add definitions for the type AR_TYPE_SETTINGS

Intro
This PR adds

The reversed packed struct type ar_settings_t for  AR_TYPE_SETTINGS.
Adds a simple CI workflow which runs make on pushes.
Updates README.md to show a CI badge.

Details

All reverse engineered fields are tested multiple times.
Maybe quickly going through the preferred formater might be necessary. (If you have any rules for your formater or you use a specific IDE I'd love to know so I don't cause extra labour like this 🙏 )
As usual please feel free to change the proposed code as you wish. ✨

Thank you for the review 🙏

Hi Ali, thanks for your ongoing contributions. I am using emacs, which, at least of the box, does not come with a (customizable, rule based) code formatter (it just does indenting). There is a clang-format extension package, which I've just installed. Will take a look at that. Does your editor support this format ?


[Text Translation] Change scopes to audiences

Description
Change scopes to audiences to align with sdk naming convention.

API change check
APIView has identified API level changes in this PR and created following API reviews.
azure-ai-translation-text


[Feature] Disable Sign Up for self hosted instances

We have deployed an Appsmith docker container in a public cloud and which is accessible publicly. It would be nice to have an option to disable sign up and add users from the admin panel. Is this option available?
Otherwise, we have to deploy it in a VPC and restrict access through a VPN. So our internal users only can access that URL.  Is this the option you suggest?
Thanks in advance.

@sijomc thank you for reaching out. Right now this is not possible but we'll consider taking this up as a feature request. We do suggest deploying appsmith behind a VPN today.
Also just to be clear, any users signing up on your instance will not have access to your databases and apps unless they are explicitly invited to the organization in which you build them

This has been asked for before as #1342 and #1362.

After the next release, please set APPSMITH_SIGNUP_DISABLED environment variable to true to disable public signups on your Appsmith instance. Note that invited users will still be able to sign up as before.

After the next release, please set APPSMITH_SIGNUP_DISABLED environment variable to true to disable public signups on your Appsmith instance. Note that invited users will still be able to sign up as before.


[SPARK-5342][YARN] Allow long running Spark apps to run on secure YARN/HDFS

Current Spark apps running on Secure YARN/HDFS would not be able to write data
to HDFS after 7 days, since delegation tokens cannot be renewed beyond that. This
means Spark Streaming apps will not be able to run on Secure YARN.
This commit adds basic functionality to fix this issue. In this patch:

new parameters are added - principal and keytab, which can be used to login to a KDC
the client logs in, and then get tokens to start the AM
the keytab is copied to the staging directory
the AM waits for 60% of the time till expiry of the tokens and then logs in using the keytab
each time after 60% of the time, new tokens are created and sent to the executors

Currently, to avoid complicating the architecture, we set the keytab and principal in the
SparkHadoopUtil singleton, and schedule a login. Once the login is completed, a callback is scheduled.
This is being posted for feedback, so I can gather feedback on the general implementation.
There are currently a bunch of things to do:

[x] logging
[x] testing - I plan to manually test this soon. If you have ideas of how to add unit tests, comment.
[x] add code to ensure that if these params are set in non-YARN cluster mode, we complain
[x] documentation
[x] Have the executors request for credentials from the AM, so that retries are possible.

Test build #27709 has started for   PR 4688 at commit 2b0d745.

This patch merges cleanly.

Test build #27709 timed out     for PR 4688 at commit 2b0d745     after a configured wait of 120m.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27709/
Test FAILed.

Test build #27730 has started for   PR 4688 at commit f8fe694.

This patch merges cleanly.

Test build #27730 has finished for   PR 4688 at commit f8fe694.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  case class UpdateCredentials(newCredentials: SerializableBuffer)

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27730/
Test FAILed.

Jenkins, retest this

Jenkins, test this please

Test build #27739 has started for   PR 4688 at commit f8fe694.

This patch merges cleanly.

Test build #27739 has finished for   PR 4688 at commit f8fe694.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  case class UpdateCredentials(newCredentials: SerializableBuffer)

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27739/
Test FAILed.

We can probably move it out to a I told class in the YARN module, and we should be ok, since this code is used only by the AM.

Test build #27769 has started for   PR 4688 at commit bcfc374.

This patch merges cleanly.

Looks like ClientSuite.scala has a bug -- it does not set YARN mode to true, so we end up getting an instance of SparkHadoopUtil, instead of YarnSparkHadoopUtil

Test build #27770 has started for   PR 4688 at commit d282d7a.

This patch merges cleanly.

Test build #27769 has finished for   PR 4688 at commit bcfc374.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  case class UpdateCredentials(newCredentials: SerializableBuffer)
public class JavaPowerIterationClusteringExample

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27769/
Test FAILed.

Test build #27770 has finished for   PR 4688 at commit d282d7a.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  case class UpdateCredentials(newCredentials: SerializableBuffer)

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27770/
Test PASSed.

Looks like the build issue was fixed by the last commit.

Test build #27828 has started for   PR 4688 at commit 41efde0.

This patch merges cleanly.

Test build #27828 has finished for   PR 4688 at commit 41efde0.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  case class UpdateCredentials(newCredentials: SerializableBuffer)

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27828/
Test PASSed.

Test build #27872 has started for   PR 4688 at commit fb27f46.

This patch merges cleanly.

Test build #27872 has finished for   PR 4688 at commit fb27f46.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  case class UpdateCredentials(newCredentials: SerializableBuffer)

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27872/
Test FAILed.

Test build #27879 has started for   PR 4688 at commit 8c6928a.

This patch merges cleanly.

Test build #27879 has finished for   PR 4688 at commit 8c6928a.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  case class UpdateCredentials(newCredentials: SerializableBuffer)

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27879/
Test PASSed.

Test build #27901 has started for   PR 4688 at commit d79b2b9.

This patch merges cleanly.

Test build #27901 has finished for   PR 4688 at commit d79b2b9.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  case class UpdateCredentials(newCredentials: SerializableBuffer)

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27901/
Test PASSed.

Generally speaking its a not a good idea to ship keytabs around.  If one is compromised then the user who gets it can do anything as that user forever.  YARN/HDFS generally uses tokens because they have a limited lifetime so if one was compromised it would be for a limited amount of time.  I know our internal paranoids would not allow this change to be rolled out internally.
This really needs to be fixed on the YARN side but unfortunately not much work there: https://issues.apache.org/jira/browse/YARN-896
That said I realize there aren't many options right now so as long as we make sure this doesn't happen automatically I don't have any better ideas (other then work on the YARN jira).

So actually one other option would be to build our own mechanism for pushing new tgt's and updating the UGI.  Storm actually has this functionality, you could look there as reference.

Thanks for taking a look, Tom!
To get new tgts, we'd still need the keytab, right? I am wondering how to get around having the keytab being shipped to the AM, since the client can go away in cluster mode and the app can keep running.
I will take a look at storm later today.

The user has to have a cron job (or similar) that does a kinit and then would push the credentials out.  This would be harder on spark as it would have to do it for each application and then you would have to create that communication protocol to send it.
I did also just notice that apache slider is actually using the keytab approach similar to what you are proposing.
Some other cons though to using the keytab approach:

some kdc's might consider it a replay attack if on lots of hosts are doing kinit all at once (ie you are using headless user type keytab)
if not using headless user keytab then you need to generate per host keytab and figure out how to distribute properly.  This is a headache for operations team to generate those and manage them.

Not sure if you got a chance to look at the patch. We are actually not logging in on every executor, but only on the Client and the AM. The client logs in, sets the tokens for the AM and the AM starts using those. At 60% of the expiry of AM's tokens (so around 4.2 days), the AM logs in and then generates tokens which are passed around the executors. So the replay attack should not be a concern, since the login happens days apart.
The keytab is actually used only by the client once and the AM once. It is also passed around securely - we copy it to the application staging directory and the AM reads it from there.  So I don't think the distribution of the keytab is actually much of an issue.

ah ok, sorry no I did not look at the code.  Thanks for the explanation.
So yes as long as all the permissions are setup properly anything copied to staging and then localized should be secure.  Its more of a risk aspect then.  If you have secure storm cluster but not using ssl someone could sniff it.  or if there is a bug with permissions then someone could get it and the keytab lasts forever, vs just getting a single token and it lasts a limited time.

Correct, the risk aspect is right. If the keytab is compromised, there is a problem. The only way to work around this is to ensure that all communication is secure - via SSL or other mechanisms. Since we are writing to the staging dir, permissions does protect it. (I am not actually localizing it in the YARN sense, I am just using the HDFS API to copy it over to the local Spark working directory though, that is again secure via permissions).

Again, since the keytab is always sent only via HDFS API, we are ok, since that is encrypted on a secure HDFS cluster. Only the delegation tokens are sent via akka, which can be secured via SSL. So even if a token is sniffed, it is accessible only for a limited time (though the root cause would be that there is no SSL, which should be taken care of)

Left a few comments, most pretty minor, but a couple that need looking after.

Test build #28096 has started for   PR 4688 at commit 0985b4e.

This patch merges cleanly.

Test build #28096 has finished for   PR 4688 at commit 0985b4e.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  case class UpdateCredentials(newCredentialsLocation: String)

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28096/
Test PASSed.

Test build #28101 has started for   PR 4688 at commit b4cb917.

This patch merges cleanly.

Test build #28101 has finished for   PR 4688 at commit b4cb917.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  case class UpdateCredentials(newCredentialsLocation: String)

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28101/
Test FAILed.

Test build #28116 has started for   PR 4688 at commit 5c11c3e.

This patch merges cleanly.

Test build #28116 has finished for   PR 4688 at commit 5c11c3e.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  case class UpdateCredentials(newCredentialsLocation: String)

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28116/
Test PASSed.

Test build #28308 has started for   PR 4688 at commit f6954da.

This patch merges cleanly.

Test build #28308 has finished for   PR 4688 at commit f6954da.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28308/
Test FAILed.

Test build #28314 has started for   PR 4688 at commit f0f54cb.

This patch merges cleanly.

Test build #28314 has finished for   PR 4688 at commit f0f54cb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28314/
Test PASSed.

Test build #28318 has started for   PR 4688 at commit af6d5f0.

This patch merges cleanly.

Test build #28318 has finished for   PR 4688 at commit af6d5f0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28318/
Test PASSed.

Test build #28336 has started for   PR 4688 at commit 2debcea.

This patch merges cleanly.

Test build #28336 has finished for   PR 4688 at commit 2debcea.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28336/
Test FAILed.

Looks like YarnClusterSuite is missing the SPARK_YARN_MODE too

The reason this is failing is because sparkConf was used without specifying defaults. Updating PR soon.

Test build #28352 has started for   PR 4688 at commit f4fd711.

This patch merges cleanly.

Test build #28352 has finished for   PR 4688 at commit f4fd711.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28352/
Test PASSed.

Test build #28360 has started for   PR 4688 at commit 9ef5f1b.

This patch merges cleanly.

Test build #28360 has finished for   PR 4688 at commit 9ef5f1b.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28360/
Test FAILed.

Test build #28362 has started for   PR 4688 at commit 55522e3.

This patch merges cleanly.

Test build #28362 has finished for   PR 4688 at commit 55522e3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28362/
Test PASSed.

Following Thomas's earlier comment

Yes, slider does keytabs. For deploying things like HBase, Accumulo & Storm, those keytabs will be around anyway.
The trouble with keytabs is you need to go to the security bit of IT and they will often turn you down.
Because once that keytab is compromised you are in trouble: and wont even know you are in trouble
The other tactic is client-side refresh of tickets, over a kerberos-authed IPC channel.
which is what Twill does
at which point you worry about that client staying up/being restarted.

This patch takes the Twill path.
YARN does refresh the AM/RM tokens, and (Hadoop 2.6+) fixes up the tokens so that a restarted
AM can have its resources localised and that initial 7 days of HDFS access. That's something to include in a functional test with a cluster set to expire tokens every hour.

Test build #28967 has started for   PR 4688 at commit 55522e3.

This patch does not merge cleanly.

Test build #28967 has finished for   PR 4688 at commit 55522e3.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28967/
Test PASSed.

Test build #28973 has started for   PR 4688 at commit 0de27ee.

This patch merges cleanly.

Test build #28973 has finished for   PR 4688 at commit 0de27ee.

This patch fails Spark unit tests.

This patch merges cleanly.

This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28973/
Test FAILed.

Test build #28979 has started for   PR 4688 at commit 42813b4.

This patch merges cleanly.

Test build #28979 has finished for   PR 4688 at commit 42813b4.

This patch passes all tests.

This patch merges cleanly.

This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28979/
Test PASSed.

Looks OK to me. The code in Client.scala is getting pretty hard to follow, would probably benefit from some cleanup later on...

Yeah, Client.scala was already pretty complex (it took me a while to figure out what was happening, I think we'd need to do a separate cleanup jira at some point)

Test build #29102 has started for   PR 4688 at commit fa233bd.

This patch merges cleanly.

Test build #29102 has finished for   PR 4688 at commit fa233bd.

This patch passes all tests.

This patch merges cleanly.

This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29102/
Test PASSed.

Test build #29124 has started for   PR 4688 at commit 62c45ce.

This patch merges cleanly.

Test build #29124 has finished for   PR 4688 at commit 62c45ce.

This patch passes all tests.

This patch merges cleanly.

This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29124/
Test PASSed.

I met the same problem in our cluster. And I have a question for this design: when AM or executor failover, is YARN able to download resources(like jar) from HDFS with old token?

On AM failover, YARN refreshes the tokens for the AM so that it the NM localizer can restart the app. when the AM collects these tokens (by the normal mechanism), they should have the normal expiry time. Provided the AM gets these to the containers when launched, the executors get current tokens too.
Life is more complex on work-preserving-AM-restart —when the containers stay up during AM failover. They don't get the new tokens automatically, so a mechanism like this patch is needed. the Spark AM doesn't attempt this (yet) —it requires all state to be persisted somewhere or rebuildable, the code to actually do the rebuilding, and a large set of tests to verify it works.

Thanks Steve for explanation.
I found that in YARN-2704(Hadoop v2.6) has the ability to refresh token as proxy user.

YARN-896 is where to go to look for issues related to long-lived services.

Since Spark executors die when the AM dies anyway, work-preserving restart of the AM is the same as a normal restart as far as spark is concerned.
So when the new AM is created, we will still create new HDFS tokens and write it out.

Test build #29175 has started for   PR 4688 at commit 61b2b27.

This patch merges cleanly.

Test build #29175 has finished for   PR 4688 at commit 61b2b27.

This patch passes all tests.

This patch merges cleanly.

This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29175/
Test PASSed.

Test build #29192 has started for   PR 4688 at commit 2f9975c.

This patch merges cleanly.

Test build #29192 has finished for   PR 4688 at commit 2f9975c.

This patch passes all tests.

This patch merges cleanly.

This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29192/
Test PASSed.

@andrewor14 @tgravescs @pwendell - Could you guys take a look? I am testing it over this week to ensure functionality works. It would be great to get any feedback you guys have in the mean time

Test build #29268 has started for   PR 4688 at commit f74303c.

This patch merges cleanly.

Test build #29268 has finished for   PR 4688 at commit f74303c.

This patch passes all tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

class AMDelegationTokenRenewer(sparkConf: SparkConf, hadoopConf: Configuration) extends Logging

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29268/
Test PASSed.

Test build #29963 has started for   PR 4688 at commit 7f1bc58.

This patch does not merge cleanly.

Test build #29963 has finished for   PR 4688 at commit 7f1bc58.

This patch fails Spark unit tests.

This patch does not merge cleanly.

This patch adds the following public classes (experimental):

  case class RegisteredExecutor(tokens: Option[SerializableBuffer])

  case class NewTokens(tokens: SerializableBuffer) extends CoarseGrainedClusterMessage

class AMDelegationTokenRenewer(sparkConf: SparkConf, hadoopConf: Configuration) extends Logging

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29963/
Test FAILed.

Test build #29976 has started for   PR 4688 at commit 0e9507e.

Test build #29976 has finished for   PR 4688 at commit 0e9507e.

This patch fails MiMa tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

  case class RegisteredExecutor(tokens: Option[SerializableBuffer])

  case class NewTokens(tokens: SerializableBuffer) extends CoarseGrainedClusterMessage

class AMDelegationTokenRenewer(sparkConf: SparkConf, hadoopConf: Configuration) extends Logging

This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29976/
Test FAILed.

@pwendell - This should not fail MiMa test since this is an internal API change, right?
@shaneknapp - Looks like this was a spurious failure - the tests didn't seem to have run. Any idea what is happening here?

Jenkins, retest this please

Test build #29983 has started for   PR 4688 at commit 0e9507e.

Test build #29983 has finished for   PR 4688 at commit 0e9507e.

This patch fails Spark unit tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

  case class RegisteredExecutor(tokens: Option[SerializableBuffer])

  case class NewTokens(tokens: SerializableBuffer) extends CoarseGrainedClusterMessage

class AMDelegationTokenRenewer(sparkConf: SparkConf, hadoopConf: Configuration) extends Logging

This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29983/
Test FAILed.

Test build #30011 has started for   PR 4688 at commit e800c8b.

Jenkins, test this please

Test build #30013 has started for   PR 4688 at commit e800c8b.

Test build #30011 has finished for   PR 4688 at commit e800c8b.

This patch passes all tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

  case class NewTokens(tokens: SerializableBuffer) extends CoarseGrainedClusterMessage

class AMDelegationTokenRenewer(sparkConf: SparkConf, hadoopConf: Configuration) extends Logging

This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30011/
Test PASSed.

Test build #30013 has finished for   PR 4688 at commit e800c8b.

This patch passes all tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

  case class NewTokens(tokens: SerializableBuffer) extends CoarseGrainedClusterMessage

class AMDelegationTokenRenewer(sparkConf: SparkConf, hadoopConf: Configuration) extends Logging

This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30013/
Test PASSed.

@tgravescs @vanzin - Could you please take a look?

I need a second look, but looks sane, other than RPC system weirdness which isn't really your fault.

Hi @harishreedharan - could you add some more documentation for this? The high level architecture here may be hard for users to see. Here are some places you might consider documenting how this works:

The security page in the Spark docs: http://spark.apache.org/docs/latest/security.html (that one could probably use some better YARN related docs anyways).
(if relevant) the Security Manager Doc: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SecurityManager.scala
In the Spark submit doc describing --keytab - I would say something like "The keytab will be copied into the distributed cache for regular token renewal." It would be good to explain very explicitly that we're copying this private credential around the cluster.

It would be good to just make it discoverable for someone who is trying to figure out how this works. Basically that you copy keytab and then do regular keytab-based logins from within the cluster.

Test build #30387 has started for   PR 4688 at commit 8a4f268.

Test build #30387 has finished for   PR 4688 at commit 8a4f268.

This patch passes all tests.

This patch does not merge cleanly.

This patch adds the following public classes (experimental):

  case class NewTokens(tokens: SerializableBuffer) extends CoarseGrainedClusterMessage

This patch adds the following new dependencies:

commons-math3-3.1.1.jar
snappy-java-<IP_ADDRESS>.jar

This patch removes the following dependencies:

commons-math3-3.4.1.jar
snappy-java-<IP_ADDRESS>.jar

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30387/
Test PASSed.

Test build #30395 has started for   PR 4688 at commit 7b19643.

Test build #30395 has finished for   PR 4688 at commit 7b19643.

This patch passes all tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

  case class NewTokens(tokens: SerializableBuffer) extends CoarseGrainedClusterMessage

This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30395/
Test PASSed.

Test build #30641 has started for   PR 4688 at commit ebb36f5.

Test build #30641 has finished for   PR 4688 at commit ebb36f5.

This patch passes all tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

  case class RegisteredExecutor(tokens: Option[SerializableBuffer])

  case class NewTokens(tokens: SerializableBuffer) extends CoarseGrainedClusterMessage

This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30641/
Test PASSed.

Hi @tgravescs - could you please take a look at this one and see if this looks good to you?

@harishreedharan what manual testing have you done with this?  Have you actually left cluster up long enough to need it to login again?  Have you killed executors after that period and make sure new ones come up successfully and can access everything?

I have done testing by making sure the app runs longer than the expiration time, but not killing the executors. I will test that scenario too and report back. In the mean time, please let me know if you have any other comments.

@harishreedharan , fyi - we are doing a feature code freeze for spark 1.4 this friday.   I think this is really close so hopefully we can get it in. Let me know if there are any questions of concerns on my comments.

I am testing the changes right now. I will update this PR soon. Thanks @tgravescs!

So I noticed that if the dfs.namenode.delegation.token.renew-interval is different from the max lifetime of the token, a lot of exceptions get thrown around with the token being expired etc - and the executors may not be able to read the new tokens. It looks like the tokens don't get renewed if HDFS is not accessed before the renew interval - so for an executor which accesses HDFS rarely enough, it may not be able to read from HDFS.
So instead of waiting till 80% of max lifetime, I wait till 0.75 * dfs.namenode.delegation.token.renew-interval to renew. This means that the hdfs-site.xml file must be in sync with the one on the namenode (my understanding is this param's value is rarely changed, so this is unlikely to be an issue at all).

Test build #31085 has started for   PR 4688 at commit 42eead4.

Test build #31085 has finished for   PR 4688 at commit 42eead4.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.
This patch removes the following dependencies:

RoaringBitmap-0.4.5.jar
activation-1.1.jar
akka-actor_2.10-2.3.4-spark.jar
akka-remote_2.10-2.3.4-spark.jar
akka-slf4j_2.10-2.3.4-spark.jar
aopalliance-1.0.jar
arpack_combined_all-0.1.jar
avro-1.7.7.jar
breeze-macros_2.10-0.11.2.jar
breeze_2.10-0.11.2.jar
chill-java-0.5.0.jar
chill_2.10-0.5.0.jar
commons-beanutils-1.7.0.jar
commons-beanutils-core-1.8.0.jar
commons-cli-1.2.jar
commons-codec-1.10.jar
commons-collections-3.2.1.jar
commons-compress-1.4.1.jar
commons-configuration-1.6.jar
commons-digester-1.8.jar
commons-httpclient-3.1.jar
commons-io-2.1.jar
commons-lang-2.5.jar
commons-lang3-3.3.2.jar
commons-math-2.1.jar
commons-math3-3.4.1.jar
commons-net-2.2.jar
compress-lzf-1.0.0.jar
config-1.2.1.jar
core-1.1.2.jar
curator-client-2.4.0.jar
curator-framework-2.4.0.jar
curator-recipes-2.4.0.jar
gmbal-api-only-3.0.0-b023.jar
grizzly-framework-2.1.2.jar
grizzly-http-2.1.2.jar
grizzly-http-server-2.1.2.jar
grizzly-http-servlet-2.1.2.jar
grizzly-rcm-2.1.2.jar
groovy-all-2.3.7.jar
guava-14.0.1.jar
guice-3.0.jar
hadoop-annotations-2.2.0.jar
hadoop-auth-2.2.0.jar
hadoop-client-2.2.0.jar
hadoop-common-2.2.0.jar
hadoop-hdfs-2.2.0.jar
hadoop-mapreduce-client-app-2.2.0.jar
hadoop-mapreduce-client-common-2.2.0.jar
hadoop-mapreduce-client-core-2.2.0.jar
hadoop-mapreduce-client-jobclient-2.2.0.jar
hadoop-mapreduce-client-shuffle-2.2.0.jar
hadoop-yarn-api-2.2.0.jar
hadoop-yarn-client-2.2.0.jar
hadoop-yarn-common-2.2.0.jar
hadoop-yarn-server-common-2.2.0.jar
ivy-2.4.0.jar
jackson-annotations-2.4.0.jar
jackson-core-2.4.4.jar
jackson-core-asl-1.8.8.jar
jackson-databind-2.4.4.jar
jackson-jaxrs-1.8.8.jar
jackson-mapper-asl-1.8.8.jar
jackson-module-scala_2.10-2.4.4.jar
jackson-xc-1.8.8.jar
jansi-1.4.jar
javax.inject-1.jar
javax.servlet-3.0.0.v201112011016.jar
javax.servlet-3.1.jar
javax.servlet-api-3.0.1.jar
jaxb-api-2.2.2.jar
jaxb-impl-2.2.3-1.jar
jcl-over-slf4j-1.7.10.jar
jersey-client-1.9.jar
jersey-core-1.9.jar
jersey-grizzly2-1.9.jar
jersey-guice-1.9.jar
jersey-json-1.9.jar
jersey-server-1.9.jar
jersey-test-framework-core-1.9.jar
jersey-test-framework-grizzly2-1.9.jar
jets3t-0.7.1.jar
jettison-1.1.jar
jetty-util-6.1.26.jar
jline-0.9.94.jar
jline-2.10.4.jar
jodd-core-3.6.3.jar
json4s-ast_2.10-3.2.10.jar
json4s-core_2.10-3.2.10.jar
json4s-jackson_2.10-3.2.10.jar
jsr305-1.3.9.jar
jtransforms-2.4.0.jar
jul-to-slf4j-1.7.10.jar
kryo-2.21.jar
log4j-1.2.17.jar
lz4-1.2.0.jar
management-api-3.0.0-b012.jar
mesos-0.21.0-shaded-protobuf.jar
metrics-core-3.1.0.jar
metrics-graphite-3.1.0.jar
metrics-json-3.1.0.jar
metrics-jvm-3.1.0.jar
minlog-1.2.jar
netty-3.8.0.Final.jar
netty-all-4.0.23.Final.jar
objenesis-1.2.jar
opencsv-2.3.jar
oro-2.0.8.jar
paranamer-2.6.jar
parquet-column-1.6.0rc3.jar
parquet-common-1.6.0rc3.jar
parquet-encoding-1.6.0rc3.jar
parquet-format-2.2.0-rc1.jar
parquet-generator-1.6.0rc3.jar
parquet-hadoop-1.6.0rc3.jar
parquet-jackson-1.6.0rc3.jar
protobuf-java-2.4.1.jar
protobuf-java-2.5.0-spark.jar
py4j-<IP_ADDRESS>.jar
pyrolite-2.0.1.jar
quasiquotes_2.10-2.0.1.jar
reflectasm-1.07-shaded.jar
scala-compiler-2.10.4.jar
scala-library-2.10.4.jar
scala-reflect-2.10.4.jar
scalap-2.10.4.jar
scalatest_2.10-2.2.1.jar
slf4j-api-1.7.10.jar
slf4j-log4j12-1.7.10.jar
snappy-java-<IP_ADDRESS>.jar
spark-bagel_2.10-1.4.0-SNAPSHOT.jar
spark-catalyst_2.10-1.4.0-SNAPSHOT.jar
spark-core_2.10-1.4.0-SNAPSHOT.jar
spark-graphx_2.10-1.4.0-SNAPSHOT.jar
spark-launcher_2.10-1.4.0-SNAPSHOT.jar
spark-mllib_2.10-1.4.0-SNAPSHOT.jar
spark-network-common_2.10-1.4.0-SNAPSHOT.jar
spark-network-shuffle_2.10-1.4.0-SNAPSHOT.jar
spark-repl_2.10-1.4.0-SNAPSHOT.jar
spark-sql_2.10-1.4.0-SNAPSHOT.jar
spark-streaming_2.10-1.4.0-SNAPSHOT.jar
spire-macros_2.10-0.7.4.jar
spire_2.10-0.7.4.jar
stax-api-1.0.1.jar
stream-2.7.0.jar
tachyon-0.6.4.jar
tachyon-client-0.6.4.jar
uncommons-maths-1.2.2a.jar
unused-1.0.0.jar
xmlenc-0.52.jar
xz-1.0.jar
zookeeper-3.4.5.jar

Build failure caused by ThreadUtils being split off Utils. Since the patch was merging cleanly, I didn't merge master back in. Doing so now.

Test build #31094 has started for   PR 4688 at commit 072659e.

Test build #31094 has finished for   PR 4688 at commit 072659e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Failures are unrelated to this patch. Other PRs are failing too: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31089/
Looks like master build might be broken or there is a Jenkins issue.

Test build #31131 has started for   PR 4688 at commit 6963bbc.

Test build #31131 has finished for   PR 4688 at commit 6963bbc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

@tgravescs - I tested this by running an app for several hours with renewal interval set to 10 mins and max life time set to 20.
I killed an executor after many hours and the new one picked up the tokens and wrote fine. Killing the AM also worked and the new AM logged into the KDC immediately and got fresh tokens.

So I noticed that if the dfs.namenode.delegation.token.renew-interval is different from the max lifetime  of the token, a lot of exceptions get thrown around with the token being expired etc - and the executors  may not be able to read the new tokens. It looks like the tokens don't get renewed if HDFS is not accessed before the renew interval - so for an executor which accesses HDFS rarely enough, it may not  be able to read from HDFS.

So instead of waiting till 80% of max lifetime, I wait till 0.75 * dfs.namenode.delegation.token.renew-interval to renew. This means that the hdfs-site.xml file must be in sync with the one on the namenode >(my understanding is this param's value is rarely changed, so this is unlikely to be an issue at all).

thanks for the updates and details on testing.
So my guess on this is that after the initial expiration period the yarn RM isn't renewing the tokens anymore since it doesn't get the updated ones (it only has the one you initially submitted the application with).  Thus in order for the token to stay good for longer then 1 day you either have to renew it or do the loginFromKeytab like you mention.
So you could change this to renew until the max lifetime and then do the loginFromKeytab.  I don't think doing the loginFromKeytab is going to add much more overhead then doing the renewal so I'm ok with leaving this doing the loginFromKeytab before the renewal period. We could always change it later if we decide.
I'd rather not use dfs.namenode.delegation.token.renew-interval config.  As you say it might not match on the gateway as compared to what the namenode is using.  You can get the renewal interval by doing a renew on the token once.  Then we can store that and do the loginFromKeytab at X% of that. Note that addDelegationTokens in obtainTokensForNamenodes will return a list of tokens that you could renew to get the period.
I'll look through the rest of the code and leave any comments.

I couldn't find a way to renew the tokens. Is there a way to do that using the token itself? The token contains the max date, but not the renewal date.

Ah, looks like the token class has a method.

OK, so I am testing with the t.renew method. It will take an hour or so (I have the renewal interval set to 10 mins). Once it is done, I will push the update.

as you say the token has a renew method.  It hands you back the expiration on that renew.  you can store that expiration.
Are you changing to renew for 7 days and then do login or just do renew ones to get the expiration?

I am just doing the renew to get the expiration. That keeps the logic the same - always login using keytab and get new tokens rather than having different code paths for renew and creating new ones

just a comment here.  We are only getting the hdfs tokens.  We also support adding hive metastore tokens and there is pr up for hbase tokens.  We should probably file a followup jira to add those in also.

It looks like tokens can be renewed only if the user is logged in via Kerberos, which is not the case for executors. So this method to calculate renewal intervals on the executor is failing. How about figuring out the renewal interval in Client.scala and then using sparkConf to pass it around? This way, we don't have to find it out on each executor.

I think thats fine.

Also, I had to hack around a bit here. Since we currently create tokens with yarn as renewer the renew() call fails. So I had to create a new token as the principal and then renew that one.  I didn't want to change the renewer of the tokens created for general use, since that might mess other things up. I am testing that change now. Updating soon.

Merged build triggered.

Merged build started.

Test build #31203 has started for   PR 4688 at commit 09fe224.

Test build #31203 has finished for   PR 4688 at commit 09fe224.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31203/
Test PASSed.

Merged build triggered.

Merged build started.

Test build #31231 has started for   PR 4688 at commit 611923a.

Test build #31231 has finished for   PR 4688 at commit 611923a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31231/
Test FAILed.

Looks like a Jenkins or build issue. Other builds are failing too: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31219/
Will restart build later

Jenkins, test this please

Merged build triggered.

Merged build started.

Test build #31244 has started for   PR 4688 at commit 611923a.

Test build #31244 has finished for   PR 4688 at commit 611923a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31244/
Test PASSed.

@tgravescs - Do you have any more comments? I'd like to get this into 1.4 if possible.

@tgravescs - Jira for Hive and HBase token updates: https://issues.apache.org/jira/browse/SPARK-7252

@harishreedharan sorry I've been swamped today, I'll look at this later tonight.

Thanks!

Merged build triggered.

Merged build started.

Test build #31395 has started for   PR 4688 at commit 36eb8a9.

Test build #31395 has finished for   PR 4688 at commit 36eb8a9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch removes the following dependencies:

RoaringBitmap-0.4.5.jar
activation-1.1.jar
akka-actor_2.10-2.3.4-spark.jar
akka-remote_2.10-2.3.4-spark.jar
akka-slf4j_2.10-2.3.4-spark.jar
aopalliance-1.0.jar
arpack_combined_all-0.1.jar
avro-1.7.7.jar
breeze-macros_2.10-0.11.2.jar
breeze_2.10-0.11.2.jar
chill-java-0.5.0.jar
chill_2.10-0.5.0.jar
commons-beanutils-1.7.0.jar
commons-beanutils-core-1.8.0.jar
commons-cli-1.2.jar
commons-codec-1.10.jar
commons-collections-3.2.1.jar
commons-compress-1.4.1.jar
commons-configuration-1.6.jar
commons-digester-1.8.jar
commons-httpclient-3.1.jar
commons-io-2.1.jar
commons-lang-2.5.jar
commons-lang3-3.3.2.jar
commons-math-2.1.jar
commons-math3-3.4.1.jar
commons-net-2.2.jar
compress-lzf-1.0.0.jar
config-1.2.1.jar
core-1.1.2.jar
curator-client-2.4.0.jar
curator-framework-2.4.0.jar
curator-recipes-2.4.0.jar
gmbal-api-only-3.0.0-b023.jar
grizzly-framework-2.1.2.jar
grizzly-http-2.1.2.jar
grizzly-http-server-2.1.2.jar
grizzly-http-servlet-2.1.2.jar
grizzly-rcm-2.1.2.jar
groovy-all-2.3.7.jar
guava-14.0.1.jar
guice-3.0.jar
hadoop-annotations-2.2.0.jar
hadoop-auth-2.2.0.jar
hadoop-client-2.2.0.jar
hadoop-common-2.2.0.jar
hadoop-hdfs-2.2.0.jar
hadoop-mapreduce-client-app-2.2.0.jar
hadoop-mapreduce-client-common-2.2.0.jar
hadoop-mapreduce-client-core-2.2.0.jar
hadoop-mapreduce-client-jobclient-2.2.0.jar
hadoop-mapreduce-client-shuffle-2.2.0.jar
hadoop-yarn-api-2.2.0.jar
hadoop-yarn-client-2.2.0.jar
hadoop-yarn-common-2.2.0.jar
hadoop-yarn-server-common-2.2.0.jar
ivy-2.4.0.jar
jackson-annotations-2.4.0.jar
jackson-core-2.4.4.jar
jackson-core-asl-1.8.8.jar
jackson-databind-2.4.4.jar
jackson-jaxrs-1.8.8.jar
jackson-mapper-asl-1.8.8.jar
jackson-module-scala_2.10-2.4.4.jar
jackson-xc-1.8.8.jar
jansi-1.4.jar
javax.inject-1.jar
javax.servlet-3.0.0.v201112011016.jar
javax.servlet-3.1.jar
javax.servlet-api-3.0.1.jar
jaxb-api-2.2.2.jar
jaxb-impl-2.2.3-1.jar
jcl-over-slf4j-1.7.10.jar
jersey-client-1.9.jar
jersey-core-1.9.jar
jersey-grizzly2-1.9.jar
jersey-guice-1.9.jar
jersey-json-1.9.jar
jersey-server-1.9.jar
jersey-test-framework-core-1.9.jar
jersey-test-framework-grizzly2-1.9.jar
jets3t-0.7.1.jar
jettison-1.1.jar
jetty-util-6.1.26.jar
jline-0.9.94.jar
jline-2.10.4.jar
jodd-core-3.6.3.jar
json4s-ast_2.10-3.2.10.jar
json4s-core_2.10-3.2.10.jar
json4s-jackson_2.10-3.2.10.jar
jsr305-1.3.9.jar
jtransforms-2.4.0.jar
jul-to-slf4j-1.7.10.jar
kryo-2.21.jar
log4j-1.2.17.jar
lz4-1.2.0.jar
management-api-3.0.0-b012.jar
mesos-0.21.0-shaded-protobuf.jar
metrics-core-3.1.0.jar
metrics-graphite-3.1.0.jar
metrics-json-3.1.0.jar
metrics-jvm-3.1.0.jar
minlog-1.2.jar
netty-3.8.0.Final.jar
netty-all-4.0.23.Final.jar
objenesis-1.2.jar
opencsv-2.3.jar
oro-2.0.8.jar
paranamer-2.6.jar
parquet-column-1.6.0rc3.jar
parquet-common-1.6.0rc3.jar
parquet-encoding-1.6.0rc3.jar
parquet-format-2.2.0-rc1.jar
parquet-generator-1.6.0rc3.jar
parquet-hadoop-1.6.0rc3.jar
parquet-jackson-1.6.0rc3.jar
protobuf-java-2.4.1.jar
protobuf-java-2.5.0-spark.jar
py4j-<IP_ADDRESS>.jar
pyrolite-2.0.1.jar
quasiquotes_2.10-2.0.1.jar
reflectasm-1.07-shaded.jar
scala-compiler-2.10.4.jar
scala-library-2.10.4.jar
scala-reflect-2.10.4.jar
scalap-2.10.4.jar
scalatest_2.10-2.2.1.jar
slf4j-api-1.7.10.jar
slf4j-log4j12-1.7.10.jar
snappy-java-<IP_ADDRESS>.jar
spark-bagel_2.10-1.4.0-SNAPSHOT.jar
spark-catalyst_2.10-1.4.0-SNAPSHOT.jar
spark-core_2.10-1.4.0-SNAPSHOT.jar
spark-graphx_2.10-1.4.0-SNAPSHOT.jar
spark-launcher_2.10-1.4.0-SNAPSHOT.jar
spark-mllib_2.10-1.4.0-SNAPSHOT.jar
spark-network-common_2.10-1.4.0-SNAPSHOT.jar
spark-network-shuffle_2.10-1.4.0-SNAPSHOT.jar
spark-repl_2.10-1.4.0-SNAPSHOT.jar
spark-sql_2.10-1.4.0-SNAPSHOT.jar
spark-streaming_2.10-1.4.0-SNAPSHOT.jar
spark-unsafe_2.10-1.4.0-SNAPSHOT.jar
spire-macros_2.10-0.7.4.jar
spire_2.10-0.7.4.jar
stax-api-1.0.1.jar
stream-2.7.0.jar
tachyon-0.6.4.jar
tachyon-client-0.6.4.jar
uncommons-maths-1.2.2a.jar
unused-1.0.0.jar
xmlenc-0.52.jar
xz-1.0.jar
zookeeper-3.4.5.jar

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31395/
Test PASSed.

Merged build finished. Test PASSed.

jenkins, test this please

Merged build triggered.

Merged build started.

Test build #31419 has started for   PR 4688 at commit 36eb8a9.

Test build #31419 has finished for   PR 4688 at commit 36eb8a9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31419/
Test PASSed.

Merged build finished. Test PASSed.

Tested the latest update, still working fine

+1, looks good.  Thanks for testing the latest, I was about to ask that.  I ran a few things with the patch but didn't configure things to expire.

This was reverted because it broke Hadoop-1. I am moving the ExecutorDelegationTokenRenewer to the yarn module and using reflection to use it the CoarseGrainedExecutorBackend class.


heroku demo app is down

The documentation contains links to https://motor-admin.herokuapp.com/demo/queries/3  (e.g. from the "Dashboards" documentation), but that app doesn't seem to be responding.
I get an Application Error when I visit that url.

@toddp now the demo is located under https://app.getmotoradmin.com/demo/ - will remove herokuapp.com links, thanks


UserAgent and forced platform question

I have several Android device UserAgent, but I see the code here forces platform identification as iOS.
$telegram_web_app = str_replace("return webAppPlatform;", 'return "ios";', $telegram_web_app);
The code is in hamsterkombat.io-telegram-web-app.php.

That code has nothing to do with bot itself, it's just simply used to make HamsterKombat accessible on Desktop.


increase hypershift node count to 6

https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-knative-serverless-operator-main-415-hypershift-e2e-hypershift-continuous/1829983318683160576
as hypershift job is also running EKB tests that normally use 6 workers ( https://github.com/openshift-knative/serverless-operator/blob/main/Makefile#L178  )

/lgtm


Status of the project and open PRs

Dear github maintainer,
there are PRs open since March 2020: https://github.com/actions/cache/pulls?page=2&q=is%3Apr+is%3Aopen
Please either remove the examples completely, if you dont plan to support them or merge/close the open PRs.
Ideally write a warning that the examples are not maintained and one should conduct the build system (of the language) as authoritative.
Its quite annoying to use wrong examples, in my case from Rust.
The current status makes me wonder how big the reliability standards of github are, when github does not manage to supervise (minimal) examples.
Best regards, Jan

Hi, we are looking into the issues/PRs actively and have a plan to bring down the number of issues/PRs in the coming months. Closing this for now. Thank you for opening.


First flutter run, error unzipping gradle

Launching lib/main.dart on Redmi 3S in debug mode...
Initializing gradle...                                   /
Oops; flutter has exited unexpectedly: "Exit code 1 from: G:\Flutter Projects\testproject\android\gradlew.bat -v:
Unzipping C:\Users\user.gradle\wrapper\dists\gradle-4.1-all\bzyivzo6n839fup2jbap0tjew\gradle-4.1-all.zip to C:\Users\user.gradle\wrapper\dists\gradle-4.1-all\bzyivzo6n839fup2jbap0tjew
Exception in thread "main" java.util.zip.ZipException: error in opening zip file
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.(ZipFile.java:219)
at java.util.zip.ZipFile.(ZipFile.java:149)
at java.util.zip.ZipFile.(ZipFile.java:163)
at org.gradle.wrapper.Install.unzip(Install.java:214)
at org.gradle.wrapper.Install.access$600(Install.java:27)
at org.gradle.wrapper.Install$1.call(Install.java:74)
at org.gradle.wrapper.Install$1.call(Install.java:48)
at org.gradle.wrapper.ExclusiveFileAccessManager.access(ExclusiveFileAccessManager.java:65)
at org.gradle.wrapper.Install.createDist(Install.java:48)
at org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:128)
at org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:61)".
Thanks

Does it work if you try again? This is actually an error from inside the gradle wrapper script, which we defer to but that isn't technically code we develop... any idea what caused the failure?

@Hixie I tried more than 5 to 6 times. but same error is coming again and again. No idea.

Closing as duplicate of https://github.com/flutter/flutter/issues/13645.


(#15100) Eliminate spurrious padding in .bs-docs-header

I agree with all the comments so far here.
Rather than dicker with container nesting at this late juncture, near sunset for BS 3.0x, let's just fix and move on.  It's docs only.
This PR addresses issue #15100 and eliminates the unnecessary horizontal scrollbar that appears for width for width >=768px and <= 782px.
Tested on three different iOS devices and FireFox/Chrome on OSX and it's fine now.
The only difference is the text in .bs-docs-header is shifted left by 15-pixels, lines are 30 px longer so wrap less in some circumstances in some viewports.
Fixes #15100.

Protip: Include "Fixes #15100" in the commit message next time, so that GitHub will auto-close the issue after the PR is merged.

Protip: Include "Fixes #15100" in the commit message next time, so that GitHub will auto-close the issue after the PR is merged.

Works in the issue/PR body, too, but not the issue/PR title.

Yes, of course.  Sorry @mdo. Done.


egolane

Can the lane detection be done just for lines on the left and right of the car?

Our lane detection method can detect multiple lanes including non-ego lanes.
Thank you!