[gym] Export of archive takes a LOT of time.

Issue Description
Using gym and export method ad-hoc or app-store takes on our project 30 minutes or more on Jenkins., but manual archive of the same project, same settings but using xCode archive takes 1/4 of the time - using MacBook Pro late 2015 with i7.
As you will se below, export action took 26 minutes.
Environment
Spec of Jenkins machine:
Mac Pro late 2013, Xeon E5 5 cores, 16 GB RAM, 512 SSD
MacOS: 10.12.1
Jenkins Version: 2.36
Xcode 8.1
✅ fastlane environment ✅
Stack

Key
Value

OS
10.12.1

Ruby
2.3.1

Bundler?
false

Git
git version 2.9.3 (Apple Git-75)

Installation Source
~/.rvm/gems/ruby-2.3.1/bin/fastlane

Host
Mac OS X 10.12.1 (16B2555)

Ruby Lib Dir
~/.rvm/rubies/ruby-2.3.1/lib

OpenSSL Version
OpenSSL 1.0.2g  1 Mar 2016

Is contained
false

Xcode Path
/Applications/Xcode.app/Contents/Developer/

Xcode Version
8.1

System Locale

Variable
Value

LANG
en_US.UTF-8
✅

LC_ALL
en_US.UTF-8
✅

LANGUAGE
en_US.UTF-8
✅

fastlane files:
`./fastlane/Fastfile`
#
# Project spec. Necessary things go here
#

	default_device = "iPhone 6s"

	devices_tests_ab = [
		"iPad Pro (12.9 inch)",
		"iPad Pro (9.7 inch)",
		"iPad Air",
		"iPhone 6s Plus",
		"iPhone 6s",
		"iPhone 5s"
	]

	schemes_tests_ab = [
	]

	schemes_tests = [
		"DataTypes",
		"GUI",
		"SportFeedsIO",
		"Store",
		"AppStore",
		"UIReusable",
		"UIDataSources",
		"App"
	]

	project_path_name = "App/App.xcodeproj"
	workspace_name = "Main.xcworkspace"
	scheme_release = "App"
	crashlytics_local_path = "App/Vendors/Crashlytics.framework/"
	sonar_name = "iOS-App"
	default_config = "Release"

	#Signing details, please fill it always!
	provisioning_adhoc = "PROVISIONING_PROFILE_SPECIFIER=\"App Ad Hoc\""
	provisioning_release = "PROVISIONING_PROFILE_SPECIFIER=\"App App Store\""

#
# the end of needed things
#

platform :ios do
	desc "Before all do this:"
	before_all do
	   	unlock_keychain(path: "login.keychain-db",
	   					password: "password")
 	end

	desc "Upload on iTunes"
	lane :deploy do
		name = scheme_release
		provisioning_profile = provisioning_release

		project_path = project_path_name
		build_number = number_of_commits

		increment_build_number(
			build_number: build_number,
			xcodeproj: project_path)

		build(export: "app-store",
			  provisioning: provisioning_profile)

		deliver(ipa: "#{name}.ipa",
				submit_for_review: false,
				skip_metadata: true,
				automatic_release: false,
				force: true)
	end

	desc "Run unit tests"
	lane :test do
		schemes = schemes_tests
		schemes.each do |scheme|
			test_generic(
				scheme:scheme,
				devices: [default_device])
		end
	end

	desc "Install all dependencies"
	lane :make do
		carthage(
			command: "bootstrap",
			platform: "iOS",
			use_binaries: true)
	end

	desc "Run build for fabric"
	lane :fabric do

		build_number = number_of_commits
		project_path = project_path_name
		workspace = workspace_name
		scheme = scheme_release
		crashlytics_path = crashlytics_local_path
		provisioning_profile = provisioning_adhoc

		version_number = get_version_number(xcodeproj: project_path)
		git_tag = "CI/v#{version_number}/##{build_number}"

		m_build_number = "Build number: #{build_number}"
		m_build_branch = "Branch: #{git_branch}"
		m_build_commit = "Commit: #{last_git_commit[:abbreviated_commit_hash]}"
		m_groups = ENV["CRASHLYTICS_GROUPS"]
		m_comments = ENV["CRASHLYTICS_NOTES"]

		message = [
			m_build_number,
			m_build_branch,
			m_build_commit,
			"GIT TAG: #{git_tag}",
			"Groups: #{m_groups}",
			"Release Notes: #{m_comments}"
		]
		.join("\n")

		increment_build_number(
			build_number: build_number,
			xcodeproj: project_path)

		build(export: "ad-hoc",
			  provisioning: provisioning_profile)

		crashlytics(
			crashlytics_path: "#{crashlytics_path}",
			api_token: "xxxxx",
			build_secret: "xxxxxx",
			notes: "#{message}",
			groups: m_groups)

		if !git_tag_exists(tag: git_tag)
			add_git_tag(tag: git_tag)
			push_git_tags
		end

		slackNotifier(message: message)
	end

	desc "Run metrics for all project modules"
	lane :metrics do
		schemes = schemes_tests
		schemes.each do |scheme|
			metrics_generic(
				scheme: scheme,
				devices: default_device)
		end
	end

	desc "Run all AB Tests"
	private_lane :test_ab do
		devices = devices_tests_ab
		schemes = schemes_tests_ab

		schemes.each do |scheme|
			test_generic(
				scheme:scheme,
				devices:devices)
		end
	end

	desc "Build project"
	private_lane :build do |options|
		cleanAll

		export = options[:export]
		provisioning = options[:provisioning]
		workspace = workspace_name
		scheme = scheme_release
		config = default_config

		gym(workspace: workspace,
			scheme: scheme,
			configuration: config,
			include_bitcode: false,
			export_method: export,
			clean: true,
			output_name: "#{scheme}.ipa")
	end

	desc "Run metrics for selected scheme in project"
	private_lane :metrics_generic do |options|

		scheme = options[:scheme]
		devices = options[:devices]
		project_path = project_path_name
		workspace = workspace_name
		crashlytics_path = crashlytics_local_path
		project_name = sonar_name

		test_generic(
			scheme:scheme,
			devices:devices)

		slather(
			workspace: "#{workspace}",
			scheme: "#{scheme}",
			proj: "#{scheme}/#{scheme}.xcodeproj",
			cobertura_xml: true,
			jenkins: true,
			build_directory: "./build/#{scheme}.DerivedData",
			output_directory: "./build/#{scheme}.reports/")
		sh("cd .. && lizard ./#{scheme}/ -l swift --xml > ./build/#{scheme}.reports/lizard-report.xml")

		swiftlint(output_file: "./build/#{scheme}.reports/swiftlint.txt", ignore_exit_status: true)

		sonar_parameters = [
			"sonar.projectKey=#{project_name}.#{scheme}",
			"sonar.projectName=#{project_name}.#{scheme}",
			"sonar.junit.reportsPath=build/#{scheme}.reports",
			"sonar.junit.include=*.junit",
			"sonar.sources=#{scheme}/#{scheme}",
			"sonar.tests=#{scheme}/#{scheme}Tests",
			"sonar.swift.lizard.report=build/#{scheme}.reports/lizard-report.xml",
			"sonar.swift.coverage.reportPattern=build/#{scheme}.reports/cobertura.xml",
			"sonar.swift.swiftlint.report=build/#{scheme}.reports/*swiftlint.txt",
			"sonar.swift.workspace=#{workspace}",
			"sonar.swift.project=#{scheme}/#{scheme}.xcodeproj"
		]

		final_sonar_parameters = "-D" + sonar_parameters.join(" -D")

		sonar(
			project_key:"#{project_name}.#{scheme}",
			project_name:"#{project_name}.#{scheme}",
			sonar_runner_args: final_sonar_parameters)
	end

    desc "Runs unit tests for selected scheme"
	private_lane :test_generic do |options|

		scheme = options[:scheme]
		devices = options[:devices]

		scan(
			scheme: scheme,
			code_coverage: true,
			clean: true,
			devices: devices,
			derived_data_path: "./build/#{scheme}.DerivedData",
			output_directory: "./build/#{scheme}.reports/")
	end

	private_lane :cleanAll do
		workspace = workspace_name
		scheme = scheme_release

		xcclean(workspace:  workspace_name,
	      		scheme: scheme)

		clear_derived_data
	end

	private_lane :slackNotifier do |options|
		message = options[:message]
		slack(
			slack_url: "https://hooks.slack.com/services/xxxx",
			channel: "xxxxi",
			message: "#{message}")
	end

	desc "Called on error"
	error do |lane, exception, options|
		slackNotifier(message: exception)
	end
end

**No Appfile found**
### fastlane gems

Gem
Version
Update-Status

credentials_manager
0.16.2
✅ Up-To-Date

spaceship
0.39.0
✅ Up-To-Date

fastlane_core
0.60.0
✅ Up-To-Date

deliver
1.16.2
✅ Up-To-Date

snapshot
1.17.0
✅ Up-To-Date

frameit
3.0.0
✅ Up-To-Date

pem
1.4.1
✅ Up-To-Date

cert
1.4.5
✅ Up-To-Date

sigh
1.12.1
✅ Up-To-Date

produce
1.4.0
✅ Up-To-Date

gym
1.13.0
✅ Up-To-Date

pilot
1.13.1
✅ Up-To-Date

scan
0.14.2
✅ Up-To-Date

supply
0.8.0
✅ Up-To-Date

match
0.11.1
✅ Up-To-Date

screengrab
0.5.6
✅ Up-To-Date

fastlane
1.111.0
✅ Up-To-Date

Loaded fastlane plugins:
No plugins Loaded
Loaded gems

Gem
Version

did_you_mean
1.0.2

executable-hooks
1.3.2

bundler-unload
1.0.2

rubygems-bundler
1.4.4

bundler
1.13.6

io-console
0.4.6

highline
1.7.8

commander
4.4.1

terminal-table
1.4.5

json
1.8.3

multipart-post
2.0.0

faraday
0.10.0

faraday_middleware
0.10.1

dotenv
2.1.1

net-ssh
3.2.0

net-sftp
2.1.2

plist
3.2.0

security
0.1.3

krausefx-shenzhen
0.14.11

slack-notifier
1.5.1

CFPropertyList
2.3.4

claide
1.0.1

colored
1.2

nanaimo
0.2.3

xcodeproj
1.4.1

rouge
1.11.1

xcpretty
0.2.4

terminal-notifier
1.7.1

public_suffix
2.0.4

addressable
2.5.0

credentials_manager
0.16.2

multi_xml
0.6.0

unf_ext
<IP_ADDRESS>

unf
0.1.4

domain_name
0.5.20161129

http-cookie
1.0.3

faraday-cookie_jar
0.0.6

babosa
1.0.2

xcode-install
2.0.9

word_wrap
1.0.0

multi_json
1.12.1

excon
0.54.0

rubyzip
1.1.7

gh_inspector
1.0.2

fastlane_core
0.60.0

fastimage
1.6.8

mini_magick
4.5.1

xcpretty-travis-formatter
0.0.4

uber
0.0.15

representable
2.3.0

retriable
2.1.0

mime-types-data
3.2016.0521

mime-types
3.1

hurley
0.2

little-plugger
1.1.4

logging
2.1.0

jwt
1.5.6

memoist
0.15.0

os
0.9.6

signet
0.7.3

googleauth
0.5.1

httpclient
2.8.3

google-api-client
0.9.20

i18n
0.7.0

thread_safe
0.3.5

tzinfo
1.2.2

activesupport
<IP_ADDRESS>

*generated on:* **2016-12-15**

Fastlane output
[13:35:03]: ▸ Touching App.app
[13:37:09]: ▸ Signing /Users/jenkins/Library/Developer/Xcode/DerivedData/Main-dxwovjwuwgnfzygmrpkrhcvwkesr/Build/Intermediates/ArchiveIntermediates/App/InstallationBuildProductsLocation/Applications/App.com.app
[13:37:36]: ▸ Touching App.app.dSYM
[13:37:37]: ▸ Archive Succeeded
[13:37:37]: Generated plist file with the following values:
[13:37:37]: ▸ -----------------------------------------
[13:37:37]: ▸ {
[13:37:37]: ▸   "method": "ad-hoc"
[13:37:37]: ▸ }
[13:37:37]: ▸ -----------------------------------------
[13:37:37]: $ /usr/bin/xcrun /Users/jenkins/.rvm/gems/ruby-2.3.1/gems/gym-1.13.0/lib/assets/wrap_xcodebuild/xcbuild-safe.sh -exportArchive -exportOptionsPlist '/var/folders/q1/45d0dl9x52g0p0r2rq9ld3680000gn/T/gym_config20161215-45654-18u2704.plist' -archivePath /Users/jenkins/Library/Developer/Xcode/Archives/2016-12-15/App\ 2016-12-15\ 13.17.52.xcarchive -exportPath '/var/folders/q1/45d0dl9x52g0p0r2rq9ld3680000gn/T/gym_output20161215-45654-pz12s5'
[14:03:54]: Compressing 32 dSYM(s)
[14:03:54]: $ cd '/Users/jenkins/Library/Developer/Xcode/Archives/2016-12-15/App 2016-12-15 13.17.52.xcarchive/dSYMs' && zip -r '/Users/jenkins/.jenkins/jobs/Mobile-iOS-App-Develop/workspace/App.app.dSYM.zip' *.dSYM

I recently had an issue with my build times increasing with each run; I'd suggest checking your keychain search path from the command line.
security list-keychains
I was using unlock_keychain, and every time it ran, it would add my login keychain to the search path.
Before I found the problem, my build was taking over 2 hours. It was due to codesigning looping through each search path entry to do the codesigning.
and try security find-identity
I resolved the issue by adding the add_to_search_list to my unlock_keychain call.
		unlock_keychain(
			path: "login.keychain",
			add_to_search_list: :replace,
			password: ENV['FASTLANE_PASSWORD']
		)

@neldridge Thank you for that, I've checked list of key chains and look like this:
    "/Users/jenkins/Library/Keychains/login.keychain-db"
    "/Users/jenkins/Library/Keychains/login.keychain-db"
    "/Users/jenkins/Library/Keychains/login.keychain-db"
    "/Users/jenkins/Library/Keychains/login.keychain-db"
    "/Users/jenkins/Library/Keychains/login.keychain-db"
    "/Users/jenkins/Library/Keychains/login.keychain-db"
    "/Users/jenkins/Library/Keychains/login.keychain-db"
    "/Library/Keychains/System.keychain"

So I have to remove those duplicate ones in list ?

Look for unlock_keychain  in your Fastfile (or maybe it's a config option in Jenkins?), and add the parameter below- fastlane will remove the duplicates by itself, and resolve the issue.
add_to_search_list: :replace

Ok now I know what to do. Fastlane action unlock_keychain does no share that knowledge with me ;)
I will test it, once again thank you for help.

@KrauseFx is this a known issue?
It may be worth changing the default behavior of unlock_keychain; I would think the expected behavior would be :replace - not adding the same keychain to the search path with every execution.

Hey, I don't have enough background information about this. What kinds of setups does this affect? What would we change? Would that have any side effects? Would that break any existing setups?

this could only happen if the keychain to be unlocked could not be found in the current listed keychains. (https://github.com/fastlane/fastlane/blob/master/fastlane/lib/fastlane/actions/unlock_keychain.rb#L35)
but even if triggered - the code executed then is exact the same as with :replace
@milch could that be related to the FastlaneCore::Helper.keychain_path  i could'nt reproduce it.
setting :path  to a full path or even setting it to just the filename (as initial issue reporter did)

@hjanuschka
Hi Helmut,
I am able to reproduce this with a new project directory and fastfile. What is the default expected behavior here? It sounds like the default behavior is :replace
Fastlane version: 2.1.1
OS X version: 10.12.1
Fastfile contents
skip_docs
opt_out_usage
fastlane_version "2.0.0"

default_platform :ios

platform :ios do

	lane :test_unlock do
		unlock_keychain(
			path: "login.keychain",
		)
	end
end

Commands to reproduce
iosbuilder:test-fastlane iosbuilder$ security list-keychains
    "/Users/iosbuilder/Library/Keychains/login.keychain-db"
    "/Users/iosbuilder/Library/Keychains/login.keychain-db"
    "/Users/iosbuilder/Library/Keychains/login.keychain-db"
    "/Library/Keychains/System.keychain"
iosbuilder:test-fastlane iosbuilder$ fastlane
[08:52:41]: ---------------------------
[08:52:41]: --- Step: opt_out_usage ---
[08:52:41]: ---------------------------
[08:52:41]: Disabled upload of used actions
[08:52:41]: -------------------------------------------------
[08:52:41]: --- Step: Verifying required fastlane version ---
[08:52:41]: -------------------------------------------------
[08:52:41]: Your fastlane version 2.1.1 matches the minimum requirement of 2.0.0  ✅
[08:52:41]: ------------------------------
[08:52:41]: --- Step: default_platform ---
[08:52:41]: ------------------------------
[08:52:41]: Welcome to fastlane! Here's what your app is setup to do:
+--------+-----------------+------------------------------+
|                 Available lanes to run                  |
+--------+-----------------+------------------------------+
| Number | Lane Name       | Description                  |
+--------+-----------------+------------------------------+
| 1      | ios test_unlock |                              |
| 0      | cancel          | No selection, exit fastlane! |
+--------+-----------------+------------------------------+
Which number would you like run?
1
[08:52:44]: Running lane `ios test_unlock`. Next time you can do this by directly typing `fastlane ios test_unlock` 🚀.
[08:52:44]: Driving the lane 'ios test_unlock' 🚀
[08:52:44]: -----------------------------
[08:52:44]: --- Step: unlock_keychain ---
[08:52:44]: -----------------------------
[08:52:44]: To not be asked about this value, you can specify it using 'password'
Keychain password: **************

+------+-------------------------------------+-------------+
|                     fastlane summary                     |
+------+-------------------------------------+-------------+
| Step | Action                              | Time (in s) |
+------+-------------------------------------+-------------+
| 1    | opt_out_usage                       | 0           |
| 2    | Verifying required fastlane version | 0           |
| 3    | default_platform                    | 0           |
| 4    | unlock_keychain                     | 3           |
+------+-------------------------------------+-------------+

[08:52:47]: fastlane.tools finished successfully 🎉
iosbuilder:test-fastlane iosbuilder$ security list-keychains
    "/Users/iosbuilder/Library/Keychains/login.keychain-db"
    "/Users/iosbuilder/Library/Keychains/login.keychain-db"
    "/Users/iosbuilder/Library/Keychains/login.keychain-db"
    "/Users/iosbuilder/Library/Keychains/login.keychain-db"
    "/Library/Keychains/System.keychain"

Please let me know if there's anything else I can help with.
-Nick

@hjanuschka Don't worry, I'm able to reproduce it - could be related to the change, could be that it's something that only happens on Sierra. I'll take a look.
@neldridge Thanks for the detailed report! I was able to quickly reproduce it using
$ security list-keychains
    ".../login.keychain-db"
    ".../System.keychain"
$ fastlane run unlock_keychain path:"login.keychain" password:$KEYCHAIN_PASSWORD
[10:48:35]: -----------------------------
[10:48:35]: --- Step: unlock_keychain ---
[10:48:35]: -----------------------------
[10:48:35]: Result: ["", "", ""]
$ security list-keychains
    ".../login.keychain-db"
    ".../login.keychain-db"
    ".../System.keychain"

@milch No problem, glad I can help out. good you know you can run functions in fastlane like that also, thanks :)
fwiw, I had this issue before upgrading to fastlane version 2. (which makes sense in the context that no code changes were made, only the conversion to a monogem).
Let me know if there's anything else I can help with.

@milch ohh nice to hear you can reproduce, seeing your output i think the problem is that list-keychains return's ".." in the path - which does not match the include?  in https://github.com/fastlane/fastlane/blob/master/fastlane/lib/fastlane/actions/unlock_keychain.rb#L35
in my case list-keychains reports full-resolved paths. strange!

@hjanuschka Oh sorry for the confusion, the ... were just put in by me to get rid of the full path.

@milch  oh well - i thought that might be the problem - let me know if can help in any way to tackle this.

@hjanuschka Thanks, will do!
In an attempt to reproduce this on my work computer this morning, I suddenly wasn't able to anymore! The only difference between this one and my personal laptop (where I was able to repro it) I can think of is that this one had a fresh install of Sierra, while my personal one was upgraded from ElCap. I'll take a look at it again tonight when I get home!

As a note, my machines are upgraded from ElCap to Sierra.

As a note, my machines are upgraded from ElCap to Sierra.

Same story with my CI machine.


Autoloading WordPress core classes

As you know core classes are always require-d in WordPress.
Here is a way to use Composer class autoloader instead: https://github.com/szepeviktor/wordpress-autoloaded
What do you think?

I surely think that would be better to use autoloader in WordPress, but I don't want to add that feature to WP Starter.
In fact, WP Starter 3 allows to install WordPress from any source. If you happen to have a WordPress installation that has autoloader enabled and comes as a Composer package, you can just use it.
If you have a script that remove all hardcoded require and replace with autoloder, you can easily integrate that in WP Starter, and if you wrap it as a WP Starter step / extension (https://github.com/wecodemore/wpstarter/blob/dev/docs/08-Custom-Steps-Development.md) that could be easily reusable.
In short, I don't dislike the idea, I just don't think that WP Starter should have that feature: if anyone wants to use the untouched official release of WordPress (or any WordPress "variant") I want they to be able to do that when using WP Starter.

Okay!


Fixing publint error and warning

The types field needs to be first in the exports object and no type is specified. This should fix all of the publint issues for the package.
Reference: https://publint.dev/<EMAIL_ADDRESS>

@mikecousins I'll take a look at this PR soon, thanks for looking into this.

@mikecousins I've finally had a chance to take a look at this PR and it makes sense to me. I've also validated this place nicely with some common test apps that have been a little tricky in the past, namely:

Next.js
Remix
Sveltekit
The classic Express app

To actually merge this PR in you'll also need to add a changefile - we use a tool called beachball for semantic versioning. If you run npm run change on this branch you'll be guided through a dialog to set the type of change (just a patch), and add a note about the change that will get included in our changelog.

closing because these changes were made in https://github.com/passageidentity/passage-node/pull/172


Media component bug in Chrome 55 (easy fix)

Issue
A strange race condition in Chrome for Windows causes
 components to display incorrectly sometimes
Details
Having a display: table-cell element whose immediate parent is not a display: table-row or display: table is invalid according to the W3C table spec:
https://www.w3.org/TR/CSS2/tables.html#table-display
However: According to the W3C spec, when an element is misparented an anonymous element should be generated around it with the proper display type. Therefore there should be no issue. And this is, in fact, the case in Internet Explorer and Firefox (did not test Safari or Opera)
A bug or race condition in Chrome causes it to improperly generate these anonymous elements when the page takes too long to render (many DOM elements on the page). The net effect looks like so:

Repro

Operating system: Windows 10
Browser: Chrome 55

In order to create this effect I had a page with many elements above the media item:

A responsive navbar
A carousel
A collapsible form with multiple form-groups and inputs
Images, text, etc

At the bottom of the page I had a card with a media inside of it:

...

...

                ...

Fix
By setting display: table on .media it generates properly every time:
.media {
    display: table;
}

Fixed:

/cc @cvrebert

@stevendesu Please post the URL of your example webpage or a link to a CodePen/JSBin/etc. example that demonstrates the problem, so that we can file a Chrome bug.

https://jsbin.com/ruhoqocumo/1/edit?html,output

Won't be an issue anymore with #21389—we're going all in on flexbox.


Incorrect statement about Basic licenses able to create test plans

There is a note within the first section of the page that states "Note: Stakeholders cannot create or manage test plans. You must have at least Basic access."  This is not true, however, as you must have Basic + Test Plans access to create/manage test plans.
"Note: Stakeholders cannot create or manage test plans. You must have at least Basic access."

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: c8e2cb79-709d-972b-6747-6d26c965fb2a
Version Independent ID: d512ea69-a97a-e5ff-a804-6691c5dfe1e6
Content: Create test plans and suites - Azure Test Plans
Content Source: docs/test/create-a-test-plan.md
Product: devops
Technology: devops-test
GitHub Login: @steved0x
Microsoft Alias: sdanie

That doc has been updated to say that "Note: Stakeholders cannot create or manage test cases. You must have at least Basic access." since basic can create test cases, and basic + test to create test plans.
https://docs.microsoft.com/en-us/azure/devops/test/create-a-test-plan?view=azure-devops#access-level


Use appdirs to determine cache directory.

Resolves issue #124.

Those tests are not failing in my development environment, investigating.

@CasperWA Just a simple mistake, should be all good now.

Those tests are not failing in my development environment, investigating.

Seems fine now after the latest push.

Do you still want me to adjust the logger file as well or is CWD ok?

Do you still want me to adjust the logger file as well or is CWD ok?

CWD is fine for now I think, then we can leave the log stuff for later. I can create an issue.


MINOR: Small enhancement to Deserializer Javadoc

I’ve implemented my own custom Deserializer and been using it with KStream.reduceByKey; I observed that reduceByKey was passing null to my implementation, but it wasn’t clear to me what my implementation was expected to do in this case. So this attempts to clarify it.
This is my original work and I license this work to the Kafka project under Kafka’s open source license (the Apache License 2.0).

LGTM

LGTM


todo(perf): prefetch GET /api/me/votes if user is signed in

Similar to #515. However, GET /api/me/votes should be prefetched if the user is signed in. This should be done wherever votes are possible: /, /users/:login and /items/:id.

@ryanponce, would this one interest you?


Create the correct healthcheck probes

Is there a related GitHub Issue?
#1701
What is this change about?
CF allows timeout seconds for an app to start, with a probe each 2 seconds. Once started, CF probes the app each 30 seconds, killing the container if a probe fails. The health-check-invocation-timeout determines how long to wait for a response for these probes.
To reproduce this functionality in korifi we need to create:

A startup probe with:

periodSeconds: 2
timeoutSeconds: health-check-invocation-timeout
failureThreshold: (timeout / 2) rounded to the next integer.

A liveness probe with:

periodSeconds: 30
timeoutSeconds: health-check-invocation-timeout
failureThreshold: 1

The readiness probe is not required as this does not correspond to CF functionality. It would stop k8s routing traffic to an app failing the readiness probe, but would not terminate it. This might be a nice-to-have in the future.
Does this PR introduce a breaking change?
No
Acceptance Steps
See #1701
Tag your pair, your PM, and/or team
@georgethebeatle

The readiness probe is not required as this does not correspond to CF functionality. It would stop k8s routing traffic to an app failing the readiness probe, but would not terminate it. This might be a nice-to-have in the future.

I assume for slow starting apps the new startupProbe we're adding will keep them from getting traffic prematurely as well, right? The readiness probe would be more around an app that went "un-ready" sometime later in its lifecycle.

Yep - liveness and readiness kick in after startup suceeds


Bumped web-components version

#2620  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @tommylau523 and the rest of your teammates on  Graphite


nyc

Please use the template provided below, when reporting bugs:
Expected Behavior
Observed Behavior
Bonus Points! Code (or Repository) that Reproduces Issue
Forensic Information
Operating System: the operating system you observed the issue on.
Environment Information: information about your project's environment, see instructions below:

run the following script:

sh -c 'node --version; npm --version; npm ls' > output.txt

share a gist with the contents of output.txt.

sh -c 'node --version; npm --version; npm ls' > output.txt

ок


feat: revisit color scheme

I was not a fan of the bright colors that were not part of the palette (cyan, magenta), so I revisited them and tried to fix a couple of visual bugs as well.
See screenshots here and in the extended self-review.
Light
Before

After

Dark
Before

After

I used https://mycolor.space/?hex=%23DA6200&sub=1 to create 2 similar colors:
Pulling the cyclic dependency link out: https://npmgraph.js.org/?q=string-width%404.2.3%2C+yargs%4016%2C+wrap-ansi%407%2C+#select=exact%3Acliui%407.0.4

I'm not totally sure about the green, but it's slightly more pleasant.
I also reverted the orange background, even though I think it's a downgrade. I'll keep playing with the colors.

Ok I think I got it:

selected: orange, thick
downstream: orange
upstream: cyan (but the direct negative of the accent)

Seems obvious in retrospect: unrelated modules are already dimmed, so why does the upstream need to use a third color?

Merging now. If you'd like I can drop unused colors separately.
To create related colors we can use oklch colors and color-mix() in the future:

https://oklch.com/


Fix atscfg comparing pointers not values

This causes the config gen to think CDNs are different when they aren't, causing it to think no servers on the same CDN are assigned, omitting a max_origin_connections entry.
Includes modifying the tests to have different pointers, so the tests will fail without this fix.
Includes changelog.
No docs, no interface change.

[x] This PR is not related to any other Issue

Which Traffic Control components are affected by this PR?

Traffic Ops ORT

What is the best way to verify this PR?
Generate config on a Mid with MaxOriginConnetions on a DS, verify max_origin_connections is inserted in header rewrite file.
If this is a bug fix, what versions of Traffic Control are affected?

master
5.0
5.1

The following criteria are ALL met by this PR

[x] This PR includes tests OR I have explained why tests are unnecessary
[x] This PR includes documentation OR I have explained why documentation is unnecessary
[x] This PR includes an update to CHANGELOG.md OR such an update is not necessary
[x] This PR includes any and all required license headers
[x] This PR DOES NOT FIX A SERIOUS SECURITY VULNERABILITY (see the Apache Software Foundation's security guidelines for details)

Additional Information

This looks good to me.

@rob05c this seems to fix https://github.com/apache/trafficcontrol/issues/5427 -- does it?

I'm not entirely clear on the scope of that case. But yes, with this fix, ATC will auto-calculate the max origin connections per mid.


Add support for MongoDB

Create a new module 'mongodb' to support MongoDB.
Add a new sample app with MongoDB.

https://github.com/Kobee1203/weedow-searchy/pull/31


Error embeding png

What were you trying to do?
Embed png file into page
How did you attempt to do it?
const pdf = await PDFDocument.create();
const image = await pdf.embedPng(file);
const page = pdf.addPage([image.width, image.height]);

file is buffer, attached

What actually happened?
2024-10-17 01:54:06.455	ERROR	Invoke Error 	{"errorType":"TypeError","errorMessage":"dV.default.decode is not a function","stack":["TypeError: dV.default.decode is not a function","    at new PNG (/node_modules/@cantoo/pdf-lib/src/utils/png.ts:51:23)","    at pngData (/node_modules/@cantoo/pdf-lib/src/utils/png.ts:41:42)","    at Function.for (/node_modules/@cantoo/pdf-lib/src/core/embedders/PngEmbedder.ts:12:21)","    at a.embedPng (/node_modules/@cantoo/pdf-lib/src/api/PDFDocument.ts:1113:40)"
What did you expect to happen?
added successfully
How can we reproduce the issue?
const pdf = await PDFDocument.create();
const image = await pdf.embedPng(file);
const page = pdf.addPage([image.width, image.height]);

file is buffer, attached
Version
2.2.3
What environment are you running pdf-lib in?
Node
Checklist

[X] My report includes a Short, Self Contained, Correct (Compilable) Example.
[X] I have attached all PDFs, images, and other files needed to run my SSCCE.

Additional Notes
No response

strangely enough it doesn't happen running with bun, only nodejs (v20, v22)

original pdf-lib doesn't encounter this issue FYI

Which version of Node did you use?

happens on node 18, 20, and 22

looks to be happening when using esm ...
import UPNG from '@pdf-lib/upng';

is an object with default

It seems like it was due to a typo in the code for esm. I'll fix and release.

To solve the issue, we would probably need to solve this problem:
https://github.com/cantoo-scribe/pdf-lib/issues/24


btc-rpc-explorer: init at 2.0.2

Motivation for this change
Adds btc-rpc-explorer to nodePackages.
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[x] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

rebased on master

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-already-reviewed/2617/290


ADD: Season POST and GET Endpoint + Database Table

GIVEN: I'm logged into the application as a Manager or Admin configuring seasons
WHEN: I click on "Seasons"
THEN: I can see the Seasons
Acceptance Criteria

Inherit from DBModel
All ID variables of the model, if any, need to be of type uint.
The name of the Season
The time (day, month, year, hour, ...) the seasons Starts
The time (day, month, year, hour, ...) the seasons Ends
A slice of type Registration as the variable Registrations
A slice of type Game as the variable Schedule
A slice of type League as the variable Leagues
API endpoint for /rosters that returns all leagues

In the folder /backend/internal/server/apis the file seasons.go should exist in the folder groups

Endpoint for /seasons includes all associated leagues
Rapidocs implemented for /seasons endpoint

I could take a shot at this one

API endpoint for /rosters that returns all leagues
Endpoint for /seasons includes all associated leagues

@tannerwheeler could you expand on these for me? Are these supposed to be seasons instead of leagues?

I changed 'API endpoint for /rosters that returns all leagues'. Rosters should have been seasons.
For the second, you should be able to see all of the Leagues when you look at the season.
Did that help?


Add coveralls

Because "encrypted environment variables are not available for pull requests from forks" from https://docs.travis-ci.com/user/encryption-keys/ , I need to merge it first to validate it. If it go south, I'll revert this merge.


Great work! However i would like to know use cases about how BLIP-Diffusion can be used in Salesforce and how will it make job easier for us users?

BLIP-Diffusion facilitates effective subject-driven image generation and editing. It is an integral part of the Generative AI initiative.

Okay Thanks , So I also wanted to know what is The meaning/full form of BLIP in BLIP-Diffusion? As BLIP is all in capital it may stand for an acronym.

@Akashdeep122 This BLIP-Diffusion model is based on the BLIP (bootstrapping Language Image Pre-training) model that has been developed by SalesForce. They are using a variant to make the BLIP-Diffusion architecture

Thank you for letting me know :)


[SPARK-15992] [MESOS] Refactor MesosCoarseGrainedSchedulerBackend offer consideration

The offer acceptance workflow is a little hard to follow and not very extensible for future considerations for offers. This is a patch that makes the workflow a little more explicit in its handling of offer resources.

Can one of the admins verify this patch?

Thanks for the pull request, but the refactoring seems to have made it more difficult to read?

I broke something in the test suite, evaluating and I'll open this back in a bit

Can one of the admins verify this patch?

ok to test

Test build #60997 has started for PR 13715 at commit 9e0aedf.

Test build #60997 has finished for PR 13715 at commit 9e0aedf.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60997/
Test FAILed.

(ping @drcrallen)


DISC: pd.DataFrame methods we specifically don't want included

#3 has a discussion of which pd.DataFrame methods should be included in the standard based on a) measures of popularity among users and b) which are common across dataframe libraries.  I'd like to try coming at it from the other direction: what existing pd.DataFrame methods can we exclude from consideration?
Throwing out some ideas here, none of these are strong opinions on my part:

metadata-centric attributes/methods

attrs
flags
set_flags

deprecated pd.DataFrame methods

tshift
slice_shift
lookup
iteritems
append

non-dunder arithmetic methods

add
radd
mul
rmul
[26 of these total]
eq
ne

Methods that don't make sense if there isn't a row-index

set_index
reset_index
sort_index
stack
unstack
swapaxes
asof
between_time
at_time
last_valid_index
first_valid_index

Methods that don't make sense if there isn't a MultiIndex

droplevel
reorder_levels
swaplevel

Dtype specific methods

explode (object)
infer_objects
sparse

Other

to_records/from_records
to_dict/from_dict
potentially many other to_foo/pd.read_foo methods/functions
style
to_period/to_timestamp/tz_convert/tz_localize/asfreq

Thanks @jbrockmendel, that is a very good question. My handwavy suggestion would be "anything that's not core to the data structure or methods to manipulate it or do basic computations on it" .Your list makes complete sense to me, and I'd add:

I/O functionality
Visualization functionality
All options/settings type APIs: https://pandas.pydata.org/docs/user_guide/options.html
Higher level statistical methods (e.g. pct_change, cov, corr, rank, spearman, kendall, pearson). It must be possible to defer such methods to statsmodels, scipy.stats or another such library, rather than requiring it to be reimplemented in every dataframe library.
Special calendar handling, e.g. BusinessHour & co, Easter, SemiMonth, etc. Very little on https://pandas.pydata.org/docs/reference/offset_frequency.html aside from DateOffset and to_offset probably.
Not sure, but probably most windowing functionality: https://pandas.pydata.org/docs/user_guide/window.html
Exceptions and warnings (as a general principle, only expected input/behavior should be specified, not responses to unexpected input - because there's an endless amount of possible unexpected inputs).
An API extension mechanism (https://pandas.pydata.org/docs/reference/extensions.html)

The above is all my sense of "not core". For a different reason I'd add eval and query - evaluating string expressions of syntax rather than the actual syntax seems like a performance optimization detail bubbled up to the end user. If you already optimize performance in a different way, or don't have something like numexpr, it probably doesn't make sense.

Thanks! Largely agree with all the suggestions so far. Just one comment:

Not sure, but probably most windowing functionality: https://pandas.pydata.org/docs/user_guide/window.html

I think windowing, and in particular, grouped-window functions, should stay. It's core to timeseries analysis and not easy for users to work around or implement themselves.

Higher level statistical methods (e.g. pct_change, cov, corr, rank, spearman, kendall, pearson). It must be possible to defer such methods to statsmodels, scipy.stats or another such library, rather than requiring it to be reimplemented in every dataframe library.

Often these are used as groupby and window functions, where other libraries typically don't have grouped implementations and calling them per group leads to very subpar performance.
I'm still a +1 on removing them at least from the v1 of the API though 😄.

Thanks for the details on grouped window functions, interesting. So it seems like an important topic to deal with at some point. It feels like something can be made more composable there - I had a quick look at how it's implemented in Pandas. For example 'spearman' is only used as:
class DataFrame:
    def corr(...):
        elif method == "spearman":
            correl = libalgos.nancorr_spearman(mat, minp=min_periods)

And libalgos.nancorr_spearman looks like a standard statistical function - 2-D array in, numerical result out (no special handling of a groupby object):
def nancorr_spearman(ndarray[float64_t, ndim=2] mat, Py_ssize_t minp=1) -> ndarray:

So that seems generalizable to any callable and correlation metric. Same for if it would take some object that is the result of calling groupby: should still be able to define what functions take in and return, so you don't have to reimplement such things over and over.
Windowing is a large topic (and has a large API surface), maybe worth splitting off into its own issue?

We had a little brainstorm on functions not to include in a call a few weeks ago. Here are some notes on what was discussed regarding APIs that would be good to exclude:

APIs dealing with row indices & implicit alignment. These are basically impossible to do in parallel distributed settings. There's also not many use cases for these - these are some, e.g. in Dask sorting on index allows for some performance optimizations. That's an implementation detail though.
row indices in general
do not allow calling general dataframe operations against groupby and windowing groups.

for apply, relevant operations need to be defined.
Outside of apply, there are various operations possible in pandas (e.g., as performing a correlation on a groupby), which are not heavily used, but requires specialization in libraries, such as cuDF. (note that in Pandas, this isn't done, but it may be needed in principle to get good performance)

.empty / .is_empty -> all agree
.bool -> all agree
.describe and .info. Useful in interactive contexts, but not by libraries
.explode -> several people find this useful (e.g. when flattening nested data), keep
iter-like methods -> wide agreement, some discussion points made:

for iter methods, would be nice if they could explicitly raise, rather than defer to library implementations. Would prefer users be guided to built-in methods.
we should not support making dataframe iterables in any way.
breaking backwards compat in e.g. pandas is tricky though. Question about what to include in standard and what not to include.

even if standard says to raise, does not mean that we cannot include in standard. There will have to be some sort of compatibility mode.

.transpose / .T -> everyone agrees
.head / .tail -> maybe:  they are used, but they overlap with iloc and other methods. On the other hand, easy to implement.
.squeeze

if there is no Series, then there's no use for .squeeze perhaps?

.combine -> agree, remove
.update -> agree, remove (indexing is better)
.select_dtypes and .exclude_dtypes -> some discussion: useful for end users, less so for libraries. likely leave out for now, but may revisit later
.convert_dtypes (converts NumPy dtypes to nullable dtypes) -> agree, remove
.mask? discussion: perhaps doesn't make sense if you are immutable; however doesn't have to be in-place. Could use where instead.
.query -> agreement that this is a big hack/mess, good to remove.

If something like this is desired, need to put some work in standardizing expression format. Some work done in Arrow. Oriented toward devs and more onerous for users.
more attempts in that direction, like numexpr, patsy
related to eval. All of this fancy query stuff is a fancy way of doing loop fusion. pandas is just deferring to numexpr.
if one wants to do loop fusion, one should just write/use a compiler.

Maybe controversial, but should loc and iloc be in this list?

Maybe controversial, but should loc and iloc be in this list?

loc sounds like a good candidate bc in the absence of a row-index i think it would be dominated by __getitem__.
iloc I'd expect to be more universally useful, though it could be replaced by something like iat+slice+take
Definitely want to avoid overloading __getitem__ (https://github.com/pandas-dev/pandas/issues/9595)

I think everything here has been addressed (or rather, kept out), so I think this can be closed, do let me know if I've misunderstood


Phi (possibly also other greek letters) behaving different in comments and code

In my Juno, \phi in code returns what in LaTeX would be \varphi and vice versa. In the comments however, \phi and \varphi return the expected symbol. The screenshot shows only the use of \phi
(Juno 0.12.6, Julia 1.5.3, UTF-8)

That's a font issue -- the italics variant switches phi and varphi. I'd recommend using JuliaMono (or any other consistent font):


Webpack-2 recursive tree-shaking/dead code elimination.

Hello, i've been trying out<EMAIL_ADDRESS>and noticed that webpack doesn't eliminate dead code recursively. Let me show you an example:
import { otherFunc1, otherFunc2 } from './otherModule';

function callOther() {
  otherFunc1();
  otherFunc2();
}

export default function () {
  if (process.env.NODE_ENV !== 'production') {
    callOther();
  }
}

Let's say process.env.NODE_ENV === 'production', so the condition is always false.

I would expect the whole if block in default function to be stripped out.
So callOther would get stripped out.
So otherFunc1 and otherFunc2 would get stripped out(they aren't used anywhere else).

Webpack does steps 1 and 2, but skips step 3, so otherFunc1 and otherFunc2 end up in the bundle anyway.
If i call otherFunc1 and otherFunc2 directly inside the if block, they don't end up in the bundle.
My webpack.config.js:
'use strict';

var webpack = require('webpack');

var env = process.env.NODE_ENV;

var config = {
  module: {
    loaders: [
      { test: /\.js$/, loader: 'babel-loader', exclude: /node_modules/ }
    ]
  },
  output: {
    library: 'bundle',
    libraryTarget: 'umd'
  },
  plugins: [
    new webpack.DefinePlugin({
      'process.env.NODE_ENV': JSON.stringify(env)
    })
  ]
};

module.exports = config;

My .babelrc:
{
  "plugins": [
    ["transform-es2015-template-literals", { "loose": true }],
      "transform-es2015-arrow-functions",
      "transform-es2015-shorthand-properties",
      ["transform-es2015-computed-properties", { "loose": true }],
      "check-es2015-constants",
      ["transform-es2015-spread", { "loose": true }],
      "transform-es2015-parameters",
      ["transform-es2015-destructuring", { "loose": true }],
      "transform-es2015-block-scoping",
      "transform-object-rest-spread",
  ]
}

And i call webpack with:
cross-env NODE_ENV=production webpack -p src/index.js dist/bundle.min.js"

Thanks!

+1

Looks like I have the same issue, was it working in the previous version ?

@maxs15 no. Just checked versions 2.0.4-beta - 2.0.7-beta. Haven't tried the previous betas, but i doubt it used to work in those.

@le0nik I think I know why this happens.
As I understand it, webpack does dead code elimination in two steps.  First it figures out which exports are being referenced, and it writes the chunk's code, only assigning exports to the exports object if they're "used" somewhere.  Then it passes that code to UglifyJS, which removes dead code.

I would expect the whole if block in default function to be stripped out.
So callOther would get stripped out.
So otherFunc1 and otherFunc2 would get stripped out(they aren't used anywhere else).

Numbers 1 and 2 are handled by UglifyJS.  Number 3 is handled by webpack.  However, since webpack is doing its part before UglifyJS, it doesn't know it can remove those functions.

If i call otherFunc1 and otherFunc2 directly inside the if block, they don't end up in the bundle.

I think you're just getting lucky here because webpack ignores code inside false conditionals.  It doesn't, however, remove that code; it waits for Uglify to take care of that.

@sokra @cspotcode
To expand on this (sorry if highjacking, as at this moment it seems appropriate), this is the Webpack output, without running -p:
/***/ },
/* 1 */
/***/ function(module, exports, __webpack_require__) {

	var _createClass = function () { function defineProperties(target, props) { for (var i = 0; i < props.length; i++) { var descriptor = props[i]; descriptor.enumerable = descriptor.enumerable || false; descriptor.configurable = true; if ("value" in descriptor) descriptor.writable = true; Object.defineProperty(target, descriptor.key, descriptor); } } return function (Constructor, protoProps, staticProps) { if (protoProps) defineProperties(Constructor.prototype, protoProps); if (staticProps) defineProperties(Constructor, staticProps); return Constructor; }; }();

	function _classCallCheck(instance, Constructor) { if (!(instance instanceof Constructor)) { throw new TypeError("Cannot call a class as a function"); } }

	var Bar = function () {
	    function Bar() {
	        _classCallCheck(this, Bar);
	    }

	    _createClass(Bar, [{
	        key: 'contructor',
	        value: function contructor() {
	            this.bar = 'classBar';
	        }
	    }]);

	    return Bar;
	}();/* unused harmony export Bar */

So Webpack clearly recognizes the export as unused (as expected).
So what is the recommendation to minify it? Uglify doesn't do that for reasons outlined above.
What I am trying to say is - since there is no working minification solution, to say "tree-shaking is working" is kind of misleading, isn't it?

@dnutels @cspotcode @le0nik:
If I understand correctly, Webpack will perform the "tree shaking" part of the equation. Tree shaking creates "dead code" which can be eliminated through a couple plugins (webpack-closure-compiler, UglifyJsPlugin). Also, I believe LoaderOptionsPlugin is the new way to specify to loaders what should be minified/optimized/etc. because of #283 https://github.com/webpack/webpack/issues/283#issuecomment-179219468.
Has anyone tested this issue out in the latest beta. I'd like to get this into tracking if this is still a problem.

@TheLarkInn
Haven't tested this yet, but can tell you that LoaderOptionsPlugin is sort of a unicorn, as I was unable to find any documentation on it aside from this: https://gist.github.com/sokra/27b24881210b56bbaff7#loader-options--minimize

I have an idea for how this can work.  UglifyJS can already do all the dead code elimination we need.  No sense re-implementing it when all we need to do is emit the right kind of code for Uglify.
ES6 exports can be thought of as global variables.  Imagine fooModule.js does export var foo = 'foo'; and barModule.js does import {foo} from './fooModule'; console.log(foo);  You can think about it like a single variable that's written in one place and accessed at the other.
var __webpack_es6_named_export__fooModule__foo__ = 'foo';
console.log(__webpack_es6_named_export__fooModule__foo__);

If webpack emits code that looks like the above, UglifyJS can perform recursive dead code elimination. (or "tree-shaking;" two words for the same thing)  Then webpack can rewrite all occurrences of __webpack_es6_named_export__... to be full accesses / assignments on the relevant exports objects.  Then UglifyJS can take the result and mangle variable names.

So, does webpack include unused imports in the bundle or not?

It does. Webpack can mark unused imports/exports, (see usedExports property on HarmonyExportSpecifierDependency (I believe)), however it's simply marking the statements and specifiers. Is it possible, maybe, but would be extremely complex and time consuming. We believe that a generic tool should be passed an AST program, to assist in analyzing and optimizing (and used/implemented in Webpack similar to UglifyJsPlugin) that any bundler etc can use. (Like babili, but more than just ES6 I think)
We'd love to have help from Flow, TS, babili, esprima, acorn, uglify, whatever, and framework teams to make one unifying tool to that can provide a High Level general purpose static program flow analysis optimizer/minifier. I know each library, parser has separate needs, but I think there could be a high-level approach at some of this, and benefits everyone.
/cc @hzoo @gaearon @danielrosenwasser @wycatz @igorminar @Rich-Harris @aryia
My super super high level thoughts:
Tool takes ES6(or mixed commonjs, etc whatever is possible)AST/Program, tool does optimization, returns either optimized AST Program for respective parsers/compilers/etc to use.

Should move discussion to #2867


please add support for custom actions

like custom icons and custom labels

This project is focussed on power/login management. Is there some particular action related to power/login management which you would like added to the menu of options? Or you want to change the language of the labels? Or something else?

I want to reload sway config, change cpu governor, change actions order ...

The actions will appear on the screen in the order that you list the *-command options in the config file.
I've added a 'reload' action, see fb0850b.
Changing cpu governor sounds like something you should bind a shortcut key to bring up a menu program (wofi or whatever) to do. Or bind a shortcut key to a script that will cycle through some list of governors, and pop up a notification telling you what the current governor is.

Just two-three actions for two-three governors, without popups or menus.

The actions will appear on the screen in the order that you list the *-command options in the config file.

Nice! Could you describe this in the readme please.

The actions will appear on the screen in the order that you list the *-command options in the config file.

Nice! Could you describe this in the readme please.

This is already described in the APPEARANCE section of the manpage.

Just two-three actions for two-three governors, without popups or menus.

Your screen real estate will get crowded --- currently there is no code to arrange the action indicators in more than a single horizontal row. So there is a fair bit of complexity needed to be added to handle both reading custom actions from the config and also adding and handling an nrows config option.
I will mark this as a feature request, but I likely will not have any time to work on it until the summer, if then.
Pull requests welcome.


Update status messages when unsupported devices are detected by the tools

cc @devoncarew Addresses the issues raised in https://github.com/flutter/flutter/pull/2049 after it had landed.

lgtm! thanks

Rebased


epic - is it usual that analyses take time?

Hi,
Just to ask, I have run epic and it seems to be stuck at this step:
Making duplicated bins unique by summing them. (File: count_reads_in_windows, Log level: INFO, Time: Wed, 09 Nov 2016 22:19:45 )
Merging ChIP and Input data. (File: helper_functions, Log level: INFO, Time: Wed, 09 Nov 2016 22:22:00 )
0.8 effective_genome_size (File: compute_background_probabilites, Log level: DEBUG, Time: Wed, 09 Nov 2016 22:24:23 )
200 window size (File: compute_background_probabilites, Log level: DEBUG, Time: Wed, 09 Nov 2016 22:24:23 )
61753836 total chip count (File: compute_background_probabilites, Log level: DEBUG, Time: Wed, 09 Nov 2016 22:24:23 )
15438459000.0 average_window_readcount (File: compute_background_probabilites, Log level: DEBUG, Time: Wed, 09 Nov 2016 22:24:23 )
It has been >24hrs now, is this usual?
My input command was as follows:
epic --treatment --treatment /pathtofolder/lib.filtered.H3K27me3_sorted.MARKDUP.bedpe --input /pathtofolder/lib.filtered.input.sorted_2_MARKDUP.bedpe chromsizes /home/anish/marked_duplicates_ChIP-seq_YM/smed_only/SmedAsxl.size.file --bigwig BIGIWG -pe --fragment-size 300 -cpu 30 --effective_genome_size 0.8
Many thanks,
Anish

No, epic should finish rather quickly. I use massive files and it takes 5
minutes, tops (optionally writing additional data such as matrixes to disk
might take a little extra time).
Your non-standard files are causing this to happen. I can't debug this
without the files unfortunately. If you have the opportunity to share them
I can try to look closer at it.
Is there a reason you are doing ChIP-Seq calling on contigs?
On Fri, Nov 11, 2016 at 12:22 AM, anishdattani<EMAIL_ADDRESS>wrote:

Hi,
Just to ask, I have run epic and it seems to be stuck at this step:
Making duplicated bins unique by summing them. (File:
count_reads_in_windows, Log level: INFO, Time: Wed, 09 Nov 2016 22:19:45 )
Merging ChIP and Input data. (File: helper_functions, Log level: INFO,
Time: Wed, 09 Nov 2016 22:22:00 )
0.8 effective_genome_size (File: compute_background_probabilites, Log
level: DEBUG, Time: Wed, 09 Nov 2016 22:24:23 )
200 window size (File: compute_background_probabilites, Log level: DEBUG,
Time: Wed, 09 Nov 2016 22:24:23 )
61753836 total chip count (File: compute_background_probabilites, Log
level: DEBUG, Time: Wed, 09 Nov 2016 22:24:23 )
15438459000.0 average_window_readcount (File: compute_background_probabilites,
Log level: DEBUG, Time: Wed, 09 Nov 2016 22:24:23 )
It has been >24hrs now, is this usual?
My input command was as follows:
epic --treatment --treatment /pathtofolder/lib.filtered.H3K27me3_sorted.MARKDUP.bedpe
--input /pathtofolder/lib.filtered.input.sorted_2_MARKDUP.bedpe
chromsizes /home/anish/marked_duplicates_ChIP-seq_YM/smed_only/SmedAsxl.size.file
--bigwig BIGIWG -pe --fragment-size 300 -cpu 30 --effective_genome_size 0.8
Many thanks,
Anish
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/endrebak/epic/issues/46#issuecomment-259837591, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AQ9I0sU9b3ZBhBLWtxiVgraJb80oKlz1ks5q86csgaJpZM4KvQMX
.

Hi,
We are using a genome that is understudied, hence no chromosome resolution but lots of contigs. We were just exploring whether epic could deal with this data.
I don't mind sharing the bedpe files - what would be the best way?
Best wishes,
Anish

Dropbox/google drive link? Thanks for this.
As far as I can see, the statistical method in SICER (and hence epic) should be able to deal with this situation. I have never used contigs myself, but I guess you can have many overlapping regions? Anyways, your FDRs will be more conservative, but otherwise epic should be able to find the most enriched regions and order them correctly.
You might want to run the script epic-effective on the fasta to compute the effective genome size.
Ps. the --fragment-size is not used for paired end data. I should update the help message.
Psps. it seems you forgot the two dashes in the --chromsizes flag :)

Hi Endre,
This is the link to the BedPe files - you can find input and sample files here.
<EMAIL_ADDRESS><EMAIL_ADDRESS>I will try again, accounting for the chromisize and fragment size mistakes.
Many thanks and best wishes,
Anish

On 11 Nov 2016, at 13:01, Endre Bakken Stovner<EMAIL_ADDRESS>wrote:
Dropbox/google drive link? Thanks for this.
As far as I can see, the statistical method in SICER (and hence epic) should be able to deal with this situation. I have never used contigs myself, but I guess you can have many overlapping regions? Anyways, your FDRs will be more conservative, but otherwise epic should be able to find the most enriched regions and order them correctly.
You might want to run the script epic-effective on the fasta to compute the effective genome size.
Ps. the --fragment-size is not used for paired end data. I should update the help message.
Psps. it seems you forgot the two dashes in the --chromsizes flag :)
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub https://github.com/endrebak/epic/issues/46#issuecomment-259952546, or mute the thread https://github.com/notifications/unsubscribe-auth/AVEqR56CBJXHaVHzTAbP4xjsODDi_pd6ks5q9GcrgaJpZM4KvQMX.

Thanks.

Would you mind me taking a small subset of these data and including them
in an automated test?

Do you have a file of the chromsizes for these bedpes? If not I can make
one myself where the length of the chromosome is equal to the end of the
rightmost read (this won't be much work).

Ps. this is first on my todo list after merging a PR I just got, but I
cannot guarantee that I will solve it quickly.
On Fri, Nov 11, 2016 at 7:35 PM, anishdattani<EMAIL_ADDRESS>wrote:

Hi Endre,
This is the link to the BedPe files - you can find input and sample files
here.
<EMAIL_ADDRESS><https://user:Cifperel5@zoo-
hydra.zoo.ox.ac.uk/>
I will try again, accounting for the chromisize and fragment size
mistakes.
Many thanks and best wishes,
Anish

On 11 Nov 2016, at 13:01, Endre Bakken Stovner<EMAIL_ADDRESS>wrote:
Dropbox/google drive link? Thanks for this.
As far as I can see, the statistical method in SICER (and hence epic)
should be able to deal with this situation. I have never used contigs
myself, but I guess you can have many overlapping regions? Anyways, your
FDRs will be more conservative, but otherwise epic should be able to find
the most enriched regions and order them correctly.
You might want to run the script epic-effective on the fasta to compute
the effective genome size.
Ps. the --fragment-size is not used for paired end data. I should update
the help message.
Psps. it seems you forgot the two dashes in the --chromsizes flag :)
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub <
https://github.com/endrebak/epic/issues/46#issuecomment-259952546>, or
mute the thread <https://github.com/notifications/unsubscribe-auth/
AVEqR56CBJXHaVHzTAbP4xjsODDi_pd6ks5q9GcrgaJpZM4KvQMX>.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/endrebak/epic/issues/46#issuecomment-260024154, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AQ9I0lN74s3Y46-r2ifNbsIFK-WCUKfLks5q9LVmgaJpZM4KvQMX
.

I tried this, and it worked on a small subset, so I guess it should work on the large files. Am running it right now.
Ps. epic runs stuff in parallell per chromosome, so you should use as many cores as possible.

Thanks for your time and help Andre.
One more question - when using effective genome size using epic-effective I get the following message:
Traceback (most recent call last):

File "./epic-effective", line 35, in
effective_genome_size(fasta, read_length, nb_cpu, tmpdir)
File "/home/anish/.local/lib/python2.7/site-packages/epic/scripts/effective_genome_size.py", line 22, in effective_genome_size
logging.info("Temporary directory: " + tmpdir)
TypeError: cannot concatenate 'str' and 'NoneType’ objects
input command was:
./epic-effective --read-length=300 --nb-cpu=32 /pathtofolder/genome

Best Anish

On 14 Nov 2016, at 10:58, Endre Bakken Stovner<EMAIL_ADDRESS>wrote:
I tried this, and it worked on a small subset, so I guess it should work on the large files. Am running it right now.
Ps. epic runs stuff in parallell per chromosome, so you should use as many cores as possible.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub https://github.com/endrebak/epic/issues/46#issuecomment-260306458, or mute the thread https://github.com/notifications/unsubscribe-auth/AVEqR9dvqdPPovEAPe-FiB50MRwd-ceuks5q-D64gaJpZM4KvQMX.

It worked for me too! Happy, happy. I love it when you are able to use a tool in an unexpected or unintended way.
The error happened because you had not set a temporary directory, and I had forgot to set a default. Fixed in 0.1.23 which is out on pypi now.

Also, if you are able to use this in a paper eventually, I'd be very happy to read it. I will almost certainly even cite it in my eventual paper to show what is possible with epic.

Yes of course - if I use this in an ultimate publication I will of course send it to you before hand.
Is there a way of changing the file path of Jellyfish used to calculate effective genome size - I couldn’t seem to work it out. This is what I get:
/bin/sh: 1: jellyfish: not found
/bin/sh: 1: jellyfish: not found
Traceback (most recent call last):
File "./epic-effective", line 35, in
effective_genome_size(fasta, read_length, nb_cpu, tmpdir)
File "/home/anish/.local/lib/python2.7/site-packages/epic/scripts/effective_genome_size.py", line 55, in effective_genome_size
shell=True)
File "/usr/lib/python2.7/subprocess.py", line 544, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command 'jellyfish stats /home/anish/tmp/SmedAsxl_genome_v1.1.nt.jf' returned non-zero exit status 127
rm: cannot remove `/home/anish/tmp/SmedAsxl_genome_v1.1.nt.jf': No such file or directory
Best,
A

On 15 Nov 2016, at 07:21, Endre Bakken Stovner<EMAIL_ADDRESS>wrote:
It worked for me too! Happy, happy. I love it when you are able to use a tool in an unexpected or unintended way.
The error happened because you had not set a temporary directory, and I had forgot to set a default. Fixed in 0.1.23 which is out on pypi now.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub https://github.com/endrebak/epic/issues/46#issuecomment-260566243, or mute the thread https://github.com/notifications/unsubscribe-auth/AVEqR-NrkpNggcsQr0etCv1GroSuMnj5ks5q-V2HgaJpZM4KvQMX.

It seems like you do not have jellyfish on your path. If you use the bash
shell, I think this should work:
PATH=/full_path/to/jellyfish_directory:$PATH epic-effective ...
I do not think you need to use epic-effective though. For most mammalian
genomes, the EGS is like 0.9 > with a read length of 100, with a read
length of 300 it is going to be close to 1 so you can probably just use
that. The only difference will be the n-th decimals of your FDR score.
On Tue, Nov 15, 2016 at 11:04 AM, anishdattani<EMAIL_ADDRESS>wrote:

Yes of course - if I use this in an ultimate publication I will of course
send it to you before hand.
Is there a way of changing the file path of Jellyfish used to calculate
effective genome size - I couldn’t seem to work it out. This is what I get:
/bin/sh: 1: jellyfish: not found
/bin/sh: 1: jellyfish: not found
Traceback (most recent call last):
File "./epic-effective", line 35, in
effective_genome_size(fasta, read_length, nb_cpu, tmpdir)
File "/home/anish/.local/lib/python2.7/site-packages/epic/
scripts/effective_genome_size.py", line 55, in effective_genome_size
shell=True)
File "/usr/lib/python2.7/subprocess.py", line 544, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command 'jellyfish stats
/home/anish/tmp/SmedAsxl_genome_v1.1.nt.jf' returned non-zero exit status
127
rm: cannot remove `/home/anish/tmp/SmedAsxl_genome_v1.1.nt.jf': No such
file or directory
Best,
A

On 15 Nov 2016, at 07:21, Endre Bakken Stovner<EMAIL_ADDRESS>wrote:
It worked for me too! Happy, happy. I love it when you are able to use a
tool in an unexpected or unintended way.
The error happened because you had not set a temporary directory, and I
had forgot to set a default. Fixed in 0.1.23 which is out on pypi now.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub <
https://github.com/endrebak/epic/issues/46#issuecomment-260566243>, or
mute the thread <https://github.com/notifications/unsubscribe-auth/AVEqR-
NrkpNggcsQr0etCv1GroSuMnj5ks5q-V2HgaJpZM4KvQMX>.

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/endrebak/epic/issues/46#issuecomment-260599153, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AQ9I0hSgfCyLzm4WYRI400MxBlPxdzC4ks5q-YOtgaJpZM4KvQMX
.


Add campaign WS & Modules

Questions
Answers

Description?
Add campaign WS & Modules

Type?
improvement

BC breaks?
no

Deprecations?
no

Fixed ticket?
N/A

Sponsor company
@PrestaShopCorp

How to test?
Tested by a PR : https://github.com/Progi1984/ga.tests.ui.pr/actions/runs/4875606125 for PrestaShop/PrestaShop#32431

Ping @PrestaShop/ui-tests-maintainers


Compatible with awesome-nova/dependent-filter ?

Hi,
I'm using the plugin https://github.com/awesome-nova/dependent-filter#dynamic-dependencies as I have operation, and inside operations, I have meters.
So when I select an operation, I want meter filter to refresh, filtering by operation.
 Anything I could do about it ?

I think this happens because tradionally in Laravel nova you apply filters to resource.. And from the filter you retrieve the back the instance of resource... However the nova dashboard has no resource associate with it.. So when your filter tries to find the resource it abort(404)

I just checked the code, and it results that when I hit :
/nova-api/meters ( meters being the resource), I get all the related resources.
But when I hit
/nova-api/my-dashboard I get a 404.
Solution might to implement an API for dashboards. What I don't don't know is if it is possible.
If you have any clue about this, I can check it out.


Bug in the code (Wrong dictionary)

Just a quick note. Thx a lot for sharing the starter code, it was useful.
There is a bug though in finding the users from user ids... Wrong dictionary is used...

I found this as well today.
Should be
def get_user_by_id(self, userid):
    return self.users_id_dict.get(userid)

It's worth fixing as this is by far the best starting point to making use of the flask-login plugin. The provided documentation is far too verbose, and seeing a coherent example like this goes a long way to making it possible.
Thanks for the effort


Update config names

Changed the config names for clarity

lambda_flatfield_coef -> smoothness_flatfield
lambda_darkfield_coef -> smoothness_darkfield
lambda_darkfield_sparse_coef -> sparse_cost_darkfield

Should be reviewed after #103.

Almost done! Sorry, I fixed a typo and seemingly dismissed your review.


Missing 'boot' partition, and 'recovery' partition contains 'boot' content with NOOBS 1.3.4

I've just made a few attempts at running rpi-clone. The image I was cloning is Raspian from NOOBS 1.3.4. I used -f to force a new SDCard to be partitioned. No errors are printed during the process. I am running the latest script directly from a github clone.
When done, the new SDCard has only one partition, named 'RECOVERY' (1.5G) and it contains the files from the original SDCard that were in the partition named 'BOOT'. The new card has no partition named 'BOOT'. Not surprising this new SDCard does not boot either. Anybody else seeing this? Thanks!
Original card:
'RECOVERY', FAT32, 1.45gb - files: recovery.img, ...
'BOOT', FAT32, 61.9mb - files: config.txt, ...
New card:
'RECOVERY', FAT32, 1.45gb - file: config.txt (this is wrong, these are the 'boot' dir files).
(missing 'BOOT' partition)

@bhamail @samighi @davthomaspilot rpi-clone doesn't support NOOBS, so you should expect errors. I have just released a new tool ClonePi that aims to fill the NOOBS gap...
https://github.com/SpoddyCoder/clonepi


[Feature Request] Compatibility/Usage of Dim darkness limit from the Perceptive module

The module Perceptive, among other things, allows you to set up a Bright and Dim darkness limit. It would be nice if this could be combined with some of the vision applications of this module.
At the moment if you want to hide a token in darkness the only way is to have the global illumination disabled, but then you also cant see the map, which is a pain. It would be nice that if the darkness level exceeds the dim darkness limit (but is below the Global Illumination Threshold that tokens either become invisible or see through for other tokens that lack something like darkvision. That way you can have characters use the darkness but people can still move their tokens around without being completely blind.

I'd probably need someone to point me out where in the Perceptive module these checks are done and how. Perhaps this is even something that can be handled on Perceptive's side.

Stale issue. Closing.


Flipped rendering for *-right and *-left

Semantics for right and left are not changed in bootstrap-rtl theme, and you need to add a custom .flip classes to a .pull-right and .pull-left entity if it needs to be rendered from right to left (instead of left to right). It would be convenient to create additional css file (e.g. bootstrap-flipped.css) so this flipping process could be automated without the .flip class.
However, due to the consistency of meanings of the words, flipping would never happen to the .pull-right and .pull-left in bootstrap-rtl.css file.

good idea

The idea we do not want to change our code by adding .flip class, we just want all the code to directly work for both directions and you already flipped all floats and all direction classes so just wondering why these classes. Anyway, good work.

Semantics of the words have been discussed several times here and on original bootstrap issue list. Please search next time.

Thank you for your explanation, in general there is no one solution for everyone and it is always easy to add your own custom css in a file to overwrite others.

Hey,
I have a web site that requires both support for RTL/LTR,
If possible, can anyone give some guides how to flip dynamically between the two ?
Thanks.

You can do it very easily. I do not know what is your server side language but the idea is simple. Always load bootstrap.css, if your site is in RTL mode, add bootstrap-rtl css

@afattahi54 So I guess the answer is, if rtl.css is loaded, I can't switch back to LTR, right?

Yes you can? If the page is reloaded from server you can only load bootstrap.css, if you want to switch page direction, with javascript you can remove the bootstrap-rtl.css with javascript as mentioned in http://stackoverflow.com/questions/24087152/remove-css-file-with-javascript

@asaf
this is my way.
Adding to html dir="rtl" or "ltr"  and then check by js
	if(document.getElementsByTagName('html')[0].dir != "ltr"){
		document.write('<link href="bootstrap-rtl.min.css" rel="stylesheet">');
	} else {
		document.write('<link href="bootstrap.min.css" rel="stylesheet">');
	}

@AlexYudaev
I belive the bootstrap.min.css must always be loaded. OR the bootstrap-rtl should have bootstrap.css on top of it

@afattahi54
you right. sorry, in my projects i just include bootstrap to bootstrap-rtl.

I use AngularJS, so use ng-if to pull in the relevant file:
<link rel="stylesheet" href="bower_components/bootstrap-rtl/dist/css/bootstrap-rtl.css" ng-if="getCurrentLanguage().rtl"/>

I also use an ng-class on the body to add an rtl class:
<body ng-class="{rtl:getCurrentLanguage().rtl}">

I then use css to set the text direction and override bootstrap classes rather than going around adding .flip mainually:
.rtl {
    direction: rtl;
}

.rtl .pull-right {
    float: left !important;
}

.rtl .pull-left {
    float: right !important;
}

.rtl .dropdown-menu-right {
    left:  0;
    right: auto;
}

.rtl .dropdown-menu-left {
    left:  auto;
    right: 0;
}

Thanks guys, I just wanted to ensure that while the rtl css loaded there's no way to revert back to LTR,
@willi1s Nice trick with the ng-if, Shame on Ember.js not having anything that completes with param level directives.
Thanks!

Though I agree this is dirty method, I am able to fix this issue with my asp.net MVC5 project with the help of Resource files. I added a resource setting with blank value inside default one and with value that is pointing to bootstrap-rtl.min.css. Then I added this into @Html.Raw which would print the css link tag either empty or tag.
Resource.resx
RtlStyleSheet : ""
Resource.ar.resx
RtlStyleSheet: <link rel='stylesheet' href='Content/bootstrap-rtl.min.css' />
Index.html
<head>
@Html.Raw(Resources.Resouce.RtlStyleSheet)
</head>

Hope this helps asp.net mvc guys.


Enable visualisation of local datasets by adding root arg

I am trying to convert my dataset to a LeRobotDataset and have implemented a conversion script.
I wanted to visualize the dataset locally before uploading or continuing, but the visualization script lacks a root arg to enable loading from a local dataset (used in the LeRobotDataset init to distinguish between local or remote)
I added this locally and then I was able to visualize the dataset in rerun

It would be convenient if this would be added to the main branch!
It is a very small change and I would be willing to make a PR for this.

@tlpss Sorry for the lack of documentation. You can already load a LeRobotDataset stored locally with:
export DATA_DIR="data"

See: https://github.com/huggingface/lerobot/blob/main/lerobot/common/datasets/lerobot_dataset.py#L38-L47
Happy yo add root arg in visualize_dataset.py. By any chance, would you have time to put up a PR? Thanks!

By any chance, would you have time to put up a PR? Thanks!

sure, I'll make a PR for this!


ByteString arrayOffset method

Motivation:
The ByteString class currently assumes the underlying array will be a complete representation of data. This is limiting as it does not allow a subsection of another array to be used. The forces copy operations to take place to compensate for the lack of API support.
Modifications:

add arrayOffset method to ByteString
modify all ByteString and AsciiString methods that loop over or index into the underlying array to use this offset
update all code that uses ByteString.array to ensure it accounts for the offset
add unit tests to test the implementation respects the offset

Result:
ByteString and AsciiString can represent a sub region of a byte[].

@trustin @nmittler @normanmaurer @buchgr @rkapsi - PTAL.

@buchgr - Here you go:
AsciiStringBenchmark.complexInsensativeCompare      10  thrpt   20  25881070.811 ± 143484.632  ops/s
AsciiStringBenchmark.complexInsensativeCompare     100  thrpt   20   3898435.486 ±  17824.358  ops/s
AsciiStringBenchmark.complexInsensativeCompare    1000  thrpt   20    485045.614 ±    367.934  ops/s
AsciiStringBenchmark.complexInsensativeCompare   10000  thrpt   20     48434.538 ±    574.461  ops/s
AsciiStringBenchmark.complexInsensativeCompare  100000  thrpt   20      4917.089 ±      5.499  ops/s
AsciiStringBenchmark.complexSensativeCompare        10  thrpt   20  39311729.069 ± 215287.538  ops/s
AsciiStringBenchmark.complexSensativeCompare       100  thrpt   20  11358090.386 ±  16428.000  ops/s
AsciiStringBenchmark.complexSensativeCompare      1000  thrpt   20   1305740.981 ±   2166.293  ops/s
AsciiStringBenchmark.complexSensativeCompare     10000  thrpt   20    141399.231 ±    176.152  ops/s
AsciiStringBenchmark.complexSensativeCompare    100000  thrpt   20     14120.667 ±      8.289  ops/s
AsciiStringBenchmark.simpleInsensativeCompare       10  thrpt   20  21860398.628 ± 275208.059  ops/s
AsciiStringBenchmark.simpleInsensativeCompare      100  thrpt   20   3055570.415 ±  53073.958  ops/s
AsciiStringBenchmark.simpleInsensativeCompare     1000  thrpt   20    318989.668 ±   9318.473  ops/s
AsciiStringBenchmark.simpleInsensativeCompare    10000  thrpt   20     22072.309 ±    102.826  ops/s
AsciiStringBenchmark.simpleInsensativeCompare   100000  thrpt   20      1314.410 ±      4.863  ops/s
AsciiStringBenchmark.simpleSensativeCompare         10  thrpt   20  38683486.474 ± 176874.275  ops/s
AsciiStringBenchmark.simpleSensativeCompare        100  thrpt   20  10618091.726 ±  31343.587  ops/s
AsciiStringBenchmark.simpleSensativeCompare       1000  thrpt   20   1367593.426 ±   1637.948  ops/s
AsciiStringBenchmark.simpleSensativeCompare      10000  thrpt   20    143448.066 ±    210.082  ops/s
AsciiStringBenchmark.simpleSensativeCompare     100000  thrpt   20     14408.861 ±     12.843  ops/s

@Scottmitch
thanks! So I think it's fair to say that sizes bigger than 100 bytes we don't care about for AsciiString. Even 100 bytes is a bit much, but that's okay. (We mostly care about strings / headers in the 5 - 50 byte range). So for 10 bytes the case sensitive compare is 2% slower, the case insensitive 16%. For 100 bytes the case sensitive is 7% slower and the case insensitive 22%. Honestly, I would still go with the simple compare. That's a microbench, not a real application.
So my suggestion (as a compromise) in the other PR was to have one version for CharSequence and one version for AsciiString. Having one for all the possible combinations of the two seems a bit much.

@netkins build

@normanmaurer @trustin @Scottmitch Before Java 7 j.l.String used to have this feature, but they removed it, afaik one reason being that it can lead to memory leaks (small substring referencing part of a big string). Just an FYI :-).

@buchgr - This was collapsed so I'll pull it out https://github.com/netty/netty/pull/3649#discussion_r28633871:

@Scottmitch

Oh, so now you changed the hash function :-). How can you proof that this new hash function produces a better hash code or at least isn't likely to have more collisions? It might evaluate faster, but it might also produce a worse hash code. That's why I was hesitant to change the hash function with my equals performance update.

Looking at j.l.String and Hotspot, they are calculating the hash code character by character and it's not intrinsified by HotSpot. I think that's also what we should do. The hashcode is being cached after all and as I said in another comment, I haven't seen AsciiString.hashCode() show up in any profiler while working on the headers.

Also how did you test the big endian version?

@buchgr

How can you proof that this new hash function produces a better hash code or at least isn't likely to have more collisions?

This is an excellent point. My initial focus was just to to take the existing AsciiString's hash code algorithm's XOR and and operations and extend that into the long/int/char realms....no mathematical proof was done but I started empirically testing.  I introduced a simple example to be able to distribute a work load of random byte hashing and collision detection and the results looked promising, but at the end of the day I didn't have any strong theoretical backing and just ended up porting Murmur3.  This hash algorithm is more widely used and understood to have a favorable performance / collision ratio.  I did some benchmarking below and seems to be about a 2x boost over Arrays.hashCode(..).
Benchmark                                       (size)   Mode  Cnt         Score        Error  Units
PlatformDependentBenchmark.arraysBytesHashCode     100  thrpt   20  12680337.891 ± 369222.871  ops/s
PlatformDependentBenchmark.arraysBytesHashCode  100000  thrpt   20     12331.054 ±    300.674  ops/s
PlatformDependentBenchmark.unsafeBytesHashCode     100  thrpt   20  21647401.177 ± 845628.253  ops/s
PlatformDependentBenchmark.unsafeBytesHashCode  100000  thrpt   20     23767.495 ±    819.359  ops/s

I haven't seen AsciiString.hashCode() show up in any profiler while working on the headers.

Despite this it may be a good idea to unify the hash codes in Netty to use more of a standard approach.  For example ByteBufUtil implements a custom hash code algorithm, and I'm not sure of its origins.  We would need to do evaluation but perhaps we could adopt Murmur3.

Also how did you test the big endian version?

Big endian has not yet been tested and needs to be done.  Any suggestions for setting up an emulation environment?

@normanmaurer @buchgr @nmittler @trustin - PTAL.  I dumped some more code here related to the hash code algorithm.  This also includes a new example....I can pull stuff out of this PR and open a new one if necessary.

@Scottmitch LGTM

@Scottmitch
Seems like you were busy this weekend! Looks awesome! :-) Will review later today. Just three things immediately popping in my head

Why don't you just fork an agreed-to-be-correct-and-fast Java implementation of murmur?
You could have just used something like https://code.google.com/p/smhasher/ to evaluate your "unsafe hash function", maybe it was fine ...
What I tried to say in my previous comment is that we probably don't have to worry about having the same hash function for little and big endian systems. We could just fall back to the old "byte-by-byte" hash function for big endian, that way we at least can test it and know it performs reasonably well.

@buchgr - Good questions :)

Murmur3_32HashFunctio from guava works with CharSequence and their known output values test which I borrowed from produced different results than the Murmur 3 C++ Reference Implementation when I built/ran it.  None of the existing implementations I looked at accounted for endianness.
Didn't know about this project....good to know.  Could have been fine...but why invent another hash algorithm when there are already ones that will do what we need.
So I still need to test but the endianness code however the code to account for endianness isn't too pervasive and what is there should be easy for JIT to optimize (assumption based on the fact that the if branches will always evaluate the same way).  That being said haven't tested it...I'm tracking down a test environment....

So I still need to test but the endianness code isn't too pervasive and what is there should be easy for JIT to optimize (assumption based on the fact that the if branches will always evaluate the same way). That being said haven't tested it...I'm tracking down a test environment....

I am more worried about the code complexity and code size. Personally, I would only add this hash function for unsafe and little endian and if one of them is not available, go with the byte by byte hash code.

@buchgr @nmittler - I have pushed the code which helps give some background for our discussion tomorrow.  The challenge was to introduce a consistent hash algorithm for case sensitive (just the byte value itself) and also case insensitive (requires translation).  These use cases we have also requires support for different data types (CharSequence, and ByteString/AsciiString).  The code is not complete (documentation needed, unit tests needed, etc...) but as I said it is a good place to start discussions for how this overlaps with (if at all) the headers updates @buchgr is investigating.
Just an FYI I still need to investigate the previous implementation of caseInsensitiveHashCode for a few reasons... 1. It doesn't look like there is any effort to make the hash "insensitive"...it looks the same as the regular hashCode(). 2. The cached value is being used if the value is AsciiString...which is the case "sensitive" value...unless everything was being translated to lowercase I'm missing something.

@trustin @normanmaurer @nmittler @buchgr - PTAL.  I extracted out the example from this PR and will submit that as another PR.
@buchgr - I think I would like to keep the comparators as they are in this PR.  After you header improvements I may take a crack at removing them completely...or at least moving them to a Util type class and not using them for Headers.

@Scottmitch
I still don't see why we need a big endian and "unsafe is not supported" version. This complicates everything and is effectively dead code. Falling back to the current hash function would work just fine. I even don't understand why we need a new hash function ...

@buchgr - I'm going to move the hash code stuff to a new PR to discuss separately....seemed to have support from a few folks but easier to discuss in isolation.
Note that case insensitive hash code is broken though...have that fixed in here...and will just submit the hash code stuff I have as a fix for that also.

@buchgr @nmittler - I pulled the hash code changes out of this PR.  The scope should now be limited to the original intention of this PR.  PTAL.

@Scottmitch LGTM, just one question about a benchmark that @buchgr was asking for.

@nmittler - Can you provide a link for the question?  I don't see it.

Sure https://github.com/netty/netty/pull/3649/files#r28617443

@nmittler yep he did https://github.com/netty/netty/pull/3649#issuecomment-94375705

@nmittler yep he did https://github.com/netty/netty/pull/3649#issuecomment-94077741

@buchgr ok great!
@Scottmitch I think you can pull the trigger on this.

@normanmaurer @trustin - Good to cherry pick?

@Scottmitch ship it :beers:

Thanks for review all!
Cherry-picked 4.1( f812180) master (f1e122a)


HOP-3801, HOP-3807 Hop-Translator crashes when exist conflicting bundle

Always allow users to enter the i18n editor without conflicting bundle.

Auto merge another commit when exist a PR?

ah, no that does not work it only links the first HOP-xxxx ticket it finds which is a shame. It does not even look at the commits only at the PR title


Breaking change: New default TLS CipherSuites for .NET on Linux

New default TLS CipherSuites for .NET on Linux
.NET, on Linux, now respects the OpenSSL configuration for default ciphersuites when doing TLS/SSL via the SslStream class or higher level operations like HTTPS via the HttpClient class. When there is not an explicitly configured default .NET on Linux uses a very restricted list of permitted ciphersuites.
Version introduced
.NET 5.0 RC1
Old behavior
.NET did not respect system configuration for default ciphersuites. The default ciphersuite list for .NET on Linux was very permissive.
New behavior
.NET on Linux respects the OpenSSL configuration for default ciphersuites when it is specified in openssl.cnf. When ciphersuites are not explicitly configured the only permitted ciphersuites are the TLS 1.3 ciphersuites and  TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,  TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, and TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256. Since this fallback default does not include any ciphersuites compatible with TLS 1.0 or TLS 1.1, these older protocol versions are effectively disabled by default.
Supplying a CipherSuitePolicy value to SslStream for a specific session will still replace the configuration file content and/or .NET fallback default.
Reason for change
Users running .NET on Linux requested that the default configuration for SslStream be changed to one that provided a high security rating from third party assessment tools.
Recommended action
The new defaults are likely to work when communicating with modern clients or to modern servers.  If you need to expand the default ciphersuite list to accept legacy clients (or contact legacy servers), either specify a CipherSuitePolicy value or change the OpenSSL configuration file.  On many Linux distributions the OpenSSL configuration file is at /etc/ssl/openssl.cnf.
This sample openssl.cnf is a minimal file that is equivalent to the .NET 5.0 on Linux default ciphersuites policy. Rather than replacing the system file, you need to merge these concepts with the file present on your system.
openssl_conf = default_conf

[default_conf]
ssl_conf = ssl_sect

[ssl_sect]
system_default = system_default_sect

[system_default_sect]
CipherString = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256

On the Red Hat Enterprise Linux, CentOS, and Fedora distributions .NET applications default to the ciphersuites permitted by the system-wide cryptographic policies. On these distributions use the crypto-policies configuration instead of openssl.cnf.
Category

[ ] ASP.NET Core
[ ] C#
[ ] Code analysis
[ ] Core .NET libraries
[ ] Cryptography
[ ] Data
[ ] Debugger
[ ] Deployment
[ ] Globalization
[ ] Interop
[ ] JIT
[ ] LINQ
[ ] Managed Extensibility Framework (MEF)
[ ] MSBuild
[X] Networking
[ ] Printing
[X] Security
[ ] Serialization
[ ] Visual Basic
[ ] Windows Forms
[ ] Windows Presentation Foundation (WPF)
[ ] XML, XSLT

Affected APIs
Not detectable via API analysis

Issue metadata

Issue type: breaking-change

cc: @wfurt @tmds

LGTM. should we mention this is applicable only to Linux with OpenSSL 1.1.x and old distributions are not affected?

cc @omajid