Allow transpilation options to be provided to Qiskit devices

The Qiskit transpile function accepts a wide range of options: https://qiskit.org/documentation/stubs/qiskit.compiler.transpile.html
We should allow keyword arguments to devices to be passed during transpilation.
This could be done by inspecting qiskit.compiler.transpile, and passing any keyword arguments that occur in the signature.

Hi @josh146 I am interested in working on this one. Can I give this one a try?
I have worked on qiskit transpiler code earlier.

Hi @sagarpahwa, this would be a great first issue for a PR 🙂

Thank you @josh146, I will start working on it.

@sagarpahwa, feel free to make a PR to this repo titled [WIP] Pass transpilation options. If you run into any issues, feel free to ask

@sagarpahwa great! Looking forward to your contribution :smiley:

Hi @josh146 , @co9olguy
I tried to understand the code and the required changes. PFB the understanding.
File to change: qiskit_device.py
location:
compiled_circuits = transpile(self._circuit, backend=compile_backend)
memory = str(compile_backend) not in self._state_backends
return assemble(
    experiments=compiled_circuits,
    backend=compile_backend,
    shots=self.shots,
    memory=memory,
)

parameters to pass:
transpile_params = [
basis_gates,
coupling_map,
backend_properties,
initial_layout,
layout_method,
routing_method,
translation_method,
seed_transpiler,
optimization_level,
pass_manager,
callback,
output_name
]

assemble_params = [
qobj_id,
qobj_header,
max_credits,
seed_simulator,
qubit_lo_freq,
meas_lo_freq,
qubit_lo_range,
meas_lo_range,
schedule_los,
meas_level,
meas_return,
meas_map,
memory_slot_size,
rep_time,
rep_delay,
parameter_binds,
parametric_pulses,
init_qubits,
**run_config
]

If my understanding is correct, then do we need to pass the additional parameters to both compile and assemble method?
Also, these will be at root level of the **kwargs coming to apply method, or we need to nest them into separate sub-dictionaries?

Hi @sagarpahwa :slightly_smiling_face:

If my understanding is correct, then do we need to pass the additional parameters to both compile and assemble method?

There might be distinct options for the transpile and the assemble functions. A first step could be wiring in options for the transpile function, as per the original idea. That solution could then be extended in a similar fashion for assemble. If there are keyword arguments that are accepted by both functions, I'd think it would be fine for now to pass the same value in both cases. :slightly_smiling_face:

Also, these will be at root level of the **kwargs coming to apply method, or we need to nest them into separate sub-dictionaries?

One approach here could be to create a transpile_options dictionary for the device. Valid options for transpile that are passed as keyword arguments when creating the QiskitDevice could then be added into this dictionary.
As @josh146 mentioned, using inspect to get the function arguments could lead to a nice solution. This would allow dynamically determining which are valid arguments (without having to keep track of them in a list) and could allow that no breaking changes are introduced to PennyLane-Qiskit even when the signature of transpile changes.
This would include obtaining the signature of transpile (and including the result in the logic):
from qiskit.compiler import transpile
import inspect

res = inspect.signature(transpile)

A similar inspection is used for the backend.run method.

Thanks @antalszava for helping out.
I went back and checked the way we are passing the parameters to the run method.

 s = inspect.signature(b.run)
 self.run_args = {}

 if "noise_model" in kwargs:
     if "noise_model" in s.parameters:
         self.run_args["noise_model"] = kwargs.pop("noise_model")

 if "backend_options" in s.parameters:
         self.run_args["backend_options"] = kwargs

With this, for qasm_simulator, a user is never able to pass this argument

validate (bool) – validate the Qobj before running (default: True).

in general,
- user won't be able to pass any argument except noise_model, qobj, backend_options
- all other args will be assed as part of backend_options
'backend_options': {'test_param': 'dummy_param', 'optimization_level': 3, 'validate': False, 'verbose': False}
My understanding is at stage 3 the code should be:
    if "backend_options" in s.parameters:
            self.run_args["backend_options"] = kwargs.pop("backend_options")

or even more, I think, we can do it in one go:

create argument setter utility function

    def set_args(source, signature, destination):
        for arg in signature:
            if arg in source:
                # don't pop args, as it might be common to run and compile
                self.destination[arg] = source[arg]

during init

    run_sig = inspect.signature(b.run)
    self.run_args = {}
    set_args(kwargs, run_sig, self.run_args)
    <!-- above method call is same as
    for arg in run_sig:
        if arg in kwargs:
            # don't pop args, as it might be common to run and compile
            self.run_args[arg] = kwargs[arg] -->

    # if user passes transpile options at time of init
    transpile_sig = inspect.signature(transpile)
    self.transpile_args = {}
    set_args(kwargs, transpile_args, self.transpile_args)
    <!-- above method call is same as
    for arg in transpile_sig:
        if arg in kwargs:
            # can pop args, but leaving it as is, as it might be common to compile and assemble (future implementation)
            self.transpile_args[arg] = kwargs[arg] -->

during apply

    transpile_sig = inspect.signature(transpile)
    # no need to initialize transpile_args, only updte them if required
    set_args(kwargs, transpile_args, self.transpile_args)

As, this will be my first contribution to pennylane, let me know if I am missing out something important, or we should do it step by step (same as transpile and assemble changes)

Hi @sagarpahwa,
The approach you've outlined looks good overall! The parsing and storing of transpile/assemble/run arguments can be further extended, as you've pointed out.

 if "backend_options" in s.parameters:
     self.run_args["backend_options"] = kwargs

The logic here assumes that everything that was further left in kwargs was meant to be passed backend_options.
As @josh146 mentioned, feel free to reach out if you'd have further questions or feel it would be good for us to go through the specific solution.

Thanks @antalszava for the review. I will do the rest of the changes.
@josh146 , It seems to me that run method argument passing needs a little amendment as well. but as discussed earlier I am making changes here only for transpiration options, assuming that we will track other changes (assemble and run) via separate threads. If everything here goes fine, would like to work upon the rest of the tasks.


1--Setup

Added:

flaskr: application folder
tests: tests folder
requirements.txt: requirements file to install packages in project directory
setup.py: pip setup file for installing StockMeetsBagel
MANIFEST.in: file to specify what to copy and exclude when installing

Closing in favor of Jenny's skeleton.

Closing in favor of Jenny's skeleton.


Need more clarification about overwriting XFF

Hi,
This document mentions "Both HTTP and HTTPS protocols (with TLS inspection) are always filled by Azure Firewall with XFF (X-Forwarded-For) header equal to the original source IP address. ".
But I guess that customers can't imagine that Azure Firewall overwrites XFF by reading this sentence. I believe that we can use a simple sentence to explain this behavior. For example, Azure Firewall overwrites XFF (X-Forwarded-For).
Best regards.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 6392b4a0-fa1e-fe5e-a31b-1eee48b860ef
Version Independent ID: 551028da-b651-30ac-0e77-9ee5eebf6bb0
Content: Azure Firewall rule processing logic
Content Source: articles/firewall/rule-processing.md
Service: firewall
GitHub Login: @vhorne
Microsoft Alias: victorh

@kongou-ae , thank you for your feedback. We have assigned this to the content owner/author for review.
@vhorne , for your review.

Thank you for your dedication to our documentation.
Unfortunately, we have been unable to review this issue in a timely manner. We sincerely apologize for the delayed response. We are closing this issue. If you feel that the problem persists, please respond to this issue with additional information.
Please continue to provide feedback about the documentation. We appreciate your contributions to our community.
#please-close


⚠ Bofrost.lu has degraded performance

In f92006b, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 2235 ms

Resolved: Bofrost.lu performance has improved in d2c4957 after 16 minutes.


fix(network): Try reparsing the socket address

Pull Request Template
Requirements
ref #250
When parsing the socket address, it will retry if it fails.
Note: There is still a problem updating the network configuration
@jiangxianliang007 Please test this branch on the k8s cluster.

PTAL @kaikai1024 @leeyr338

Is it normal for ci to fail?

Is it normal for ci to fail?

No. Check format.

Checking clippy failed.

error[E0308]: mismatched types
--> cita-network/src/connection.rs:226:41

Have you built the project before pulling request? This is not a lint error!
CI is really important for this situation.

Is it normal for ci to fail?

Have you checked the CI logs before you asked?
It is OK if you just confused with the CI logs, ask and someone will answer it, but, if you just are lazy ...

Have you checked the CI logs before you asked?
It is OK if you just confused with the CI logs, ask and someone will answer it, but, if you just are lazy ...

I checked the previous pr and found that many ci failed, so I raised this question.

Have you checked the CI logs before you asked?
It is OK if you just confused with the CI logs, ask and someone will answer it, but, if you just are lazy ...

I checked the previous PR and found that many ci failed, so I raised this question.

Because the circleCI was not ready that time.

Tested through the K8S cluster

Tested through the K8S cluster

We'll no release a patch version, That's right?


preserve location.key only if it was present in the original input object

Don't add key: undefined to the location if the input does not have a location key.
See https://github.com/ReactTraining/react-router/commit/90d4832b57ce5a4f7e0c128923d93108173c6b61, https://github.com/ReactTraining/react-router/issues/3956

What's keeping this from being merged? It blocks #4107 and #4106. Can I do something to help?

@ryanflorence needs to take a look at it. He and I are both terribly busy, so our bandwidth is limited. It's a people problem, not a code problem :)

This PR is outdated now. Thanks for the contribution, @jochenberger. It was just a little too early :)


Are you gonna add cases soon?

On the commits page I saw that you had added case keys, so I automatically assume that this is going to happen. Thanks in advance Daniel.
(WILL CLOSE ISSUE WHEN SAID COMMIT BE PUBLISHED)

I hope so.


E220 Keyword not capitalized - False flag for PtrSafe?

I tried to run the dev version of the linter on the dev branch of one of my repos here.
It looks like the E220 falsly flags the keyword PtrSafe as not being correctly capitalized in this line of this file (10,21):
    Private Declare PtrSafe Function SystemParametersInfo Lib "user32" _

Fixed in dev. Thanks for testing and for the feedback!

Great, thanks! I was thinking of making another issue for that, but you're too quick 😅.

LongPtr (and potentially LongLong) seems to be causing some issues as well.

Thanks. Fixed


Changed the Tokenizer to only increment position iff a character was read

Some Tokenizer flows call next() after EOF has already been reached, which leaves position in the wrong place. This fixes that, and fixes a few of the test cases.

Travis seems to have trouble. I'm not able to restart any builds. I'll try again later.


fix BDouble init?(_ nStr: String) when String has scientific notation

Hi!
I think there is bug when init BDouble when String has scientific notation
So i fixd some logic and added tests
My changes passed all the Unit tests
Bug Example
       // MARK: Fail
        let a = BDouble("+2.46e+3")
        let b = BDouble(2460)
        XCTAssertEqual(a, b)
        XCTAssertEqual(a?.decimalDescription, b.decimalDescription)

        let c = BDouble("-2.46e+1")
        let d = BDouble(-24.6)
        XCTAssertEqual(c, d)

        XCTAssertEqual(c?.decimalDescription, d.decimalDescription)

        // MARK: Success
        let e = BDouble("-2.46e-2")
        let f = BDouble(-0.0246)
        XCTAssertEqual(e, f)
        XCTAssertEqual(e?.decimalDescription, f.decimalDescription)

@piggy-park Thanks for your contribution :)


Fix type defitions for Spec#returns

Reading through the sources it seems to me that Spec#returns(...) will
either return something directly or (if the response is not yet there)
it will continue building the spec. This is not shown in the type
definitions.
In this way you can use it like this if you already have a response.
const x  = spec.returns<number>('res.body') // number

But will continue the spec if you don't give a type argument:
const x = spec.returns('res.body') // Spec

Thanks for the approval! Any idea when this could get merged and released by any chance? It would help me a lot :-)

Thanks for the approval! Any idea when this could get merged and released by any chance? It would help me a lot :-)

Will release it by this weekend.


HDDS-1849. Implement S3 Complete MPU request to use Cache and DoubleBuffer.

This is dependent on HDDS-1856.
To get CI run, posted HDDS-1856 changes + HDDS-1849  in this PR.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
42
Docker mode activated.

_ Prechecks _

+1
dupname
2
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 27 new or modified test files.

_ trunk Compile Tests _

+1
mvninstall
589
trunk passed

+1
compile
363
trunk passed

+1
checkstyle
73
trunk passed

+1
mvnsite
0
trunk passed

+1
shadedclient
864
branch has no errors when building and testing our client artifacts.

+1
javadoc
164
trunk passed

0
spotbugs
420
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
607
trunk passed

_ Patch Compile Tests _

+1
mvninstall
579
the patch passed

+1
compile
375
the patch passed

+1
javac
375
the patch passed

+1
checkstyle
66
the patch passed

+1
mvnsite
0
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
655
patch has no errors when building and testing our client artifacts.

+1
javadoc
153
the patch passed

+1
findbugs
619
the patch passed

_ Other Tests _

-1
unit
302
hadoop-hdds in the patch failed.

-1
unit
1728
hadoop-ozone in the patch failed.

+1
asflicense
37
The patch does not generate ASF License warnings.

7400

Reason
Tests

Failed junit tests
hadoop.ozone.client.rpc.TestOzoneAtRestEncryption

hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures

hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis

hadoop.ozone.client.rpc.TestOzoneRpcClient

hadoop.ozone.om.TestScmSafeMode

hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures

hadoop.ozone.client.rpc.TestSecureOzoneRpcClient

hadoop.ozone.client.rpc.TestWatchForCommit

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1181/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1181

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux c62a4a1946ea 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 8053085

Default Java
1.8.0_212

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1181/1/artifact/out/patch-unit-hadoop-hdds.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1181/1/artifact/out/patch-unit-hadoop-ozone.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1181/1/testReport/

Max. process+thread count
4500 (vs. ulimit of 5500)

modules
C: hadoop-ozone/ozone-manager U: hadoop-ozone/ozone-manager

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1181/1/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
42
Docker mode activated.

_ Prechecks _

+1
dupname
2
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 27 new or modified test files.

_ trunk Compile Tests _

+1
mvninstall
599
trunk passed

+1
compile
353
trunk passed

+1
checkstyle
64
trunk passed

+1
mvnsite
0
trunk passed

+1
shadedclient
795
branch has no errors when building and testing our client artifacts.

+1
javadoc
147
trunk passed

0
spotbugs
414
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
603
trunk passed

_ Patch Compile Tests _

+1
mvninstall
527
the patch passed

+1
compile
352
the patch passed

+1
javac
352
the patch passed

+1
checkstyle
70
the patch passed

+1
mvnsite
0
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
619
patch has no errors when building and testing our client artifacts.

+1
javadoc
150
the patch passed

+1
findbugs
616
the patch passed

_ Other Tests _

-1
unit
294
hadoop-hdds in the patch failed.

-1
unit
2977
hadoop-ozone in the patch failed.

+1
asflicense
41
The patch does not generate ASF License warnings.

8406

Reason
Tests

Failed junit tests
hadoop.ozone.client.rpc.TestOzoneAtRestEncryption

hadoop.ozone.client.rpc.Test2WayCommitInRatis

hadoop.ozone.om.TestScmSafeMode

hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1181/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1181

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux cc0c9a4a0c41 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 8053085

Default Java
1.8.0_212

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1181/2/artifact/out/patch-unit-hadoop-hdds.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1181/2/artifact/out/patch-unit-hadoop-ozone.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1181/2/testReport/

Max. process+thread count
3725 (vs. ulimit of 5500)

modules
C: hadoop-ozone/ozone-manager U: hadoop-ozone/ozone-manager

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1181/2/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

/retest

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
121
Docker mode activated.

_ Prechecks _

+1
dupname
0
No case conflicting files found.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 3 new or modified test files.

_ trunk Compile Tests _

+1
mvninstall
873
trunk passed

+1
compile
400
trunk passed

+1
checkstyle
73
trunk passed

+1
mvnsite
0
trunk passed

+1
shadedclient
918
branch has no errors when building and testing our client artifacts.

+1
javadoc
197
trunk passed

0
spotbugs
521
Used deprecated FindBugs config; considering switching to SpotBugs.

+1
findbugs
774
trunk passed

_ Patch Compile Tests _

+1
mvninstall
671
the patch passed

+1
compile
417
the patch passed

+1
javac
417
the patch passed

+1
checkstyle
86
the patch passed

+1
mvnsite
0
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedclient
717
patch has no errors when building and testing our client artifacts.

+1
javadoc
171
the patch passed

+1
findbugs
723
the patch passed

_ Other Tests _

-1
unit
346
hadoop-hdds in the patch failed.

-1
unit
278
hadoop-ozone in the patch failed.

+1
asflicense
39
The patch does not generate ASF License warnings.

6984

Reason
Tests

Failed junit tests
hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse

Subsystem
Report/Notes

Docker
Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1181/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/1181

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux 8a0fd756d256 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / e5c4131

Default Java
1.8.0_212

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1181/3/artifact/out/patch-unit-hadoop-hdds.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-1181/3/artifact/out/patch-unit-hadoop-ozone.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-1181/3/testReport/

Max. process+thread count
514 (vs. ulimit of 5500)

modules
C: hadoop-ozone/ozone-manager U: hadoop-ozone/ozone-manager

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-1181/3/console

versions
git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1

Powered by
Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

TestOzoneManagerDoubleBufferWithOMResponse is passing locally. Other test failures are not related to this patch.
Thank You @arp7 for the review.
I will commit this to the trunk.


Revamp view behaviour

Current view behaviour when viewing vendor/event would display:

vendor/event filtered list on the right,
event/vendor associated to in body

This makes it hard to assign new events to vendor and vice versa since there's no list to refer to.
Proposed Change:

should be event/vendor filtered list on the right instead
unassigning events associated listed in nested list should work (maybe by tracking their index one to one as compared to the left list?)

This should be one of the biggest feature change for 1.4


WCF Test Client does not support TLS 1.2

I am unable to use the WCF Test Client after disabling TLS 1.0 and lower:
An error occurred while making the HTTP request to URL. This could be due to the fact that the server certificate is not configured properly with HTTP.SYS in the HTTPS case.
Tested against version 15.0.27825.0

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 599983a3-8520-1f76-61c6-4b77ef804492
Version Independent ID: 37629297-b13c-bd53-aeac-5646c6bdc06f
Content: WCF Test Client (WcfTestClient.exe)
Content Source: docs/framework/wcf/wcf-test-client-wcftestclient-exe.md
Product: dotnet-framework
GitHub Login: @dotnet-bot
Microsoft Alias: dotnetcontent

@zhenlan can you or someone on your team help with this?

@mconnew

Is there any update on this?


Change assert() to CHECK [phab:D695]

Created by zarko at 2016-02-05 20:48:57:
assert() doesn't always run but CHECK does. Since our assertions are generally cheap, let's ensure that they run regardless of build configuration.

shahms accepted this revision at 2016-02-05 20:52:54

jdennett wrote at 2016-02-05 20:53:59:
Could the change description mention why this is being done?

zarko closed via commit b8e958cd09290b923343083f0f66a099414ab78b at 2016-02-05 20:55:02


Resolver-side shading

This is a proposal for a large feature that I think the JVM ecosystem needs, and coursier seems like a promising place to do it.
The idea is basically to resolve conflicting transitive-dependencies by creating shaded copies of each version of a conflicted library, and rewriting class-files in [other dependencies that depend on the conflicted one] to point at the shaded version they wanted.
This is different from normal shading workflows because it happens at resolution time; today, we generally hope that upstream libraries shade commonly-problematic deps (e.g. guava) before publishing, but that's usually not done, and anyway is the wrong place to solve the problem: the right set of things to shade from a given library depends on what else is on the classpath, which is different in every downstream library that uses it.
Here are relevant illustrations from an unconference presentation I gave about this idea at NEScala 2017:
"Before" / Today:

"After" / Future:

If this makes sense to ppl then I'd like to help build this here, after dreaming about it for a long time. I have a lot of thoughts about how to do it that we can discuss.
Also, if there is prior art I may have missed, pointers are welcome!
To pre-empt one possible comment: a lot of people I've talked to have said that they thought JDK9 modules solve this (and even that the point of JDK9 modules was to solve this!), but my impression is that they don't; quoting the spec:

It is not a goal of the module system to solve the version-selection problem, which is best left to build tools and container applications.

Looking forward to thoughts here (and also hoping to discuss/work on it with folks at Scala Spree NYC tomorrow 😀)

Thanks @stuhood, that all makes sense.

While I would of course love for someone to implement it in a build system ("our" build system) rather than in a dependency resolver (which should generally do one thing, well), solving it in coursier seems like it would more efficiently help the ecosystem-in-general.

I hear that; to me, this is part of "doing resolving well" that, coincidentally, no resolvers do today 😄

For bonus points: to avoid creating "too many" clones of libraries, you might consider hashing/comparing the public APIs of different versions of libraries to determine whether they are API compatible before cloning them. So for example: if v1 and v2 are API compatible, you'd choose one of them rather than cloning/shading. There is prior art for this in rust's cargo.

gtk about this in rust, and yea there are a lot of interesting possible optimizations like this.
FWIW, in the specific case you've cited, my default position is to not take that shortcut, bc it still results in some code being run against code that it wasn't built against. v1 and v2 being API-compatible doesn't mean they do the same thing (indeed, presumably they do something different, otherwise why are they two different versions?), and I am approaching this problem with a conservative posture about preserving "correctness" in the sense of "not running code against dependencies other than those which it was built/tested against".
Fingerprinting all the way down to individual methods, and the transitive-closures of other classes and methods they reference, to identify cases where the byte-code is identical, and doing deduplication like you've described, is definitely "in scope" for some (later) version of this effort, I think.

In which module(s) of coursier would you see this added?
IMO, it can be an extra feature of the sbt-shading plugin (which would shade the B and C dependencies, while at the same time shading in them the v1 and v2 of D), and / or of the fetch command in the CLI (which would generate some extra JARs with shading / renaming applied, and print those instead of the downloaded JARs).
I guess the latter case would be more straightforward to implement. But sbt users would only benefit from it via the sbt plugin…

yea @alexarchambault I'm imagining this being somewhere more "core" to coursier than the sbt-plugin; accessing it via the CLI, and across any build tools that use coursier (e.g. pants) is a goal!
The basic interface I'm imagining is:

input: list of Maven coords

typically: the direct dependencies of a library being compiled/run (regardless which build tool is being used)
letting a user feed dependencies in from the coursier CLI could also be useful

output: a "safe" classpath containing those dependencies and their trans-deps

in the common case this would be a list of local-filesystem paths, like SBT's fullClasspath today
some of these would be unmodified .jars under ~/.{ivy2,m2,coursier}, also like today
others would be rewritten JARs under target; they are specific to a project and are outputs of build tasks

(this framing came out of discussions with @jvican and @smarter at the Spree on Tuesday)
Notes on publishing
Here's a bunch of thoughts about different cases we'd run in to when attempting to synthesize "safe classpaths" for libraries and publish them for downstream consumption.
Easy (+ common?) case: conflicting private dependencies
When a conflicted library (D in the images above) is a "private" dependency (not exposed in the public API) of other dependencies (B, C above) of a library we are developing (A), we can publish A as usual, and anything downstream can choose to use resolver-side shading to deal with [the {B,C}→D conflict they now inherit from A] or just take their chances with "legacy" conflict-resolution 😀
Backwards-compatibility with existing conflict-resolution strategies comes for free here.
Note also that a downstream user of A may have other dependencies that bring in Dv3 and Dv4, or other paths to Dv1/Dv2, and that's fine, they can build their own safe classpath, and are not required to rewrite Dv1 and Dv2 references inside B and C in the exact same way as we did when building/testing A, since the published A artifacts contain no direct references to D, renamed or otherwise.
Harder case: conflicting public dependencies
In other cases, A will end up with references – in its byte-code – to D1 and/or D2, in which case downstream libraries need to know that the rewriting happened, and how.

Here, the conflicted class (Opt), is exposed in the public APIs of B and C (resemblance to Spark's past struggles with guava's Optional being exposed in its public API is not entirely coincidental 😀) and, importantly, A calls methods .map on Dv1 and Dv2, so its published bytecode will contain references to those synthetic classes/methods.
In this case, I presume we will stuff some metadata into suitable fields of the published POM / ivy.xml, and downstream libraries will have to provide A with similarly-renamed Dv1 and Dv2 on the classpath.
Additionally, two .jars could be published for A (one with a classifier like -shaded or -safe, though the former is already used in the wild to mean something different; example), so that downstream libraries can opt in or out of needing to abide by these rules, vs deal with the conflicts on D in the old way.
Importantly, we can save on this trouble if A only actually uses ≤1 version of D directly: if B and C publicly expose Dv1 and Dv2 classes (resp.), but A only ever directly uses Dv1 via B, then Dv1 (and B) can be left un-rewritten, Dv2 alone can be rewritten, A will be published with references to plain D (and should probably include D (v1) as a direct dependency), and we can fall-back to the easy / backwards-compatible case above.
Impossible case → compile error
If A actually tries to pass something from Dv1 to something that expects a Dv2, naively rewriting things as above will give a compile-time error, which seems basically ideal:

Additional/Clarifying functionality: support directly depending on >1 version of same library
To resolve the compiler error above, we want to let A convert between a Dv1 Opt and a Dv2 Opt, however it's generally impossible to write such a function today because it requires depending on and using two classes with the same FQN from two different versions of the same "library" (Maven {group,artifact} pair).
Allowing direct dependencies on multiple versions of the same library naturally falls out of a lot of what's discussed above, and is in itself a powerful feature that naturally would have a home in a tool like this.

Closing this for now, as it seems no one's working on it right now. But feel free to keep discussing things here, or to send related PRs anytime.


Variable anomalies

Environment: Spigot 1.11.2, ScoreboardStats 0.10.1, aSkyBlock <IP_ADDRESS> (all latest), and a long list of up-to-date plugins, sent via PM. Most variables report correctly, but some do not. If I need to configure differently, or If I am experiencing conflicts with other plugins, please identify, so I can take appropriate corrective action.

exp 49848
xp_to_level 1201 or 940 or 7 in three different worlds
island 46

aSkyBlock:is level reports /is level 15 :: verified detail with /asadmin level
The value 46 is not stored anywhere that I can find. What interface is used to derive it?
essentials:exp reports 49,843 exp (level 122) and needs 470 more exp to level up.
Experience is close, but why a difference? And why the wildly different values for level up?

PlaceholderAPI + Expansion/ASkyBlock reports island level accurately.

ScoreboardStats + PlaceholderAPI + Expansion/{Essentials,Player,Statistics,Vault}

return the same value for money balance: PAPI %vault_eco_balance% = SS %money%
return the same value for experience: PAPI %player_total_experience% = SS %exp%. PAPI %player_exp% = 0; not sure why that should be.
return the same values for experience-to-level: PAPI %player_exp_to_level% = SS %xp_to_level%. That is, experience-to-level is different in the three tested worlds, as reported previously.

Any insight appreciated.

return the same value for experience: PAPI %player_total_experience% = SS %exp%. PAPI %player_exp% = 0; not sure why that should be.

A bug in the Bukkit API

The value 46 is not stored anywhere that I can find. What interface is used to derive it?

This plugin uses the ASkyBlockAPI with the variable %island_level%

Thanks. You might want to track that down, but I'll use the obvious workaround.


Web Worker Status

There was some initial discussion awhile ago regarding threading and using web workers to accomplish that task. What's the current status? I have a use case where they would be helpful. There are a number of NodeJS implementations of web workers. Could some of those be adapted? Some clues as to where to start would be appreciated. Thanks!

I found out a way to "emulate" Web Workers using the setInterval function, but I'm still working on the design.

If you are emulating them you don't need setInterval, since you already are
using a single events queue. To properly inplement them we need to use
threads on the C side, or better going fully down the road and implement
preemptive threads, that can be done using interrupts. Next step would be
processes with memory isolation, but they are out of runtime.js design.
El 23/6/2017 2:59 AM, "SpaceboyRoss01"<EMAIL_ADDRESS>escribió:

I found out a way to "emulate" Web Workers using the setInterval function,
but I'm still working on the design.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/runtimejs/runtime/issues/148#issuecomment-310541708,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAgfvv7T5rd9Ptjy6LedDe00cqbFAjk6ks5sGw37gaJpZM4OCaiD
.

I'm emulating Web Workers and Multitasking using the setIntervals

Do you have the code somewhere? On the other hand, I'm not sure how much
important this would be, since Node.js already have support for having
several isolates using a single events queue, so we could use something
similar... The biggest problem would be if someone makes an infinite loop
that would block the full system, but having several isolates and using
interrupts to move to another one would do the trick...
El 23/6/2017 7:55, "SpaceboyRoss01"<EMAIL_ADDRESS>escribió:

I'm emulating Web Workers and Multitasking using the setIntervals
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/runtimejs/runtime/issues/148#issuecomment-310578704,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAgfvrIAwgIcj-ue8mZ9cjFBcew8Pp6yks5sG1NEgaJpZM4OCaiD
.

This is the experimental code:
function main() {
    console.log("Hello, World");
}

var running = false;
var pid = setInterval(() => {
    if(running) return;
    running = true;
    main();
    while(running);
    clearInterval(pid);
},1000)

Can you explain what are you trying to achieve with that? It goes directly to a blocking infinite loop and do nothing more...

It is multitasking. To kill the task, set running equal to false, then the loop ends and kills the task fully with the clearInterval function.

 Do you have the full source code of the proof of concept somewhere?

No, I only have that piece of code

It's going to be pretty hard and time consuming to implement, because requires working with multiple CPUs and/or allowing time sharing multitasking. This will dramatically increase complexity of the codebase.
Runtime runs bare-metal, there is no linux(or other kernel) to provide luxuries like threads and multitasking for us :)
I think being single-threaded is fine for the OS that runs in the cloud on virtual CPUs, i.e you can run a separate runtime.js instance per physical CPU and get "multitasking" that way.

Runtime runs bare-metal, there is no linux(or other kernel) to provide luxuries like threads and multitasking for us :)

That's why I'm talking about of concepts like setjmp/lngjmp :-)

I think being single-threaded is fine for the OS that runs in the cloud on virtual CPUs, i.e you can run a separate runtime.js instance per physical CPU and get "multitasking" that way.

Single threaded is not ok, but it's fine. What we would need is to have several concurrent tasks, and I think "emulate" threads and processes (I think this is how greenlets works) on a single CPU/thread is good enough for now. Far for ideal, but cooperative multitasking would be a good first aproach before moving to preemptive threads/processes so far nobody does a blocking loop, and we could move later in the future to multi-CPU multitasking.
Yes, I'm still willing to port NodeOS to runtime.js and I needs processes some way or another :-P

Fair enough, also it makes sense for WASM, which needs multithreading. Definitely doable but requires some refactoring and code cleanup. I agree, cooperative multitasking is a good start.
As WASM becomes more and more important, I'm thinking we can rename this project to runtime.wasm and start building multi-language cloud platform :)

also it makes sense for WASM, which needs multithreading

Not yet, but it's planned...

As WASM becomes more and more important, I'm thinking we can rename this project to runtime.wasm and start building multi-language cloud platform :)

Please no, runtime.js is cool as a name :-P A wasm-based and optimized OS similar to Inferno with dis-assembler would be cool thouhgt, but I think it would be a different project... There are already wasm VM written in C, so they could be used as basis, or also integrate llvm to compile to hardware in real time, hum...

Well, V8 is wasm VM too, the engine is the same :)
JS should still be supported of course, just thinking it shouldn't be locked to js-only. I have a couple ideas for the new language that will target WASM and be low level enough for high performance code.

the web assembly piece is really interesting, it provides a library OS that is language agnostic, maybe one of the few unikernels that can claim that!
@iefserge are you talking about implementing your own language? and just for my own clarification, is it easier to all provide threading for wasm vs js, or is the complexity the same?

If we change the name, it should be [something].js, of course :-)

@iefserge you mentioned in some older issues (#75)  that background tasks can be run effectively, provided they are non blocking. What would a good implementation look like for running background tasks on a single thread? can there be true background tasks in a single threaded environment or just asynchronous tasks ?

Having WASM integration will definitely help runtime.js (mainly because pointers). Maybe we could take something like TurboScript that has a similar syntax to JavaScript and add some runtime.js integration (direct memory access and custom malloc, free, etc.).

@facekapow
Having WASM integration will definitely help runtime.js (mainly because pointers).
does this mean no copying between WASM and JS, just passing pointers to the same object?

What would a good implementation look like for running background tasks on
a single thread? can there be true background tasks in a single threaded
environment or just asynchronous tasks ?
Being the system single thread and with an unified memory, they can just
call to setInterval and setTimeout to push events to the events queue and
exec themselves as usual in Javascript...

I've recently been reading about wasm because I don't fully understant what it is. Is wasm just low-level assembly access through JavaScript?

That can help with specific things.

@piranna that's what I was thinking, so thanks for confirming.
@facekapow thanks for the info
@SpaceboyRoss01 can you explain that further?

I found how to use a custom memory manager in Runtime.js by reading and writing at a specific section of memory to allocate and free memory for specific tasks, like multitasking.

@SpaceboyRoss01 WASM is cross platform assembly via a JavaScript engine. It has no native system access, it was created to achieve near native speeds for web code. But, since there is a way to interface with WASM through JavaScript, we can provide access to runtime's native functions.

@facekapow That is what I was asking about what WASM is.

I've recently been reading about wasm because I don't fully understant what it is. Is wasm just low-level assembly access through JavaScript?

It's a bytecode specification, similar to the one of the Java VM or Python. In fact, I have been using its opcodes to define the instructions of a CPU of my own :-)

@aafrey Well no, I just mean that it could help with managing pointers, since in an operating system you have to deal with memory directly. The fact that JavaScript doesn't have pointers makes this much more difficult.

Pointers are not needed to manage memory data, you can use structs and MemoryViews. In fact, that's the way you do it with wasm, and it's pretty easy and secure since you work with arrays instead of pointers operations and you can access only the memory area of the MemoryView.

@piranna Well yes, in plain JavaScript you can use Uint arrays with ArrayBuffers for specific memory areas backing them to deal with memory. But what I mean is, languages like C and C++, Rust, TurboScript, and others that compile to WebAssembly and use pointers can help make memory easier to work with.

If you read the WebAssembly specification, you'll see that there're no
pointers arithmetic at all at low level, and memory regions need to be
defined in advance and are accesed and checked the same way UIntArrays are
done ;-) In fact, is dictated that compilers must take this in account when
compiling to WebAssembly as a target platform :-)
El 23/06/2017 18:40, "facekapow"<EMAIL_ADDRESS>escribió:

@piranna https://github.com/piranna Well yes, in plain JavaScript you
can use Uint arrays with ArrayBuffers for specific memory areas backing
them to deal with memory. But what I mean is, languages like C and C++,
Rust, TurboScript, and others that compile to WebAssembly and use pointers
can help make memory easier to work with.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/runtimejs/runtime/issues/148#issuecomment-310714785,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAgfvnXmgIdell8mapMbhXdl6pAviJydks5sG-qAgaJpZM4OCaiD
.

after doing some additional reading, it looks like threads are coming to wasm eventually (https://github.com/WebAssembly/design/issues/1073)
would there be any benefit in beginning that process now with runtime? I know a number of libraries handle web workers with a polyfill when they are not supported by a browser. Maybe we could do something similar. build out a worker class, runtime.worker, that would use lightweight threads (setTimeout) initially, but once threads are a reality we have something that already works. so we build the polyfill first lol

@piranha Yeah, I just found that out for myself reading through some source code for TurboScript. Damn. It made my hopes of writing device drivers (e.g. USB host controller support) much harder (I know runtime is a server OS, but you can't blame me for wanting to have traditionally native OS things written in web technologies).

Why do you think makes it more difficult? Low level needs to be done with
C/C++, but over it everything can be done in high level languages... If you
talk about ports, just expose in() and out() functions. Take a look on the
Python portio module, with that I did several years ago an I2C driver to
override some Linux security measurements...
El 24/06/2017 00:10, "facekapow"<EMAIL_ADDRESS>escribió:
@piranha https://github.com/piranha Yeah, I just found that out for
myself reading through some source code for TurboScript. Damn. It made my
hopes of writing device drivers (e.g. USB host controller support) much
harder (I know runtime is a server OS, but you can't blame me for wanting
to have traditionally native OS things written in web technologies).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/runtimejs/runtime/issues/148#issuecomment-310784071,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAgfvvCVvLsy3n_g_UHDxIZjuY-jFtUDks5sHDfxgaJpZM4OCaiD
.

The problem is that EHCI does everything in memory, no ports. So the convenience of being able to overlay a struct onto an area of memory simplifies it so much. Oh well.

@facekapow how is wasm enabled? in the runtimejs: {} property in package.json?

I don't find any problem about exposing an UIntArray at the exact memory
address EHCI needs...
El 24/06/2017 00:39, "facekapow"<EMAIL_ADDRESS>escribió:

The problem is that EHCI does everything in memory, no ports. So the
convenience of being able to overlay a struct onto an area of memory
simplifies it so much. Oh well.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/runtimejs/runtime/issues/148#issuecomment-310788153,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAgfvv-Bzkofpa85SevFmKa2y1SOrK1qks5sHD6MgaJpZM4OCaiD
.

@aafrey No, in runtime.json put:
{
  "v8": {
    "flags": "--expose_wasm"
  }
}

@piranna Yeah, well I've been having some trouble with code not executing if I try to modify memory via a Uint8Array.

Take in account, Memory objects in wasm are expected to not be executable...
El 24/06/2017 02:37, "facekapow"<EMAIL_ADDRESS>escribió:

@aafrey https://github.com/aafrey No, in runtime.json put:
{
"v8": {
"flags": "--expose_wasm"
}
}
@piranna https://github.com/piranna Yeah, well I've been having some
trouble with code not executing if I try to modify memory via a Uint8Array.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/runtimejs/runtime/issues/148#issuecomment-310801567,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAgfvupQAVAQ8GHwY_LkAYCnVdPvtbizks5sHFo7gaJpZM4OCaiD
.

closing since conversation died off. feel free to re-open if this comes up again :)


bug #1152: Fix for indexing small docs

Purpose

...

Does this introduce a breaking change?

[ X] Yes
[ ] No

Pull Request Type
What kind of change does this Pull Request introduce?
It solves the problem with indexing small documents, where amount of text is less than section_overlap. In this case it's enought to check if  amount of text is less than max_section_length and return only one section without additional processing.
This approach assumes that the section_overlap < max_section_length

[X] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[ ] Other... Please describe:

How to Test

Get the code
Index document whith amount of text less than max_section_length

git clone [repo-address]
cd [repo-name]
git checkout [branch-name]
npm install

Test the code

Index document whith amount of text less than max_section_length
What to Check
Verify that the following are valid
After indexing document whith amount of text less than max_section_length, check if document content has been added to the search index
Other Information

@marekjakimiuk1  thank you for your contribution!

@marekjakimiuk1 thank you for your contribution! The PR has been edited slightly to reduce diff size and help with git blame in the future

@mattgotteiner Thank you too!


Java unable to read real-time output of youtube-dl

Please follow the guide below

You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your issue (like this: [x])
Use the Preview tab to see what your issue will actually look like

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2018.05.26. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

[x] I've verified and I assure that I'm running youtube-dl 2018.05.26

Before submitting an issue make sure you have:

[x] At least skimmed through the README, most notably the FAQ and BUGS sections
[x] Searched the bugtracker for similar issues including closed ones
[x] Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

What is the purpose of your issue?

[x] Bug report (encountered problems with youtube-dl)
[ ] Site support request (request for adding support for a new site)
[ ] Feature request (request for a new functionality)
[ ] Question
[ ] Other

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2018.05.26
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.17134
[debug] exe versions: ffmpeg N-84014-g9e6b269, ffprobe N-84014-g9e6b269
[debug] Proxy map: {}
Usage: youtube-dl.exe [OPTIONS] URL [URL...]

PROBLEM:
I'm trying to use Java to embed youtube-dl into my application. However, I cannot capture the output of the program until after the program has completely executed. The output is returned to me in one large batch. Here is my code.
               BufferedReader bufferedReaderStdOut = null;
               BufferedReader bufferedReaderStdErr = null;

               boolean isError = false;

               try {
                   System.out.println("Starting video download thread...");

                   String[] cmdArgs = {
                           App.config.settings.youtubeDlBin,
                           "-o %(title)s.%(ext)s",
                           "-f worstvideo[ext=mp4]",
                           url
                   };

                   ProcessBuilder procBuilder = new ProcessBuilder(cmdArgs);
                   procBuilder.directory(new File(App.config.settings.videoCollectionDir));
                   Process proc = procBuilder.start();

                   bufferedReaderStdOut = new BufferedReader(new InputStreamReader(proc.getInputStream()));
                   bufferedReaderStdErr = new BufferedReader(new InputStreamReader(proc.getErrorStream()));

                   System.out.println("Checking for program response...");

                   String stdOutLine = bufferedReaderStdOut.readLine();
                   String stdErrLine = bufferedReaderStdErr.readLine();
                   while(stdOutLine != null || stdErrLine != null) {
                       String logLine = "";
                       if(stdOutLine != null) {
                           System.out.println(stdOutLine);
                           logLine += stdOutLine + "\n";
                       }

                       if(stdErrLine != null) {
                           System.out.println(stdErrLine);
                           logLine += stdErrLine + "\n";
                           isError = true;
                       }

                       // On-the-fly class to pass data into runnable
                       class UpdateWindowLog implements Runnable {
                           private String line;

                           public UpdateWindowLog(String logLine) {
                               line = logLine;
                           }

                           @Override
                           public void run() {
                               textLog.setText(textLog.getText() + line);
                           }
                       }
                       Platform.runLater(new UpdateWindowLog(logLine));

                       stdOutLine = bufferedReaderStdOut.readLine();
                       stdErrLine = bufferedReaderStdErr.readLine();
                   }

                   proc.waitFor();
               } catch(Exception ex) {
                   // TBD
                   isError = true;
                   ex.printStackTrace();
               } finally {
                   System.out.println("Finished.. closing buffer.");
                   if(bufferedReaderStdOut != null) {
                       try {bufferedReaderStdOut.close();} catch(Exception ex) {}
                   }
               }

               // Create an on-the-fly custom Runnable that can receive data input into its thread
               class CloseWindowTask implements Runnable {
                   private boolean isError;

                   public CloseWindowTask(boolean error) {
                       isError = error;
                   }

                   @Override
                   public void run() {
                       Alert alert;

                       if(isError) {
                           alert = Utilities.showSimpleAlert("An error occurred!");

                       } else {
                           //System.out.println("Close dialog window...");
                           //closeWindow();

                           alert = Utilities.showSimpleAlert("Completed Successfully!");
                       }

                       alert.show();
                   }
               }
               Platform.runLater(new CloseWindowTask(isError));

               return null;
           }
       };

       // Execute the task described above
       Thread thread = new Thread(task);
       thread.setDaemon(true);
       thread.start();
   }

So what? How is this even related to youtube-dl? If youtube-dl returned it as one chunk you'd get the same  single chunk when running from command line but you get it gradually. So fix your code.

Whoa, why so rude? Nice to meet you too.
Please see:
https://github.com/MrS0m30n3/youtube-dl-gui/issues/192
https://stackoverflow.com/questions/13530429/read-realtime-output-of-a-command
There appears to be an issue with the application treating console output and piping from another application different. I will admit I may be wrong about that but from what i've researched seems to suggest the issue is coming from the source application, e.g. youtube-dl. If the program was actually piping the data in real-time the Java code I have provided above should be able to retrieve in real-time as well.
I would ask that you look at this for more than a fraction of a second before closing out the ticket. Any assistance would be appreciated. Thanks.

If something works everywhere apart from your code then it pretty much obvious whose code is broken. Yes, I've just wasted 15 minutes to check if this works properly in dummy Qt/C++ UI app running youtube-dl.exe via QProcess as well as in python script via subprocess (I don't do Java anymore, sorry). Both worked real-time just fine. So as already said fix your code.
 Thanks.


IScroll 5 allow selecting text inside a div

Hi
I am implementing a reader with iscroll
I whant to add a special mode "Select Text"
in this mode i would like to stop scrollin on mousemove
How can i acomplish that?
Thanks
Nicolas

in iScroll v5.2.0, use options.preventDefault, set to false makes content selection works. options.preventDefaultException can also do this.

could you give me an example of options.preventDefault?


비슷한 기능들 통합

edit_filter
html_filter
interwiki
title_filter
email_filter

https://github.com/2du/opennamu/issues/386

ㅇㅋ


Portal displays only 25 Flights Seen

The portal Flights Seen page only displays the last 25 flights seen, and will not allow you to page to display more flights. Using MySQL database with advanced features turned on.

Hi, thanks for reporting this and I will need to check the code to confirm the expected behaviour.. in the mean time would you mind sharing a screenshot?

I had to do two screenshots to get top and bottom of the list:
Screen Shot 2017-02-27 at 27 Feb 17-0709 1.PDF
Screen Shot 2017-02-27 at 27 Feb 17-0709.PDF

Hi,
Looking at flights.php I can see the following in the code:
    // Items per page.
    $itemsPerPage = 25;

And then in the template there is code to add links to the additional pages, if required:

                {for pageNumber eq 1 to page:pageLinks}

{pageNumber}

                {/for}

Can you try scrolling down on the middle of the page to see if these links are at the bottom after the table of the last 25 flights?
If there is no sign of these links can you right click on the middle of the page, go to view source and share that?

The tabular section in the middle doesn't scroll independently from the rest of the page. No links at the bottom. Is there something in particular you want to see from View Source? In Safari that brings up the Web Inspector which has a ton of data. Anything in particular you want to see? It does show some errors, do you want to see those or the code itself?

 If you see anything interesting, let me know…

thanks again for the details and screenshots, it looks like the pagination script (http://botmonster.com/jquery-bootpag/) may not be installed... would you mind sharing the output of the following command:
ls -l ~/adsb-receiver/build/portal/html/templates/default/assets/js/

Looks like it isn’t. Is that a prerequisite that should install with the script or something that should be there in a standard Raspbian build?
total 156
-rw-r--r-- 1 pi pi 37045 Feb 24 12:16 bootstrap.min.js
-rw-r--r-- 1 pi pi  3884 Feb 24 12:16 date.format.js
-rw-r--r-- 1 pi pi  7983 Feb 24 12:16 graphs.js
-rw-r--r-- 1 pi pi 86709 Feb 24 12:16 jquery-3.1.1.min.js
-rw-r--r-- 1 pi pi  4330 Feb 24 12:16 jquery.sidebar.js
-rw-r--r-- 1 pi pi  1355 Feb 24 12:16 notifications.js
-rw-r--r-- 1 pi pi  3543 Feb 24 12:16 system.js

On Feb 28, 2017, at 0847 , Romeo-Golf<EMAIL_ADDRESS>wrote:
thanks again for the details and screenshots, it looks like the pagination script (http://botmonster.com/jquery-bootpag/ http://botmonster.com/jquery-bootpag/) may not be installed... would you mind sharing the output of the following command:
ls -l ~/adsb-receiver/build/portal/html/templates/default/assets/js/
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub https://github.com/jprochazka/adsb-receiver/issues/322#issuecomment-283076369, or mute the thread https://github.com/notifications/unsubscribe-auth/AX8rqCaUHpl8HSKLsNE43WUQaOQ0Ibz3ks5rhEGjgaJpZM4MMnfA.

To be honest ive only been helping out with the project for a few months and have yet to delve deeply into the portal code; I can see that the jquery.bootpag.min.js file is required by the advanced portal but doesnt appear to be installed by the install script :(
The following command will download the required file from github, please note this is a single line:
wget -O ~/adsb-receiver/build/portal/html/templates/default/assets/js/jquery.bootpag.min.js https://raw.githubusercontent.com/botmonster/jquery-bootpag/master/lib/jquery.bootpag.min.js
Would you mind testing this and confirming that it resolves the reported issue?.. if so I should be able to push the fix out to my 2.6.0 branch pretty quickly :)

Well, I tried that. No joy. Then I get another error:

It is also throwing errors about the flight notification code. I think it must be unfinished. You can put an N-number in to the notification preference page, but it doesn't seem to go anywhere. Here is the reoccurring error that shows on the Flights Seen page:

And note the CSS advisory in yellow:

Thanks for the updates and unfortunately it may well be a case that the install time disclaimer that "the advanced portal features are still in development" means that there is still some outstanding work on these features...
Which hopefully @jprochazka will get a chance to finish off for the 2.6.0 release; however it may be worth trying my 2.5.1 release, which is a bugfix only release based on the current 'stable' version 2.5.0, and thus should result in an Advanced Portal install that at least does what it  supposed to :)
Also FYI from what I can see it may be a jquery version issues that is preventing the pagination code from working, I will need to do some more digging but it does seem like what is described here:
https://geeksprogrammings.blogspot.com/2015/01/Uncaught-TypeError-undefined-is-not-a-function.html

If it's helpful for script diagnostics to mention: I have had had Flights pagination work successfully on 2 clean builds where I cloned from master. I also chose MySQL option on Rpi3 installed through the advanced features.

@patmont thanks for the info, it looks @jprochazka was trying to use the bootpag jQuery script to condense the screen real estate required to display links to many hundreds, or potentially thousands, of historical flights logged by an advanced portal install with this commit: 043349c916abe0736005e4c1583da78adcefaa80
Unfortunately the bootpag script looks to have been written for jQuery version 1, while its version 3 that is used by the other portal functionality... hence the Uncaught Type Error 'undefined is not a function' reported when accessing the flights.php in the 2.6.0 branch.
As my contributions have mostly been around the decoder/feeder components I have only tried 2.6.0 on my Pi's but I will see if I can find some time to spin up another vm for a 2.6.0 branch install with the advanced portal installed so I can try to help out with issues such as this one :)
Its also worth highlighting that my experience with jQuery is minimal so if there is anyone who can provide any assistance with either getting the current bootpag script to work, or suggest a suitable replacement that plays nice with jQuery 3, that would be greatly appreciated...
@jdwall how is your familiarity with jQuery?  ;)

The code making up the pagination on this page had been updated on my end at one point however I need to go through and find it again or rewrite it. It used totally different pagination code doing away with showing EVERY page available. I will be making this a priority and should find its way into the upcoming 2.6.0 release.


settings / features / MFA

the part in the info text that says "More info HERE" turn it in to:
"For more information, see our help page on MFA.
... keep the rest of the text around it.

ready for the next release


Add _restclient README with example. Fulfill requirement in #62

Closes #62
Additions

-restclient README.md with example

Testing

Tested the example locally. Will not work until #67 is merged.

Checklist

[x] PR has an informative and human-readable title
[x] PR is well outlined and documented. See #12 for an example
[x] Changes are limited to a single goal (no scope creep)
[x] Code can be automatically merged (no conflicts)
[x] Code follows project standards (see CONTRIBUTING.md)
[x] Passes all existing automated tests
[x] Any change in functionality is tested
[x] New functions are documented (with a description, list of inputs, and expected output) using numpy docstring formatting
[x] Placeholder code is flagged / future todos are captured in comments
[ ] Reviewers requested with the Reviewers tool :arrow_right:

Merging since #67 was merged.


mantra create should download Meteor if not found

I just installed mantra-cli and tried to create a project. The directory was created but nothing else happened. I realized it was because I didn't have Meteor installed. It'd be great if Mantra either downloaded Meteor for you, or if it notified you that Meteor was not installed.

Thanks for the suggestion. I think downloading Meteor is the out of the scope of Mantra CLI. Also, it will be tricky to support Windows.
It'd be good if we can output errors back to the user instead of halting silently. I can't think of a good way to do this. But now we have a verbose mode (doc), so that people can debug what is going on.


Add validation test for cookie values enclosed in double-quoted characters

Test added as per https://dart-review.googlesource.com/c/sdk/+/63920
This test compliments the fix in #33765 along with changes made here.

The dartfmt command needs to be run on new or modified files, so they are formatted automatically and canonically.
This PR was added to the Gerrit CL https://dart-review.googlesource.com/c/sdk/+/63920

@whesse Does "This PR was added to the Gerrit CL ..." mean that this PR should be closed?

According to @whesse, this can be closed. Re-open if you disagree.


Add Assent library

@rubengees sorry can't merge. Your PR fails the travis-ci build. Your change must not contain a ' inside the xml ;)  Please update, so I can merge it in ;)
Thanks

Whoops, didn't look out for those.
But fixed.
Maybe add checks to the generator website you have?

@rubengees yeah probably would help ;)


CCC does not get cleared when CONFIG_BT_KEYS_OVERWRITE_OLDEST is enabled

Describe the bug
When CONFIG_BT_KEYS_OVERWRITE_OLDEST is enabled, a warning gets thrown by bt_gatt saying no space to store CCC cfg.
To Reproduce
Steps to reproduce the behavior:

git apply prj.conf.patch && west build -b nrf52_pca10040 samples/bluetooth/peripheral_hr
west flash
bond with the peripheral using nrfconnect and subscribe to the CCC notification
disconnect and perform step 3. with a different device (bluetoothctl)
disconnect and perform step 3 again with the first device. The CCC cannot be written to the device and the error message appears

Expected behavior
Expected is that the CCC can be normally subscribed to without warning messages from bt_gatt
Impact
This leads to all notifications being broken once in this situation. If the device is power cycled the state is returned to normal.
Screenshots or console output
[00:00:55.689,941] <inf> hrs: HRS notifications enabled
Disconnected (reason 0x13)
[00:01:51.021,789] <inf> hrs: HRS notifications disabled
Connected
[00:02:06.110,473] <inf> bas: BAS Notifications enabled
[00:03:09.010,406] <inf> hrs: HRS notifications enabled
[00:03:38.810,302] <wrn> bt_ctlr_hci: type: 0x12
Disconnected (reason 0x13)
[00:03:51.810,699] <inf> bas: BAS Notifications disabled
[00:03:51.810,760] <inf> hrs: HRS notifications disabled
Connected
[00:03:58.308,197] <wrn> bt_ctlr_hci: type: 0x14
[00:04:11.032,073] <wrn> bt_gatt: No space to store CCC cfg

Environment (please complete the following information):

OS: manjaro
Toolchain gcc-arm-non-eabi-9-2019-q4-major
Commit SHA or Version used: commit 9518bd19b7a736bd649fa2fc8d8d52967c0b486f (HEAD, tag: zephyr-v2.2.0, tag: v2.2.0)

Additional context
Short discussion with Joakim revelead this could be an issue in CONFIG_BT_KEYS_OVERWRITE_OLDEST, as it's quite a new feature
patch:
diff --git a/samples/bluetooth/peripheral_hr/prj.conf b/samples/bluetooth/peripheral_hr/prj.conf
index 3dfa8796b8..f65f1a0913 100644
--- a/samples/bluetooth/peripheral_hr/prj.conf
+++ b/samples/bluetooth/peripheral_hr/prj.conf
@@ -8,3 +8,7 @@ CONFIG_BT_GATT_BAS=y
 CONFIG_BT_GATT_HRS=y
 CONFIG_BT_DEVICE_NAME="Zephyr Heartrate Sensor"
 CONFIG_BT_DEVICE_APPEARANCE=833
+
+CONFIG_BT_KEYS_OVERWRITE_OLDEST=y
+CONFIG_BT_WHITELIST=y
+CONFIG_BT_MAX_PAIRED=1
\ No newline at end of file

@joerchan I redid the test as follows:
Set-up:
nrfjprog --eraseall && west flash

Connect using smartphone with (nrfconnect).
Bond using nrfconnect.
Subscribe to notifications.
Disconnect
Corresponding output from device:
*** Booting Zephyr OS build zephyr-v2.2.0  ***
Bluetooth initialized
Advertising successfully started
[00:00:00.004,974] <inf> bt_hci_core: HW Platform: Nordic Semiconductor (0x0002)
[00:00:00.004,974] <inf> bt_hci_core: HW Variant: nRF52x (0x0002)
[00:00:00.004,974] <inf> bt_hci_core: Firmware: Standard Bluetooth controller (0x00) Version 2.2 Build 0
[00:00:00.005,615] <inf> bt_hci_core: Identity: db:5c:ca:0c:c9:c2 (random)
[00:00:00.005,615] <inf> bt_hci_core: HCI: version 5.1 (0x0a) revision 0x0000, manufacturer 0x05f1
[00:00:00.005,615] <inf> bt_hci_core: LMP: version 5.1 (0x0a) subver 0xffff
Connected
[00:00:34.359,222] <wrn> bt_ctlr_hci: type: 0x14
Disconnected (reason 0x13)
Connected
[00:00:58.149,353] <wrn> bt_ctlr_hci: type: 0x14
[00:01:15.943,328] <inf> hrs: HRS notifications enabled
Disconnected (reason 0x13)
[00:01:22.963,684] <inf> hrs: HRS notifications disabled

Connect device 2 (bluetoothctl) and perform
Pair
Subscribe
Disconnect
as follows:
[bluetooth]# scan on
Discovery started
[CHG] Controller 00:28:F8:F8:D6:6B Discovering: yes
[NEW] Device DB:5C:CA:0C:C9:C2 Zephyr Heartrate Sensor
[bluetooth]# scan off
[bluetooth]# connect DB:5C:CA:0C:C9:C2
Attempting to connect to DB:5C:CA:0C:C9:C2
[CHG] Device DB:5C:CA:0C:C9:C2 Connected: yes
Connection successful
[NEW] Primary Service (Handle 0x81fe)
        /org/bluez/hci1/dev_DB_5C_CA_0C_C9_C2/service0001
        00001801-0000-1000-8000-00805f9b34fb
        Generic Attribute Profile
[NEW] Characteristic (Handle 0x81fe)
.
.
.
.
[Zephyr Heartrate Sensor]# pair
Attempting to pair with DB:5C:CA:0C:C9:C2
[CHG] Device DB:5C:CA:0C:C9:C2 Paired: yes
Pairing successful
[Zephyr Heartrate Sensor]# menu gatt
[Zephyr Heartrate Sensor]# select-attribute 00002a37-0000-1000-8000-00805f9b34fb
[Zephyr Heartrate Sensor:/service0019/char001a]# notify on
[CHG] Attribute /org/bluez/hci1/dev_DB_5C_CA_0C_C9_C2/service0019/char001a Notifying: yes
Notify started
[CHG] Attribute /org/bluez/hci1/dev_DB_5C_CA_0C_C9_C2/service0019/char001a Value:
  06 66                                            .f
[CHG] Attribute /org/bluez/hci1/dev_DB_5C_CA_0C_C9_C2/service0019/char001a Value:
  06 67                                            .g
[CHG] Attribute /org/bluez/hci1/dev_DB_5C_CA_0C_C9_C2/service0019/char001a Value:
  06 68                                            .h
[Zephyr Heartrate Sensor:/service0019/char001a]# back
[Zephyr Heartrate Sensor:/service0019/char001a]# disconnect
Attempting to disconnect from DB:5C:CA:0C:C9:C2
[CHG] Attribute /org/bluez/hci1/dev_DB_5C_CA_0C_C9_C2/service0019/char001a Value:
  06 6b                                            .k
[CHG] Attribute /org/bluez/hci1/dev_DB_5C_CA_0C_C9_C2/service0019/char001a Value:
  06 6c                                            .l
[CHG] Device DB:5C:CA:0C:C9:C2 ServicesResolved: no
Successful disconnected

Corresponding output from device:
Connected
[00:01:56.086,242] <inf> bas: BAS Notifications enabled
[00:02:32.113,067] <inf> hrs: HRS notifications enabled
Disconnected (reason 0x13)
[00:02:38.937,927] <inf> bas: BAS Notifications disabled
[00:02:38.937,957] <inf> hrs: HRS notifications disabled

Connect again with device 1 (nrfconnect)
Automatically bonded using LTK's
Unable to subscribe to notifications
Output from device:
Connected
[00:02:48.763,641] <wrn> bt_ctlr_hci: type: 0x14
[00:03:08.458,831] <wrn> bt_gatt: No space to store CCC cfg

I am still seeing this issue occasionally, but I am using bluetooth mesh. I connect to the bearer successfully but the bearer does not open. And my device is outputting the following:
bt_gatt: No space to store CCC cfg
I'm not exactly sure how to reproduce. It seems to be happening pretty randomly and I can't pinpoint exactly when it happens.

@johnsmithecon Please find out how to reproduce your issue and open a new bug-report if you can


docs: improve readme

Add link to the RFC 6265 (better descriptions of available options).

Thanks for the link!


Adds Filter Exclusion Name character limit & related UI

This PR address issue #13140 by doing the following:

Truncates exclusion names over 20 characters.
Adds text underneath the input bar to note the remaining characters.

Note: This has one string (for the "characters remaining" text) that is externalized in nls/root.
Questions:
I'm having trouble unit testing this, which I've left out of this PR for now. This is my first time working on the project, and I believe I'm making a mistake causing my unit tests to not be considered when I try to run them from the testing interface. None of the console.log statements I've added to my unit test pass through. My process has been:

Add a unit test in FileFilters-test.js
Reload Brackets
Test the FileFilters unit test section.

Travis says:
bomanimc has NOT submitted the contributor license agreement. See http://dev.brackets.io/brackets-contributor-license-agreement.html

Please sign the agreement before I can merge the code.

@zaggino Just signed the form! My bad on that.

Thanks for the PR @bomanimc , nice work.


More test coverage

Codecov Report

Merging #19 into master will increase coverage by 1.97%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master      #19      +/-   ##
==========================================
+ Coverage   46.67%   48.64%   +1.97%
==========================================
  Files          18       18
  Lines        1217     1217
==========================================
+ Hits          568      592      +24
+ Misses        631      607      -24
  Partials       18       18

Impacted Files
Coverage Δ

chttp/chttp.go
67.58% <0%> (+2.06%)
:arrow_up:

db.go
87.96% <0%> (+9.72%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a4fa1dc...59a55b9. Read the comment docs.


Slide completed progress

Clear and concise description of the problem
Make it possible to determine how far a slide is dragged/moved to its next position. So for example request this value with 'swiper.slides[0].fractionalProgress'. So if the slide isn't moving / snapped, this value is always 0. But if the slide moves (manual by dragging or automatically) to its next position then this value will increase slowly from 0 (start position) to 1 (end position). So this means this value will be a floating fractional value like 0.304 or something. Also see this illustration for more details:

The documentation states that the slide progress can be requested with the property 'progress' when 'watchSlidesProgress' is enabled. But this value doesn't give me the desired results.
My Swiper uses the following configuration:
:slides-per-view="1"
:space-between="30"
centeredSlides
loop
:navigation="{
nextEl: '.swiper-button-next',
prevEl: '.swiper-button-prev',
}"
:pagination="{ el: '.swiper-pagination', clickable: true }"
@slideChange="onSlideChange"
class="slideshow"
:breakpoints="breakpoints()"
Use case:
I want to make the title of the first (previous) slide a color between black and white. So the value 0 should be completely white and 1 is completely black. The values in between determine how far the color deviates to white/black.
Suggested solution
Add another property to the slides like 'swiper.slides[0].fractionalProgress'. See the explanation in the description above.
Alternative
No response
Additional context
No response
Validations

[X] Follow our Code of Conduct
[X] Read the docs.
[X] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

Would you like to open a PR for this feature?

[ ] I'm willing to open a PR

Swiper v9 comes with fully reworked and now different loop functionality. If you have similar issues in Swiper 9, open a new issue with a CodeSandbox showing the issue.


Fix missing empty string as first element in serialized StringTable

Avoid making StringTable inherit from Array, this seems dangerous as one could push directly without updating the internal structures
StringTable constructor with no argument inserts the empty string '' as first element and makes sure it will be serialized
During decoding ProfileInput.stringTable is a string[], that is then converted to a StringTable in Profile constructor
If Profile is constructed without defining ProfileInput.stringTable, an empty StringTable (without '' as first element) will be constructed

Already fixed this in #5.

Already fixed this in #5.

I updated my PR and resolved conflicts.
I am new to JS/TS so don't hesitate to suggest improvements

Nothing on-hand, no. But in a case like this V8 may produce a shape which only includes the sub-class so all lookups become heap lookups rather than potentially stack lookups.

Yet another iteration:

new StringTable() constructs a StringTable that contains only the empty string. Hence Profile({}).encode() contains only the serialization of a StringTable with '' as unique element.
new StringTable(emptyTableToken) constructs an empty table. emptyTableToken is a constant, at first I did not export it but I ended up needing it for the tests.


Fix warnings on julia 0.4

Also closes #15.

0.3 Coverage failure should be fixed by https://github.com/JuliaLang/METADATA.jl/pull/3432 .

thanks!


Clearer distinction between Ergo and JS archive

Test coverage improvements

Coverage increased (+2.5%) to 91.041% when pulling d35703be573211374c092e13b837958cc5d6c70e on jeromesimeon:js-template-studio-mvp into 6e42837348a4010f15cb27d77a489a598c519337 on accordproject:master.


Conform to fontconfig spec.

Currently, font can be changed by using from vim
set guifont=fontname:fontsize

The parsing is trivial: find the first :, what is on the LHS is the font name, on the RHS is the font size. The font name is then directly passed to the Evas_Textblock.
Turns out that things are not that pretty in real life, as it is possible to specify fonts with great details with fontconfig. And Evas_Textblock actually parses  the font as fontconfig formats.
See the fontconfig user manual for details.
So instead of having our own minimal format above, we should just use the fontconfig spec. We just need to force users to specify a font size. For example, to select DejaVu Sans Mono with font size 12 and style Book, this should give:
set guifont=DejaVu\ Sans\ Mono-12:style=Book

This requires to:

[ ] change the parsing: font size is available after the first dash (what happens if a font contains a dash??)
[ ] pass everything to the Evas_Textblock
[ ] update the wiki

Thanks @pcouderc12 for the report.

Notes for the wiki:

EFL implementation of the format is here
Paths to PCF files are also allowed (that's what terminology does) (X11 Portable Compiled Font data) -> /usr/local/share/terminology/fonts/terminus-20-bold.pcf


[Enhancement #1280] Change to try-with-resource

Fixes #1280.
Motivation
Making code clear
Modifications
try-with-finally changed to try-with-resource.
Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented? -
If a feature is not applicable for documentation, explain why? -
If a feature is not documented yet in this PR, please create a followup issue for adding the documentation -

Sorry for the whole mess with titles and previous commits

Please fix the checkStyle error, if you don't know how to do with the code check style you can connect with the EventMesh helper. @zdb11


Passed -> Matched, for clarity

Signed-off-by: Fabian von Feilitzsch<EMAIL_ADDRESS>

I think tests may be failing at least in part because the tests are wrong, going to double check

This looks good to me thank you this is going to make everything much clear

@shawn-hurley is that approval to merge?

I think so yeah


gEvent install failure in run_tests_python_macos_opt_gevent

Happened in https://source.cloud.google.com/results/invocations/33114acf-c33d-4dca-862e-6fc082d532cc/targets/grpc%2Fcore%2Fpull_request%2Fmacos%2Fgrpc_basictests_opt/log
+ /Volumes/BuildData/tmpfs/src/github/grpc/workspace_python_macos_opt_gevent/py34_gevent/bin/python3.4 -m pip install gevent
Collecting gevent
  Could not fetch URL https://pypi.python.org/simple/gevent/: There was a problem confirming the ssl certificate: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:600) - skipping
  Could not find a version that satisfies the requirement gevent (from versions: )
No matching distribution found for gevent

Seems like a dup of #14815.  Feel free to re-open if it is a separate issue.


question for abstracter

Hi，auther，
I'd like to ask you some questions，I ran around the server with the code you gave me on github, and the results were similar. But the reason you said abstracter higher than the unified model was because grandtruth was used for training and testing.
But shouldn't two stages train abstracter via threshold  sentences？I don't understand this part, and in the final Table table-2,  how did the result of your two-stage be produced, did you use the Abstracter Table in Github？How can I get it.
And the threshold you set it is 0.5,  random or optimized?
Thanks for your help！

I think the higher scores in abstracter just for pretraining a bestmodel. And the best abstracter model is used in end2end, right?  But the two stag's ROUGE score how to get ?

Hi,
In the two-stage experiment, we use pretrained extractor and pretrained abstracter listed in this github. The pretrained models are provided in here.
The abstracter takes extracted sentences with sentence-level attention βn > 0.5, where β is obtained from the pretrained extractor, as input during two-stages training. Since the input for the abstracter is different, we finetune the abstracter for 10k iterations. The two-stage results came from the pretrained extractor + finetuned abstracter.
In the end-to-end experiment, we use the same pretrained extractor and pretrained abstracter. But we finetune both the extractor and abstracter by using our end-to-end method.
You can refer to my paper for more details.

How can I get the two-stage's ROUGE Result, now I get the ROUGE results for rewrier is like to the Abstracter results in your Github table Abstracter (model of 81000 iteration).
If I need to change the model ckpt_path for two-stage's results?
Now I have the best pretrianed extractor and abstracter, how to proceed two-stages experiment. In end2end.sh?

Sorry that I didn't provide the code of two-stage training.
You can run the end-to-end scripts to do this. But you need to modify the code to fix the extractor and use the abstracter's loss functions during training.


Added The scraper for Humboldt with successful pre-commit run

Hey @zstumgoren please review this one as well Thanks.

@naumansharifwork Checks look good!  I should have asked on the last go-round, but could you post a snippet in this thread showing the directory tree structure produced in the cache directory (e.g. using tree command, along with a small sample of records in the final JSON output from scrape_meta? A main goal for us is to retain information about the linkage between assets (videos, audio, pdfs, etc.) and the associated case information, and a visual inspection of the output is one easy way to verify that everything works as expected.

@zstumgoren here is a screenshot of tree

Also i am uploading meta json
ca_humboldt_pd.json

Hey, @zstumgoren Thanks for the feedback,
For Panel Code:
I was grouping the records in their respective folders for the PDF documents we have a Panel Code e.g
832.7 (b)(1)(A)(i)
but for the audio/video files we don't have panel code mentioned, They have some case code like this example below but it does not appears to be the date

So i thought it might be panel code as well, i can name them as case id but this case id will not be available for the pdf type of documents,
Parent Page:
i understand about the first part that i should give the cache file location,
For the download page as the download and the name of document is available on the next page so we are first downloading the download page in cache as well so should i provide its relative path?
File Tree:
For saving the files we are already grouping the documents in their respective penal code

for the audio/video files they are also grouped in their respective case folder e.g

All these files will go to the 17-3004 folder,
File names are preserved the same as we get when we download them.
Let me know if we need to change anything in file tree,

@naumansharifwork A few follow-ups to your most recent questions:
Panel Code
In the JSON, does this refer to Penal Code? If so, we should replace with correct spelling -- penal_code -- and only fill it out when that information is available. Or you can drop the field, since I'm not certain we'd use that downstream in our processing.
Parent page vs. download page

For the download page as the download and the name of document is available on the next page so we are first downloading the download page in cache as well so should i provide its relative path?

Sure, that'd be a good strategy to to handle as:

download_page - relative local path in cache
parent_page - higher-level index page that leads to the download_page

File tree with case-number folders
After reviewing the page a bit more, it appears that the unique "ID" for each case does indeed appear to be case number. These case numbers appear to have a PDF report on the Documents tab, and where available, corresponding audio and video and other assets on the Audio/Video tab.
So the key thing for us would be to use this unique ID to organize the files in separate folders. To expand on my earlier example, for 201504289, you might have the below structure:
ca_humboldt_pd/
ca_humboldt_pd/SB-1421-AB-748-Information.html # parent_page

ca_humboldt_pd/201504289/ # This folder contains case file and related audio/video listed on the "parent_page"
ca_humboldt_pd/201504289/201504289.pdf # NOTE: THIS IS NEW and corresponds to the file on the Documents tab
ca_humboldt_pd/201504289/201504289-085-4 Redacted.wav

It sounds like you're already doing this to group audio/video by case, but grouping the police reports (ie the PDFs) under separate penal code folders. Is that correct?
If so, we should get rid of the penal code directories and just save  the case file PDF inside the appropriate case folder as illustrated above, using its case number + .pdf suffix (or whatever the actual file name of the PDF is based on the URL). That type of structure will greatly help with our downstream processing.
For the penal code, it's fine to add that as a key in the JSON file for each record, where available, and leave blank where it's not available. Or as mentioned above, also fine to drop the field for now. If you decide to keep it in the JSON, we should correct the spelling from panel_code to penal_code.

ca_humboldt_pd.json
Hey please check json now
also i am attaching the tree screenshot now

@naumansharifwork Folder structure looks spot on! Thanks! JSON looks good overall, just one minor note: It appears the download_page and parent_page are identical:
        "parent_page": "ca_humboldt_pd/SB-1421-AB-748-Information.html",
        "download_page": "ca_humboldt_pd/SB-1421-AB-748-Information.html",

If we keep download_page, that should be the relative path to a local copy of the HTML from the NextURL page where the file can be dowloaded (i.e. based on the links harvested from the Audio/Video tab).

@zstumgoren download page is same as parent page for the cases where we get the download link actually from the parent page (for document types), EIther we can keep it same or if you want i can remove the download page key for the records where its not required.

@zstumgoren download page is same as parent page for the cases where we get the download link actually from the parent page (for document types), EIther we can keep it same or if you want i can remove the download page key for the records where its not required.

Gotcha. Sure, let's remove it for the cases where parent_page and download_page are identical. Once that change is done, I think we should be ready to merge. Thanks!

ca_humboldt_pd.json
Hey @zstumgoren done.

Looks great! Merging. Huge thanks!


ip2location-c: 8.4.1 -> 8.5.1

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/chrislim2888/IP2Location-C-Library/releases.
meta.description for ip2location-c is: "Library to look up locations of host names and IP addresses"
meta.homepage for ip2location-c is: "https://www.ip2location.com/developers/c"
meta.changelog for ip2location-c is: ""
Updates performed

Version update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found 8.5.1 with grep in /nix/store/cvxpf9c0yasx5k2mlxski9bxqagpk8w6-ip2location-c-8.5.1
found 8.5.1 in filename of file in /nix/store/cvxpf9c0yasx5k2mlxski9bxqagpk8w6-ip2location-c-8.5.1

Rebuild report (if merged into master) (click to expand)

2 total rebuild path(s)

2 package rebuild(s)

First fifty rebuilds by attrpath
ip2location-c
ipv6calc

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/cvxpf9c0yasx5k2mlxski9bxqagpk8w6-ip2location-c-8.5.1 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A ip2location-c https://github.com/r-ryantm/nixpkgs/archive/df97dbf9a840e508160471bfe2b9c24816dd6ac9.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/cvxpf9c0yasx5k2mlxski9bxqagpk8w6-ip2location-c-8.5.1
ls -la /nix/store/cvxpf9c0yasx5k2mlxski9bxqagpk8w6-ip2location-c-8.5.1/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

2 packages built:

ip2location-c
ipv6calc

Maintainer pings

Result of nixpkgs-review pr 183386 run on x86_64-linux 1

2 packages built:

ip2location-c
ipv6calc


Change http to https for security links

Which problem is this PR solving?

Fix http link to https link for secure.

Short description of the changes

Modify http to https in the links of project. Have some http links does not redirect into https.

I'll merge without DCO since these are not source code files


[SPARK-40271][FOLLOWUP][PYTHON] fix the minor exception message

What changes were proposed in this pull request?
This PR is followup for https://github.com/apache/spark/pull/37722.
Why are the changes needed?
The original PR was merged without applying the review suggestion that fix the error message.
Does this PR introduce any user-facing change?
Yes, the error message is fixed.
How was this patch tested?
The existing tests should cover

cc @HyukjinKwon @zhengruifeng FYI

thank you @itholic so much!

Merged into master, thank you all


Error when creating kitty

On the latest v2.0 branch, when creating kitty from polkadot-JS app, it returns in the browser dev-console:
RPC-CORE: submitAndWatchExtrinsic(extrinsic: Extrinsic): ExtrinsicStatus:: 1002: Verification Error: Execution: Could not convert parameter `tx` between node and runtime: Error decoding field CheckEra.0: Client(Execution(ApiError("Could

The runtime console:
2019-11-27 16:34:06 Idle (0 peers), best: #50 (0x95d8…39c0), finalized #48 (0x45ff…1a14), ⬇ 0 ⬆ 0
2019-11-27 16:34:09 Failed to submit extrinsic: Extrinsic verification error: Execution: Could not convert parameter `tx` between node and runtime: Error decoding field CheckEra.0

I and @kaichaosun encounter this issue.

We've been discussing this issue in the Substrate Technical (Public) Riot chatroom.
It's because all the dependencies are being build from an old commit dated 25th October in the 'master' branch of the Substrate repo. I've got it to work by using the latest commit instead (i.e. progressively going through all the commits since then and getting it to compile), and then checking that it was consistent with the https://github.com/paritytech/substrate/tree/master/bin/node-template
I'll transfer the changes I had to make across to this template

Solution: https://github.com/ltfschoen/substratekitties/tree/v2.0
Noting that this fix likely won't get merged https://matrix.to/#/!HzySYSaIhtyWrwiwEV:matrix.org/$1574898806602KyvWj:matrix.parity.io?via=matrix.parity.io&via=matrix.org&via=web3.foundation

stale


Checkbox

As a user, I need a checkbox component that will match the style of radio but also allow for validation.  The checkbox should allow for an intermediate state (-) along with (no check) and (check).

[x] checked state
[x] unchecked state
[x] intermediate state [-]
[ ] validation
[x] primary
[x] urgent (red)
[x] disabled

Not exactly sure what validation means in this context, validation occurs outside of the checkbox component


Update the PGP signing key

Key would expire in July of this year; avoid having builds fail for trying to sign with an expired key, use the updated keyring with the new expiration date in 2022.

I was working on a version branch, but since this is all travis, i don't see any reason to go through that formality.

I would have done something with a version branch but the core issue is that this logic is only ever used on master, and if we do use this logic on a version branch by changing when it's invoked, then we push artifacts and it might as well have been master.  In fact, it would be worse, as there would be published artifacts without a record in master.


Add Campaign Field Length Feedback

Related to #58 and #254.
AS A: logged-in user
WHEN I: type over the character limit in the campaign fields
I WANT TO: receive visual feedback that I've typed over the limit. (See example from the Registration form)
Over Max Characters in Campaign Field

Registration Validation Feedback

Since those textareas won't let you type over the character limit (and therefore won't trigger an invalid form state like some other inputs will), maybe we can implement a dynamic character counter for those fields? Instead of 'Max 1000 characters' it could be '70/1000' or '930 characters remaining'


Fix esc-cli workflow

Proposed changes
Explicitly use bash for a shell command

Unreleased product version (optional)

Related issues (optional)

Never mind, I just noticed that the failing workflow was already using bash


Python 3 compatibility and t-batch caching.

The first 3 commits address python 3 compatibility and remove unnecessary imports.
The final commit is an incomplete tbatching optimization. We don't need to recompute tbatches for every epoch, so it makes sense to do some type of caching. Also, we don't need to recompute them for every run either, assuming we can load the entire dict of tbatches into memory and do random access on each dict (needed to account for user changes to timespan).
However, the current code isn't set up to incorporate these changes easily, because chunks of t-batches are computed on-the-fly, trading off with the corresponding chunk of the epoch. So one would have to compute the "start" and "end" points of each tbatch chunk so that the epoch chunk can access the right tbatches.
The code in the 4th commit is not just unoptimized, but buggy. In the first epoch, the tbatch dicts keep growing, as args.cache_tbatches=True removes tbatch reinitialization. But the epoch still iterates over the full length of the tbatch dicts. There are a two competing ways one could fix this:

Revert to reinitializing the tbatch chunk every time, but save the tbatch chunks to disk.

Tell the epoch where to access the tbatch dict, instead of starting from the beginning.

2 seems easier, but I will leave that to the code maintainers' judgement :)

Hi @jpalowitch, I can help finishing&testing cached t-batch feature as synced-up with Prof. Kumar last week, are you still interested/in-need of it?

Hi @jpalowitch, I can help finishing&testing cached t-batch feature as synced-up with Prof. Kumar last week, are you still interested/in-need of it?

Yes, thanks!


Create single release script?

Having to do manual releases now relies on the release manager to correctly execute a series of manual scripts in the correct order. It would be useful to have all the scripts executed using a single "mega-script" to simulate the automation that came from Travis.

Gone as far as I think we can in #299. Will close as soon as that is merged.


[Feature]  How to change colors on 'VOL' and other indicators?

Feature Description
Is it possible to change the color of the volume bars?
To Do
styles param in createTechnicalIndicator seems to be ignored.

Maybe the level is wrong.

Sorry what do you mean by that?


Use a good programming language

Current Implementation
The current implementation is done with Java.
Suggested Enhancement
Pick a modern programming language instead.
Expected Benefits
Developer productivity, developer happiness, a million dollars saved in NullPointerExceptions. Another few 100k in ClassDefNotFoundException.

never