[feature request] add bit operators support in AQL

such  as &,|, >>, they are very useful in some situations, I hope arangodb can support them.

I actually thought about this the other day, but couldn't think of a good
use case. Instead of a bit field for instance, you should use an object
instead - which will be self-describing and therefore more suitable for a
schema-less database.
Can you provide a real use case?
You can always implement these operations as AQL user-defined functions
using JavaScript BTW.

Ya sure. We usually use this to store states to gain performance and less storage requirement, It's not against object and scheme-less. imagine you have tons of states on each object, just use one property to remember all the states are much easier to maintain.
It can use to do math as well.
Also as you mentioned, we can achieve bit operation with JavaScript function. Just think it would be more convenient that AQL could support it innative.
Thanks.

Redis has
SETBIT http://redis.io/commands/setbit
GETBIT http://redis.io/commands/getbit
I guess bit fileds can't be exposed to clients for now because JSON doesn't support them.
But I agree, to have a arbitrary field of bytes aka. binary data is sometimes handier than workaround.
For example #1110, for now I encode the filepath into base64. To have binary data that isn't touched / converted by ArangoDB would make things easier.
To support binary data maybe document insert, update, replace functions need some option to convert a key explicit to binary. JSON return would be then somethings like 0x... or base64.
INSERT {binData:'0x0123456789abcdef`} IN myCol {binary:['binData']}

then sub sequential aql queries:

// IS_BIT_ONE(value, bit offset)
// IS_BIT_ZERO(value, bit offset)
FOR doc IN myCol
  FILTER IS_BIT_ZERO(doc.binData, 0)
  RETURN doc

link #107

A problem with bitfields and binary data in general would be that there is no such type in JSON.
Though we could internally store some binary data in ArangoDB, there is no established way to get binary data into or out of it via a JSON interface.
Having our own representation of "binary" in the JSON interface would be incompatible with all other clients (at least, other clients wouldn't support this because it would be an ArangoDB-only extension).
Using a convention like "'0x' means hexadecimal" (or "'0b' means binary") would work well for numbers, but I don't think we should try to peek into the first bytes and treat value as hex or binary if starting with "0x". For example this would make it impossible to store a UTF-8 string starting with the sequence '0x'. It would always be treated as a hex value then.

And the approach with a option {binary:['binData']} tread the content of key binData as binary data. Has to be 0x… or a base64 encoded string.

Wouldn't that mean that you can't use an attribute "binary" for non-binary
data, and that you could only have a single binary blob per object?

@Simran-B : I think so. Whenever trying to handle binary data with standard JSON, you are down to specific conventions, e.g. "strings that start with '0x' contain binary data", "if an attribute is named 'binary' it will contain binary data" etc. Which would do the job, but it is not ambiguous and may come as a surprise for some.

I mean tell ArangoDB explicitly tread the content of key x as binary data. So that default behavior is never changed.
// insert binData as string '0x0123456789abcdef'
INSERT {binData:'0x0123456789abcdef'} IN myCol

// inserts binData as binary
INSERT {binData:'0x0123456789abcdef'} IN myCol {binary:['binData']}

// option binary: convert the content of every key in [] into binary representation
{binary:[]}

@baslr: understood. This would work by enhancing all AQL operations such as INSERT, UPDATE, REPLACE with the binary option.
However, I don't think this would be sufficient. For example, if there's binary data in the database already, how would you treat it when it is accessed from a query, e.g. from a FILTER condition or an arbitrary expression?
For example, for FILTERs, there would be no "natural" place to put the binary option, e.g.
FILTER binData == '0x0123456789abcdef' /* binary?? */
FILTER someThingBinary /* binary?? */ < CONCAT(otherBinaryValue /* binary?? */, 'foo') /* binary?? */
So I think we should not yet extend some of the operations (INSERT, UPDATE etc.) because it's completely undefined how other operations and in general expressions should handle binaries. Apart from that, there is no sane way to return binary in JSON. It can be encoded somehow, but then the result would need to contain some information which parts of it are binary values.
For mixed results as they can be produced by AQL this will not be trivial. Imagine the following result:
{
  "result": [
    { "_key": "abc", "binData": "0x1234..." },
    { "_key": "def", "someBinary": "0x1234", "otherBinaryValue": "0x1234..." },
    { _key: "123", value: 1 }
  ]
}

How would this convey the information about its binary values (and potentially their encoding)?

Hi, thanks for the discussion about the bit operation. But I guess we could make it simpler? As what we can do in JavaScript.
In JavaScript there is no byte type, only Number type, but still offer this kind of operation.  that's enough though.

@dsonet you cant access the bits of Number without converting it to a Buffer. (10).toString(2) will give you the binary representation but not the underlying bits. And you have only 64 bits / 8 bytes.
const buf = new Buffer(8);
buf.writeDoubleBE/LE(myNumber, 0);
buf[0] // access first byte
...

@jsteemann I wouldn't allow concat hex data to a string. since the string could be invalid afterwards.
FILTER MATCH_HEX('ff', doc.binData) // omit 0x
FILTER CONTAINS_HEX('dadada', doc.binData)
CONCAT_HEX('dadada', doc.binData)

but some lower level datatype would be quite cool and fast?
Datatype Hex | Byte defined as a sequence of bytes.
Output to JSON could be ether hex or base64.

@baslr Ya you are right, but like you said it's underlying. So it's does have to expose to outside.
For now just can operate numbers is enough, like in mysql has BIT_COUNT,BIT_OR function. It's just a more efficient way to operate numbers.
But if  you want to offer complete byte type input and out, maybe can have a look in to CBOR? But to be honest I don't think it's super valuable to arangodb. Like handle binary files, I'd more prefer to use S3 or some other services that does offer CDN,etc ....

But since nowadays JavaScript can manipulate binary data in most environments, so offer binary data access in arango is also reasonable. My only concern is don't make it over complicated, keep it simple and fast.
Thanks for all your great work.

@Simran-B I found better use cases here. Please take a look. https://dev.mysql.com/doc/refman/5.7/en/bit-functions.html
Thanks.

@dsonet ArangoDBs internal document representation is actually https://github.com/arangodb/velocypack and vpack supports binary blobs. Maybe I will program a poc for binary data support if I'm step through the code.

@baslr I see velocypack is for storage, COBR is more towards transporting.

vpack can also be used to stream data. They are both data formats. So you can store it on disk or transport it. Just like protocol buffers. https://en.wikipedia.org/wiki/Comparison_of_data_serialization_formats

I just remember another use case: store password hashes or more general store hashes.

Okay, so there are actually two requests:

Support for binary operations in AQL (binary and, or, xor, not, nand; bit shift and roll)
Support for storing binary data in the database (BLOBs) and ways to transport this in and out of ArangoDB

My thoughts:

can be implemented as user-defined AQL functions. JavaScript provides the means to do this.
Is complicated with JSON as transportation format, and there is currently no good proposal how to handle binary data in AQL. For small chunks of data, base64 can be used already. For large BLOBs, it might be better to not support this through the REST interface, but the upcoming binary communication protocol only - for those who really need to store lots of binary data, it should be acceptable to use this interface instead, even if it's slightly more complicated than HTTP communication. But this needs more discussion with the team.

You can save below code to a file and run it like arangosh --javascript.execute path/to/file.js to register a set of binary op user functions:
const aqlfunctions = require('@arangodb/aql/functions');

const binaryFunctions = {
  and: (a, b) => a & b,
  or: (a, b) => a | b,
  xor: (a, b) => a ^ b,
  not: (a) => ~a,
  leftShift: (a, b) => a << b,
  rightShift: (a, b) => a >> b,
  rightShiftZeroFill: (a, b) => a >>> b
};

for (key in binaryFunctions) {
  if (binaryFunctions.hasOwnProperty(key)) {
    // AQL style function names: BINARY::RIGHT_SHIFT_ZERO_FILL()
    const name = 'BINARY::' + key.split(/(?=[A-Z])/).join('_').toUpperCase();
    aqlfunctions.register(name, binaryFunctions[key], true);
  }
}

The implementation is pretty straightforward but should work fine. Note that AND, OR and NOT are reserved keywords in AQL and you therefore have to call these functions like this:
 `BINARY::AND`(2, 3) // 2 (note the backticks around the function name)

Bit operators are a needed functionality for the DFIR (Digital Forensics and Indecent Response) community. We often have documents from logs that have flags as integers that we need to filter for. This is best done with operators such as AND and OR:
Here is an example of what is needed:
FOR record in usn_records
FILTER record.reason_flag & 256
RETURN record
While user functions can be used, I am not sure how it hurts performance. But, because bit operators are standard on many other databases, I think this is something AQL should account for.

I'm in the same boat as @devgc.  If I'm going to use arango as a back end for my digital forensics tools and research (and I really want to), bitfields are a "must have".

My above posted JavaScript implementation plus a yet-to-be-coded C++ equivalent could be added without much hassle as functions (BINARY_AND, BINARY_OR etc.).
But to support the use case presented by @devgc, we would have to extend the AQL syntax with operators (&, |, etc.) and implement optimizations to utilize indexes - although I'm not sure, if that is even possible with the existing indexes. Hash indexes store a hashed value, not the original value, thus we can't perform bitwise operations on them to test for e.g. value & 256. It would be possible with a skiplist index I guess, but might not perform that well...


WIF Fix bug where touch detection ignores touch start position

Current status after the 'Make _pointerDownPosition nullable' commit:

Add a startPosition to the HandleTouched method
Went through all calling code to add that argument. In some cases the MapControl did not have one yet. I added it in the most simple way.
Did not think everything through, but just got it to compile, more careful analysis is necessary
All the code involved screams 'REFACTOR ME!', but this is not something we can do in v4. There should not be too many changes. This is the main thing on the v5 roadmap.

Thanks for looking into this, @pauldendulk.
I just tested the branch at revision afd2d4198d9d1cf85261c5b0a855a7eabdad51fc, and it seems that actually none of the widgets show any reaction (it is not possible to trigger them by tapping/clicking).
You wrote that you did not think everything through yet. Apparently more adjustments are needed.

just tested the branch at revision afd2d41, and it seems that actually none of the widgets show any reaction (it is not possible to trigger them by tapping/clicking).
You wrote that you did not think everything through yet. Apparently more adjustments are needed.

Thanks for testing. What happened: The _pointerDownPosition was set after the check for widget touch. If the pointer down was on a widget the method returned and _pointerDownPosition was not set. I now moved this before the handling check for all platforms. I tested this only with mouse in WPF. Perhaps you could test this with touch on Android.
With all the different flows of touch in the different platforms it is likely there is still some problem somewhere. Perhaps we should set _pointerDownPosition to null in some scenarios. I guess every time there is a multi touch somewhere between touch down and touch up we should set _pointerDownPosition to null.

What happened: The _pointerDownPosition was set after the check for widget touch. If the pointer down was on a widget the method returned and _pointerDownPosition was not set. I now moved this before the handling check for all platforms. I tested this only with mouse in WPF. Perhaps you could test this with touch on Android.

Thanks for the update. I just tried 5fb4984e070507e87a2e6a25cda33dff998cbd32 on Android, and it seems to work pretty well. The widgets are functional again, and the bug reported in #2262 appears to be gone. Will try to do some more testing later.

On iOS (via Mapsui.Forms), the widgets work as well, but after tapping a widget (in a 'normal' fashion), the touch-navigation is still broken (or again?) in the same way described in #2262. After tapping a widget for the first time, single-finger dragging performs a zoom operation. After tapping a widget for the second time, single-finger dragging does not do anything any more.

We are keen to see this fix as I think we have been hit by this bug. The 4.0.0 release worked as expected (but had the rotation bug and the tiling spacing bug), but 4.1.0 and 4.1.1 (which fixed those bugs) seem to have broken the long touch events for us under Android.

So a colleague has tested this and it seems to only partially work for us. We will hopefully be able to explain more shortly - trying to pin down what is going wrong.

@pauldendulk yes - Xamarin Forms, but specifically only Android. We currently don't have an iOS version of the app in question.

To us - it looks like in the MapControl, where the SKTouchAction.Released is handled, there are no touches on a long press. So the fix we have is to remove the check from the outer if, then we inject a TouchEvent by setting the releaseTouched to a value if the TryRemove(..) fails. Don't know if that helps. My colleague has demonstrated it just now and it seems to work.  So - circa
(L118) We added a new field to record the time the initial Pressed event took place.
    private long _pointerDownTicks;

(L310) We store the time when we also set the Pressed location.
            if (e.ActionType == SKTouchAction.Pressed)
            {
                _pointerDownPosition = location;
                _pointerDownTicks = ticks;
            }

{L343) Moved the 'touchRemoved' check from the if statement so we can take action when it fails.
(L346 - L351) If we didn't receive a 'releasedTouch' TouchEvent from the _touches.TryRemove we create one with the Id of the Released, the  _pointerDownPosition and _pointerDownTicks;
We don't add this to _touches collection as we still want to enter the if on L353, but as we now have the 'releasedTouch', the code to check for long press should work.
           else if (e.ActionType == SKTouchAction.Released)
            {
                var touchRemoved = _touches.TryRemove(e.Id, out var releasedTouch);
                if (!touchRemoved)
                {
                    // There was no touch to remove so create a releasedTouch event to use in long tap checking
                    releasedTouch = new TouchEvent(e.Id, new MPoint(_pointerDownPosition?.X ?? 0, _pointerDownPosition?.Y ?? 0),
                        _pointerDownTicks);
                }

                if (_touches.Count == 0)
                {
                    // Is this a fling?

This seems to solve the issue we had at any rate.

Is there any update on this bug?

@memsom The long press seems a separate problem. I created a new issue for it: https://github.com/Mapsui/Mapsui/issues/2313.
What happens is that the ticks used to compare for long press are set on every move, not just on touch down. This was indeed in line with the fix that you suggested.

I will merge this one, as it is at least an improvement. The long press issue will be in a separate fix. Any remaining issues need to be fixed later.


Listen on IPv4 as well as IPv6

I'm using the matchbox.service template from contrib and it only comes up listening on IPv6. This prevents talking to the gRPC api from less capable networks.
matchbox.service
[Unit]
Description=CoreOS matchbox Server
Documentation=https://github.com/coreos/matchbox

[Service]
Environment="IMAGE=quay.io/coreos/matchbox"
Environment="VERSION=v0.6.1"
Environment="MATCHBOX_ADDRESS=<IP_ADDRESS>:8080"
Environment="MATCHBOX_RPC_ADDRESS=<IP_ADDRESS>:8081"
Environment="MATCHBOX_LOG_LEVEL=debug"
ExecStartPre=/usr/bin/mkdir -p /etc/matchbox
ExecStartPre=/usr/bin/mkdir -p /var/lib/matchbox/assets
ExecStart=/usr/bin/rkt run \
  --net=host \
  --inherit-env \
  --trust-keys-from-https \
  --mount volume=data,target=/var/lib/matchbox \
  --mount volume=config,target=/etc/matchbox \
  --volume data,kind=host,source=/var/lib/matchbox \
  --volume config,kind=host,source=/etc/matchbox \
  ${IMAGE}:${VERSION}
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet <IP_ADDRESS>/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp0s20f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 0c:c4:7a:e5:48:16 brd ff:ff:ff:ff:ff:ff
3: enp0s20f1: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond0 state DOWN group default qlen 1000
    link/ether 0c:c4:7a:e5:48:17 brd ff:ff:ff:ff:ff:ff
4: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 0c:c4:7a:e5:48:16 brd ff:ff:ff:ff:ff:ff
    inet <IP_ADDRESS>/31 brd <IP_ADDRESS> scope global bond0
       valid_lft forever preferred_lft forever
    inet <IP_ADDRESS>/31 brd <IP_ADDRESS> scope global bond0
       valid_lft forever preferred_lft forever
    inet6 2604:1380:4:1e00::1/127 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::ec4:7aff:fee5:4816/64 scope link
       valid_lft forever preferred_lft forever

netstat -tlnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp6       0      0 :::8081                 :::*                    LISTEN      1069/matchbox
tcp6       0      0 :::22                   :::*                    LISTEN      1/systemd
tcp6       0      0 :::8080                 :::*                    LISTEN      1069/matchbox

I'm fully expecting that I might be doing something wrong related to rkt when launching the matchbox container, but I haven't had much experience with practical IPv6 usage.

I imagine you're following the Installing on CoreOS docs based on your matchbox.service unit, except with Environment="MATCHBOX_RPC_ADDRESS=<IP_ADDRESS>:8081" and the log level set directly.
Running the same matchbox.service on a CoreOS stable 1409.7.0 box,
$ netstat -nltp
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp6       0      0 :::8081                 :::*                    LISTEN      -
tcp6       0      0 :::22                   :::*                    LISTEN      -
tcp6       0      0 :::8080                 :::*                    LISTEN      -

But its displayed like that because its listening on IPv4 and IPv6.
$ curl <IP_ADDRESS>:8080
matchbox
$ curl -g -6 http://[::1]:8080
matchbox

$ ss -nlt -6
State       Recv-Q Send-Q       Local Address:Port                      Peer Address:Port
LISTEN      0      128                     :::8081                                :::*
LISTEN      0      128                     :::22                                  :::*
LISTEN      0      128                     :::8080                                :::*

I think this stackoverflow answer is a nice writeup.

I'll have to try again. I had a problem reaching the gRPC service from my remote machine. Nothing should have been blocked, but a 2nd look over is warranted. Matchbox not listening on IPv4 seemed unlikely, but was all I had going.

Ok, lemme know if you can't get https://coreos.com/matchbox/docs/latest/deployment.html#verify to succeed.

Just set it up again and it looks like I had two errors from separate programs that lead me to believe I wasn't reaching the endpoint.

Using the terraform matchbox provider I had set the endpoint to https://matchbox:8081, this should have been matchbox:8081. The error returned was context deadline exceeded.

When I tried the openssl verify on the matchbox host it worked just fine. When I tried it from my provisioning computer I got 14675:error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version:/BuildRoot/Library/Caches/com.apple.xbs/Sources/OpenSSL098/OpenSSL098-64.50.6/src/ssl/s23_clnt.c:593:. This upon further inspection is about openssl trying an unsupported TLS/SSL version.

The openssl that comes bundled with macOS Sierra can not connect to the matchbox gRPC endpoint for verification. I tried with the homebrew installed openssl with success, but that version is not installed in a way to override the system version. Someone has to explicitly seek out to use that version.

What version of terraform-provider-matchbox are you using? I've made some improvements since the initial release that should show better error messages.
I'm on Linux and not too familiar with the macOS default openssl vs homebrew. Is it the version being old? The server is fairly strict about TLS certificate properties https://github.com/coreos/matchbox/blob/master/matchbox/tlsutil/info.go#L52

It is 0.2.1 for the terraform provider.
$ openssl version
OpenSSL 0.9.8zh 14 Jan 2016
$ ~/.homebrew/opt/openssl/bin/openssl version
OpenSSL 1.0.2l  25 May 2017

I don't think the bundled macOS version has TLS 1.2.

Ok, maybe I add a more explicit check in terraform-provider-matchbox for if someone puts an http or https protocol on the rpc endpoint so its easier to notice.

Hah, yeah just so used to everything dealing with terraform having an HTTP endpoint that I just typed it without thinking. I was checking the hostname for errors but didn't even think of the scheme for a while. Does gRPC have a scheme? grpc://?

grpc.Dial(target string, opts...) expects a host:port target per their example client.


get too many frames

I used codes as follows to get a buffer of batch frames of video. In reality, the video has 88 frames, however, I receive 89 frames actually.
process = (
ffmpeg
.input(file_path)
.output('pipe:', format='rawvideo', pix_fmt='rgb24', s="224x224")
.run_async(pipe_stdout=True, pipe_stderr=True)
)
while True:
    in_bytes = process.stdout.read(224 * 224 * 3 * batch_size)
    if not in_bytes:
        break

    _buffer = np.frombuffer(in_bytes, np.uint8)

    in_frame = (_buffer.reshape([-1, 224, 224, 3]))
    _real_size = in_frame.shape[0]
    print("_real_size:+++", _real_size )

it possibly reads the last empty array as a frame


Mention that x-amz-checksum-algorithm is not available

Which Cloudflare product does this pertain to?
R2
Existing documentation URL(s)
https://developers.cloudflare.com/r2/api/s3/api/
What changes are you suggesting?
After quite a bit of debugging I realised that we cannot use R2 yet because of this missing feature. Essentially, the lack of this feature means that clients must calculate a hash, which for some use-cases is a huge memory burden as it means reading the entire stream into memory before passing it on to R2.
Definitely worth noting in the docs that this header is not available on R2.
Additional information
https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html

Duplicated in #10623
(closing this one b/c that one will be fresher / more visible as it's closer to the top)


Memory consumption

Hello,
I'd like to ask if there are any tips to reduce memory consumption of the fmlrc2 process? I'm running it on human data, my FASTA is 111Gb and the index generated with the short reads is 69Gb, but the tool while running is taking 700Gb of RAM. Why does it take that much RAM? Would it be advisable to change some default settings or perhaps reduce the coverage of the short-reads when creating the indes?
Best regards,
Jonatan

Hello,
The short answer is that fmlrc2 (and it's predecessor fmlrc) both made the intentional choice to sacrifice memory consumption in favor of speed by default.  It's basically using an uncompressed index that typically consumes ~12 bits per symbol.  However, since it's uncompressed it means random accesses (and therefor queries) can be performed very quickly.
At this time, we don't really have a workaround in fmlrc2.  However, if this is a major block you can revert to fmlrc v1 with the -i option which will "build a sampled FM-index instead of bit arrays".  You can further reduce the memory consumption on that option by increasing the -F parameter.  However, this approach will be quite a bit slower and it will only get slower as you increase that -F parameter.
At the end of the day, it's a trade-off either way.  The community as a whole has tended towards sacrificing memory for speed, so that's what we did in fmlrc2 by default.  I have it on the backlog to add some options for those who want to make a different choice, but I'm not sure when I'll get around to implementing those in fmlrc2.

Hello,
How much more would be the time increase in changing from using fmlrc2 to using fmlrc1? Because as of now it's been running for 2 days using 20 CPUs and it looks like it'll take 2-3 more days to finish. And with it taking 700Gb of RAM we don't really see it feasible to implement it in our pipeline. It all will depend on how much improvement we get from the correction.
I'm still dumbfounded at how much RAM it uses when the inputs don't even account for 200Gb. Perhaps we should've gone with less coverage for the short-reads seeing as this is what you mentioned in an issue in fmlrc1 repo is what increases the memory.

How much more would be the time increase in changing from using fmlrc2 to using fmlrc1?

I ran a quick test using the E. coli benchmark data, and it ran in 48m40.583s wall clock with 344m30.815s CPU time on the same test laptop (seems like roughly 20x slower).  I use the default of -F 8, but you might want to increase that to at least 10 since your data seems to be very high coverage.  Additionally, you could split the long-reads into sub-files to further parallelize if the memory footprint becomes more manageable.

Perhaps we should've gone with less coverage for the short-reads seeing...

I'm not sure how much data you have on either the short- or long-read side of things, and I'm also not entirely sure what "normal" is for human data (the majority of our tests for fmlrc were non-human, so even at high (100x) coverage, the data was comparably small).  If you look at Ratatosk, they use:
"The short reads used are Illumina paired-end reads of length 151 bases with a mean coverage of 42x in the Icelandic trios and 61x in the Ashkenazim trio."
However, even with those datasets, the memory usage for fmlrc (this is v1) is still quite high as reported by their Supplement and even higher for the Ratatosk runs.  Seems like with human data, this is probably the norm.
Additionally, if you look at some of the other recent comparators, specifically this image, most of these are comparable or just worse on memory consumption.  However, none of these were tested with human data, so take caution interpreting it too far.

And with it taking 700Gb of RAM we don't really see it feasible to implement it in our pipeline.

Honestly, if you're memory limited, I think you're going to have a hard time running any of the higher accuracy tools.    Ratatosk (which was built specifically for human data) seems to use more memory than fmlrc v1.  Some of the tools from the survey paper are more memory efficient, but they come at a significant accuracy cost and I'm unsure of their performance on human data (fmlrc included, we never explicitly tested human).  Ultimately, you're probably going to have to make a tough decision about how much you care about accuracy v. compute costs, and then use that to guide whether you are looking for alternate tools/methods or alternate compute approaches (e.g. cloud or increasing your on-prem footprint).
Sorry these are probably not the answers you're hoping for, but I think that's just the state of the art currently.  Long-term (and really not that far away), I think all of most of these tools will disappear in favor of HiFi or similar style datasets.

Additionally, you could split the long-reads into sub-files to further parallelize if the memory footprint becomes more manageable.

Do you mean use the same short-read index but on subsets of the long-reads data? Is there any risk of this process causing an overcorrection of the long-reads?
We are not exactly really limited by memory, but we just have a 755Gb RAM cluster with 128CPUs. So it's not ideal for me to use the full resources. Unfortunately our sysadmin caused the process to stop so I'm trying the v1 with the -i option as you mentioned and using 90 cores. Probably going to take a week at this rate.
Would you perhaps recommend to better use Ratatosk since I'm handling human data and they've tested it with that type of data in mind? We really need the most accuracy we can get since we plan on calling for structural variants.

Do you mean use the same short-read index but on subsets of the long-reads data? Is there any risk of this process causing an overcorrection of the long-reads?

Yes exactly.  In fmlrc (v1 and v2), all long reads are corrected independently so you can split your long read file into as many subfiles as you want and recombine them later.  You should get identical answers as if you handed it the original file simply because the reads are all corrected independent of each other.
I would definitely recommend at least reading through the Ratatosk paper to determine if it's right for you (FWIW, I trust the authors of the paper/tool and they would likely be willing to help debug any issues you encounter as well).  There are trade-offs (i.e. Ratatosk is more costly from a compute perspective, but according to the paper, also more accurate), so again it's dependent on what's right for your specific experiment/pipeline.

Closing due to inactivity, feel free to re-open if you have more questions!


fix logging with Sphinx 2+

Fixes logging compat in Sphinx 2+, see https://github.com/NextThought/sphinxcontrib-programoutput/issues/29.

Thanks for reporting this, and thanks for offering the fix! There were other broken tests that needed to be tracked down, so we have #31 that resolves it all.


Package import failing in widgets/core/chooser.py

I'm attempting to use the<EMAIL_ADDRESS>branch, on macOS (10.15.7).  The example I'm working with needs a FileDrop, but I get this error:
Traceback (most recent call last):
  File "/Users/anj/Documents/workspace/mpt/venv/bin/mpt", line 33, in <module>
    sys.exit(load_entry_point('bl-mpt==1.1.6', 'console_scripts', 'mpt')())
  File "/Users/anj/Documents/workspace/mpt/venv/lib/python3.9/site-packages/gooey/python_bindings/gooey_decorator.py", line 128, in inner2
    return payload(*args, **kwargs)
  File "/Users/anj/Documents/workspace/mpt/venv/lib/python3.9/site-packages/bl_mpt-1.1.6-py3.9.egg/mpt/__main__.py", line 112, in main
  File "/Users/anj/Documents/workspace/mpt/venv/lib/python3.9/site-packages/gooey/python_bindings/gooey_decorator.py", line 95, in run_gooey
    from gooey.gui import application
  File "/Users/anj/Documents/workspace/mpt/venv/lib/python3.9/site-packages/gooey/gui/application.py", line 16, in <module>
    from gooey.gui.containers.application import GooeyApplication
  File "/Users/anj/Documents/workspace/mpt/venv/lib/python3.9/site-packages/gooey/gui/containers/application.py", line 15, in <module>
    from gooey.gui.components.console import Console
  File "/Users/anj/Documents/workspace/mpt/venv/lib/python3.9/site-packages/gooey/gui/components/console.py", line 6, in <module>
    from .widgets.basictextconsole import BasicTextConsole
  File "/Users/anj/Documents/workspace/mpt/venv/lib/python3.9/site-packages/gooey/gui/components/widgets/__init__.py", line 3, in <module>
    from .textfield import TextField
  File "/Users/anj/Documents/workspace/mpt/venv/lib/python3.9/site-packages/gooey/gui/components/widgets/textfield.py", line 4, in <module>
    from gooey.gui.components.widgets.core.text_input import TextInput
  File "/Users/anj/Documents/workspace/mpt/venv/lib/python3.9/site-packages/gooey/gui/components/widgets/core/__init__.py", line 1, in <module>
    from . chooser import Chooser, FileChooser, FileSaver, DirChooser, DateChooser, TimeChooser, MultiFileChooser, MultiDirChooser, ColourChooser
  File "/Users/anj/Documents/workspace/mpt/venv/lib/python3.9/site-packages/gooey/gui/components/widgets/core/chooser.py", line 11, in <module>
    from gui.util.filedrop import FileDrop
ModuleNotFoundError: No module named 'gui'

i.e. this import line fails: https://github.com/chriskiehl/Gooey/commit/402f0b0c8ab51b9cedae214f7f9d2c333705c0e7#diff-208ba1453473b5eace297a6fcab13fd74b9399a3ed8b23cd8c981c9e77f020b2R11
If I change it to from gooey.gui.util.filedrop import FileDrop it works fine.
I realise the 1.0.5-release branch is a work in progress, so please accept my apologies and close this if it's not appropriate.

Whoops! Thanks for the report. I accidentally introduce this mistake all the time on the dev branches due to how PyCharm autocompletes imports. I usually don't catch it until I start packaging things.
Will fix shortly!

This has now been fixed on the 1.0.5 branch!


Fails to import

I try to import a mod and this pops up
org.mozilla.javascript.WrappedException: Wrapped net.zhuoweizhang.mcpelauncher.MissingTextureException: The item icon jpkemty:0 does not exist (SuperJetPack.js#16)
at org.mozilla.javascript.Context.throwAsScriptRuntimeEx(Unknown Source)
at org.mozilla.javascript.MemberBox.invoke(Unknown Source)
at org.mozilla.javascript.FunctionObject.call(Unknown Source)
at org.mozilla.javascript.Interpreter.interpretLoop(Unknown Source)
at script(SuperJetPack.js:16)
at org.mozilla.javascript.Interpreter.interpret(Unknown Source)
at org.mozilla.javascript.InterpretedFunction.call(Unknown Source)
at org.mozilla.javascript.ContextFactory.doTopCall(Unknown Source)
at org.mozilla.javascript.ScriptRuntime.doTopCall(Unknown Source)
at org.mozilla.javascript.InterpretedFunction.exec(Unknown Source)
at net.zhuoweizhang.mcpelauncher.ScriptManager.initJustLoadedScript(ScriptManager.java:267)
at net.zhuoweizhang.mcpelauncher.ScriptManager$ParseThread.run(ScriptManager.java:192)
at java.lang.Thread.run(Thread.java:841)
Caused by: net.zhuoweizhang.mcpelauncher.MissingTextureException: The item icon jpkemty:0 does not exist
at net.zhuoweizhang.mcpelauncher.ScriptManager$NativeModPEApi.setItem(ScriptManager.java:3317)
at java.lang.reflect.Method.invokeNative(Native Method)
at java.lang.reflect.Method.invoke(Method.java:515)
... 11 more

That's an issue you should send to the creator of that mod.

I don't think it's the creator of the mod because it doesn't import for all of the mods
Sent from my Samsung Galaxy Tab®4
-------- Original message --------
From: Black-DeathPE<EMAIL_ADDRESS>Date:05/29/2016  1:55 PM  (GMT-08:00)
To: zhuowei/MCPELauncher<EMAIL_ADDRESS>Cc: mr-modz<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [zhuowei/MCPELauncher] Fails to import (#1035)
That's an issue you should send to the creator of that mod.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

No it's because you need to install the texture pack first then the script

Yes, you need them too but it doesn't still works...
Idk whymy coded custom items isn't works.
Trying to import Effect swords mod... then it show errors.

Oh wait! They need var things!!

I try to work it then it actually do!


Question about RsFolder parameters

Hi guys,
I've got a design question:
What was the reason to break with the PowerShell standard for parameters designating paths and introducing the name "RsFolder" for parameters pointing at a folder only?
Cheers,
Fred

We had a long debate with @SQLvariant about it and he consulted with additional MVPs and decided that was the more intuitive name for it as in some places you have the path for the local disk folder and then the RsFolder for those in SSRS, a good example for that case is Write-RsCatalogItem

Hm ... well, intuitive is a rather subjective thing. I'm one person, you asked many, so odds are you're right on that front (especially considering that I'm not the core audience for this module, coming very much from the PowerShell - rather than the dba - side of things).
However you now have created a naming conflict:

When targeting folders versus targeting content elements in RS (such as individual catalog items).
You've still got the path/itempath aliases, both for folders as well as content (unless you are willing to commit serious breaking changes)
You'll still conflict with the file system parameters (only now it's the aliases conflicting, rather than the main name). Would have gone with 'OutputPath' and 'InputPath' for the FS paths, given how they are the exception, not the rule.

I'll still argue it's the worse approach, even from an intuition perspective, because now people have to relearn habits:

Dbas will have to learn that the rest of PowerShell uses 'path' for paths, when they want to apply their PowerShell skills to something other than Reporting Services.
PowerShell users that have to pick up Reporting Services have to get used to the other terminology
(The fact, that aliases exist helps in not breaking existing scripts, however they are a lot worse to discover, so in most cases they won't help with interactive human usage).

Ah well, done is done, but now that I've had my say I can take it with equanimity ;)
Cheers,
Fred


Nullpointer Exception with onPanUpdate() calling a CustomPainter.

I have a simple Scaffold with a GestureDetector that has a CustomPaint child drawing a rectangle.
Now if I call onPanUpdate() and want to call another CustomPainter for example to paint over the rectangle based on the touches position a NullPointer Exception is thrown.
The method 'drawPoints' was called on null.
Receiver: null
Tried calling: drawPoints(Instance of 'PointMode', Instance(length:1) of '_GrowableList', Instance of 'Paint')

Here is a small Demo Code to illustrate the Issue, the position of the touchpoints from onPanUpdate are correctly passed to the painter, yet an exception is still thrown.
import 'package:flutter/material.dart';
import 'dart:ui' as ui;

void main() {
  runApp(MyApp());
}

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Crash',
      theme: ThemeData(
        primarySwatch: Colors.blue,
      ),
      home: MyHomePage(),
    );
  }
}

class MyHomePage extends StatefulWidget {
  @override
  _MyHomePageState createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  Canvas crashCanvas;

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text("crash"),
      ),
      body: Center(
        child: GestureDetector(
          child: CustomPaint(size: Size(200, 200), painter: MyPainter()),
          onPanUpdate: (details) {
            MyPainterDetails(details).paint(this.crashCanvas, Size(100, 100));
          },
        ),
      ), // This trailing comma makes auto-formatting nicer for build methods.
    );
  }
}

class MyPainterDetails extends CustomPainter {
  DragUpdateDetails details;
  MyPainterDetails(this.details);

  @override
  void paint(Canvas canvas, Size size) {
    print(details.globalPosition.dx.toString() +
        "," +
        details.globalPosition.dy.toString());
    canvas.drawPoints(
        ui.PointMode.points,
        [Offset(details.globalPosition.dx, details.globalPosition.dy)],
        Paint()..color = Colors.red);
  }

  @override
  bool shouldRepaint(CustomPainter old) {
    return true;
  }
}

class MyPainter extends CustomPainter {
  @override
  void paint(Canvas canvas, Size size) {
    canvas.drawRect(
        Offset(0, 0) & Size(200, 200), Paint()..color = Colors.orange);
  }

  @override
  bool shouldRepaint(CustomPainter old) {
    return true;
  }
}

The first time the Exception is throwns this also shows up:
When the exception was thrown, this was the stack
#0      Object.noSuchMethod (dart:core-patch/object_patch.dart:54:5)
#1      MyPainterDetails.paint
package:flutter_application_1/main.dart:57
#2      _MyHomePageState.build.<anonymous closure>
package:flutter_application_1/main.dart:40
#3      DragGestureRecognizer._checkUpdate.<anonymous closure>
package:flutter/…/gestures/monodrag.dart:436
#4      GestureRecognizer.invokeCallback
package:flutter/…/gestures/recognizer.dart:182
...
Handler: "onUpdate"
Recognizer: PanGestureRecognizer#f3a83
    debugOwner: GestureDetector
    start behavior: start

This also happens when using the "Touchable" package, which provides a very similar idea of drawing inside other shapes using the onPanUpdate function, maybe the error there can also help to narrow down the issues, see:
https://github.com/nateshmbhat/touchable/issues/19

Hi @Marvin10193
Thanks for filing the issue
Reproduced the issue using the following minimal code sample

minimal code sample
import 'package:flutter/material.dart';
import 'dart:ui' as ui;

void main() {
  runApp(MyApp());
}

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Crash',
      theme: ThemeData(
        primarySwatch: Colors.blue,
      ),
      home: MyHomePage(),
    );
  }
}

class MyHomePage extends StatefulWidget {
  @override
  _MyHomePageState createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  Canvas crashCanvas;

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text("crash"),
      ),
      body: Center(
        child: GestureDetector(
          child: CustomPaint(size: Size(200, 200), painter: MyPainter()),
          onPanUpdate: (details) {
            MyPainterDetails(details).paint(this.crashCanvas, Size(100, 100));
          },
        ),
      ), // This trailing comma makes auto-formatting nicer for build methods.
    );
  }
}

class MyPainterDetails extends CustomPainter {
  DragUpdateDetails details;
  MyPainterDetails(this.details);

  @override
  void paint(Canvas canvas, Size size) {
    print(details.globalPosition.dx.toString() +
        "," +
        details.globalPosition.dy.toString());
    canvas.drawPoints(
        ui.PointMode.points,
        [Offset(details.globalPosition.dx, details.globalPosition.dy)],
        Paint()..color = Colors.red);
  }

  @override
  bool shouldRepaint(CustomPainter old) {
    return true;
  }
}

class MyPainter extends CustomPainter {
  @override
  void paint(Canvas canvas, Size size) {
    canvas.drawRect(
        Offset(0, 0) & Size(200, 200), Paint()..color = Colors.orange);
  }

  @override
  bool shouldRepaint(CustomPainter old) {
    return true;
  }
}

logs
                    CodeSign /Users/tahatesser/StudioProjects/master_flutter/build/ios/Debug-iphonesimulator/Runner.app (in
                    target 'Runner' from project 'Runner')
                        cd /Users/tahatesser/StudioProjects/master_flutter/ios
                        export
                        CODESIGN_ALLOCATE\=/Volumes/Extreme/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/
                        usr/bin/codesign_allocate

                        Signing Identity:     "-"

                        /usr/bin/codesign --force --sign - --entitlements
                        /Users/tahatesser/Library/Developer/Xcode/DerivedData/Runner-fozhnrbljlzmspdvyyycqobvucew/Build/Inter
                        mediates.noindex/Runner.build/Debug-iphonesimulator/Runner.build/Runner.app.xcent --timestamp\=none
                        /Users/tahatesser/StudioProjects/master_flutter/build/ios/Debug-iphonesimulator/Runner.app

                    ** BUILD SUCCEEDED **
[ +123 ms]  └─Compiling, linking and signing... (completed in 4.7s)
[   +1 ms] Xcode build done.                                           20.7s
[   +4 ms] executing: rsync -av --delete
/Users/tahatesser/StudioProjects/master_flutter/build/ios/Debug-iphonesimulator/Runner.app
/Users/tahatesser/StudioProjects/master_flutter/build/ios/iphonesimulator
[ +510 ms] building file list ... done
                    Runner.app/
                    Runner.app/Info.plist
                    Runner.app/PkgInfo
                    Runner.app/Runner
                    Runner.app/Base.lproj/
                    Runner.app/Base.lproj/LaunchScreen.storyboardc/
                    Runner.app/Base.lproj/LaunchScreen.storyboardc/01J-lp-oVM-view-Ze5-6b-2t3.nib
                    Runner.app/Base.lproj/LaunchScreen.storyboardc/Info.plist
                    Runner.app/Base.lproj/LaunchScreen.storyboardc/UIViewController-01J-lp-oVM.nib
                    Runner.app/Base.lproj/Main.storyboardc/
                    Runner.app/Base.lproj/Main.storyboardc/BYZ-38-t0r-view-8bC-Xf-vdC.nib
                    Runner.app/Base.lproj/Main.storyboardc/Info.plist
                    Runner.app/Base.lproj/Main.storyboardc/UIViewController-BYZ-38-t0r.nib
                    Runner.app/Frameworks/
                    Runner.app/Frameworks/App.framework/
                    Runner.app/Frameworks/App.framework/App
                    Runner.app/Frameworks/App.framework/_CodeSignature/CodeResources
                    Runner.app/Frameworks/App.framework/flutter_assets/AssetManifest.json
                    Runner.app/Frameworks/App.framework/flutter_assets/FontManifest.json
                    Runner.app/Frameworks/App.framework/flutter_assets/NOTICES.Z
                    Runner.app/Frameworks/App.framework/flutter_assets/kernel_blob.bin
                    Runner.app/_CodeSignature/CodeResources

                    sent 35368670 bytes  received 408 bytes  23579385.33 bytes/sec
                    total size is 101920179  speedup is 2.88
[   +6 ms] executing: xcrun simctl install D5A130FB-A402-4C16-8F33-7806BCE32682
/Users/tahatesser/StudioProjects/master_flutter/build/ios/iphonesimulator/Runner.app
[ +717 ms] executing: /usr/bin/plutil -convert json -o -
/Users/tahatesser/StudioProjects/master_flutter/build/ios/iphonesimulator/Runner.app/Info.plist
[  +12 ms] Exit code 0 from: /usr/bin/plutil -convert json -o -
/Users/tahatesser/StudioProjects/master_flutter/build/ios/iphonesimulator/Runner.app/Info.plist
[   +1 ms]
{"CFBundleName":"master_flutter","DTXcode":"1250","DTSDKName":"iphonesimulator14.5","UILaunchStoryboardName":"LaunchScreen","
CFBundleIcons~ipad":{"CFBundlePrimaryIcon":{"CFBundleIconFiles":["AppIcon20x20","AppIcon29x29","AppIcon40x40","AppIcon60x60",
"AppIcon76x76","AppIcon83.5x83.5"],"CFBundleIconName":"AppIcon"}},"DTSDKBuild":"18E182","CFBundleDevelopmentRegion":"en","CFB
undleVersion":"1","BuildMachineOSBuild":"20E241","DTPlatformName":"iphonesimulator","CFBundlePackageType":"APPL","UIMainStory
boardFile":"Main","CFBundleSupportedPlatforms":["iPhoneSimulator"],"CFBundleShortVersionString":"1.0.0","CFBundleInfoDictiona
ryVersion":"6.0","CFBundleExecutable":"Runner","DTCompiler":"com.apple.compilers.llvm.clang.1_0","UISupportedInterfaceOrienta
tions~ipad":["UIInterfaceOrientationPortrait","UIInterfaceOrientationPortraitUpsideDown","UIInterfaceOrientationLandscapeLeft
","UIInterfaceOrientationLandscapeRight"],"MinimumOSVersion":"9.0","CFBundleIdentifier":"com.nevercode.triage","UIDeviceFamil
y":[1,2],"DTPlatformVersion":"14.5","CFBundleSignature":"????","CFBundleIcons":{"CFBundlePrimaryIcon":{"CFBundleIconFiles":["
AppIcon20x20","AppIcon29x29","AppIcon40x40","AppIcon60x60"],"CFBundleIconName":"AppIcon"}},"DTXcodeBuild":"12E262","LSRequire
sIPhoneOS":true,"UISupportedInterfaceOrientations":["UIInterfaceOrientationPortrait","UIInterfaceOrientationLandscapeLeft","U
IInterfaceOrientationLandscapeRight"],"UIViewControllerBasedStatusBarAppearance":false,"NSBonjourServices":["_dartobservatory
._tcp"],"DTPlatformBuild":"18E182","NSLocalNetworkUsageDescription":"Allow Flutter tools on your computer to connect and
debug your application. This prompt will not appear on release builds."}
[   +5 ms] executing: xcrun simctl launch D5A130FB-A402-4C16-8F33-7806BCE32682 com.nevercode.triage --enable-dart-profiling
--enable-checked-mode --verify-entry-points --observatory-port=0
[ +307 ms] com.nevercode.triage: 35206
[        ] Waiting for observatory port to be available...
[ +537 ms] Observatory URL on device: http://<IP_ADDRESS>:64222/aGCMUubKHP8=/
[  +12 ms] Caching compiled dill
[ +216 ms] Connecting to service protocol: http://<IP_ADDRESS>:64222/aGCMUubKHP8=/
[ +221 ms] Launching a Dart Developer Service (DDS) instance at http://<IP_ADDRESS>:0, connecting to VM service at
http://<IP_ADDRESS>:64222/aGCMUubKHP8=/.
[ +117 ms] DDS is listening at http://<IP_ADDRESS>:64227/q3qE5iElKc8=/.
[  +92 ms] Successfully connected to service protocol: http://<IP_ADDRESS>:64222/aGCMUubKHP8=/
[  +27 ms] DevFS: Creating new filesystem on the device (null)
[  +22 ms] DevFS: Created new filesystem on the device
(file:///Users/tahatesser/Library/Developer/CoreSimulator/Devices/D5A130FB-A402-4C16-8F33-7806BCE32682/data/Containers/Data/A
pplication/E460CE6C-503D-41E4-9B1A-4DDD619A7CF0/tmp/master_flutterjBXV4r/master_flutter/)
[   +3 ms] Updating assets
[ +111 ms] Syncing files to device iPhone 12...
[   +2 ms] <- reset
[        ] Compiling dart to kernel with 0 updated files
[   +5 ms] <- recompile package:master_flutter/main.dart 52c91741-6f4b-42b5-b203-f924a0ea3387
[        ] <- 52c91741-6f4b-42b5-b203-f924a0ea3387
[ +146 ms] Updating files.
[        ] DevFS: Sync finished
[   +2 ms] Syncing files to device iPhone 12... (completed in 154ms)
[        ] Synced 0.0MB.
[   +1 ms] <- accept
[   +9 ms] Connected to _flutterView/0x7fba7801ae20.
[   +5 ms] Flutter run key commands.
[   +3 ms] r Hot reload. 🔥🔥🔥
[   +1 ms] R Hot restart.
[        ] h List all available interactive commands.
[        ] d Detach (terminate "flutter run" but leave application running).
[        ] c Clear the screen
[        ] q Quit (terminate the application on the device).
[        ] Running with unsound null safety
[        ] For more information see https://dart.dev/null-safety/unsound-null-safety
[   +1 ms] An Observatory debugger and profiler on iPhone 12 is available at: http://<IP_ADDRESS>:64227/q3qE5iElKc8=/
[ +771 ms] DevTools activation throttled until 2021-05-18 04:24:14.657303.
[ +523 ms] The Flutter DevTools debugger and profiler on iPhone 12 is available at:
                    http://<IP_ADDRESS>:9103?uri=http%3A%2F%2F<IP_ADDRESS>%3A64227%2Fq3qE5iElKc8%3D%2F
[+5001 ms] flutter:<PHONE_NUMBER>9414062,455.0
[  +82 ms]
           ══╡ EXCEPTION CAUGHT BY GESTURE ╞═══════════════════════════════════════════════════════════════════
           The following NoSuchMethodError was thrown while handling a gesture:
           The method 'drawPoints' was called on null.
           Receiver: null
           Tried calling: drawPoints(Instance of 'PointMode', Instance(length:1) of '_GrowableList', Instance
           of 'Paint')

           When the exception was thrown, this was the stack:
           #0      Object.noSuchMethod (dart:core-patch/object_patch.dart:63:5)
           #1      MyPainterDetails.paint (package:master_flutter/main.dart:56:12)
           #2      _MyHomePageState.build.<anonymous closure> (package:master_flutter/main.dart:39:39)
           #3      DragGestureRecognizer._checkUpdate.<anonymous closure> (package:flutter/src/gestures/monodrag.dart:445:55)
           #4      GestureRecognizer.invokeCallback (package:flutter/src/gestures/recognizer.dart:193:24)
           #5      DragGestureRecognizer._checkUpdate (package:flutter/src/gestures/monodrag.dart:445:7)
           #6      DragGestureRecognizer.handleEvent (package:flutter/src/gestures/monodrag.dart:298:9)
           #7      PointerRouter._dispatch (package:flutter/src/gestures/pointer_router.dart:94:12)
           #8      PointerRouter._dispatchEventToRoutes.<anonymous closure>
(package:flutter/src/gestures/pointer_router.dart:139:9)
           #9      _LinkedHashMapMixin.forEach (dart:collection-patch/compact_hash.dart:397:8)
           #10     PointerRouter._dispatchEventToRoutes (package:flutter/src/gestures/pointer_router.dart:137:18)
           #11     PointerRouter.route (package:flutter/src/gestures/pointer_router.dart:123:7)
           #12     GestureBinding.handleEvent (package:flutter/src/gestures/binding.dart:440:19)
           #13     GestureBinding.dispatchEvent (package:flutter/src/gestures/binding.dart:420:22)
           #14     RendererBinding.dispatchEvent (package:flutter/src/rendering/binding.dart:287:11)
           #15     GestureBinding._handlePointerEventImmediately (package:flutter/src/gestures/binding.dart:374:7)
           #16     GestureBinding.handlePointerEvent (package:flutter/src/gestures/binding.dart:338:5)
           #17     GestureBinding._flushPointerEventQueue (package:flutter/src/gestures/binding.dart:296:7)
           #18     GestureBinding._handlePointerDataPacket (package:flutter/src/gestures/binding.dart:279:7)
           #22     _invoke1 (dart:ui/hooks.dart:182:10)
           #23     PlatformDispatcher._dispatchPointerDataPacket (dart:ui/platform_dispatcher.dart:282:7)
           #24     _dispatchPointerDataPacket (dart:ui/hooks.dart:96:31)
           (elided 3 frames from dart:async)

           Handler: "onUpdate"
           Recognizer:
             PanGestureRecognizer#e95bc
           ════════════════════════════════════════════════════════════════════════════════════════════════════

Reproduced on the following channels

stable
master

✅
✅

Check flutter doctor -v outputs for each channel below

flutter doctor -v
[✓] Flutter (Channel stable, 2.0.6, on macOS 11.3.1 20E241 darwin-x64, locale
    en-GB)
    • Flutter version 2.0.6 at /Users/tahatesser/Code/flutter_stable
    • Framework revision 1d9032c7e1 (3 weeks ago), 2021-04-29 17:37:58 -0700
    • Engine revision 05e680e202
    • Dart version 2.12.3

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
    • Android SDK at /Volumes/Extreme/SDK
    • Platform android-30, build-tools 30.0.3
    • ANDROID_HOME = /Volumes/Extreme/SDK
    • Java binary at: /Applications/Android
      Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.8+10-b944.6916264)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS
    • Xcode at /Volumes/Extreme/Xcode.app/Contents/Developer
    • Xcode 12.5, Build version 12E262
    • CocoaPods version 1.10.1

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 4.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.8+10-b944.6916264)

[✓] VS Code (version 1.56.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.22.0

[✓] Connected device (4 available)
    • Taha’s iPhone (mobile) • 00008020-001059882212002E            • ios
      • iOS 14.6
    • iPhone 12 (mobile)     • D5A130FB-A402-4C16-8F33-7806BCE32682 • ios
      • com.apple.CoreSimulator.SimRuntime.iOS-14-5 (simulator)
    • macOS (desktop)        • macos                                • darwin-x64
      • macOS 11.3.1 20E241 darwin-x64
    • Chrome (web)           • chrome                               •
      web-javascript • Google Chrome 90.0.4430.212

• No issues found!

[✓] Flutter (Channel master, 2.3.0-2.0.pre.240, on macOS 11.3.1 20E241
    darwin-x64, locale en-GB)
    • Flutter version 2.3.0-2.0.pre.240 at /Users/tahatesser/Code/flutter_master
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision e364e30c90 (11 hours ago), 2021-05-16 19:40:00 -0700
    • Engine revision 193be59942
    • Dart version 2.14.0 (build
      2.14.0-edge.a527411e5100a0a4f48c4009087a1b988aa784af)

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
    • Android SDK at /Volumes/Extreme/SDK
    • Platform android-30, build-tools 30.0.3
    • ANDROID_HOME = /Volumes/Extreme/SDK
    • Java binary at: /Applications/Android
      Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.8+10-b944.6916264)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS
    • Xcode at /Volumes/Extreme/Xcode.app/Contents/Developer
    • Xcode 12.5, Build version 12E262
    • CocoaPods version 1.10.1

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 4.2)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.8+10-b944.6916264)

[✓] IntelliJ IDEA Community Edition (version 2021.1.1)
    • IntelliJ at /Volumes/Extreme/IntelliJ IDEA CE.app
    • Flutter plugin version 55.1.5
    • Dart plugin version 211.7233

[✓] VS Code (version 1.56.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.22.0

[✓] Connected device (4 available)
    • Taha’s iPhone (mobile) • 00008020-001059882212002E            • ios
      • iOS 14.6
    • iPhone 12 (mobile)     • D5A130FB-A402-4C16-8F33-7806BCE32682 • ios
      • com.apple.CoreSimulator.SimRuntime.iOS-14-5 (simulator)
    • macOS (desktop)        • macos                                • darwin-x64
      • macOS 11.3.1 20E241 darwin-x64
    • Chrome (web)           • chrome                               •
      web-javascript • Google Chrome 90.0.4430.212

• No issues found!

In both of these examples crashCanvas is never initialized, so it will be null. Also, you cannot just call the paint method of a CustomPainter. A CustomPainter is usually used as an argument to a CustomPaint widget, see https://master-api.flutter.dev/flutter/widgets/CustomPaint-class.html.

I know this is not the place to ask for help with personal code,
but even if the Canvas would be correctly initialized, then still currently there does not exist a smart/easy way to allow drawing inside another CustomPaint object using a gestureDetector as we cannot simply call the paint method of a CustomPainter as you explained, correct?


Gallery, Shrine, Tablet: Layout of Shrine home

Shrine on a Nexus 9:

This is probably the same as https://github.com/flutter/flutter/issues/5135.

Or the same as https://github.com/flutter/flutter/issues/5134.


bump dependencies

@BalduinLandolt Before I make a proper release, I wondered if the other dependencies were up to date, because dependabot is configured to only update security-relevant dependencies. I realized that the dependencies were very much outdated.

@BalduinLandolt Before I make a proper release, I wondered if the other dependencies were up to date, because dependabot is configured to only update security-relevant dependencies. I realized that the dependencies were very much outdated.
How reliable are the automated tests? I haven't tried out if this PR breaks something...

I can't really say, but if I had to guess, I wouldn't trust the tests too much. But at the same time, the ark resolver is so simple that I doubt there is much one can break by changing dependencies. So I think it should be fine (and otherwise we re-deploy the old version) - maybe you can test a couple of different ARKs manually, after it's deployed?


Bug: Not bug, but Fix Documentation, please

In the validation section:
less_than  ->  Fails if field is greater than or equal to the parameter value or not numeric.
less_than_equal_to  ->  Fails if field is greater than the parameter value or not numeric.
must be reverse!

Not a doc error. I think you are confused.
Let us say you have a param value of 5 and test values of 5 and 6:
// not the official syntax though
// TESTVAL OPERATOR[PARAMVAL]
5 less_than[5] // false
6 less_than[5] // false
5 less_than_equal_to[5] // true
6 less_than_equal_to[5] // false

As @paulbalandan pointed out - documentation is fine.


Option to delete some/all of the history commands

Thanks for the good work that has been put into this plugin, it has become an essential part of my daily workflow and a real life saver!
It would be nice to have an option to delete some/all of the command history due to typos, temporary tests, security concerns & other various reasons, to keep the workflow more focused. Right now we have to go to the workspace.xml to modify it, which is not a very pleasant maneuver to do regularly.

+1 on this! Myself and other colleagues have been wanting this feature as well.

Any updates on this @DecoyRS? Would still really like to have this feature added if possible. Would save a lot of headache of manually having to find and edit the config file all the time

Also it would be better to have a button to toggle the history recording behavior, when toggled off, any commands used will not be recorded into history.


Kong 0.10.3 failed to use cassandra cluster

The problem seems localized on creation (POST) through the admin kong's API.
I can't tell if this can also happen for kong's frontend and/or other methods.
The problem can be reproduced for apis, consumers and plugins endpoints (at least).
We detect the problem when testing production containers (smartOS for cassandra + LX debian 8/jessie for Kong) on the servers.
Summary
For all the tests we only used one Kong instance.
Tests done with 3 configurations:
1- cassandra alone (=> no cluster mode activated)
2- cassandra x3 Network topology (=> cluster replication strategy)
3- cassandra x3 Simple strategy (=> cluster replication strategy)
Testing tools:
1- kongfig (only 22 APIs creation, but also have consumers and plugins init)
2- bash script capable of creating X apis endpoints in kong (only create APIs endpoints)
Note: same observation have been made with a bash script for consumers creation (consumer and jwt + acls configurations) which means there is a risk when creating consumers :(
Use case cassandra alone:
Whatever the used tool no fails happened.
What we can observe is after the creation of the 200nth endpoint the creation slows down.
The bash were able to create 2000 APIs endpoint without any issue other than the creation slow down.
Use case cassandra x3 Network topo (replication factor 2, dc1:2)
Kongfig:
We can observe error 500 coming back from kong randomly (means sometime it's on the first creation request, sometime later).
Bash script:
It acts like the kongfig tool.
Bash script with sleep of 1s between creations: Better but still fails. Prabably not related to high requests pressure.
Use case cassandra x3 Simple strategy  (replication factor 2)
Kongfig:
same observation as for network topo strategy replication factor 2
Bash script:
same observation as for network topo strategy replication factor 2
Bash script with sleep of 1s between creations:
Endpoint #288 failed to create.
For one run the endpoint #1 also failed.
For one run the 300 endpoints were created correctly.
Use case cassandra x3 Network topo (replication factor 1, dc1:1)
Kongfig:
Worked well.
Bash script:
Worked well.
Use case cassandra x3 Simple strategy  (replication factor 1)
Kongfig:
Worked well.
Bash script:
Worked well.
Use case cassandra x3 Network topo (replication factor 1, 2 DataCenters, dc1:1 and dc2:1)
Kongfig:
Failed with random 500 errors
Bash script:
Failed with random 500 errors
Use case cassandra x3 Network topo (replication factor ~2, 2 DataCenters, dc1:2 and dc2:1)
Kongfig:
Failed with random 500 errors
Bash script:
Failed with random 500 errors.
Happens more often with this configuration.
Use case cassandra x3 Network topo (replication factor ~2, 2 DataCenters, dc1:2,dc2:1, DCAwareRoundRobin => dc2)
Kong is configured with a dc aware round robin + is member of dc2 which hold only 1 Cassandra.
Kongfig:
Worked well.
Bash script:
Fail later, usually after more 100+ APIs endpoints creation.
KONG 0.11.0rc2 Use case cassandra x3 Network topo (replication factor ~2, 2 DataCenters, dc1:2 and dc2:1)
Tested with Kong performing a round robin and DC aware round robin over cassandra instances.
Kongfig:
Worked well.
Bash script:
Worked well.
APIs usage test:
After patching the file /usr/local/share/lua/5.1/kong/core/plugins_iterator.lua with the 0.11.0rc1 version of the file, the APIs worked well.
Note: Migrations (Cassandra schema preparation) have to do be ran manually, but it works better than with 0.10.3.
No migration failed during the multiple tries I did.
Example of error obtained when receiving the 500
For all cases (apis, plugins, consumers) it was related to a nil value into a marshall_event
2017/07/31 13:27:39 [error] 8964#0: *49591 lua coroutine: runtime error: /usr/local/share/lua/5.1/kong/dao/schemas/plugins.lua:55: attempt to index local 'plugin_t' (a nil value)
stack traceback:
coroutine 0:
        /usr/local/share/lua/5.1/kong/dao/schemas/plugins.lua: in function 'marshall_event'
        /usr/local/share/lua/5.1/kong/dao/dao.lua:68: in function 'event'
        /usr/local/share/lua/5.1/kong/dao/dao.lua:134: in function 'insert'
        /usr/local/share/lua/5.1/kong/api/crud_helpers.lua:167: in function 'post'
        /usr/local/share/lua/5.1/kong/api/routes/apis.lua:48: in function </usr/local/share/lua/5.1/kong/api/routes/apis.lua:47>
coroutine 1:
        [C]: in function 'resume'
        /usr/local/share/lua/5.1/lapis/application.lua:393: in function 'handler'
        /usr/local/share/lua/5.1/lapis/application.lua:130: in function 'resolve'
        /usr/local/share/lua/5.1/kong/api/init.lua:65: in function 'handler'
        /usr/local/share/lua/5.1/lapis/application.lua:130: in function 'handler'
        /usr/local/share/lua/5.1/lapis/application.lua:163: in function </usr/local/share/lua/5.1/lapis/application.lua:159>
        [C]: in function 'xpcall'
        /usr/local/share/lua/5.1/lapis/application.lua:159: in function 'dispatch'
        /usr/local/share/lua/5.1/lapis/nginx.lua:214: in function 'serve'
        content_by_lua(nginx-kong.conf:161):10: in function <content_by_lua(nginx-kong.conf:161):1>, client: <IP_ADDRESS>, server: kong_admin, request: "POST /apis/3a8b608a-5a72-4813-b35f-c3dac7dc69a6/plugins HTTP/1.1", host: "kong-01.ms.virtua.ch:8001"

Steps To Reproduce

Check that gist for information on what I played with: https://gist.github.com/thochra/fbfe8c42072a91fcf934d9b0cd737d74

Additional Details & Logs

Kong version 0.10.3
Operating System: docker kong:0.10.3

For information moved on 0.11.0rc2 because the issue seems to not happening there.

Closing seems as per your comment, this seems to be resolved in Kong 0.11. Thanks!


[FEATURE REQUEST] Please create binary release

Hello! Thanks for pint.
I try add pint to https://github.com/devops-works/binenv
But project dont have binary release.
Please create binary release.

https://github.com/cloudflare/pint/releases/tag/v0.1.1


YARN-11139. Wait one millisecond in DelegationTokenRenewerPoolTracker

Description of PR
Added a wait statement to reduce the CPU usage
How was this patch tested?
The issue owner already tested it. You can see the results over there
For code changes:

[x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
1m 34s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+0 :ok:
codespell
0m  0s

codespell was not available.

+0 :ok:
detsecrets
0m  0s

detect-secrets was not available.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
42m 38s

trunk passed

+1 :green_heart:
compile
1m 18s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
compile
1m  5s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
checkstyle
1m  4s

trunk passed

+1 :green_heart:
mvnsite
1m 13s

trunk passed

+1 :green_heart:
javadoc
1m  8s

trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
0m 55s

trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
2m 22s

trunk passed

+1 :green_heart:
shadedclient
24m 27s

branch has no errors when building and testing our client artifacts.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 57s

the patch passed

+1 :green_heart:
compile
1m  4s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javac
1m  4s

the patch passed

+1 :green_heart:
compile
0m 56s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
javac
0m 56s

the patch passed

+1 :green_heart:
blanks
0m  0s

The patch has no blanks issues.

+1 :green_heart:
checkstyle
0m 46s

the patch passed

+1 :green_heart:
mvnsite
0m 59s

the patch passed

+1 :green_heart:
javadoc
0m 45s

the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
0m 42s

the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

+1 :green_heart:
spotbugs
2m  4s

the patch passed

+1 :green_heart:
shadedclient
24m 30s

patch has no errors when building and testing our client artifacts.

_ Other Tests _

+1 :green_heart:
unit
101m  4s

hadoop-yarn-server-resourcemanager in the patch passed.

+1 :green_heart:
asflicense
0m 43s

The patch does not generate ASF License warnings.

211m 39s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4465/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/4465

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets

uname
Linux 2e0e28371c1a 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8 19:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / 08cb7bbd83388f7b686a66570fb28fff4be423b2

Default Java
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu<IP_ADDRESS> /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4465/1/testReport/

Max. process+thread count
893 (vs. ulimit of 5500)

modules
C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4465/1/console

versions
git=2.25.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.


kgo.LiveProduceConnection option for low-latency settings

We are trying to use franz-go in a low-latency setting.
With a normal, steady flow of produced messages, we're seeing latencies of 5ms through the system which is nice.  However, there are two cases where messages have 40-120ms latency because they need to wait for a TCP connect and a SSL handshake and auth:

The first message sent triggers broker.loadConnection() and creates the broker.cxnProduce connection on the brokers before sending.
There is a lull in messages for longer than cfg.connIdleTimeout. This causes the broker.cxnProduce to be reaped and scenario 1 reapplies on the next message.

A config option like kgo.LiveProduceConnection that would create broker.cxnProduce on init and shield it from the connection reaper, would solve our problem.

Thanks for following up!
Ensuring connection to all brokers would be necessary for our use case, yes.  The ideal is to send an "apiRequest" or other heartbeat-type message to each broker on startup and at a regular interval.
We've done a lot more testing on our end, and found that the brokers also have connection reapers defaulting to 10 minutes. While these can be configured off, it is only via undocumented behavior and some Kafka providers (most importantly ours) doesn't allow it.
Our topics have 12 partitions on a cluster of 3 brokers.
Currently we have it working well by producing dummy messages every 9 minutes and sending one each to an arbitrary partition on each of the brokers.  This maintains the connection indefinitely and we are getting low latency behavior.  It would however be more clean if we could maintain a connection without piling unnecessary messages on the topics.

Have you tried adjusting ConnIdleTimeout? Is that an option?

Some of our producers' traffic patterns have days and hours between messages, so even the max value of 15 minutes is insufficient.
If kgo's configuration tolerance were extended to high values or even 0=disabled, we can use it like we currently use librdkafka. That is, heartbeat messages every 30 minutes to avoid the server's connection reaper's max tolerance of 60 minutes.
But our hope is to be able to maintain live connections even without generating messages on a heartbeat topic. :)
Allowing us to send ApiRequest messages to individual brokers in the cluster would be an alternative that gives us the ability to solve our problem.

PTAL at the function introduced in this branch:
https://github.com/twmb/franz-go/commit/d0d94b23f8b3cbc9b735a81facb0d5569972fe9d


AbsractQueryBuilder moved to org.workflow.core.service

SonarQube analysis reported 4 issues

 1 major
 2 minor
 1 info

Watch the comments in this conversation to review them.
3 extra issues
Note: The following issues were found on lines that were not modified in the pull request. Because these issues can't be reported as line comments, they are summarized here:

 DefaultAppRequestBuilder.java#L42: Remove this unused method parameter "userProfile".
 DefaultAppRequestBuilder.java#L29: Remove the declaration of thrown exception 'com.wso2telco.core.dbutils.exception.BusinessException', as it cannot be thrown from constructor's body.
 DefaultAppRequestBuilder.java#L143: Complete the task associated to this TODO comment.

SonarQube analysis reported 3 issues

 1 major
 1 minor
 1 info

Watch the comments in this conversation to review them.
2 extra issues
Note: The following issues were found on lines that were not modified in the pull request. Because these issues can't be reported as line comments, they are summarized here:

 DefaultAppRequestBuilder.java#L41: Remove this unused method parameter "userProfile".
 DefaultAppRequestBuilder.java#L142: Complete the task associated to this TODO comment.


Scheduling messages - Semantics and middleware

We now have two different PR's for scheduling messages.
Before picking an implementation, we should think about middleware semantics.
e.g.
Let's say we have a tracing middleware.
And now someone tries to schedule a message in 10 seconds.
Should that message go through the middleware?
Should the trace be continued from the original point forward to the receiver of the scheduled message?
This relates to #95  #69

cc @jrouaix @raskolnikoov @cpx86

Another thing to consider.
If we want to scheduler to use the middleware.
Lets say an actor schedules a message in 10 seconds, then the actor is stopped.
Should the message still be sent?
Should the message still go through the now dead middleware?

yep so, after comments from @cpx86 (https://github.com/AsynkronIT/protoactor-dotnet/pull/69#issuecomment-286994606) the implementation I proposed is using the SenderMiddleware.
Basically some extensions to the IContext (https://github.com/Bee-Htcpcp/protoactor-dotnet/blob/c94d5dea71e639e6902e10271cfa56f1adec5514/src/Proto.Scheduling/SchedulingExtensions.cs) wrapping the message to a "DelayMessage" instance, and tell to self this message.
Then the middleware implementation catching this DelayMessage and use the IScheduler implementation given to Schedule the message.
 Because in my opinion no one but the actor should delay something to it. (https://github.com/AsynkronIT/protoactor-dotnet/pull/69#issuecomment-286750465)
The activation is not yet covered, right now, a message delayed to a stopped actor will end in the dead letters (tested here : https://github.com/Bee-Htcpcp/protoactor-dotnet/blob/c94d5dea71e639e6902e10271cfa56f1adec5514/tests/Proto.Scheduling.Tests/InMemorySchedulerTests.cs#L59)
I proposed to have 2 pipes : one Transcient & one Persisted, right now they are implemented just the same but we could introduce slight changes.
I don't know yet how this will integrate with "named actors" "cluster activation" in my opinion :
https://github.com/AsynkronIT/protoactor-dotnet/pull/69#issuecomment-286767424

If a transient delayed message comes and the actor is no more alive, the message should be dropped, not even appearing in the dead letters
If a Persisted message comes and the actor is no more alive, the actor should be respawned and state restored (don't know how to implement it yet)

@rogeralsing

Someone tries to schedule a message in 10 seconds. Should that message go through the middleware? Should the trace be continued?

Depends on whether the tracer supports asynchronous messages, and it's mostly up to the tracer middleware to try to decide what to do about it.

The actor is stopped. Should the message still be sent? Should the message still go through the now dead middleware?

No. If the actor is stopped and someone calls e.g. context.Tell I think the best would be if it throws an error, so whoever is trying to send the message can detect that the context is actually dead.
@jrouaix

If a transient delayed message comes and the actor is no more alive, the message should be dropped, not even appearing in the dead letters

I think it would still be useful to put the messages in dead letter, since there might be subscribers to the dead letter event stream that are interested in knowing when this happens.

If a Persisted message comes and the actor is no more alive, the actor should be respawned and state restored (don't know how to implement it yet)

I think due to how supervision works, this might not be possible to do. If an actor died because it failed repeatedly, that means the supervisor decided that there's no way to revive the actor, either because the error was unrecoverable or because the actor was restarted repeatedly but still kept failing. So if an actor is no longer alive, I think it should stay that way or be respawned manually.

I like the idea when it's an extension to IContext instead of an interface that we need to pass to the actor and every scheduler needs to act like the other scheduler library out there. For example if I like to use Quartz or FluentScheduler I could use them in the same actor with these example usage:
context.FluentScheduler.Schedule<MyJob>().ToRunNow().AndEvery(2).Seconds();

or
IJobDetail job = JobBuilder.Create<HelloJob>()
                    .WithIdentity("job1", "group1")
                    .Build();

ITrigger trigger = TriggerBuilder.Create()
    .WithIdentity("trigger1", "group1")
    .StartNow()
    .WithSimpleSchedule(x => x
        .WithIntervalInSeconds(10)
        .RepeatForever())
    .Build();

context.Quartz.ScheduleJob(job, trigger);

I very much like the idea of not having a shared interface and instead having usecase specific impls like @raskolnikoov describe
duplication is far cheaper than the wrong abstraction.

Well, the more I think about it, the more I think you're right. Too risky too bound the core librairy with some weak abstraction.
But this feature is still pretty needed, how do you want to implement it ?
As discussed here (https://github.com/AsynkronIT/protoactor-dotnet/issues/99) where do you want we push PR ?


[6.0] use unified overload data for initial overload variants

Explanation: Uses a new SymbolKit implementation of overload grouping to fix some bugs about overload handling.
Scope: Bug fixes for an experimental feature
Issue: rdar://127130136
Original PR: https://github.com/apple/swift-docc/pull/907
Risk: Low. The major implementation happened in SymbolKit, and the resulting behavior change only happens for the experimental overloads feature.
Testing: Automated testing has been added to ensure the expected behavior.
Reviewer: @patshaughnessy @mayaepps @d-ronnqvist

@swift-ci Please test

@swift-ci Please test


Bump: Cap md_toc requirement below 9.0.0

Change Summary
Cap requirement of md_toc below 9.0.0
This is due to API change in 9.0.0: cleaner module imports. For example: md_toc.build_toc becomes md_toc.api.build_toc.
Temp fix to unblock CI until we get #3837 reviewed and merged!
Component(s) name
All
How to test
CI Passes all tests

Closing this in favor of #3837


Allowing :only and :except regular expressions to match over request.fullpath - closes #184

...lpath - closes #184
this way I can do this via url parameter

This would be a breaking change for users with regexes using $ or ^. I am marking this as 1.0, but I think this looks good.

This is pretty old. I'm going to close it but if anyone wants to revive it, I'm happy to reopen it later on.


Simplify scoped manual repository

This PR inlines the arguments of SectionAssociationMarshaller and ManualPublishTaskAssociationMarshaller making the code a bit easier to follow. It's still not really clear to me why these marshallers exist but hopefully reducing some of the indirection will eventually let us work out where their behaviour really belongs.

Thanks @floehopper! I've addressed your feedback (and the discussion we had in person) in 	aa8e2ad.

I accidentally merged aa8e2ad which I had intended to fixup into 02c2361. I don't think it's a big enough problem to warrent rewriting master, but apologies!


Adds a first batch of technical documentation

Coverage remained the same at 95.46% when pulling ec967c55cb7fba5257c7c1e9f7d3e0ca418946ab on tech_doc into eb1ab9a2205e5023afac096229afe88470788d3f on master.


Change README.md links to English-language Amazon site

Since the rest of this document is in English, I switched the amazon links to e.g. https://aws.amazon.com/en/eks/ from https://aws.amazon.com/es/eks/

@GeoffMaciolek thank you!


Remove warning which is adding too much noise

Remove warning message
WARNING  autotvm:compile_engine.py:256 Cannot find config for target=llvm -keys=cpu -link-params=0, workload=('conv2d_NCHWc.x86', ('TENSOR', (1, 1, 32, 32), 'float32'), ('TENSOR', (8, 1, 5, 5), 'float32'), (1, 1), (0, 0, 0, 0), (1, 1), 'NCHW', 'NCHW', 'float32'). A fallback configuration is used, which may bring great performance regression.

I'm super happy with this, but I'll leave it to @icemelon9 and @merrymercy in case they have strong objections.

Well I don't really want this warning gone completely because it really delivers useful information especially when you are optimizing the performance. On the other hand, making them more concise or dump them to a log file are discussable.
p.s. auto-scheduler also has a similar warning, so if we decided to make a change, we should cover both.

I agree with @comaniac. While this warning is annoying during non-perf related development, I do check the absence of this warning after I apply a tuned log, to make sure that no default config is applied.
How about turn this into DEBUG?

I think we should keep the error message, but maybe we can improve it. Maybe say something like "tune your model for better performance" and then link to a page on tuning?

In summary:
(A) by @masahi: Turn the warning to DEBUG so that only the advance developers would see it by purpose. We could of course add a paragraph of how to enable this message in the tutorial.
(B) by @tkonolige: Improved the message to make it more informative. This is also a valuable direction so I try to make it more concrete. First, we remove the last scary sentence from the message:
WARNING autotvm:compile_engine.py:256 Cannot find config for target=llvm -keys=cpu -link-params=0, workload=('conv2d_NCHWc.x86', ('TENSOR', (1, 1, 32, 32), 'float32'), ('TENSOR', (8, 1, 5, 5), 'float32'), (1, 1), (0, 0, 0, 0), (1, 1), 'NCHW', 'NCHW', 'float32').

Second, when we are about to display this warning for the first time, we also display the message as Tristan mentioned:
One or more config are missing. Please tune your model for better performance (link).

(C) by myself: Combine both options. That is, we only show the overall warning once and put the detail schedule missing warnings to DEBUG.
One or more config are missing. Please tune your model for better performance (link), or use DEBUG logging level to see more details.

@rkimball @mbrookhart @masahi @tkonolige what do you folks think?

(C) sounds like a good option. I don't think we should say "config" in the error message. New users will not know what we mean. Maybe say "operators" instead?

(C) sounds like a good option. I don't think we should say "config" in the error message. New users will not know what we mean. Maybe say "operators" instead?

"missing operator" sounds like you need to add a new operator to TOPI, which is misleading. Instead, "effective schedule" is the most precise word, but simply using "schedule" is also good to me.

One or more operators have not been tuned. Please tune your model for better performance (link), or use DEBUG logging level to see more details.

One or more operators have not been tuned. Please tune your model for better performance (link), or use DEBUG logging level to see more details.

Ah sorry for misunderstanding. This message is good to me.

Another option is to detect when a user provides a tuning log. In that scenario the intention is for log entries to be applied, and the user likely should be notified on stdout that one more entries was not applied. Conversely if no log is provided, option (C) is likely safe.

Another option is to detect when a user provides a tuning log. In that scenario the intention is for log entries to be applied, and the user likely should be notified on stdout that one more entries was not applied. Conversely if no log is provided, option (C) is likely safe.

I don't think this can be achieved easily. This warning pops up in the fallback dispatch context, which only knows that the schedule is missed, but the fallback context has no idea whether there's another context beforehand.

Changed to print @tkonolige's message once if any of the kernels are untuned. If you enable DEBUG logging then the old, detailed message is printed.

@rkimball Can we actually fix the grammar of the original message, it is also poorly worded and one of my least favorite messages in the code base.
For example:
Cannot find tuning records for:
    target = `llvm -keys=cpu -link-params=0, workload=('conv2d_NCHWc.x86'`
    key = ('TENSOR', (1, 1, 32, 32), 'float32'), ('TENSOR', (8, 1, 5, 5), 'float32'), (1, 1), (0, 0, 0, 0), (1, 1), 'NCHW', 'NCHW', 'float32')
TVM will apply the fallback schedule configuration which may substantially impact performance.


ResetInactive

Hello,
uint32_t resetPeriod = 250;
I was wondering why reset period was kept this short? I tried with higher values and had no problems.
Thanks

There are some open points in your issue report: File? Line of code? Reset period of what?

Reset period of the DVM 1000, also named as "inactivity period" on some files. I don't think line of code relates as this a general question about the chip.
Thank you

Got many of these.
To maximize polling rate i keep the reset period low at 75ms to reduce time the system waits end of the reset_period. Try 500 and 1000 ms and nothing changed.
Why this happened?
Both tag and anchor have this issue:


add AutoAck support for Server

I don't know whether this is the cause but when running: go get github.com/micro/go-grpc, I'm getting this error:
# github.com/micro/go-plugins/server/grpc
../../../micro/go-plugins/server/grpc/grpc.go:532:19: sb.Options().AutoAck undefined (type server.SubscriberOptions has no field or method AutoAck)
../../../micro/go-plugins/server/grpc/subscriber.go:39:10: unknown field 'AutoAck' in struct literal of type server.SubscriberOptions

go get -u probably fixes that for you


Thumbnails

Description
Adds thumbnails to account list

Type of change (remove all that don't apply)

New feature (non-breaking change which adds functionality)

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

[X] Manually tested all use cases this could alter

Test Configuration:

OS: Windows

Browser: Chrome

Frontend: (remove all that don't apply)

[X] Development build

Backend: (remove all that don't apply)

[ ] No backend needed to test this
[ ] deployed production build
[ ] deployed development build
[X] deployed experimental build

Checklist: (remove all that don't apply)

[X] I added corresponding unit tests (if API code)
[X] I have performed a self-review of my own code
[X] My changes generate no new linting warnings or console warnings
[x] I have updated the CHANGELOG.md to include any changes made in this PR (add WIP to the top, if there is none already)

After dev merge:
Desktop

Mobile (profile picture hidden)


Add join us inline form

A section or popup which will have text box to enter mail id, this will help users to join the Google group
something similar to this one http://pune.python.org.in/

This is the form which we need to include somewhere,

Subscribe to <?= $group_title ?>

<form action="http://groups.google.com/group/<?= $group_name ?>/boxsubscribe">
<input name="email"/>
<input type="submit" value="Subscribe" name="sub"/>
<a href="http://groups.google.com/group/<?= $group_name ?>">Browse Archives
</form>

source: https://stackoverflow.com/a/18789315/6202405

Though we entered email id in form, we need to enter email id again after visiting http://groups.google.com/group/<?= $group_name ?>/boxsubscribe Any solution?

That answer is old, something at Google end must have been changed now. Better would be just pass that link and let the user fill out email address. Instead of inline form we will have a button or something.


Scraping without programming

https://docs.google.com/presentation/d/1zMvbeYhZZmv_PBsXIJAvwjr9i2UujEgp76DjQOA_-u4/edit#slide=id.p

I agree


Error provisioning for a single non-admin user

I was trying to sign up using a different tenant, regular user account. Instead of a consent dialog got this:
Sorry, but we’re having trouble signing you in.
We received a bad request.
AADSTS70001: Application with identifier XXX was not found in the directory YYY.onmicrosoft.com

Hey Darek. Is the application XXX set as a multi-tenant application in the azure portal?
What was the domain of the user you attempted to sign in as? Was it YYY.onmicrosoft.com?

Yes and yes.
Provisioning with an admin account works.
Sent from my Windows Phone

From: Danny<EMAIL_ADDRESS>Sent: 4/16/2015 10:11 AM
To<EMAIL_ADDRESS>Cc: Darek<EMAIL_ADDRESS>Subject: Re: [WebApp-MultiTenant-OpenIdConnect-DotNet] Error provisioning for a single non-admin user (#8)
Hey Darek. Is the application XXX set as a multi-tenant application in the azure portal?
What was the domain of the user you attempted to sign in as? Was it YYY.onmicrosoft.com?
—
Reply to this email directly or view it on GitHubhttps://github.com/AzureADSamples/WebApp-MultiTenant-OpenIdConnect-DotNet/issues/8#issuecomment-93787336.

Does tenant YYY allow regular users to consent to applications?  There is a config switch to set this on the Configure tab of the tenant in the Azure portal.
If that switch is set correctly, please provide a correlationId and timestamp for the error and I can take a deeper look.

Thanks, I checked the settings and they should be able to consent.
Additional technical information:
Correlation ID: 98b9b314-b558-430d-a0e1-790459bad062
Timestamp: 2015-04-16 06:10:49Z
AADSTS70001: Application with identifier d0a17f8c-4463-47c2-ae03-04601528c5a3 was not found in the directory fbtest.onmicrosoft.com
From: Danny Strockis<EMAIL_ADDRESS>Sent: Thursday, April 16, 2015 11:28 AM
To: AzureADSamples/WebApp-MultiTenant-OpenIdConnect-DotNet
Cc: Darek Grala
Subject: Re: [WebApp-MultiTenant-OpenIdConnect-DotNet] Error provisioning for a single non-admin user (#8)
Does tenant YYY allow regular users to consent to applications? There is a config switch to set this on the Configure tab of the tenant in the Azure portal.
If that switch is set correctly, please provide a correlationId and timestamp for the error and I can take a deeper look.
—
Reply to this email directly or view it on GitHubhttps://github.com/AzureADSamples/WebApp-MultiTenant-OpenIdConnect-DotNet/issues/8#issuecomment-93808710.

Ping

I'm having the same problem using NEW azure portal, when I register a new native application with all the permissions needed and I try to obtain the access token using:
        AuthenticationResult authResult = null;
        string resource = "https://management.core.windows.net/";
        var authenticationContext = new AuthenticationContext("https://login.windows.net/" + tenantId);
        try
        {
            authResult = authenticationContext.AcquireToken(resource,
                appId,
                new UserCredential(
                    partnerCenterUser,
                    partnerCenterPassword
                )
            );
        }
        catch (Exception ex)
        {
            throw;
        }
        return authResult;

I obtain the error message:
AADSTS70001: Application with identifier '4f7d5168-f754-42ed-a43d-1f0363ede30e' was not found in the directory tenantresellernew30.onmicrosoft.com
Trace ID: 182d19d7-68a7-4a48-bee1-f87486d31c66
Correlation ID: 75ef5bbb-06f9-4a06-a384-45860dae2568
Timestamp: 2016-10-13 07:32:25Z
I'm using sandbox account to perform the tests, but also in production we are facing this problem.
Any help is welcome because we are stucked and we need to integrate azure in our multi-tenant environment.

I have the same problem, does anyone have clue?

Hi!
I had this same problem and was able to resolve it. In the NEW portal you need to edit the manifest of the native application and set "availableToOtherTenants" flag to true. By default the NEW portal sets it to false (the old portal used to set it to true by default).
HTH

Thanks that helped resolve the problem

Hi all,
I was able to run this sample successfully from a commercial AD tenant and login with users from multiple tenants. However, when I try to register the app in a government tenant, I'm having this same problem as the original poster.

I have followed all the steps within the instructions.
I've taken the specific steps outlined for making it work in Azure Government.
Once completed, I can login with a user account in the government tenant that the app is registered but when attempting to login with a user from a different AD tenant I get the error (Application with identifier XXX was not found in the directory YYY).

Tenant Y allows users to consent to apps accessing company data on their behalf.
I have double-checked the app registration and the "availableToOtherTenants" flag is true.
Here is error including the correlation ID from my last attempt:
Correlation ID: f7b94d33-0aa9-4ea4-b2b2-13184d9d0f37
--
Timestamp: 2017-12-21 14:58:19Z
AADSTS70001: Application with identifier '9240c326-325a-48e2-8382-0efde95ee41e' was not found in the directory sasdomain1.onmicrosoft.com

Any help would be greatly appreciated.

Thanks that worked for me.

The sample has been heavily updated, including steps

register and consent the apps in multiple tenants
to deploy this in the sovereign clouds.
to deploy this on Azure
Please give the updated the sample a try and let us know on this thread if you have more issues.


Tautomer enumeration should remove stereo in all tautomers

The enumeration code does a good job of removing stereo from atoms/bonds directly affected by a tautomer transformation, but it should really remove stereo from all bonds/atoms that are affected in any tautomer.
Here's a simple example:
m = Chem.MolFromSmiles('COC(=O)[C@@H](N)CO')
tauts = enum.Enumerate(m)

The results are:

@mcs07 does this assessment seem reasonable to you?

Yeah, that's a good point.


Things need to be adjusted on every different phone

Hello,
I have 4 different Android devices and I am trying to use flutter_screenutil to make the UI responsive, but for all 4 different devices, I need to manually adjust the font, widget size etc. Is this how it is expected to work?
I was under impression that if I set UI on one of the phone size correctly, it should work correctly on any other phone or Screensize & all the UI will be responsive.
Please correct me if my understanding is wrong or if I am doing something wrong. It doesn't make sense to make adjustment for all different device as I think it beats the purpose of this utility. I am also new to Flutter - so if something basic is missing, please correct me.
I have following 4 devices:
Nokia:
I/flutter (15670): Device width px:1080.0
I/flutter (15670): Device height px:2154.0
I/flutter (15670): Device pixel density:2.625
TAB:
I/flutter (31840): Device width px:600.0
I/flutter (31840): Device height px:976.0
I/flutter (31840): Device pixel density:1.0
Mi Phone 1:
I/flutter ( 6127): Device width px:720.0
I/flutter ( 6127): Device height px:1280.0
I/flutter ( 6127): Device pixel density:2.0
Mi Phone 2:
I/flutter (24742): Device width px:1080.0
I/flutter (24742): Device height px:1920.0
I/flutter (24742): Device pixel density:3.0
Thanks for your help

AB

This is how it is set, used the first phone width and height for reference.
Widget build(BuildContext context) {
ScreenUtil.init(context,
width: 1080, height: 2154, allowFontScaling: false);

ScreenUtil.init(context,
width: (UI Design Width) , height: (UI Design Height), allowFontScaling: false);
It’s not the size of your phone, the size of the design draft is used here