KAFKA-12226: Prevent source task offset failure when producer is overwhelmed

Jira
When a task fails to commit offsets because all outstanding records haven't been ack'd by the broker yet, it's better to retry that same batch. Otherwise, the set of outstanding records can grow indefinitely and all subsequent offset commit attempts can fail. By retrying the same batch, it becomes possible to eventually commit offsets, even when the producer is unable to keep up with the throughput of the records provided to it by the task.
Two unit tests are added to verify this behavior.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@gharris1727 @tombentley @chia7712 anyone got a moment? 😃

Yeah, that's fair. I found https://github.com/apache/kafka/blob/2caef070d4199877337f59399563354e41b203f7/connect/runtime/src/test/java/org/apache/kafka/connect/runtime/distributed/DistributedHerderTest.java#L550-L639 which seems applicable here. I'll refactor the tests to use a similar approach. Thanks Greg!

The only consistent test failure for the last run appears unrelated to the changes here. See https://github.com/apache/kafka/pull/10140/checks?check_run_id=1917700907 and https://github.com/apache/kafka/pull/10077/checks?check_run_id=1877979511 for other instances of the same failure.
org.apache.kafka.connect.integration.InternalTopicsIntegrationTest.testCreateInternalTopicsWithDefaultSettings
org.opentest4j.AssertionFailedError: Condition not met within timeout 30000. Didn't find the topics [connect-storage-topic-connect-cluster-1, connect-config-topic-connect-cluster-1, connect-offset-topic-connect-cluster-1] ==> expected: <true> but was: <false>

Yeah, the unit tests for the worker classes in general can be a little gnarly. The boilerplate segments (for things like setting up the converter, transformation chain, task context, headers, topic tracking, offset buffering, status tracking, and performing metrics assertions) are in line with the other tests in this class, though. The only difference with the new unit test here is that we want to test task behavior under some pretty fine-grained circumstances, which is accomplished right now by setting up latches and awaiting them to ensure that the task (which is running on a separate thread) has reached certain points in its lifecycle and not gone any further. If you have suggestions for how to improve that I'm all ears!
I'm not sure what you're referring to with task restart--that's not tested for at the moment, and any tests for that would likely be dependent on the success (or lack thereof) of any prior offset commit attempts, which are already tested for. Can you clarify or provide a brief example?

I did not look at the tests closely yet but I thought sourceTask.stop(); (https://github.com/apache/kafka/pull/10112/files#diff-b8da514e331b4d8bff623626ee31a45048b753911193b564a9b65f02b97f53c8R922) was a key part of the test. Maybe it doesn't!
I hope to take another look some time next week.

Ah, gotcha! Yeah, SourceTask::stop isn't really a key part of the functional changes here or the tests added. The idea is just to prevent high-throughput source tasks whose producers are unable to keep up with them from entering a death spiral where they stop being able to commit offsets completely.
This is because right now, when an offset commit attempt fails because the current batch of source records wasn't flushed to Kafka in time, all backlogged records that were read from SourceTask::poll during that time get added to the batch, which just compounds the problem.
For example:

The source task generates 10000 records and the worker starts sending those to Kafka.
An offset commit is triggered on a separate thread and the worker waits for up to offset.flush.timeout.ms milliseconds for all of those records to be ack'd by the broker. In the meantime, on the main thread, the worker continues polling the source task for data.
The worker times out while waiting the original batch of 10000 records to be flushed; let's say that 5000 managed to be written successfully but 5000 remain unacknowledged. Additionally, during this time, the worker managed to poll an additional 1000 records from the task.

At this point, the current behavior is:

Abort the offset commit, and start a new batch consisting of the 5000 unacknowledged records from the previous batch and the 1000 records polled from the task during the failed offset commit attempt.
Continue adding records to that batch until the next offset commit attempt is triggered.

If the task is generating a steady throughput of 10000 records per offset commit attempt, and the worker's producer is only able to write 5000 of those before the offset commit attempt times out, the worker will never be able to successfully commit offsets for the task, even though there are plenty of records that have been sent to and ack'd by the broker.
The proposed behavior in the PR is:

Abort the offset commit, and keep the old batch of the 5000 unacknowledged records. Add the 1000 records polled during the failed offset commit attempt to a backlog.
Continue adding newly-polled records from the task to that backlog.
On the next offset commit attempt, only wait to flush out the records from the active batch (i.e., the 5000 unacknowledged records from step 4), and only write offsets for that batch.
If successful, use the backlog of records as the new batch of records. Otherwise, keep the same batch of records and continue adding newly-polled records to the backlog.

First of all, thanks for trying to fix this issue, @C0urante.
And thanks for your insight, @hachikuji. I agree that it seems like we should not have to block the offset commits until the full batch of records has been written to Kafka.
I suspect the current logic was written this way because it's the simplest thing to do, given that the source partition map and offset map in the source records are opaque, meaning we can't sort them and have to instead rely upon the order of the source records returned by the connector. And because the producer can make progress writing to some topic partitions while not making progress on others, it's possible that some records in a batch are written before earlier records in the same batch.
The bottom line is that we have to track offsets that can be committed using only the order of the records that were generated by the source task. The current logic simply blocks committing offsets until each "batch" of records is completely flushed. That way we can commit all of the offsets in the batch together, and let the offset writer rely upon ordering to use only the latest offset map for each partition map when we tell it to flush.
But, flushing offsets requires synchronization, and the current logic switches between the outstandingMessages and outstandingMessagesBacklog buffers to track the "batches" of records that have to complete for offset commits. It's really sort of a mess.
@hachikuji wrote:

The patch gets around the problem by relaxing offset.flush.timeout.ms a little bit. Rather than treating expiration of records as a fatal error, we continue to allow more time for outstandingMessages to be drained. This ensures that we do not have to wait for the messages from outstandingMessagesBacklog which are added while the flush is in progress.

That's my understanding, too. And maybe I don't grasp the subtleties of the fix, but it seems like the fix won't necessarily help when a producer is consistently slow. In such cases, the outstandingMessages will fill with the records sent to the producer since the previous commit offsets, and as soon as we start committing offsets all records then get sent to outstandingMessagesBacklog. If the producer writes records significantly slower than the source task generates them, then outstandingMessagesBacklog could be larger than outstandingMessages by the time the offsets for outstandingMessagesBacklog are finally committed, especially if we're blocking offset commits even longer with this change. So while we're able to eventually commit those first offsets, if the backlog is larger than it will likely take longer for the producer to flush those records than it took the producer to flush the first batch.
Fortunately, we do have back pressure to not let this get this too out of control: when the producer's buffer fills up, the worker source task's thread will block (up to max.block.ms) on calls to producer.send(...), and the worker source task will retry any sends that fail after that timeout. And since this same thread that calls poll(), the worker source task will eventually slow calls to poll().
But I think we can change how offsets are flushed such that we don't have to wait for the producer, and instead we can simply flush the latest offsets for records that have been successfully written at that point. We just need a different mechanism (other than the two "outstanding" lists and the flush-related flags) to track the offsets for the most recently written records.
One way to do that is to use a single concurrent queue that bookkeeps records in the same order as generated by the source task, but in a way that allows us to track which records have been acked and tolerates those records being acked in any order.
For example, we could replace the outstandingMessages and outstandingMessagesBacklog fields in WorkerSourceTask with something like this:
   private final Queue<SubmittedRecord> submittedRecords = new ConcurrentLinkedQueue<>();

An element is appended to this queue just before the record is sent to the producer, and the SubmittedRecord class allows us to track which of these records has been acknowledged:
    protected static class SubmittedRecord {
        private final SourceRecord record;
        private final AtomicBoolean acked = new AtomicBoolean();
        public SubmittedRecord(SourceRecord sourceRecord) {
            record = Objects.requireNonNull(sourceRecord);
        }
        public void acknowledge() {
            acked.set(true);
        }
        public boolean isAcknowledged() {
            return acked.get();
        }
        public SourceRecord record() {
            return record;
        }
    }

and where acknowledge() is called from the producer callback and the commitOffsets() method can safely call isAcknowledged() and record() from the commit thread. The sendRecords() method would add a SubmittedRecord to the end of the queue for each record that will be sent to the producer:
    private boolean sendRecords() {
        ...
        for (final SourceRecord preTransformRecord : toSend) {
            ...
            SubmittedRecord submittedRecord = new SubmittedRecord(record);
            if (!submittedRecords.offer(submittedRecord)) {
                // If a blocking queue, then retry using the existing mechanism in WorkerSourceTask
                log.warn("{} Failed to add record to buffer. Backing off before retrying", this);
                toSend = toSend.subList(processed, toSend.size());
                lastSendFailed = true;
                counter.retryRemaining();
                return false;
            }
            ...

and then have the producer callback call the SubmittedRecord.acknowledge() method:
            try {
                ...
                producer.send(
                    producerRecord,
                    (recordMetadata, e) -> {
                        if (e != null) {
                            ...
                        } else {
                            submittedRecord.acknowledge();
                            ...
                        }
                    });

This effectively replaces the outstandingMessages, outstandingMessagesBacklog and flushing flag, and it simplifying the logic in the sendRecords() to not have to know which of those to use.
Then here's the big change: in commitOffsets(), we can dequeue all acked records, then take the snapshot of offsets, and immediately flush offsets without waiting for the producer. And by using a concurrent queue, we don't even need to synchronize between the sendRecords() method adding to the back of the queue and the commitOffsets() pulling from the front of the queue.
    public boolean commitOffsets() {
        ...
        // Dequeue all submitted records that have been
        while (!submittedRecords.isEmpty()) {
            SubmittedRecord next = submittedRecords.peek();
            if (!next.isAcknowledged()) {
                // This record is not yet acknowledge, so we can't continue processing any more offsets
                break;
            }
            submittedRecords.poll();
            // The record is acknowledged, so add the offsets to the offset writer
            // Offsets are converted & serialized in the OffsetWriter
            SourceRecord record = next.record();
            offsetWriter.offset(record.sourcePartition(), record.sourceOffset());
        }
        ...
        synchronized (this) {
            boolean flushStarted = offsetWriter.beginFlush();
            if (!flushStarted) {
                ...
            }
        }

        // Now we can actually flush the offsets to user storage.
        Future<Void> flushFuture = offsetWriter.doFlush((error, result) -> {
            if (error != null) {
                log.error("{} Failed to flush offsets to storage: ", WorkerSourceTask.this, error);
            } else {
                log.trace("{} Finished flushing offsets to storage", WorkerSourceTask.this);
            }
        });
        ...
    }

I've shown the snippet above using a non-blocking queue of unlimited size. I think we could do this because the existing WorkerSourceTask logic already handles the possibility that the producer.send(...) blocks when its buffer is full, up to max.block.ms before throwing a retriable exception, and then retrying the send if needed. Since this happens on the same thread that calls SourceTask.poll(), this existing logic already has the backpressure that is based upon the producer setting and that prevents the source task getting too far ahead of the producer.
Alternatively, we could use a blocking queue, but this would require an additional worker configuration, which is not ideal and can't be backported.
@C0urante, WDYT?

My previous suggestion simply dequeues all completed records until an unacked record is found. This is really straightforward, but we could try to do better.
We could dequeue all records except those that have a source partition that has not been acked. For example, let's say we have enqueue 4 records when commitOffsets() is called:
1. SubmittedRecord{ SourceRecord{...partition=P1,offset=O1...}, acked=true}
2. SubmittedRecord{ SourceRecord{...partition=P1,offset=O2...}, acked=true}
3. SubmittedRecord{ SourceRecord{...partition=P1,offset=O3...}, acked=false}
4. SubmittedRecord{ SourceRecord{...partition=P2,offset=O1...}, acked=true}
5. SubmittedRecord{ SourceRecord{...partition=P2,offset=O2...}, acked=true}
6. SubmittedRecord{ SourceRecord{...partition=P1,offset=O4...}, acked=false}
7. SubmittedRecord{ SourceRecord{...partition=P2,offset=O3...}, acked=true}
8. SubmittedRecord{ SourceRecord{...partition=P2,offset=O4...}, acked=true}
9. SubmittedRecord{ SourceRecord{...partition=P2,offset=O5...}, acked=true}
10. SubmittedRecord{ SourceRecord{...partition=P2,offset=O6...}, acked=false}

This might happen if records 4, 5, and 7-10 were written to different topic partitions than records 1, 2, 3, and 6, and the producer is stuck on the latter partitions.
With the simplistic logic, we'd only dequeue record 1 and 2, we'd add the offset for these two records to the offset writer, and we'd flush offset partition=P1,offset=O2. But the connector has made significantly more progress on source partition partition=P2, and it would be good to also flush offset partition=P2,offset=O7.
We could do that by dequeuing acked records only if their source partition map does not match a previously unacked message. This way, we'd end up with the following remaining in the queue (using the same record numbers as before):
3. SubmittedRecord{ SourceRecord{...partition=P1,offset=O3...}, acked=false}
6. SubmittedRecord{ SourceRecord{...partition=P1,offset=O4...}, acked=false}
10. SubmittedRecord{ SourceRecord{...partition=P2,offset=O6...}, acked=false}

This minor change will dramatically improve the ability to commit offsets closer to what has actually be acked.

Also, with my proposed approach, the offsetWriter is only used (directly or indirectly) within the commitOffsets() method. That means we could actually do without the synchronized(this) block around the offsetWriter.beginFlush().

@rhauch Overall that looks good to me. It's an elegant solution to the tricky problem you noted about the opacity of task-provided source offsets w/r/t ordering.
I'm a little worried about offset commits taking longer and longer with the more sophisticated approach you proposed (where we would unconditionally iterate over every record in the batch, instead of only until the first unacknowledged record). It's true that there would be natural back pressure from the producer as its buffer.memory gets eaten up, but with the default of 32MB, it still seems possible for a large number of unacknowledged records to build up. If this does happen, then offset commits may end up exceeding the offset.flush.timeout.ms for the worker, which may cause issues with the current model where a single shared, worker-global thread is used for offset commits of all tasks.
If this is a valid concern and we'd like to take it into account for now, I can think of a couple ways to handle it off the top of my head:

Use the simpler approach that blocks offset commits across the board if a single record remains unacknowledged for a long period of time (which may realistically be a problem if a single partition out of many is unavailable for some reason).
Enable concurrent offset commits by multiple tasks.
Instead of a single dequeue per task, use a ConcurrentMap<Map<String, ?>, Queue<SubmittedRecord>> that stores a single dequeue per unique source partition. This would allow us to iterate over the bare minimum number of records for every single offset commit and not spend time, for example, on accumulated records for unavailable Kafka partitions. We'd still have to iterate over those records eventually if the Kafka partition came back online, but that iteration would only ever occur once, instead of once for every offset commit.

I think option 3 may be warranted, although it's still possible that offset commits take a long time if 32MB worth of records end up getting queued. Option 2 may be worth implementing or at least considering as a follow-up item to handle this case.
Thoughts?

@C0urante, thanks for the feedback on my suggestion. I like your option 3, because it does allow the iteration to stop on each source partition as soon as it encounters the first unacknowledged record in each queue. I also think that the behavior with the suggested approach and your option 3 is still a lot better than the current situation.
One question, though: you mention that it might be a problem if iterating over the submitted records takes longer than offset.flush.timeout.ms. But IIUC the offset.flush.timeout.ms would actually not be used anymore, as there actually are no timeouts as the offset commit thread doesn't block anymore. So, worst case, if task A has a ton of submitted records that have to be iterated over (e.g., fast producer and fast source task), it might slow the committing of offsets for other tasks. (Again, this is not any worse than the current behavior.) But your option 2 would help with this at the risk of using more threads, and so we may also want to consider this to help ensure that no slowly-proceeding producer of a task blocks other offset commits.
Of course, another option might be to incur the iteration on the worker source task thread. That would essentially move the use of the queue(s) to the worker source task thread, tho we still need to get the offsets to the offset commit thread and so would likely have to keep the synchronization blocks around the offset writer snapshot. On one hand, that's putting more work onto the worker source task thread and making the offset thread super straightforward (snapshot and write); on the other it's putting the onus on the worker source task thread.
Thoughts?

I also think that the behavior with the suggested approach and your option 3 is still a lot better than the current situation.

Agreed 👍

IIUC the offset.flush.timeout.ms would actually not be used anymore, as there actually are no timeouts as the offset commit thread doesn't block anymore.

That's mostly correct--we wouldn't be waiting on a blocking operation while iterating through the dequeue(s), although we might still choose to block on the actual write to the offset topic in the same way that we currently do just for the sake of metrics and allowing users to monitor the health of the connection between the Connect worker and the offsets topic. Not a huge deal though, and the point that we wouldn't be blocking on the task's producer is still valid.
I think the issue is less that we'd end up timing out and more that we'd end up violating the guarantee that's provided right now by the framework that each task gets to take up only offset.flush.timeout.ms milliseconds per offset commit attempt before aborting the attempt and yielding control to the next task. A dequeue-based approach may actually be worse than the current behavior in that regard if there's no check in place to ensure that iterating over the dequeue doesn't exceed the offset flush timeout. Probably worth the tradeoff, but we can probably satisfy both objectives with your suggestion:

another option might be to incur the iteration on the worker source task thread.

I think this'd be great, especially with the snapshotting logic you mention, which should basically eliminate any blocking between the two threads except to prevent race conditions while simple operations like clearing a hash map or assigning a new value to an instance variable take place.
One thing that gave me pause initially was the realization that we'd be double-iterating over every source record at this point: once to transform, convert, and dispatch the record to the producer, and then once to verify that it had been acknowledged while iterating over the dequeue it's in. But I can't imagine it'd make a serious difference with CPU utilization given that transformation, conversion, and dispatching to a producer are likely to be at least an order of magnitude more expensive than just checking a boolean flag and possibly inserting the record's offset into a hash map. And memory utilization should be very close to the existing approach, which already tracks every single unacknowledged record in the outstandingMessages and outstandingMessagesBacklog fields.
I think this buys us enough that my earlier-mentioned option 2 (multiple threads for offset commits) isn't called for, since the only blocking operation that would be performed during offset commit at this point is a write to the offsets topic. If the offsets topic is unavailable, it's likely that the impact would be the same across all tasks (unless the task is using a separate offsets topic, which will become possible once the changes for KIP-618 are merged), and even if not, things wouldn't be made any worse than they already are: the offset flush timeout would expire, and the next task in line would get its chance to commit offsets.
@rhauch If this is all agreeable I think we're ready to start implementing. Since you've provided a lot of the code yourself I'm happy to let you take on that work if you'd like; otherwise, I'll get started and see if I can have a new PR with these changes out by early next week.

@C0urante wrote:

@rhauch If this is all agreeable I think we're ready to start implementing. Since you've provided a lot of the code yourself I'm happy to let you take on that work if you'd like; otherwise, I'll get started and see if I can have a new PR with these changes out by early next week.

Sounds good to me! I'm looking forward to your new PR; please link here and ping me. Thanks!

@rhauch (and, if interested, @hachikuji) new PR is up: https://github.com/apache/kafka/pull/11323


Build and publish an rpm

Resolves # . (This is applicable only if this pull request relates to a GitHub issue, delete the line otherwise)
Notes:
*
*
*

Codecov Report

Merging #7 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master       #7   +/-   ##
=======================================
  Coverage   85.62%   85.62%
=======================================
  Files          10       10
  Lines         160      160
  Branches        9        9
=======================================
  Hits          137      137
  Misses         19       19
  Partials        4        4

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 86fb352...8ed4e14. Read the comment docs.


Behavioral difference between builder.DialogAction.send() and session.Prompt.text() in a LUIS .onBegin()

I've been trying to understand why one will work in a LUIS dialog .onBegin() and the other won't. Could you help? Semantically, they seem like they would serve the same purpose, but the behavior is different.
I am using a beginDialog() to a luis dialog from another luis dialog without getting a user input in the previous dialog. The reasoning behind this is to make each luis dialog self contained in a directed, funnel type interaction. To create a funnel otherwise would require a waterfall step prompting the user for the intent handlers in the previous dialog then calling the new dialog which has the intent handlers.
[some luis dialog].onBegin(builder.DialogAction.send('blah blah blah'));
The line above will wait for a user input and then send that input to luis to process an intent which will then trigger the appropriate handler.
[some luis dialog].onBegin( function( session ) {    return session.Prompt.text("blah blah blah"); });
If you use a closure along with the session prompt, the prompt call will never be executed and tries to go directly to luis without an input.

Closing this issue due to age. @jeffreyhyun - please open a new issue if this is not fixed and needs further discussion. Thanks!


Bump cucumber-jvm versions

@mlvandijk the latest archetype will hit maven central in a few hours.

Awesome. Feel free to merge once it's available: https://mvnrepository.com/artifact/io.cucumber/cucumber-archetype


VS Code Extension not updated since 2018-9-20, 17:48:28

Title pretty much says it all.
Are there any plans to update the VS Code extension?
Community attempts have been made to provide updated versions of the extension, however due to the fact that Quicktype has >1.8m downloads, most people won't even consider other extensions.
Published          2017-12-23, 21:41:40
Last released     2018-9-20, 17:48:28
Identifier           quicktype.quicktype

Would love for someone to contribute this!

@dvdsgl Would that even be possible for someone outside of the core team?
AFAIK only you have access to the quicktype publisher account?

Last update is 28/09/2023 as of now

We're now up to date with automated publishing to keep VS Code current.


Meteor 1.0.4 Fix

This fix is necessary to support Meteor 1.0.4, it relied on a public document method rawCollection instead of privates one.
It would nice to make a new release of this package to avoid breaking Meteor applications up to 1.0.3 version.

+1 for merging this!

+1 this should be merged.

Should I fork this repo with the fix? The author seems dead.
It kind of annoying having a copy of this pull request in production apps.


Fix equals method not allowing override

The current implementation does not allow the overriding of the FlatInterval.equals method since internally some FlatInterval objects are created to optimise data handling. As such, if one does override the equals method with the new properties, the search method brakes.
Possible solutions:

Stop creating FlatInterval objects inside the I2BplusTree and just use/ move around the valid times;
Create a new method (e.g. intervalEquals) that just makes the comparison of the intervals. Then, this method is used for comparison with the FlatInterval objects inside the tree, while the equals method is used for comparison between objects introduced by the user.

Hello. Is this bug/missing feature still around? I'm a little confused because overwriting equals is currently shown in the usage notes of the readme. If the bug is still around, a note in the readme might be a good idea.
I will be using I2Bplus-tree for a programming project in the context of my bachelor's thesis, and I'm open to contributing. Introducing the intervalEquals method seems like a good + fairly easy fix.

Hey @Nikovic thanks for your question - I need to take a deeper look into it since it has been a while
I'll get back to you once I do so


xray watches resource

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description
Hi, I'd like to be able to configure watch in XRay using terraform.
I have found a go package for xray which is  based on go-artifactory.
I'm wondering if you would consider including that package in this provider, or do you think its more appropriate to make a separate provider?
Cheers!
New or Affected Resource(s)

artifactory_xray_watches

Potential Terraform Configuration
resource "artifactory_xray_watches" "example" {
  name        = "example-watches"
  description = "example description"
  active      = true

  project_resource {
    type       = "repository"
    bin_mgr_id = "art-prod-eu"
    name       = "libs-release-local"

    filters = [
      {
        type  = "regex"
        value = ".*"
      }
    ]
  }

  policy {
    name = "default-policy"
    type   = "security"
  }
}

Migrated -> https://github.com/jfrog/terraform-provider-artifactory/issues/6


[BUG]  Instagram logo is not visible

Problem Description
Summary
The Instagram logo at the bottom left corner is not visible
Screenshots

Device
This bug is visible on all devices

the logo is visible

This bug is not reproducible on my devices as well. I'll close this issue now


Invalid pesterExplorer.TestRootDirectory on opening new projet

So I've opened a new project, it asked me to scan things I pressed Always - and then I was greeted with invalid path + cannot debug (I guess I did it before PowerShell loaded)

I already had:     "pesterExplorer.testRootDirectory": "Tests", in my config from earlier projects. When I pressed Always "pesterExplorer.autoDiscoverOnOpen": true this was added which is expected. Weirdly enough Tests path exists.

Along with this error it detected some other tests - not sure where from 😂

The error appears on reopening the project again and again

What happens if you remove the pesterExplorer.testRootDirectory from your settings? You don't need it. It should automatically discover Tests

After removal, everything works correctly. However, you can set Tests from GUI and it breaks again. Something must be wrong with this setting.

This is what happens when I code too fast... I really need to add tests. Anyway, this is fixed in 0.0.14


Out of memory when using readStream on big files

Hello,
I tried to download 50GB File from Google Cloud Platform to my local machine.
I used GCP's getReadStream function, and I saw that the memory usage was increasing dramatically until the process crashed.
What is the best practice for using your readStream? Or is there any workaround for that problem?
Thanks!

Hey @trample25. What version of our library are you using? And would you mind sharing the minimum code required to reproduce the issue? (You can assume I have a 50GB file to work with)

Hi @stephenplusplus, thank you for your quick response.
I'm using version 2.0.0
`const { createWriteStream } = require('fs-extra');
const {Storage} = require('@google-cloud/storage');
async function main() {
const storage = await getStorage();
const bucket= 'my bucket';
const testFile1 = '50GBFile';
const readStream = storage.bucket(bucket).file(testFile1).createReadStream();
const writeStream = createWriteStream(testFile1);
readStream.pipe(writeStream);`

Please monitor the memory usage while performing this test. It seems like the readstream buffers all the data into the memory.
Thanks!

So far, I haven't been able to reproduce. Watching top, the memory usage sticks around ~50 MB, but that value would likely fluctuate depending on the resources available to your specific machine and process. Node's internals should be figuring all of that out automatically.
We don't buffer data into memory. Instead, it is passed through using normal stream mechanics, where data is only fetched when the destination is ready to receive it. Then when that receiver clears it, it asks for more, we get more, etc.
Can you run and monitor just this script, and paste the output (you can use my bucket and file!)?
const fs = require('fs-extra');
const {Storage} = require('@google-cloud/storage');

const testFile1 = '50GBFile';

const storage = new Storage()
const bucket = storage.bucket('stephen-has-a-new-bucket');
const file = bucket.file(testFile1)

const readStream = file.createReadStream();
const writeStream = fs.createWriteStream(testFile1);

const startingMemoryUsage = getMemoryUsage()
var lastMemoryUsage
var timeLastLogPrinted

function getMemoryUsage() {
  lastMemoryUsage = Math.ceil(process.memoryUsage().heapUsed / 1024 / 1024)
  return lastMemoryUsage
}

function printMemoryUsage() {
  if ((Date.now() - timeLastLogPrinted) < 3000) {
    // Only log every 3 seconds
    return
  }

  if (lastMemoryUsage === getMemoryUsage()) {
    // Nothing new to report
    return
  }

  console.log(`Memory usage relative to starting memory: ${lastMemoryUsage - startingMemoryUsage} MB`)
  timeLastLogPrinted = Date.now()
}

readStream
  .on('data', printMemoryUsage)
  .pipe(writeStream)

My output

$ node .
Memory Usage: -2 MB
Memory Usage: 3 MB
Memory Usage: 0 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: 2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: 2 MB
Memory Usage: 3 MB
Memory Usage: 4 MB
Memory Usage: 3 MB
Memory Usage: -2 MB
Memory Usage: 3 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: -2 MB
Memory Usage: 0 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: 4 MB
Memory Usage: 0 MB
Memory Usage: -2 MB
Memory Usage: 1 MB
Memory Usage: 2 MB
Memory Usage: 3 MB
Memory Usage: 0 MB
Memory Usage: -1 MB
Memory Usage: 2 MB
Memory Usage: 3 MB
Memory Usage: 1 MB
Memory Usage: 2 MB
Memory Usage: 3 MB
Memory Usage: 2 MB
Memory Usage: 3 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: 1 MB
Memory Usage: 0 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: -3 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: -2 MB
Memory Usage: 0 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: -2 MB
Memory Usage: 0 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: 0 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: 1 MB
Memory Usage: -3 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: 1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: 1 MB
Memory Usage: -2 MB
Memory Usage: 0 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: -2 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: 0 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: -2 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: -3 MB
Memory Usage: 1 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: 1 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: -3 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: -1 MB
Memory Usage: 1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: 0 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: 1 MB
Memory Usage: -1 MB
Memory Usage: 1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: -1 MB
Memory Usage: 1 MB
Memory Usage: -3 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: -2 MB
Memory Usage: 0 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: -2 MB
Memory Usage: 0 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: 1 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: 1 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: 0 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: 0 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: 0 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: 0 MB
Memory Usage: 1 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: -2 MB
Memory Usage: 1 MB
Memory Usage: 0 MB
Memory Usage: -1 MB
Memory Usage: 0 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: 5 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -3 MB
Memory Usage: -2 MB
Memory Usage: -1 MB
Memory Usage: -3 MB
Memory Usage: -1 MB
Memory Usage: -2 MB
Memory Usage: -1 MB

Hi @stephenplusplus, thanks again for your professional and detailed response.

Thanks!

Thanks for running that! It seems like the stream internals are actually working. Note the various drops where it goes from 58 MB -> 5 MB, 108 MB -> 6 MB, etc. That would mean the memory usage of your system built up until it needed a breath, and flushed the rest out of the internal buffer, reclaiming that space.
As far as why your memory usage grew to those much higher levels than my test run, that could be due to different factors that differentiate our testing environments. So, I'm not sure I should be reading into that too much.
What would be bad is if that memory usage grew and grew, and never flushed down, but that's not what we're seeing in this test case. So we have to figure out what made my test case different from your initial script that ran until it crashed. Would you be able to create a sample repository that I can clone and run with that code that wasn't releasing memory?

Hi @stephenplusplus, Actually the first crash happened due to download of 5 files of 10GB, then I tried it with 1 file of 50GB and it also crashed (and now I see that sometimes it works fine even though it still uses lots of memory). Unfortunately I can't share the actual code that didn't release memory, but I can reproduce it again easily with a simple script :
for(var i =1; i<11; i++){
    let bucket = 'my-bucket';
    let testFile1 = `file${i}`;
    let testFile2 = `10FilesOf10GB/file${i}`;
    let readStream = storage.bucket(bucket).file(testFile2).createReadStream(options);
    let writeStream = createWriteStream(testFile1);
    readStream.pipe(writeStream);
  }

When I run this code, the memory usage goes to 100% and I get ESOCKETTIMEDOUT error.
If it matters, I use node version 8.12
Thank you very much!

I couldn't reproduce with 10 concurrent streams downloading 50 GB files each. The memory usage stayed around the same as before. Though, having that many streams running without being aware of each other could be a problem. The file system will also get hit pretty hard with that demand.
In the snippet below, I've introduced async to manage the flow control to 5 parallel streams at once. It will also log the memory usage as a percentage of the available memory. Would you mind running this and reporting back what you find?
Also, how are you monitoring memory usage?
Thanks!
const async = require('async')
const fs = require('fs-extra');
const {Storage} = require('@google-cloud/storage');

const testFile1 = '50GBFile';

const storage = new Storage()
const bucket = storage.bucket('stephen-has-a-new-bucket')
const file = bucket.file(testFile1)

const readStream = file.createReadStream()
const writeStream = fs.createWriteStream(testFile1)

const startingMemoryUsage = getMemoryUsage()
var lastMemoryUsage
var lastAvailableMemoryPercentage
var timeLastLogPrinted

function getMemoryUsage() {
  const memoryUsage = process.memoryUsage()
  lastMemoryUsage = Math.ceil(memoryUsage.heapUsed / 1024 / 1024)
  lastAvailableMemoryPercentage = (memoryUsage.heapTotal / memoryUsage.heapUsed).toFixed(2)
  return lastMemoryUsage
}

function printMemoryUsage() {
  if ((Date.now() - timeLastLogPrinted) < 1000) {
    // Only log ~ once per second
    return
  }

  if (lastMemoryUsage === getMemoryUsage()) {
    // Nothing new to report
    return
  }

  console.log(`
Memory Stats:
  Available memory relative to starting memory: ${lastMemoryUsage - startingMemoryUsage} MB
  Usage relative to available memory: ${lastAvailableMemoryPercentage}% `)

  timeLastLogPrinted = Date.now()
}

// arg1: How many times should this function run
// arg2: How many parallel functions should run at once
async.timesLimit(10, 5, (n, next) => {
  file.createReadStream()
    .on('error', next)
    .on('data', printMemoryUsage)
    .pipe(fs.createWriteStream(`file${n}`))
    .on('error', next)
    .on('finish', next)
}, (err) => {
  if (err) throw err
  console.log('Done downloading!')
})

I'm using the top command in linux in order to monitor the memory usage.
This is what I see at the row of the node process before it gets killed:
(PID)2890   (RES)14g   (%MEM)97.7   (COMMAND)node
And this is the output of your latest script (please pay attention to the error in the end):
node script1.js

Memory Stats:
  Available memory relative to starting memory: 2 MB
  Usage relative to available memory: 1.73%

Memory Stats:
  Available memory relative to starting memory: 8 MB
  Usage relative to available memory: 2.08%

Memory Stats:
  Available memory relative to starting memory: 15 MB
  Usage relative to available memory: 1.81%

Memory Stats:
  Available memory relative to starting memory: 10 MB
  Usage relative to available memory: 2.30%

Memory Stats:
  Available memory relative to starting memory: 23 MB
  Usage relative to available memory: 1.65%

Memory Stats:
  Available memory relative to starting memory: 27 MB
  Usage relative to available memory: 1.63%

Memory Stats:
  Available memory relative to starting memory: 30 MB
  Usage relative to available memory: 1.63%

Memory Stats:
  Available memory relative to starting memory: 36 MB
  Usage relative to available memory: 1.56%

Memory Stats:
  Available memory relative to starting memory: 33 MB
  Usage relative to available memory: 1.77%

Memory Stats:
  Available memory relative to starting memory: 42 MB
  Usage relative to available memory: 1.59%

Memory Stats:
  Available memory relative to starting memory: 43 MB
  Usage relative to available memory: 1.64%

Memory Stats:
  Available memory relative to starting memory: 48 MB
  Usage relative to available memory: 1.58%

Memory Stats:
  Available memory relative to starting memory: 49 MB
  Usage relative to available memory: 1.65%

Memory Stats:
  Available memory relative to starting memory: 63 MB
  Usage relative to available memory: 1.40%

Memory Stats:
  Available memory relative to starting memory: 58 MB
  Usage relative to available memory: 1.60%

Memory Stats:
  Available memory relative to starting memory: 67 MB
  Usage relative to available memory: 1.47%

Memory Stats:
  Available memory relative to starting memory: 68 MB
  Usage relative to available memory: 1.52%

Memory Stats:
  Available memory relative to starting memory: 74 MB
  Usage relative to available memory: 1.47%

Memory Stats:
  Available memory relative to starting memory: 85 MB
  Usage relative to available memory: 1.35%

Memory Stats:
  Available memory relative to starting memory: 87 MB
  Usage relative to available memory: 1.36%

Memory Stats:
  Available memory relative to starting memory: 92 MB
  Usage relative to available memory: 1.34%

Memory Stats:
  Available memory relative to starting memory: 86 MB
  Usage relative to available memory: 1.48%

Memory Stats:
  Available memory relative to starting memory: 99 MB
  Usage relative to available memory: 1.35%

Memory Stats:
  Available memory relative to starting memory: 94 MB
  Usage relative to available memory: 1.45%

Memory Stats:
  Available memory relative to starting memory: 98 MB
  Usage relative to available memory: 1.44%

Memory Stats:
  Available memory relative to starting memory: 111 MB
  Usage relative to available memory: 1.31%

Memory Stats:
  Available memory relative to starting memory: 115 MB
  Usage relative to available memory: 1.30%

Memory Stats:
  Available memory relative to starting memory: 108 MB
  Usage relative to available memory: 1.42%

Memory Stats:
  Available memory relative to starting memory: 122 MB
  Usage relative to available memory: 1.30%

Memory Stats:
  Available memory relative to starting memory: 118 MB
  Usage relative to available memory: 1.36%

Memory Stats:
  Available memory relative to starting memory: 124 MB
  Usage relative to available memory: 1.33%

Memory Stats:
  Available memory relative to starting memory: 123 MB
  Usage relative to available memory: 1.36%

Memory Stats:
  Available memory relative to starting memory: 125 MB
  Usage relative to available memory: 1.37%

Memory Stats:
  Available memory relative to starting memory: 134 MB
  Usage relative to available memory: 1.31%

Memory Stats:
  Available memory relative to starting memory: 130 MB
  Usage relative to available memory: 1.37%

Memory Stats:
  Available memory relative to starting memory: 140 MB
  Usage relative to available memory: 1.30%

Memory Stats:
  Available memory relative to starting memory: 137 MB
  Usage relative to available memory: 1.36%

Memory Stats:
  Available memory relative to starting memory: 146 MB
  Usage relative to available memory: 1.30%

Memory Stats:
  Available memory relative to starting memory: 143 MB
  Usage relative to available memory: 1.35%

Memory Stats:
  Available memory relative to starting memory: 149 MB
  Usage relative to available memory: 1.32%

Memory Stats:
  Available memory relative to starting memory: 159 MB
  Usage relative to available memory: 1.26%

Memory Stats:
  Available memory relative to starting memory: 151 MB
  Usage relative to available memory: 1.34%

Memory Stats:
  Available memory relative to starting memory: 158 MB
  Usage relative to available memory: 1.31%

Memory Stats:
  Available memory relative to starting memory: 156 MB
  Usage relative to available memory: 1.34%

Memory Stats:
  Available memory relative to starting memory: 160 MB
  Usage relative to available memory: 1.33%

Memory Stats:
  Available memory relative to starting memory: 164 MB
  Usage relative to available memory: 1.33%

Memory Stats:
  Available memory relative to starting memory: 166 MB
  Usage relative to available memory: 1.32%

Memory Stats:
  Available memory relative to starting memory: 171 MB
  Usage relative to available memory: 1.31%

Memory Stats:
  Available memory relative to starting memory: 181 MB
  Usage relative to available memory: 1.25%

Memory Stats:
  Available memory relative to starting memory: 172 MB
  Usage relative to available memory: 1.33%

Memory Stats:
  Available memory relative to starting memory: 175 MB
  Usage relative to available memory: 1.33%

Memory Stats:
  Available memory relative to starting memory: 189 MB
  Usage relative to available memory: 1.25%

Memory Stats:
  Available memory relative to starting memory: 180 MB
  Usage relative to available memory: 1.33%

Memory Stats:
  Available memory relative to starting memory: 190 MB
  Usage relative to available memory: 1.28%

Memory Stats:
  Available memory relative to starting memory: 185 MB
  Usage relative to available memory: 1.32%

Memory Stats:
  Available memory relative to starting memory: 194 MB
  Usage relative to available memory: 1.28%

Memory Stats:
  Available memory relative to starting memory: 191 MB
  Usage relative to available memory: 1.32%

Memory Stats:
  Available memory relative to starting memory: 199 MB
  Usage relative to available memory: 1.28%

Memory Stats:
  Available memory relative to starting memory: 207 MB
  Usage relative to available memory: 1.24%

Memory Stats:
  Available memory relative to starting memory: 210 MB
  Usage relative to available memory: 1.24%

Memory Stats:
  Available memory relative to starting memory: 212 MB
  Usage relative to available memory: 1.25%

Memory Stats:
  Available memory relative to starting memory: 206 MB
  Usage relative to available memory: 1.30%

Memory Stats:
  Available memory relative to starting memory: 207 MB
  Usage relative to available memory: 1.31%

Memory Stats:
  Available memory relative to starting memory: 208 MB
  Usage relative to available memory: 1.31%

Memory Stats:
  Available memory relative to starting memory: 222 MB
  Usage relative to available memory: 1.24%

Memory Stats:
  Available memory relative to starting memory: 225 MB
  Usage relative to available memory: 1.24%

Memory Stats:
  Available memory relative to starting memory: 227 MB
  Usage relative to available memory: 1.24%

Memory Stats:
  Available memory relative to starting memory: 218 MB
  Usage relative to available memory: 1.31%

Memory Stats:
  Available memory relative to starting memory: 220 MB
  Usage relative to available memory: 1.31%

Memory Stats:
  Available memory relative to starting memory: 224 MB
  Usage relative to available memory: 1.30%

Memory Stats:
  Available memory relative to starting memory: 225 MB
  Usage relative to available memory: 1.31%

Memory Stats:
  Available memory relative to starting memory: 239 MB
  Usage relative to available memory: 1.24%

if (err) throw err
           ^

Error: ESOCKETTIMEDOUT
    at ClientRequest.<anonymous> (/disk1/data/eyal/source/node_modules/request/request.js:816:19)
    at Object.onceWrapper (events.js:313:30)
    at emitNone (events.js:106:13)
    at ClientRequest.emit (events.js:208:7)
    at TLSSocket.emitTimeout (_http_client.js:707:34)
    at Object.onceWrapper (events.js:313:30)
    at emitNone (events.js:106:13)
    at TLSSocket.emit (events.js:208:7)
    at TLSSocket.Socket._onTimeout (net.js:422:8)
    at ontimeout (timers.js:498:11)

The difference between my test and yours might be related to the environment we are using?
What OS do you use and which node version?

Hi @stephenplusplus @kinwa91 , do you have any updates regarding this issue?
Thanks!

ping @stephenplusplus

Sorry for the delay. @trample25, would you mind updating to the latest release, 2.3.4, and running the script again? I have run it, and my results are similar to my other runs-- hovering around -4 to +4 MB memory usage relative to my starting point.
Regarding the ESOCKETTIMEDOUT error, I think that was an unrelated issue that has since been resolved. Let me know if that comes back, though.

OS: macOS Mojave, 10.14.2
Node: 10.11.0

Greetings @trample25!  If you're still running into issues, please do let us know.  We're going to close this out for now.

Hi guys, I'm running into this issue when using the file.createReadStream method. It seems this problem was introduced in version 2.2.0 and is present in all version since then (including version 2.5.0). I'm using version 2.1.0 to avoid running into this issue.
My specific use case:

I'm streaming files that are between 100MB and 200MB in size.
I'm using the node-throttle module.
I'm piping the stream into the HTTP response of Express.js

bucket
    .file(filename)
    .createReadStream({ start, end })
    .on("error", err => {
        if (err.code === "ESOCKETTIMEDOUT") {
            logger.warn("Error reading file stream", err);
        } else {
            logger.error("Error reading file stream", err);
        }
    })
    .pipe(throttle)
    .pipe(res);

Some measurements I made using the memory-usage module:

v2.1.0:

v2.2.0:

v2.4.0:

v2.5.0:


Serverless failing to deploy when "serverless-python-requirements" is installed

Hello,
Given I've just been deploying my serverless application without issue
When I installed the "serverless-python-requirements" package
Then I get the following error when deploying
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Injecting required Python packages to package...

  Error --------------------------------------------------

  nodestream is not supported by this platform

If I uninstall the package I can deploy again.
Any thoughts on why this is happening?
"serverless-python-requirements version: "4.3.0"

Well... what platform are you on?

That might help alright:
  Your Environment Information -----------------------------
     OS:                     darwin
     Node Version:           10.9.0
     Serverless Version:     1.38.0

I'm going to close this "issue" because it was nothing to do with your package. Thank you 🙏 for taking the time to reply to such a vague issue though 🙄
For anyone who might come across this post the problem was caused by a mismatch of npm modules installed globally and locally on the host system.
My resolution was to remove the local node_modules folder and run the globally installed modules instead. Your mileage may vary.


(Investigate) Volume intellisense--path suggestions

Investigate if it's possible to suggest file paths in volume mappings (host-side only, container side is impossible)

One thing I'm not sure about is if you can switch into another completion, while inside a completion already, and then switch back. For volumes we have a completion to help them with syntax, and IMO this is more important than one to help them with file paths (which they are likely to copy/paste anyway). If we have to choose one or the other, I'd choose syntax.

One thing I'm not sure about is if you can switch into another completion, while inside a completion already, and then switch back.

Can you clarify what you mean here? Wouldn't the text caret determine whether you want syntax support or path support?

@bwateratmsft Oh, I see what you mean now. It can kind of work...? 🤔 See below...

Paste the attached JavaScript code into the Monaco playground.
Click the "Run" button.
Place the text caret in the editor on the right-hand side.
Invoke Intellisense.
Accept "volume" with enter.
Invoke Intellisense.
Select "/host/path" with enter.
Tab to "containerPath".
Select "/container/path" with enter.
Tab to "rw".

function createDependencyProposals(word, range) {
    if (word.word === "hostPath") {
        return [
            {
                label: '/host/path',
                insertText: '/host/path',
                range: range
            }
        ];
    }
    if (word.word === "containerPath") {
        return [
            {
                label: '/container/path',
                insertText: '/container/path',
                range: range
            }
        ];
    }
	return [
		{
			label: 'volume',
			insertText: '${1:hostPath}:${2:containerPath}:${3|rw,ro|}$0',
			insertTextRules: monaco.languages.CompletionItemInsertTextRule.InsertAsSnippet,
			range: range
		}
	];
}

monaco.languages.registerCompletionItemProvider('plaintext', {
	provideCompletionItems: (model, position) => {
		const word = model.getWordUntilPosition(position);
		const range = {
			startLineNumber: position.lineNumber,
			endLineNumber: position.lineNumber,
			startColumn: word.startColumn,
			endColumn: word.endColumn
		};
		return {
			suggestions: createDependencyProposals(word, range)
		};
	}
});

monaco.editor.create(document.getElementById('container'), { language: 'plaintext' });


Apply PHP 7.4 syntax and typed property

Signed-off-by: Abdul Malik Ikhsan<EMAIL_ADDRESS>

Q
A

QA
yes

Description
Since composer.json require php 7.4, php 7.4 syntax can be applied with typed properties.

For final class, typed properties for protected is allowed if no inherit of parent or used by its final class parent.
Otherwise, update only private properties.

@Ocramius All green 🎉 it is ready now.


Added CircleCi configuration

Fixes #3

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@bf57f5c). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master       #4   +/-   ##
=========================================
  Coverage          ?   74.62%
=========================================
  Files             ?        8
  Lines             ?      331
  Branches          ?        0
=========================================
  Hits              ?      247
  Misses            ?       70
  Partials          ?       14

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bf57f5c...75343f1. Read the comment docs.


doesNotMatchKey not working

Hi!
I'm trying to do this query on my own hosted Parse server but I get error con shell:
Code:
 - (PFQuery *)queryForTable {     PFQuery *bann = [PFQuery queryWithClassName:@"Banned"];     PFQuery *query = [PFQuery queryWithClassName:self.parseClassName];     query.cachePolicy = kPFCachePolicyCacheThenNetwork;     query.cachePolicy = kPFCachePolicyNetworkElseCache;         [query whereKey:@"qUser" doesNotMatchKey:@"bannedUser" inQuery:bann];         [query orderByAscending:@"createAt"];         return query;    }
Get on shell:
**[Error]: improper usage of $dontSelect (Code: 102, Version: 1.12.0) **
Thanks For help.

Hey @dasoga, sounds very strange, as this should work...
Let's debug this together, here are few questions that would help:

Are you using parse.com or self-hosted Parse Server?
What does your object schema look like for both Banned and the objects you are querying for (can't see the class name).
Could you instrument your app with Network Debug Tool from this guide. The most important piece is how the query is being sent to the server...
Does this reproduce if you are issuing a cURL request with the same data?

Confident we can uncover the problem together...

Hi @nlutsenko

Are you using parse.com or self-hosted Parse Server?

self-hosted Parse Server

What does your object schema look like for both Banned and the objects you are querying for (can't see the class name).
I have the class "Question" (questions uploaded for users) and class "Banned" (for users banned). And I want to do a query for question without user banned.

Could you instrument your app with Network Debug Tool from this guide. The most important piece is how the query is being sent to the server...

Request Body : {"where":{"qUser":{"$dontSelect":{"query":{"className":"Banned"},"key":"bannedUser"}}},"order":"createAt","_method":"GET","limit":"10"} 2016-03-08 18:07:16.600 alertatrasero[14595:6171749] [Error]: improper usage of $dontSelect (Code: 102, Version: 1.12.0) 2016-03-08 18:07:16.600 alertatrasero[14595:6171750] URL : http://<IP_ADDRESS>:1337/parse/classes/Question 2016-03-08 18:07:16.601 alertatrasero[14595:6171750] Status Code : 400 2016-03-08 18:07:16.601 alertatrasero[14595:6171750] Headers : {     "Access-Control-Allow-Headers" = "X-Parse-REST-API-Key, X-Parse-Javascript-Key, X-Parse-Application-Id, X-Parse-Client-Version, X-Parse-Session-Token, X-Requested-With, X-Parse-Revocable-Session, Content-Type";     "Access-Control-Allow-Methods" = "GET,PUT,POST,DELETE,OPTIONS";     "Access-Control-Allow-Origin" = "*";     Connection = "keep-alive";     "Content-Length" = 52;     "Content-Type" = "application/json";     Date = "Wed, 09 Mar 2016 00:07:11 GMT";     Etag = "\"1510158381\"";     "X-Powered-By" = Express; }

Does this reproduce if you are issuing a cURL request with the same data?

When I do a CURL to get the objects of Banned class everything is correct, same to the Question class. I don't know how can I do CURL with "doesNotMatchKey" parameter.
Thanks for response!

I think I've found the issue - this is the relevant piece of code that gets triggered.
I am going to double check whether removing a !query.where is going to impact anything, but I think that's the fix... Overall, as a workaround - add any condition to your banned query and it should start working.

Excellent work @nlutsenko I added any condition to Banned query and it works!
For this moment I'll keep this query this way
Thanks a lot for your help.


Ayoub Laaoufi  username: aylaaoufi

PR Merged🎉
Thank you for your contribution!
Your efforts help make our project better for everyone!!!


NETOBSERV-1336: DSCP mapping

Description
update netobserv package to get dscp field
Dependencies
https://github.com/netobserv/netobserv-ebpf-agent/pull/158
https://github.com/netobserv/network-observability-console-plugin/pull/401
Checklist
If you are not familiar with our processes or don't know what to answer in the list below, let us know in a comment: the maintainers will take care of that.

[x] Will this change affect NetObserv / Network Observability operator? If not, you can ignore the rest of this checklist.
[ ] Is this PR backed with a JIRA ticket? If so, make sure it is written as a title prefix (in general, PRs affecting the NetObserv/Network Observability product should be backed with a JIRA ticket - especially if they bring user facing changes).
[ ] Does this PR require product documentation?

[ ] If so, make sure the JIRA epic is labelled with "documentation" and provides a description relevant for doc writers, such as use cases or scenarios. Any required step to activate or configure the feature should be documented there, such as new CRD knobs.

[ ] Does this PR require a product release notes entry?

[ ] If so, fill in "Release Note Text" in the JIRA.

[ ] Is there anything else the QE team should know before testing? E.g: configuration changes, environment setup, etc.

[ ] If so, make sure it is described in the JIRA ticket.

QE requirements (check 1 from the list):

[x] Standard QE validation, with pre-merge tests unless stated otherwise.
[ ] Regression tests only (e.g. refactoring with no user-facing change).
[ ] No QE (e.g. trivial change with high reviewer's confidence, or per agreement with the QE team).

/label qe-approved

/lgtm

Closed in favor of https://github.com/netobserv/flowlogs-pipeline/pull/518


Nix buildVimPlugin compatibility & support

I've tried a simple nix flake setup for this plugin, but I guess I'm missing some dependencies:
In https://github.com/Anderssorby/lean.nvim/tree/acs/add-nix-flake running nix build
vimplugin-lean.nvim> unpacking sources
vimplugin-lean.nvim> unpacking source archive /nix/store/lvrmjq6x8hxq07v5gn4wbyi9dwfsr4ap-fzjpvalizgp1l9w71ij4ni3nd026c0ln-source
vimplugin-lean.nvim> source root is fzjpvalizgp1l9w71ij4ni3nd026c0ln-source
vimplugin-lean.nvim> patching sources
vimplugin-lean.nvim> configuring
vimplugin-lean.nvim> no configure script, doing nothing
vimplugin-lean.nvim> building
vimplugin-lean.nvim> build flags: SHELL=/nix/store/wv35g5lff84rray15zlzarcqi9fxzz84-bash-4.4-p23/bin/bash
vimplugin-lean.nvim> nvim --noplugin -u scripts/minimal_init.lua -c "lua require'lean'.setup"{}""
vimplugin-lean.nvim> =11;?(BError detected while processing /build/fzjpvalizgp1l9w71ij4ni3nd026c0ln-source/
vimplugin-lean.nvim> scripts/minimal_init.lua:
vimplugin-lean.nvim> line    1:
vimplugin-lean.nvim> E194: No alternate file name to substitute for '#': vim.opt.display = 'lastline'
vimplugin-lean.nvim> line    2:E303: Unable to create directory "/homeless-shelter" for swap file, recovery imp
vimplugin-lean.nvim> ossible: permission denied
vimplugin-lean.nvim> "directory" [New File]Cannot open file "directory"
vimplugin-lean.nvim> E303: Unable to create directory "/homeless-shelter" for swap file, recovery impossible: permission denied
vimplugin-lean.nvim> "=" [New File]Cannot open file "="
vimplugin-lean.nvim> E303: Unable to create directory "/homeless-shelter" for swap file, recovery impossible: permission denied
vimplugin-lean.nvim> "''" [New FileCannot open file "''"
vimplugin-lean.nvim> E480: No match: opt
vimplugin-lean.nvim> line    4:
vimplugin-lean.nvim> E492: Not an editor command: local __file__ = debug.getinfo(1).source:match("@(.
vimplugin-lean.nvim> *)$")
vimplugin-lean.nvim> line    5:

see https://github.com/NixOS/nixpkgs/pull/144282

Is there what more to do in this repo after that PR in the nixpkgs repo? Or given that it's now merged are you saying we should close this ticket?

Going to close since I assume that was the implication above but if that's not the case and there's something left to be done let me know!

@Julian There is more to be done since that PR didn't solve the problem right away. I'm also using a different function to build the vim plugin which doesn't read the dependencies attribute apparently. I've not looked at this too much yet.

Got it. OK, and the request here is basically to add a nix flake to the repo?

Yes, so I can load it into my env without having to wait for nixpkgs.

Assuming this is even still relevant (given it seems nixpkgs stays relatively up to date with the repo here from what I can tell), and given I don't personally know how to do this or have access to a nixOS machine anymore, going to close this, but if you or anyone is still interested a PR to do whatever is needed is definitely fine with me.


Improve facts management

Replace legacy facts with their equivalent syntax;
Avoid usage of top-scope variables to access facts.

This is intended to fix the module when disabling legacy facts in puppet
with puppet config set include_legacy_facts false.

This is still relevant.

This is still relevant.

This is still relevant

Hello, any news about this subject ?
Thanks.

Hey,
I'm also curious if this PR will be merged. Or are you not planning to support Puppet 8 at all?
Thanks a lot!


Restart/Poweroff

Have to search the proper way to emit a reboot or poweroff from the interface.
Then put some big red push buttons somewhere.

Use the "reboot" and "poweroff" commands

Note for myself: Think about an Ajax view to receive commands and another Ajax to receive "ping".
Interface will send reboot/poweroff command, the view will perform it. Obviously the server wont be able to answer to request since it will be shutted off when the command is executed.
So the interface should assume the box is rebooting/powered off if the ajax request fail on network error (unreachable host or somewhat). For reboot command, the interface would have to send a ping request until it is answered.

:+1:

:heart:

Note: be strict on available command, don't let execute everything

With the 4.0.0, systemV is now used for init process.
The commands you should use now are : reboot and shutdown -h
That will allow the pin 5/6 wakeup

I'd love to see options to reboot and shutdown the Pi, as I was often stuck in an emulator after messing with the pad configuration, and I have no keyboard.
It would be nice to also be able to go back into ES this way although a reboot would do the job.

I just spent the last half hour looking for this feature. Now I see it is not implemented.


Change accumelated count

Is it possible to change accumulated count of markers to 5 - 10 - 15 instead of 1 - 2 - 3 ?
Thank you

@sydgis
How about using the offset parameter ?
https://github.com/adoroszlai/leaflet-distance-markers#options

Using offset: 5000 would show markers at 5-10-15 km, but with labels 1-2-3.

Thanks to @matzepan this is now possible with the following options:
distanceMarkers: {
	offset: 5000,
	textFunction: function(distance, i, offset) {
		return distance/1000;
	}
}


the openid_connect gem is restricted to version 0.12.0

If we want to use a more recent version of the openid_connect gem (1.1.5 is the current latest), we need to update the gem dependency for openid_connect.
I created a pool request to address it: https://github.com/m0n9oose/omniauth_openid_connect/pull/2

@clucas We maintained a fork as new master of this library. You can found this version here.


Features/khr lights punctual

support for KHR_lights_punctual

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Ajc demo suggestions

The Demo looks awesome. I made a few suggestions:

I ran the notebook from a venv to verify the required packages and created an installable REQUIREMENTS.txt file for people who want to reuse this.
I added installation and usage instructions to the README
I made some suggestions for the formatting of the notebook (see a4f5155)

My only residual comments are:

I would specifically call out what cell types each of the genes you plot is meant to identify in the notebook.
I would add a cell at the end of the notebook that links to the data browser with some kind of suggestion like "find more data here".

👌 Nice job.

Thanks @ambrosejcarr !


ci: add govulncheck

https://github.com/golang/govulncheck-action
govulncheck ./...
govulncheck is an experimental tool. Share feedback at https://go.dev/s/govulncheck-feedback.

Using go1.20.7
 and<EMAIL_ADDRESS>with vulnerability data from https://vuln.go.dev (last modified 2023-08-02 20:33:39 +0000 UTC).

Scanning your code and 658 packages across 111 dependent modules for known vulnerabilities...

(no vulns 😎)

Codecov Report

Merging #1110 (7c8a258) into main (83589fd) will not change coverage.
The diff coverage is n/a.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
@@           Coverage Diff           @@
##             main    #1110   +/-   ##
=======================================
  Coverage   49.19%   49.19%
=======================================
  Files          44       44
  Lines        3653     3653
=======================================
  Hits         1797     1797
  Misses       1624     1624
  Partials      232      232


vue-text-highlight 2.0.5 fails to resolve

Hi,
I upgraded to 2.0.5 and start getting the following error:
This dependency was not found:

vue-text-highlight in ./src/globalComponents.js

To install it, you can run: npm install --save vue-text-highlight

With 2.0.2 it worked, and I noticed that in this latest version, dist folder is missing.

Please try upgrading to v2.0.6.
Both v2.0.4 and v2.0.5 are broken build due to some changing behaviour in npm.


Prepare for Azure Identity Extensions 1.1.1

As title.

API change check
API changes are not detected in this pull request.


fix: prevent losing rocket punch cooldown after ultimate (for real this time) - [closed]

In GitLab by @hikiru on Nov 10, 2023, 17:58
Merges punch_cooldown_ult_fix -> staging
Fixed the slam issue from last merge attempt, I tested all abilities after ult and found no issues.

In GitLab by @hikiru on Nov 10, 2023, 18:04
added 2 commits
21aeee1b - Fixed punch cooldown without flying slams91057651 - Update punch.opy
Compare with previous version


Bug: Input elements emit change events twice

Paper-Input, Paper-Textarea and most likely other form components emit a value change event twice. This leads to all validators running twice and a major performance decrease.
export class AppComponent {

  private form: FormGroup;

  constructor(private fb: FormBuilder) {
  }

  ngOnInit() {
    this.form = this.fb.group({
      name: ['']
    });

    this.form.valueChanges
        // .debounceTime(1) this would fix it
        .subscribe((value) => console.log('value changes', value));
  }

}

<form [formGroup]="form">
  <paper-input formControlName="name" label="Name"></paper-input>
</form>

Here the output on Chrome console
2016-10-13 11:42:47.698 app.component.ts:26 value changes Object {name: "s"}
2016-10-13 11:42:47.702 app.component.ts:26 value changes Object {name: "s"}

This issue is related to https://github.com/PolymerElements/paper-input/issues/421
Apparently both the IronFormElementBehavior and PaperInputBehavior declare the value property enforcing two change events

  const formElementDirective:any = Directive({
    selector: name,
    providers: [
      {
        provide: NG_VALUE_ACCESSOR,
        useExisting: forwardRef(() => formElementDirective),
        multi: true
      }
    ],
    host: (isCheckedElement ? { '(checkedChange)': 'onValueChanged($event)' } : {
     // this would fix the issue, but not sure if this is reasonable for all circumstances
     '(keyup)': 'onValueChanged($event.target.value)'
   })
  })

@cschroeter I think we could close this issue, right? Apparently it's an polymer issue...

@BorntraegerMarc I think so :)


"handleHttps" Will cause connection leaks

		if targetOK && clientOK {
			go copyAndClose(ctx, targetTCP, proxyClientTCP)
			go copyAndClose(ctx, proxyClientTCP, targetTCP)
		}

If the client actively closes the connection, and the remote server does not write any data, and the inactivity timeout is not configured. At this time, only the first copyAndClose will end (io.EOF occurs on the client).

Can you open a pull request to fix this issue?

Fixed by https://github.com/elazarl/goproxy/pull/256


Class 'Pheanstalk\PheanstalkInterface' not found

Laravel Version: 5.7.25
PHP Version:7.2
Database Driver & Version:

Description:
with php artisan queue:listen command i receive
   Symfony\Component\Debug\Exception\FatalThrowableError  : Class 'Pheanstalk\PheanstalkInterface' not found

  at /Users/aymanelarian/xxx/xxxxxx/vendor/laravel/framework/src/Illuminate/Queue/Connectors/BeanstalkdConnector.php:35
    31|     protected function pheanstalk(array $config)
    32|     {
    33|         return new Pheanstalk(
    34|             $config['host'],
  > 35|             $config['port'] ?? PheanstalkInterface::DEFAULT_PORT,
    36|             $config['timeout'] ?? Connection::DEFAULT_CONNECT_TIMEOUT,
    37|             $config['persistent'] ?? false
    38|         );
    39|     }

  Exception trace:

  1   Illuminate\Queue\Connectors\BeanstalkdConnector::pheanstalk(["beanstalkd", "localhost", "default"])
      /Users/aymanelarian/xxx/xxxxxx/vendor/laravel/framework/src/Illuminate/Queue/Connectors/BeanstalkdConnector.php:22

  2   Illuminate\Queue\Connectors\BeanstalkdConnector::connect(["beanstalkd", "localhost", "default"])
      /Users/aymanelarian/xxx/xxxxxx/vendor/laravel/framework/src/Illuminate/Queue/QueueManager.php:157

Steps To Reproduce:
run cmd after composer require "pda/pheanstalk" and config queue setting  then listen to beanstalkd Queue it fail

it seem that namespace changed to Pheanstalk\Contract\PheanstalkInterface; and  that mean laravel not optimised with the latest Pheanstalk

Pheanstalk 4 support will be introduced in 5.8, however it could be possible to add support to 5.7 while still maintaining 3.x support if @taylorotwell okay with that.
BTW I been using pheanstalk 4 on Laravel 5.7 for a week now and running smoothly.

5.7 will go into security mode only on the 4th of March but we could indeed add this in the meantime so people on 5.7 can still use it. I'll ping taylor about it.

@crynobone if you can send something in for 5.7 which doesn't breaks anything you can do so :)

Gonna close this then since it's not really a bug or anything.

@driesvints See #27622


Remove BootstrapModule

The extra layer of two injectors, simulated or not, introduces unnecessary complexity into the API.  The proposal here is to merge all the functionality in BootstrapModule (lifecycle listeners, autobind classpath scanning, configuration, etc.) into modules that are loaded by the main injector.

On second thought, it may not be possible to remove BootstrapModule due to it's use in LifecycleInjector.bootstrap.  Instead PR #186 deprecates access to all generic bindings so that BootstrapModule is more specific in its use as a way to customize the LifecycleInjectorBuilder and not as a way to specify application bindings.


Metadata search doesn't work for table columns in DB2

System information:

DBeaver version 21.2.0 UE

Connection specification:

Database name and version DB2 Express
Driver name DB2 LUW

Describe the problem you're observing:
Metadata search doesn't find table columns.
Steps to reproduce, if exist:

Open Metadata Search and try to find an existing column.

Expected: Object is found
Actual: Nothing is found

Fixed.

Verified


Why is the drawing efficiency very slow after using controlnet? I use a 4090 graphics card, use image2image, 720P, and it takes an average of 7-8 seconds for a picture, and an average of 2~3it/s. I use 3 controlnets to superimpose, Is there any way to speed it up?

Anyone can help me

I use Tesla V100, also made 7~8 second per case.


Alibaba.dingtalk

[x] Have you signed the Contributor License Agreement?
[x] Have you checked that there aren't other open pull requests for the same manifest update/change?

[x] Does your manifest conform to the 1.0 schema?

Note: <path> is the name of the directory containing the manifest you're submitting.

Microsoft Reviewers: Open in CodeFlow

This failed previously.  Rerunning with the automation to set baseline.

/azp run

This failed previously. Rerunning with the automation to set baseline.

The installer will fail with exit code 1233 during my local test, but actually it DID successfully installed the package.

I think it relates to autolaunching dingding cilent when the install process finished.


feat: automatic dialect detection

The dialect of the database that a DatabaseClient is connected to can be automatically detected:

DatabaseClient#getDialect() has been added. This method always returns the dialect of the underlying database. It will do so by executing a query that detects the dialect. This query can also be executed and cached automatically in the background during startup (see below).
SessionPoolOptions#setAutoDetectDialect(true) will cause the dialect detection query to be executed in the background automatically when a new client is created. This is disabled by default, except for when a Connection API connection (or anything that depends on that, such as JDBC) is opened. The reason for this default behavior is that a normal Spanner instance does normally not need to know what the dialect of the underlying database is, while the Connection API does. This reduces the number of times the detection query will be executed during production use.

Closing in favor of #1677


Fix adding defaultCssClass

Because method class() is not used for css.
For work with attributes are there magic methods add, get and set.
In this case addClass('class') is better solution, because your solution could make array string conversion problem in some case.
I don`t make pull request offten. Sorry, if there is some mistakes in usage.
David

Actually, this is a good PR! Using addClass is much easier. If you find any other similar issues, don't hesitate to create another one.
To be honest, I started to dislike magic accessors. I would prefer appendAttribute('class', $value) in the future, but for now this is sufficient.
Thank you for your contribution.


feat: initial structure + first approach

Description

Use Typescript
Use Jest as test runner
Generate a first simple cli to understand how ink and meow works

DISCLAIMER
The relevant part of this PR is the first structure and the tests approach. For now, there isn't an example of any feature of the cli and also it has't the final project structure (there are no hooks and things like that 😅)
Related issue(s)

Looks good so far, I would not worry much about project structure as this can always be improved later, most important is to have first hook, first simple functionality like validation and then the rest will be easier and others will know how to contribute further

Looks good so far, I would not worry much about project structure as this can always be improved later, most important is to have first hook, first simple functionality like validation and then the rest will be easier and others will know how to contribute further

Working on it!! 😛

I was thinking of an approach if we were going to use meow for parsing CLI commands
#!/usr/bin/env node
import React from 'react';
import {render} from 'ink';
import meow from 'meow';
import App from './ui';
import Generate from './generate';

const cli = meow(`
	Usage
	  $ generate <asyncapi> <template>

	Options
		--name  Your name

	Examples
	  $ learnINK --name=Jane
	  Hello, Jane
`, {
	flags: {
		name: {
			type: 'string'
		}
	}
});

let [command, ...args] = cli.input;

switch(command) {
	case "generate":
		render(<Generate file={cli.input[1]} />);
}

We can make different ink components for every command and run them like this.
or we could also create a single UI component where we can check for all the different commands and render different UI accordingly.

@jotamusik according to you what should we go with.

yeah @Souvikns!! that's the idea, this is one of the steps that will drive to the approach you mention 😋
Thanks for the feedback!!  🤗

so what is missing, when do you feel comfortable to get some code review. I guess that just some cleanup is needed and we can also slowly start splitting the initial issue into separate issues for separate detaild discussion (if needed).
I want asyncapi context -f <filepath> so much that you can't imagine 😄

so what is missing, when do you feel comfortable to get some code review. I guess that just some cleanup is needed and we can also slowly start splitting the initial issue into separate issues for separate detaild discussion (if needed).
I want asyncapi context -f <filepath> so much that you can't imagine 😄

we can have a separate command namely context and have operations to add, remove or change the filename of the spec file. then if the context file is not set then we can throw an error or ask to set the context.

@jotamusik I have a doubt about the argument parser meow
I was playing around with commander and meow in general, I don't have much experience with meow.  Could we achieve this with meow through any plugin or do we need to write specific code for this.
commander
when I use asyncapi --help
Usage: asyncapi [options] [command]

Options:
  -V, --version                             output the version number
  -h, --help                                display help for command

Commands:
  generate [options] <asyncapi> <template>

and when I use asyncapi generate --help
Usage: asyncapi generate [options] <asyncapi> <template>

Options:
  -i, --install  installs the template and its dependencies (defaults to false)
  -h, --help     display help for command

It generates help option for all the commands , this helps users as different commands have different options that it supports.
meow
when I use asyncapi --help
  Usage
  $ generate <asyncapi> <template>

  Options
  -V, --version    output the version number
  -h, --help       display help for command

and when I use asyncapi generate --help the auto help command does not work but it returns this object
{
  input: [ 'generate' ],
  flags: { help: true },
  unnormalizedFlags: { help: true },
  pkg: {
    name: 'mew',
    version: '1.0.0',
    description: '',
    main: 'index.js',
    scripts: { test: 'echo "Error: no test specified" && exit 1' },
    keywords: [],
    author: '',
    license: 'ISC',
    dependencies: { 'cli-meow-help': '^2.0.2', commander: '^7.2.0', meow: '^9.0.0' },
    readme: 'ERROR: No README data found!',
    _id<EMAIL_ADDRESS>  },
  help: '\n' +
    '  Usage\n' +
    '  $ generate [options] <asyncapi> <template>\n' +
    '\n' +
    '  Options\n' +
    '  -V, --version    output the version number\n' +
    '  -h, --help       display help for command\n',
  showHelp: [Function: showHelp],
  showVersion: [Function: showVersion]
}

we can create our own custom help string for each commands though, but is there a way of doing this automatically in meow.

@jotamusik I have a doubt about the argument parser meow
I was playing around with commander and meow in general, I don't have much experience with meow. Could we achieve this with meow through any plugin or do we need to write specific code for this.
commander
when I use asyncapi --help
....

As far as I know, we can't do this with meow. I maybe wrong though.
But this makes me wonder, should we use the commander approach by doing cli <some-action> <options> (ex. cli generate) or just use flags/options for everything(like cli --generate)?
What are your thoughts? @jotamusik @derberg @Souvikns

Looks like meow want's to support autogeneration of help -> https://github.com/sindresorhus/meow/issues/80
no strong opinion for commander or yargs instead of meow. Gatsby CLI uses for example ink with yargs 🤷🏼
@aayushmau5 have a look at https://github.com/asyncapi/cli/issues/1 to see how we want to design commands

I was playing with meow a little bit and I think we can create our own help functions and make it just like commander from checking the command and the help flag
const meow = require('meow');
const mewHelp = require('cli-meow-help');

const cli = meow(`
Usage
$ generate [options] <asyncapi> <template>

Options
-V, --version    output the version number
-h, --help       display help for command
`);

const commands = {
    "generate": `
Usage
asyncapi generate [options] <asyncapi> <template>

Options
-i, --install   installs the template and its dependencies
-h, --help      display help for command
`
}
const helperText = (command, flag) => {
    if(commands[command] && flag){
        console.log(commands[command])
    }
}

helperText(cli.input[0], cli.flags.help)

@jotamusik hey, where are we with this one. Do you need some help? Just want to make sure you are not trying to be too perfect with first PR. Keep in mind we can improve further and once we get this one merged we will also enable more people to help and improve further.

Hi! @derberg I'm working on a first minimal feature of validation, so still WIP without trying to be perfect 😅
I know it's more time than expected, but it's because I have been busy the past weeks

I'm having some issues with the tests of the validation component and I think that it's related with the useEffect() and the component lifecycle.
If anyone can take a look, it would be great! 😛

The useEffect is not working when the component is called in the test environment. There is some problem with the ink-testing-library I found this https://github.com/vadimdemedes/ink-testing-library/issues/3

@jotamusik
setTimeout(() => {
      expect(lastFrame()).toBe(`File: /home/jorgengo/dev/asyncapi/cli/test/specification.yml - was validated successfully!`);
    }, 2000);

wrapping the test with setTimeout is working.

ohhh!!! so much thank you!! 🤗 now I can finish this!! 👏🏻

Finally, the first working and refined version has come!! Please, take a look and feel free to comment 😛

In CommandsRouter.spec.tsx the last test case is failing. @jotamusik is that intended as of now. When I tried running with async-await then it was passing.

@Souvikns could you please take a screenshot because in my local enviroo

Yeah sure, 1 test case is failing

but it is passing when I am making the test function async
test('route to validate component when validate command passed', async () => {
    const { lastFrame } = renderAux({
      input: ['validate'],
      help: 'Help Message',
      flags: {
        context: 'test/specification.yml',
        watch: false,
      },
    });

    // NOTE: This is needed to be able to match the test result file name due to its dependency with the file system
    const file = new SpecificationFile('test/specification.yml');

    setTimeout(() => {
      expect(lastFrame()).toBe(chalk.green(`File: ${file.getFileName()} - was validated successfully!`));

    }, 200);
  });

test('route to validate component when validate command passed', (done) => {
    const { lastFrame } = renderAux({
      input: ['validate'],
      help: 'Help Message',
      flags: {
        context: 'test/specification.yml',
        watch: false,
      },
    });

    // NOTE: This is needed to be able to match the test result file name due to its dependency with the file system
    const file = new SpecificationFile('test/specification.yml');

    setTimeout(() => {
      expect(lastFrame()).toBe(chalk.green(`File: ${file.getFileName()} - was validated successfully!`));

    }, 200);
    done();
  });

this is also working, I think the issue with the testing library is causing this.

test('route to validate component when validate command passed', (done) => {
    const { lastFrame } = renderAux({
      input: ['validate'],
      help: 'Help Message',
      flags: {
        context: 'test/specification.yml',
        watch: false,
      },
    });

    // NOTE: This is needed to be able to match the test result file name due to its dependency with the file system
    const file = new SpecificationFile('test/specification.yml');

    setTimeout(() => {
      expect(lastFrame()).toBe(chalk.green(`File: ${file.getFileName()} - was validated successfully!`));

    }, 200);
    done();
  });

this is also working, I think the issue with the testing library is causing this.

Could you please check the test result when you increase the waiting time?? 🤔

Yes the tests were passing when I increased the waiting time.

Yes the tests were passing when I increased the waiting time.

Thanks!! What was the amount of time that let you pass the tests?? To make the change and make the commit 👅

Yes the tests were passing when I increased the waiting time.

actually, I was testing before and I had changed some code previously so it was passing and I thought it was passing because I had increased the time.
setTimeout(() => {
      expect(lastFrame()).toBe(chalk.green(`File: ${file.getFileName()} - was validated successfully!`));
    }, 200);
    done();

Actually Increasing the time does not help, but I was calling done() method after the setTimeout

I think that we are done for this first version!! 🤔


Why does this.status = 404 turn into 405?

Hi,
I am curious why setting this.status = 404 is filtered to return a 405 error instead?
Thanks

A 405 is returned if a route matched the path but not the method. Can you paste an example of the code that reproduces this?

I want to return a 404 status from my API when a resource is not found. I am setting the status like this.
this.status = 404;
in router.js on line 361 the 404 status gets explicitly filtered into a 405 status. I was just wondering why?
here is the block from router.js
if (!this.status || this.status === 404) {
  this.matched.forEach(function (route) {
    route.methods.forEach(function (method) {
      allowed[method] = method;
    });
  });

I'm having the same problem. Example code:
app.get('/404', function* () {
  this.status = 404;
  this.body = "Not found";
});

Thank you very much.


Dev Test: Virtual Workshop

@amanda-phet, @kathy-phet, @ariel-phet, @mattpen, @KatieWoe the Virtual Workshop is ready for dev testing. This will be shared with a client on 12/30. Please document issues in https://github.com/phetsims/website-meteor/issues and link to this issue.
Assigning @ariel-phet for prioritization.

General Dev Test

What to Test

Click every single button.
If there is sound, make sure it works.
Try to include browser version numbers

Focus and Special Instructions
Please make sure all links work as expected, the page navigation works appropriately in both desktop and mobile view (the column on the left side only, not the top nav bar), and that the responsive layout handles the content without looking buggy. Sometimes the animations can be slow to respond as the page is loading, so please look out for performance issues too.

General Dev Test Platforms

[x] Latest macOS, Chrome and Safari
[x] Latest iOS, Safari
[x] Windows 10, all browsers (IE dropped)
[x] Latest Chrome OS, Chrome

These issues should have either use the labels "status:ready-for-qa" or "status:ready-for-review." If it is ready for
QA then close the issue if fixed. If ready for review then leave open and assign back to the developer.

Link(s)

Virtual Workshop

FAQs for QA Members

There are multiple tests in this issue... Which test should I do first?
Test in order! Test the first thing first, the second thing second, and so on.

How should I format my issue?

We typically assign the developer who opened the issue in the QA repository.

My question isn't in here... What should I do?
You should:

Consult the QA Book.
Google it.
Ask Katie.
Ask a developer.
Google it again.
Cry.

Since the QA tests in progress is up to 4, it might not be possible to get to this before the end of 2020.
@ariel-phet suggested that some of us bang on the page for a bit so that we can feel ok publishing it before the grant deadline, but do a more comprehensive QA test once they finish up some other in-progress testing.

Speaking with @kathy-phet the plan will be to do a lighter test on this page, and get it published on the website. When there is more space for QA we will do a more thorough test before announcing on social media and such.
@oliver-phet @amanda-phet @DianaTavares please test drive the page for something around 30 minutes of testing, I will do the same on Windows (and maybe my phone)
@phet-steele could you also take a test drive (please limit your test to no more than 1 hour)

@chrisklus testing for this initial round is complete.
I suggest triaging with @amanda-phet for the first publication (where we will make the workshop live on our site, but not yet advertise to the public).

@ariel-phet did you notice no issues on windows? I know @chrisklus and @kathy-phet discussed a severe issue that came up, but I don't see an issue about it (I'm sure @chrisklus can make it, but I wondered if you were involved in the discussion at all).

All issues that came from this dev test specific to the virtual workshop have been completed. I'll open a new QA issue when ready.