Immediate cancellation of RPC calls leads to HTTP/2 RST packets for uninitialized streams

What version of gRPC and what language are you using?
From Nuget.org:
Grpc 1.22, Grpc.Core 1.22
What operating system (Linux, Windows,...) and version?
Microsoft Windows 10
What runtime / compiler are you using (e.g. python version or version of gcc)
.NET 4.7.2
Microsoft C# Tools 2.10.0
(as distributed with VS2017 v. 15.9.8)
What did you do?
We implemented a service that bundles and distributes long running client requests (using an internal protocol) to multiple upstream services. Said client requests occur in (sometimes rather large) batches of service method calls.
In the process of forwarding those calls, the intermediate service occasionally cancel newly created upstream calls without waiting for any replies (since some situation downstream or within the service renders the call unnecessary).
In this case, we encountered GOAWAY signals from the service further upstream (implemented in Java), which seems to be the way Netty handles HTTP/2 RST signals for uninitialized streams.
I am able to consistently reproduce the issue using code akin to:
var service = new MyService.MyServiceClient(channel);
for (int i = 0; i < 20; ++i)
{
    Task.Run(async () => {
        using (var cancel = new CancellationTokenSource())
        using (var call = service.MyMethod(new MyRequest(), cancellationToken: cancel.Token))
        {
            var responses = call.ResponseStream;
            cancel.Cancel();
            while (await responses.MoveNext());
        }
    });
}

What did you expect to see?
For any call that has not been communicated to the server yet, GRPC.Core should either initialize the according HTTP/2 stream before resetting it, or (probably more desirable for performance) any calls that are cancelled before being processed should not trigger any HTTP/2 communication.
What did you see instead?
Using Wireshark, we were able to determine that GRPC.Core seems to send out HTTP/2 RST packages for streams that are not yet initialized (cf. packages 19 and 20), which causes the Netty-based upstream service to return a GOAWAY signal, rendering the entire connection unusable. Section 6.4 of RFC 7540 describes this as illicit behavior:

RST_STREAM frames MUST NOT be sent for a stream in the "idle" state. If a RST_STREAM frame identifying an idle stream is received, the recipient MUST treat this as a connection error (Section 5.4.1) of type PROTOCOL_ERROR.

(cf. also the stream state diagram of section 5.1).
Capture (abbreviated:)
No.     Time           Source                Destination           Protocol Length Info
     16 0.119815       [client]              [server]              GRPC     198    HEADERS[5]: POST /[service]/[service_method], DATA[5] (GRPC) (PROTOBUF)
     17 0.120278       [server]              [client]              HTTP2    156    HEADERS[5]: 200 OK, DATA[5]
     18 0.120665       [client]              [server]              TCP      60     59650 → 22212 [ACK] Seq=778 Ack=305 Win=525056 Len=0
     19 0.121142       [client]              [server]              HTTP2    145    RST_STREAM[5], RST_STREAM[7], RST_STREAM[9], RST_STREAM[11], RST_STREAM[13], RST_STREAM[15], WINDOW_UPDATE[0]
     20 0.121368       [client]              [server]              HTTP2    67     RST_STREAM[17]
     21 0.121568       [server]              [client]              HTTP2    94     GOAWAY[0]

Complete capture

No.     Time           Source                Destination           Protocol Length Info
      1 0.000000       [client]              [server]              TCP      66     59650 → 22212 [SYN, ECN, CWR] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
      2 0.000023       [server]              [client]              TCP      66     22212 → 59650 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=128
      3 0.000459       [client]              [server]              TCP      60     59650 → 22212 [ACK] Seq=1 Ack=1 Win=525568 Len=0
      4 0.000878       [client]              [server]              HTTP2    153    Magic, SETTINGS[0], WINDOW_UPDATE[0], PING[0]
      5 0.000889       [server]              [client]              TCP      54     22212 → 59650 [ACK] Seq=1 Ack=100 Win=29312 Len=0
      6 0.001136       [client]              [server]              GRPC     392    HEADERS[1]: POST /[service]/Check, WINDOW_UPDATE[1], DATA[1] (GRPC) (PROTOBUF), WINDOW_UPDATE[0]
      7 0.001141       [server]              [client]              TCP      54     22212 → 59650 [ACK] Seq=1 Ack=438 Win=30336 Len=0
      8 0.001973       [server]              [client]              HTTP2    94     SETTINGS[0], WINDOW_UPDATE[0]
      9 0.002438       [server]              [client]              HTTP2    80     SETTINGS[0], PING[0]
     10 0.002707       [server]              [client]              HTTP2    173    HEADERS[1]: 200 OK, DATA[1]
     11 0.002791       [client]              [server]              TCP      60     59650 → 22212 [ACK] Seq=438 Ack=67 Win=525312 Len=0
     12 0.056455       [client]              [server]              TCP      60     59650 → 22212 [ACK] Seq=438 Ack=186 Win=525312 Len=0
     13 0.119339       [client]              [server]              GRPC     237    SETTINGS[0], HEADERS[3]: POST /[service]/[service_method], DATA[3] (GRPC) (PROTOBUF), WINDOW_UPDATE[0], PING[0]
     14 0.119551       [client]              [server]              HTTP2    67     RST_STREAM[3]
     15 0.119624       [server]              [client]              HTTP2    71     PING[0]
     16 0.119815       [client]              [server]              GRPC     198    HEADERS[5]: POST /[service]/[service_method], DATA[5] (GRPC) (PROTOBUF)
     17 0.120278       [server]              [client]              HTTP2    156    HEADERS[5]: 200 OK, DATA[5]
     18 0.120665       [client]              [server]              TCP      60     59650 → 22212 [ACK] Seq=778 Ack=305 Win=525056 Len=0
     19 0.121142       [client]              [server]              HTTP2    145    RST_STREAM[5], RST_STREAM[7], RST_STREAM[9], RST_STREAM[11], RST_STREAM[13], RST_STREAM[15], WINDOW_UPDATE[0]
     20 0.121368       [client]              [server]              HTTP2    67     RST_STREAM[17]
     21 0.121568       [server]              [client]              HTTP2    94     GOAWAY[0]
     22 0.121611       [server]              [client]              TCP      54     22212 → 59650 [FIN, ACK] Seq=345 Ack=882 Win=32512 Len=0
     23 0.121629       [client]              [server]              HTTP2    67     RST_STREAM[19]
     24 0.121809       [client]              [server]              HTTP2    67     RST_STREAM[21]
     25 0.122028       [server]              [client]              TCP      54     22212 → 59650 [RST, ACK] Seq=346 Ack=908 Win=32512 Len=0
     26 0.122058       [client]              [server]              TCP      60     59650 → 22212 [ACK] Seq=908 Ack=346 Win=525056 Len=0
     27 0.122075       [server]              [client]              TCP      54     22212 → 59650 [RST] Seq=346 Win=0 Len=0
     28 0.122184       [client]              [server]              HTTP2    67     RST_STREAM[1]
     29 0.122186       [server]              [client]              TCP      54     22212 → 59650 [RST] Seq=346 Win=0 Len=0

The Netty service produces the following stack trace:

Netty stack trace

[2019-07-16 10:02:03,218] [INFO ] [io.grpc.netty.NettyServerTransport.connections] [GrpcWorkerExecutor-9] Transport failed
io.netty.handler.codec.http2.Http2Exception: Stream 7 does not exist
        at io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:85)
        at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.verifyStreamMayHaveExisted(DefaultHttp2ConnectionDecoder.java:605)
        at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onRstStreamRead(DefaultHttp2ConnectionDecoder.java:367)
        at io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onRstStreamRead(Http2InboundFrameLogger.java:80)
        at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readRstStreamFrame(DefaultHttp2FrameReader.java:516)
        at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:260)
        at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160)
        at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
        at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:148)
        at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:378)
        at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:438)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:617)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:534)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.base/java.lang.Thread.run(Thread.java:834)

Anything else we should know about your project / environment?
Note: As this seems to be a timing problem and cancellations generally lead to exceptions within the C# gRPC API, I am best able to reproduce the issue without using the Visual Studio debugger. The debugger seems to slow things down considerably whenever any exceptions are in flight (I guess because it's signalling them all to the IDE and blocking the debugged process).

Also, in case anyone wonders about the [square brackets] within the capture above: I was ordered to redact system-internal information.
The /[service]/Check call is our internal health checking method, which I am leaving open in order to detect the GOAWAY signal within my test client (since all open calls are terminated when a GOAWAY signal is received).

Since the stale bot has marked this issue, I'd like to reiterate that this still is quite a big problem for our specific use case (although it likely is not all that typical for GRPC). We're using GRPC for continuously streaming live data, and the individual subscriptions can be triggered by user input, which means there's always the possibility of triggering this bug by accident. We also have to balance subscriptions between multiple connections with potentially small throughput (i.e. server capacity is not the problem), which can also lead to the same behavior.
It's also important to note that the delay between opening a request and cancelling it must not necessarily be that low in order to trigger this bug; as long as the request did not lead to an initialized HTTP/2 stream (which depends on the load of the library's internal task queue), this seems to happen rather deterministically. That means that if a user opens a couple of hundreds of subscriptions and cancels them again, they can still trigger this manually.
In either case, the entire connection to the server becoming unusable (along with any open subscriptions) is kind of a worst case scenario. So, for now, I had to avoid this issue by introducing a rather unfortunate hack: Our clients will not cancel any RPC call for which they have not received at least one response (since that is the only way I can be sure that the call has actually made it through the C implementation's internal queue).

Although this triggers in the C# API, the underlying issue should be investigated as part of the C-core HTTP/2 transport. Reassigning to @yashykt - can you try to repro this behavior in a unit-test if it's deterministic as reported?

@yashykt any update?

This bug is still existing in the latest C++ code of grpc-core, which is impacting all libraries basing on grpc-core, not only C#.
The cause is that:
current design is using grpc_chttp2_transport.outbuf to send the payload of all streams in the transport. The logic of flushing data into grpc_chttp2_transport.outbuf is in grpc/src/core/ext/transport/chttp2/transport/writing.cc
grpc_chttp2_begin_write_result grpc_chttp2_begin_write(
    grpc_chttp2_transport* t) {
  WriteContext ctx(t);
  ctx.FlushSettings();
  ctx.FlushPingAcks();
  ctx.FlushQueuedBuffers();
  ctx.EnactHpackSettings();

  if (t->flow_control->remote_window() > 0) {
    ctx.UpdateStreamsNoLongerStalled();
  }

  /* for each grpc_chttp2_stream that's become writable, frame it's data
     (according to available window sizes) and add to the output buffer */
  while (grpc_chttp2_stream* s = ctx.NextStream()) {
    StreamWriteContext stream_ctx(&ctx, s);
    size_t orig_len = t->outbuf.length;
    stream_ctx.FlushInitialMetadata();
    stream_ctx.FlushWindowUpdates();
    stream_ctx.FlushData();
    stream_ctx.FlushTrailingMetadata();
    if (t->outbuf.length > orig_len) {
      /* Add this stream to the list of the contexts to be traced at TCP */
      s->byte_counter += t->outbuf.length - orig_len;
      if (s->traced && grpc_endpoint_can_track_err(t->ep)) {
        grpc_core::ContextList::Append(&t->cl, s);
      }
    }
    if (stream_ctx.stream_became_writable()) {
      if (!grpc_chttp2_list_add_writing_stream(t, s)) {
        /* already in writing list: drop ref */
        GRPC_CHTTP2_STREAM_UNREF(s, "chttp2_writing:already_writing");
      } else {
        /* ref will be dropped at end of write */
      }
    } else {
      GRPC_CHTTP2_STREAM_UNREF(s, "chttp2_writing:no_write");
    }
  }

  ctx.FlushWindowUpdates();

  maybe_initiate_ping(t);

  return ctx.Result();
}

When grpc_chttp2_begin_write, it FlushQueuedBuffers of grpc_chttp2_transport and then tries to iterate all data of streams which need to write data. However, the ctx.NextStream() cannot always return each stream because grpc_chttp2_transport.outbuf has a target_write_size limitation(1MB). see code below
  grpc_chttp2_stream* NextStream() {
    if (t_->outbuf.length > target_write_size(t_)) {
      result_.partial = true;
      return nullptr;
    }

    grpc_chttp2_stream* s;
    if (!grpc_chttp2_list_pop_writable_stream(t_, &s)) {
      return nullptr;
    }

    return s;
  }

/* How many bytes would we like to put on the wire during a single syscall */
static uint32_t target_write_size(grpc_chttp2_transport* /*t*/) {
  return 1024 * 1024;
}

This means in every invocation to grpc_chttp2_begin_write, it cannot make sure every stream is flushed in first stream iteration.
When some streams is not flushed in last round, the code will trigger next round by invoking grpc_chttp2_begin_write in write_action_end_locked so as to flush the left streams.
However, each time by invoking grpc_chttp2_begin_write, it will invoke grpc_chttp2_begin_write which will ctx.FlushQueuedBuffers() each time. If upper code call grpc_chttp2_cancel_stream in chttp2_transport.cc, it will append the rst_stream frame into grpc_chttp2_transport.qbuf which will be flushed to grpc_chttp2_transport.outbuf in the begining of next grpc_chttp2_begin_write. This could cause ths rst_stream of some streams to be flushed ahead of the header frame of that stream.
In summary, so when the load of requests is high, the output is limited by the target_write_size limitation(1MB) and has to seperate the streams into several round of grpc_chttp2_begin_write, and some rst_stream frame is sent shortly after the stream is created, this bug will happen.
From the code perspective, maybe this is caused by seperating the rst_stream from the StreamWriteContext object. If to handle rst_stream in the StreamWriteContext class, like FlushInitialMetadata, maybe this could be handle easily

I think this is still a bug. @ctiller

any update?

I think realistically, the chances that this will get fixed in the future is very low:

the issue has been open for quite a long time
Grpc.Core has been using C-core from the LTS 1.46.x branch for the past year, so coming back to that code and fixing C-core issues has gotten more difficult.
Grpc.Core is very close to getting deprecated (we're still evaluating the data, but the officially announced maintenance period is now over so even though we haven't made the deprecation official yet, we are likely to do that soon).

Since it's very unlikely that this will get fixed, I'll close this issue as "not actionable".


[BUG] Fast keys for new building are annoying and work even outside Settlers

Bug
Keypress anywhere in Windows starts a building in Settlers.
Description
Only in the community patch (4.66.1528) some keys on the keyboard initiate a builing in the Settlers game. This is very annoying because it happens when you type a message in Settlers, but even more annoying is the fact that pressing the keys works everywhere within Windows. When I type an e-mail, a building is initiated.
Discovered keys so far

B starts building a Stone mine
I starts building a Stonecutter's hut
J starts building a Sawmill
K starts building a Forrester's hut
M starts building a Medium residence
N starts building a Small residence
U starts building a Woodcutter's hut
V starts building a Large Residence

To Reproduce
Steps to reproduce the behavior:

Start Settlers Community Patch or Community Patch PTR (using Community Patch Launcher 0.0.1)
Click on Single Player
Click on Choose Map
Select any map with enough space to build
Click Ok on the Map Settings screen
Click Ok on the Game Settings screen
Click on any other program in Windows to set the focus away from Settlers
Type any of the keys listed above

Expected behavior

Keys typed outside the Settlers programs should never be captured within Settlers!
Keys should neither be captured in the message box in Settlers

The Key is only avaible in the closed Beta. This will be removed asap.

Hey! This is not a Bug from the Launcher, this has something to do with the Hotkey Plugin made by @WizzardMaker.
He will have a look at it asap.
Greetings

I think it is a very dangerous plugin. This means that Settlers can capture all our keyboard actions and send any and all information somewehere without us knowing it.

The plugin does not save any keyboard actions. He only reacts to a amount of keys. I know that u are quite confused about this.
Wizzard will try to fix this asap.
Greetings

If you want more information about this, you can wait here until WizzardMaker arrives or visit our Discord Server: http://discord.gg/Vhs3e5N

Hey Ausgeben and Wizzard: Asap promises soon. Now, 4 days later and after syncing patches the problem remains.

The problem is on the to do list for the hotkeys plugin
I can't give an ETA though

I understand. It's very annoying. Just now I tried the history edition through the community patch launcher, same problem. And even starting the history edition directly has the same problem. Whilst typing this message, Settlers begins building all kinds of buildings.

Select the He via the Launcher and then start the game


Feat: Generate Random color cards on the audio section as Spotify

Describe the feature
Generate Random color cards on the audio section as Spotify which is user experience
I am GSSOC24 Contributor
Please assign me, I would like to work on it
Add ScreenShots
https://github.com/Satyam1923/Spring-Music-Player/assets/121508628/9575216d-adcd-416b-9469-8f7b4b8444a6
Record

[X] I agree to follow this project's Code of Conduct
[X] I have checked the existing issues
[X] I'm a GSSOC'24 contributor
[X] I want to work on this issue

@SrinivasDevolper see don't use random color i'll give you some 6 colors if song photo is near them you can change it with that color fine.

Ok sure, I can add 6 colors, you can share me, and I will do it well

use above colors.


Investigate limitations and expand ability to construct objects within functions to be JAX-transformed

Currently, the hamiltonian operator validation step will raise an error if attempting to JAX-transform a function in which a hamiltonian/lindblad model or a Solver is constructed. Setting validate=False bypasses this, and in some cases leads to being able to JAX-transform a function in which one of these objects is constructed. However, some cases that should work don't, e.g. passing in operators=[Operator.from_label('X')] to the constructor leads to an error, which seems to stem from a tracing issue with to_array.
An example of a case that works is in test.dynamics.solvers.test_solver_classes.TestSolverJax.test_transform_through_construction_when_validate_false (currently in PR #30 but should be merged soon). We should add more tests of this form with different input types, or identify what particular part of construction is causing an error and add jit tests for that (e.g. if to_array is the problem, add tests for using to_array within a function and try transforming it).

Closing as this is open-ended with no plans to work on it.


Fix a typo in Comment

od --> of

This change is

LGTM, thanks

Please sign the CLA

@piosz  signed, thanks.


And project support pass a tracker?

Describe the feature
Pass a tracker instead of global one

Motivation
I want use other ServiceName of tracker like "mysql" "postgres"

Does it look like you would have 2 more tracer clients in your project? @greatcat-taihe Of course, you can pass the specified tracer, just add an option for gorm.opentracing. But i wonder do you have different tracers in one server and why?

Does it look like you would have 2 more tracer clients in your project? @greatcat-taihe Of course, you can pass the specified tracer, just add an option for gorm.opentracing. But i wonder do you have different tracers in one server and why?

I use jaeger, after give gorm a tracker, jaeger dependency look more clear. Database will be a independent service

I got it


Pull source as git submodules by default

duniverse pull will now clone the source code and add each of
the vendored packages as a git submodule. It does this by adding
the submodule commit objects after the source pulls have been done.
This is a lot faster than the default git submodule add which
requires setting up remote tracking branches for each of the
remotes.
This adds a --no-submodules option to pull to just do a
source code clone.

It can't be on init, since the submodule generation requires pulling the source trees. Git submodules work by generating a special git link object based on the commit ref of the checked out submodule, and so they can't work without a checkout of the individual modules.
I'm thinking about the common case use here -- most beta testers of duniverse have been fairly clear that they don't want to check in the source code directly, so git submodules are the next best thing.  Your point about messing with the git history is important: however, this command can be reverted by simply git reset --hard as it only stages the files.
There is also an important missing piece of this PR: to delete unused gitmodules by comparing the present modules to the requested ones. This implies duniverse initially will not support projects with other external submodules, but this seems an acceptable limitation for a first release.

I meant that the option is passed to init to be stored in the dune-get file, so the user doesn't have to pass this option on every pull. Why is that not possible ?

That makes sense.  I'm happy with submodules being the default recorded value on init, since that's the workflow we want to promote.


Infer repo name from server url

Fixes https://github.com/git-tfs/git-tfs/issues/1201

Thanks! 👍

@0x53A Seems good. Leave a comment when you think it's ready to merge...

Ready from my side

Ok. Thanks


Clean-up lib/i18n

Make everything into classes, and remove duplicated and identical code.
@Mehrad0711 please look carefully at this. I recommend using the "ignore white space" option of the diff (-w in the command-line). Also check: some isGoodNumber implementations include Persian numbers, even for languages that use the latin alphabet.
Japanese also might need work, because it's an ideographic language but it's treated like an alphabetic one.

looks good. Need to update the isGoodNumber for some of the languages. Chinese numbers should be added to the Chinese package.
Also, Spanish and Japanese isGoodSentence can be improved. Shall I fix those in this PR?

Please make a separate PR. This one is already going to cause conflicts.
I'm still unclear on the design/purpose of isGoodNumber by the way - shouldn't numbers always be in ASCII digits? The tokenizer normalizes the numbers in words (incl. Chinese characters) to numbers in ASCII digits.


Add toggle selector and event to render

We can click/hover to render from Client
For examples applied cases

Load more items
Load detail of information
Load for rarely used data

Issue: #75

Thanks for PR @ThanhKhoaIT!
It looks excellent, I would only add more text to the README about the usage :)

Thanks for replied, @nikolalsvk ❗❗❗
I have updated README

Hey guys, I saw the attached issue / this PR, realized it was exactly what I was looking for! So thanks to both @ThanhKhoaIT and @nikolalsvk
I've been testing the changes and just wanted to share some findings:

In it's current form, the toggle feature does not work when RenderAsync.config.turbolinks = true. The conditional for turbolinks fires first and triggers the partial to render, regardless of the toggle config. This is an easy fix by changing the conditional to <% if turbolinks && !toggle %>.
It should be noted that if you're using render_async with toggle inside of a loop, the selector must be named accordingly. This is obvious but might be worth mentioning in the README.

Otherwise, this is freaking awesome. Seriously, thank you both for your work!


WIP: shorten compile time with nospecialize

This is a branch that, for the moment, is mostly for https://github.com/JuliaLang/julia/issues/35131. Steps to reproduce: after deving this branch, do
julia> using LoopVectorization

shell> pwd
/home/tim/.julia/dev/LoopVectorization/test

julia> include("broadcast.jl")
(T, #= /home/tim/.julia/dev/LoopVectorization/test/broadcast.jl:7 =# @__LINE__()) = (Float32, 7)
(T, #= /home/tim/.julia/dev/LoopVectorization/test/broadcast.jl:7 =# @__LINE__()) = (Float64, 7)
(T, #= /home/tim/.julia/dev/LoopVectorization/test/broadcast.jl:7 =# @__LINE__()) = (Int32, 7)
(T, #= /home/tim/.julia/dev/LoopVectorization/test/broadcast.jl:7 =# @__LINE__()) = (Int64, 7)
Test Summary: | Pass  Total
broadcast     |   74     74
Test.DefaultTestSet("broadcast", Any[], 74, false)

julia> include("/home/tim/src/julia/nspecializations.jl")   # just if you want a quick count
allmethods (generic function with 1 method)

julia> nspecializations(first(methods(LoopVectorization.abstractparameters)))
33

To be clear, the point of abstractparameters is to allow one to write certain methods without a where clause and still be able to access the parameters. I am assuming that having the where will force specialization independent of a @nospecialize. (If that's not correct, I'd rather get rid of abstractparameters and go back to the way they were before.) So it's ironic that abstractparameters itself seems to get inferred for many different input types.

Running the test suite prints a lot of times. How noisy are Travis runs?
I can also run the tests locally to compare as well.

Do you mean, can one use the time as a guide to progress? Or, might I have left a stray @show or something in a previous PR? I get a lot of output too.

Actually, this might already be helping: if memory serves it used to be ~205s for me to run the whole test suite, this run came out as 177. Of course that's completely uncontrolled (what else was my machine doing at the same time? was I plugged into power?), but something to keep an eye on.

Yes, I was wondering if we could use Travis to track progress.
The latest run on master, Julia 1.1, Linux: 687 seconds
This PR, Julia 1.1, Linux: 666 seconds

Here's results from a more controlled experiment where I ran each twice:

master:

213.179568
211.746881

this PR:

212.729866
214.747268

I don't know what happened with that one fast run, but so far this isn't helping. Of course, if we've identified a crack in the control of specializations (which I suspect we have), the conclusion might change.
These tests were done on Julia 1.3.

Would it be better to ditch the precompile file when timing this? That file shows a huge list of methods from the broadcast file I don't want it specializing on, such as specific static sizes or patterns of true/false in LowDimArray.
On a couple day old build of Julia master, I got, with LoopVectorization master
255.619275 seconds (743.55 M allocations: 32.386 GiB, 2.51% gc time)
255.793652 seconds (743.55 M allocations: 32.386 GiB, 2.61% gc time)
255.626683 seconds (743.55 M allocations: 32.386 GiB, 2.59% gc time)
This PR, after merging master
253.579559 seconds (739.90 M allocations: 32.215 GiB, 2.64% gc time)
253.793647 seconds (739.90 M allocations: 32.215 GiB, 2.61% gc time)
252.885642 seconds (739.90 M allocations: 32.215 GiB, 2.59% gc time)
I've disabled all turboing on this desktop (i.e., it'll run at the same speed regardless of how many cores are under load, or how long it has been working [although speeds will change as a function of work, so clocks vary with sse/avx/avx512/idle, but that should be consistent within a given workload]; but that does mean that the max speed it I have it configured to run at is actually 0.2 GHz slower than the stock max single core boost).
There's still a fair bit of noise, but this PR does already look a bit faster on my computer. It seems to allocate a little less memory.

Great observation. This PR is very much partial, so it might get even better. Still, I have a sense that something isn't right, so I'll wait a bit and see if Jeff discovers anything else.

Superseded by #97


[Azure][Track 2 SDK] Migrate Azure Service Bus PubSub

Migrate Azure Service Bus PubSub to SDK github.com/Azure/azure-sdk-for-go/sdk/messaging/azservicebus

@ItalyPaleAle @halspang have so far looked into this.


[FLINK-14231] Handle the pending processing-time timers to make endInput semantics on the operator chain strict

What is the purpose of the change
This pull request handles its processing-time timers after the operator is closed, to ensure that there is no longer output triggered by the timers before the "endInput" methods of its downstream operators in the chain are called.
For the operator chain in a task, such as OP1 - > OP2 - > ..., after the (source/network) input of OP1 are finished, the operators on the chain are closed in the following order:

quiesce ProcessingTimeService of OP1 to prevent the pending timers from firing, but wait the timers in running to finish.
call OP1#close()
call OP2#endInput()
quiesce ProcessingTimeService of OP2 to prevent the pending timers from firing, but wait the timers in running to finish.
call OP2#close()
...

Brief change log

Adds the quiesce() method to ProcessingTimeServiceImpl, which prevents the pending timers from firing, but allows the timers in running to finish.
Add StreamOperatorWrapper that properly handles the close, endInput and processingtime-quiesced of an operator.
Changes StreamTask to use StreamOperatorWrapper  to close operators,  making endInput semantics on the operator chain strict.

Verifying this change
This change added tests and can be verified as follows:

Added test for ProcessingTimeServiceImpl that validates the quiesce logic.
Added tests for StreamOperatorWrapper that validates the closes of the operators on the operator chain.
Extended test for SourceStreamTask to verify that the endInput semantics on the chain is still strict when there is timers that trigger output.
Extended test for OneInputStreamTask to verify that the endInput semantics on the chain is still strict when there is timers that trigger output.
Extended test for TwoInputStreamTask to verify that the endInput semantics on the chain is still strict when there is timers that trigger output.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

Thanks for reviewing @rkhachatryan . I have updated the PR based on the latest master.

I've updated  the PR description and code to address comments. Thanks. @rkhachatryan

I've updated the PR code to address comments. Please review it again, thanks. @rkhachatryan

Can we avoid adding null or null wrapper into the chain?
If not, can we use null object pattern?
Most of them were added previously, so probably it can be done in a separate PR.

+1

The latest update of PR code is to fix the failure on Azure.

Thanks for reviewing @pnowojski @rkhachatryan . I have updated the PR code.

The latest update of PR code is to fix the failure on travis.

Thanks for updating @sunhaibotb ,
I've reviewed the changes - please see the comments.

I've updated the PR code to address comments. Thanks. @rkhachatryan

Thanks @sunhaibotb ,
Looks good to me.

Latest update to resolve conflicts with master.

Latest two updates are to resolve conflicts with master.

Thanks for reviewing and merging @pnowojski @rkhachatryan .

Thanks for the contribution @sunhaibotb


Moving to using URI.t in both Page and the HTTP Client

Closes https://github.com/nietaki/crawlie/issues/19

Coverage increased (+0.06%) to 97.674% when pulling 00a919899753e6032521feef5defe8fbe17478e5 on uri-page into 6b13553e484d490847659f96f57f26d9a9b511f8 on master.


Setting repository credentials with -P not working

Hi,
I'm trying to publish my charts to chartmuseum with the helmPublish command.
I have set up my build.gradle like below, but can't get it to read the properties form command line like this:
./gradlew helmPublish -Pchartmuseum_username=myuser -Pchartmuseum_password=mypass
helm {
    ...
    publishing {
        repositories {
            chartMuseum('mycharts') {
                url = uri('// removed')
                credentials {
                    username = findProperty('chartmuseum_username')
                    password = "findProperty('chartmuseum_password')
                }
            }
        }
    }
}

I have also tried a variety of other methods like "${chartmuseum_username}", getProeprty('chartmuseum_username'), System.property('chartmuseum_username') and so on. None have worked for me.
If I insert my username and password directly in the build.gradle file, it works fine. It also works fine if I define chartmuseum_username and chartmuseum_password in my global gradle.properties file.
Any idea what's going on? I'm using the exact same approach for publishing maven artifacts, and that works fine.

Hey @bsgrd 👋
Just a heads up that this codebase is no longer actively maintained. As the original author recommends, you are welcome to open an issue here instead.
Our maintainers and contributors will be happy to support! ❤


[CLI]: Can't pickle local object 'TorchHistory.add_log_hooks_to_pytorch_module..'

Describe the bug

The wandb.watch() causes the exception below when we want to save the model using torch.save(). There exists workaround (passing to torch.save() an argument: pickle_module=dill) but when log_graph=True is used in the wandb.watch(), there is another unsolvable error. You can see also ultralytics bug report.
What does the wandb.watch store directly in the model and why? Would be possible to save such information in another memory (e.g. on a disk)?
I found that this is let's say old problem:

https://github.com/wandb/wandb/issues/961#issue-598509912

from ultralytics import YOLO
import wandb

def watch_model(trainer):
    wandb.watch(trainer.model, log="all", log_freq=100, log_graph=True)

# Load a model
model = YOLO("yolov8n.pt")  # load a pretrained model (recommended for training)
model.add_callback("on_train_start", watch_model)

# Use the model
model.train(data=f"datasets/custom.yaml", cache=True, epochs=1)  # train the model

AttributeError                            Traceback (most recent call last)
/var/tmp/pbs.15486198.meta-pbs.metacentrum.cz/ipykernel_195945/3018824876.py in
2
3 # Use the model
----> 4 model.train(data=f"datasets/custom.yaml", cache=True, epochs=1)  # train the model

~/.local-carnivore_id_v1.2.sif/lib/python3.8/site-packages/ultralytics/yolo/engine/model.py in train(self, **kwargs)
369             self.model = self.trainer.model
370         self.trainer.hub_session = self.session  # attach optional HUB session
--> 371         self.trainer.train()
372         # Update model and cfg after training
373         if RANK in (-1, 0):

~/.local-carnivore_id_v1.2.sif/lib/python3.8/site-packages/ultralytics/yolo/engine/trainer.py in train(self)
191                 ddp_cleanup(self, str(file))
192         else:
--> 193             self._do_train(world_size)
194
195     def _setup_ddp(self, world_size):

~/.local-carnivore_id_v1.2.sif/lib/python3.8/site-packages/ultralytics/yolo/engine/trainer.py in _do_train(self, world_size)
370                 # Save model
371                 if self.args.save or (epoch + 1 == self.epochs):
--> 372                     self.save_model()
373                     self.run_callbacks('on_model_save')
374

~/.local-carnivore_id_v1.2.sif/lib/python3.8/site-packages/ultralytics/yolo/engine/trainer.py in save_model(self)
413
414         # Save last, best and delete
--> 415         torch.save(ckpt, self.last)
416         if self.best_fitness == self.fitness:
417             torch.save(ckpt, self.best)

/opt/conda/lib/python3.8/site-packages/torch/serialization.py in save(obj, f, pickle_module, pickle_protocol, _use_new_zipfile_serialization)
378         if _use_new_zipfile_serialization:
379             with _open_zipfile_writer(opened_file) as opened_zipfile:
--> 380                 _save(obj, opened_zipfile, pickle_module, pickle_protocol)
381                 return
382         _legacy_save(obj, opened_file, pickle_module, pickle_protocol)

/opt/conda/lib/python3.8/site-packages/torch/serialization.py in _save(obj, zip_file, pickle_module, pickle_protocol)
546     pickler = pickle_module.Pickler(data_buf, protocol=pickle_protocol)
547     pickler.persistent_id = persistent_id
--> 548     pickler.dump(obj)
549     data_value = data_buf.getvalue()
550     zip_file.write_record('data.pkl', data_value, len(data_value))

AttributeError: Can't pickle local object 'TorchHistory.add_log_hooks_to_pytorch_module..'

Additional Files
No response
Environment
WandB version:
0.15.3
OS:
linux
Python version:
3.8
Versions of relevant libraries:
ultralytics==8.0.105
dill==0.3.6
Additional Context
No response

Hi @vyskocj, I'll take this to our engineering team to investigate. In the meantime could you try wandb.unwatch() before you save the model if that works for your workflow?

Thank you for your fast response. @nate-wandb Yes, this is working, but if a user (like me) is using some Python package for training the models, it requires intervention into the installed scripts or some wrappers.
In my opinion, it would be better (if the reported issue of wandb.watch() could not really be solved in another way) to have a function designed for hooks - i.e. the user would have to periodically call a function that would send the data/model to wandb (as it is done for logging), instead of doing everything itself (as it is done now). I hope that you will be able to find a great solution.

Hi @vyskocj, sorry for the delay. For your above suggestion, are you looking for something similar to our WandbLogger class for PyTorch Lightning?

It is ok, meanwhile, I have created a pull request to Ultralytics, in which this problem occurred, to save the model via dill (instead of the original pickle). So from my side, the problem is solved by a walkaround.
And according to your question, I would say yes if the WandbLogger and its watch method allow torch.save without problems (using the standard pickle library). It is true that there is a workaround, however, new users will go through the same walk that I had. However, it will be unpleasant for them if their program falls on this after days of training.
If there is no possibility to guarantee this (saving model using torch.save without exception), I highly recommend adding a warning or at least any information into your documentation to wandb.watch that it may cause issues with saving model using pickle -> 'wandb.unwatch' has to be called before saving the model to avoid an error. Because I did not know about this method until you told me about it.

Ok, glad you have a workaround in the meantime. I'll submit this request to the engineering team and see if we can solve this from our side. I'll follow back up once I have an update.

@nate-wandb I also get this with a project that uses Pytorch Lightning.
ERROR:__name__:Can't pickle local object 'TorchHistory.add_log_parameters_hook.<locals>.<lambda>'
Traceback (most recent call last):
  File "/home/josh/ -- .py", line 898, in train
    trainer.fit(model=self.model, datamodule=data_module)
  File "/opt/conda/envs/tcd/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 608, in fit
    call._call_and_handle_interrupt(
  File "/opt/conda/envs/tcd/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 36, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
  File "/opt/conda/envs/tcd/lib/python3.10/site-packages/lightning/pytorch/strategies/launchers/multiprocessing.py", line 113, in launch
    mp.start_processes(
  File "/opt/conda/envs/tcd/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 189, in start_processes
    process.start()
  File "/opt/conda/envs/tcd/lib/python3.10/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/opt/conda/envs/tcd/lib/python3.10/multiprocessing/context.py", line 288, in _Popen
    return Popen(process_obj)
  File "/opt/conda/envs/tcd/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/opt/conda/envs/tcd/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/opt/conda/envs/tcd/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/opt/conda/envs/tcd/lib/python3.10/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'TorchHistory.add_log_parameters_hook.<locals>.<lambda>'

The model is a semantic segmentation model from Torchgeo, but I underneath it's from here. This wasn't an issue in the past, but seems to be related to upgrading Lightning.

@jveitchmichaelis thank you for raising this as well. Interesting that in PTL this only fails for multiple GPU's. Are you saving the model or is this happening without a model save? I think PTL might be pickling to disk to pass the model between processes but I'll have to look into this know for sure.

@nate-wandb  - wandb.unwatch - worked well for me and allowed me to save the whole model!

@nate-wandb wandb.unwatch works if I need to save the model once at the end of the process, but I'm saving the best performing model after each epoch. Any trick on how to make this work? unwatching, saving, and then rewatching throws this error:
File "<path>/wandb/wandb_torch.py", line 114, in add_log_parameters_hook
    self._hook_handles["parameters/" + prefix] = hook
TypeError: list indices must be integers or slices, not str

@nate-wandb I'm also getting the same erorr as jveitchmichaelis above. In this case, I'm training a VAE model with PTL and also using Hydra to manage runs. I'm training the models on supercomputing nodes with 4 GPUs each and want to utilize multiprocessing to speed up training, but I am crashing out with the following error
12 Traceback (most recent call last):
13   File "/global/u2/d/dknguyen/gen_ai/cdvae/cdvae/run.py", line 166, in main
14     run(cfg)
15   File "/global/u2/d/dknguyen/gen_ai/cdvae/cdvae/run.py", line 154, in run
16     trainer.fit(model=model, datamodule=datamodule)
17   File "/global/common/software/nersc/pm-2022q4/sw/pytorch/2.0.1/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 770, in fit
18     self._call_and_handle_interrupt(
19   File "/global/common/software/nersc/pm-2022q4/sw/pytorch/2.0.1/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 721, in _call_and_handle_interrupt
20     return self.strategy.launcher.launch(trainer_fn, *args, trainer=self, **kwargs)
21   File "/global/common/software/nersc/pm-2022q4/sw/pytorch/2.0.1/lib/python3.9/site-packages/pytorch_lightning/strategies/launchers/spawn.py", line 78, in launch
22     mp.spawn(
23   File "/global/common/software/nersc/pm-2022q4/sw/pytorch/2.0.1/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 239, in spawn
24     return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
25   File "/global/common/software/nersc/pm-2022q4/sw/pytorch/2.0.1/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes
26     process.start()
27   File "/global/common/software/nersc/pm-2022q4/sw/pytorch/2.0.1/lib/python3.9/multiprocessing/process.py", line 121, in start
28     self._popen = self._Popen(self)
29   File "/global/common/software/nersc/pm-2022q4/sw/pytorch/2.0.1/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
30     return Popen(process_obj)
31   File "/global/common/software/nersc/pm-2022q4/sw/pytorch/2.0.1/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__
32     super().__init__(process_obj)
33   File "/global/common/software/nersc/pm-2022q4/sw/pytorch/2.0.1/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
34     self._launch(process_obj)
35   File "/global/common/software/nersc/pm-2022q4/sw/pytorch/2.0.1/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 47, in _launch
36     reduction.dump(process_obj, fp)
37   File "/global/common/software/nersc/pm-2022q4/sw/pytorch/2.0.1/lib/python3.9/multiprocessing/reduction.py", line 60, in dump
38     ForkingPickler(file, protocol).dump(obj)
39 AttributeError: Can't pickle local object 'TorchHistory.add_log_parameters_hook.<locals>.<lambda>'
40 Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.


Fix broken Maven coordinates in README.md

This PR fixes broken Maven coordinates in the README.md.

Thanks, looks like this already got cleaned up somewhere else.  I appreciate the PR

@markpollack Thanks for the feedback, but the README.md doesn't seem to have been updated. Due to the change of the group ID on 0.8.0-SNAPSHOT, it should be either org.springframework.ai with 0.8.0-SNAPSHOT or org.springframework.experimental.ai with 0.7.1-SNAPSHOT. Otherwise, one of the following failures would happen:
* What went wrong:
Execution failed for task ':compileJava'.
> Could not resolve all files for configuration ':compileClasspath'.
   > Could not find org.springframework.experimental.ai:spring-ai-openai-spring-boot-starter:0.8.0-SNAPSHOT.
     Required by:
         project :

* What went wrong:
Execution failed for task ':compileJava'.
> Could not resolve all files for configuration ':compileClasspath'.
   > Could not find org.springframework.ai:spring-ai-openai-spring-boot-starter:0.7.1-SNAPSHOT.
     Required by:
         project :

Am I missing something?


ja: Make docs/concepts/configuration/manage-resources-containers.md follow v1.19 of the original text

Close https://github.com/kubernetes/website/issues/26860

/assign
/cc @kakts
/approve

/lgtm


Is there any way get current ip in each update?

Hello. I have model has name, url and ip.
And I want to make current ip will be inputted in ip when creating or updating.
But you know request.remote_ip can not use in model.
And I fount rails_admin config, too.
I try this code
# rails_admin.rb
config.model 'Creator' do
    include_all_fields
    edit do
      field :ip, :hidden do
        default_value request.remote_ip
      end
    end
  end

But it's not work. Exactly, noting was inputted.
And I have one more question. default_value is only works on create.
Dose any method works on update, too?

you can access the request object through bindings[:controller]
default_value for update is not supported. You can try something inside your model though.


Creating pull request of hacks I needed to run sample.py on cuda

When I run even the basic sample.py script under examples on a cuda enabled machine, I still get errors that not all the vectors are cuda vectors (some are still cpu). You can ignore my edits in sample.py, but I did annotate in each place where I needed to add vector = vector.cuda() to make sample.py run. This occurs even when torch.device('cuda') is called, which should not be the case in PyTorch 0.4.0+.
Thank you, and feel free to follow up with any questions.

Hi @DavidLKing did you try the develop branch? The necessary changes for moving tensors to CUDA have already been made there. Master branch is yet to be updated.

Hi @pskrunner14! I did, but it still fails when I try to run sample.py with the same error:
(pytorch-seq2seq) david@Arjuna:~/bin/git/pytorch-seq2seq$ python examples/sample.py --train_path data/toy_reverse/train/data.txt --dev_path data/toy_reverse/dev/data.txt
/home/david/miniconda2/envs/pytorch-seq2seq/lib/python3.6/site-packages/torch/nn/functional.py:52: UserWarning: size_average and reduce args will be deprecated, please use reduction='elementwise_mean' instead.
  warnings.warn(warning.format(ret))
2018-11-11 15:23:53,307 root         INFO     Namespace(dev_path='data/toy_reverse/dev/data.txt', expt_dir='./experiment', load_checkpoint=None, log_level='info', resume=False, train_path='data/toy_reverse/train/data.txt')
/home/david/miniconda2/envs/pytorch-seq2seq/lib/python3.6/site-packages/torch/nn/functional.py:52: UserWarning: size_average and reduce args will be deprecated, please use reduction='sum' instead.
  warnings.warn(warning.format(ret))
/home/david/miniconda2/envs/pytorch-seq2seq/lib/python3.6/site-packages/torch/nn/modules/rnn.py:38: UserWarning: dropout option adds dropout after all but last recurrent layer, so non-zero dropout expects num_layers greater than 1, but got dropout=0.2 and num_layers=1
  "num_layers={}".format(dropout, num_layers))
2018-11-11 15:23:56,082 seq2seq.trainer.supervised_trainer INFO     Optimizer: Adam (
Parameter Group 0
    amsgrad: False
    betas: (0.9, 0.999)
    eps: 1e-08
    lr: 0.001
    weight_decay: 0
), Scheduler: None
Traceback (most recent call last):
  File "examples/sample.py", line 129, in <module>
    resume=opt.resume)
  File "/home/david/miniconda2/envs/pytorch-seq2seq/lib/python3.6/site-packages/seq2seq-0.1.6-py3.6.egg/seq2seq/trainer/supervised_trainer.py", line 186, in train
  File "/home/david/miniconda2/envs/pytorch-seq2seq/lib/python3.6/site-packages/seq2seq-0.1.6-py3.6.egg/seq2seq/trainer/supervised_trainer.py", line 103, in _train_epoches
  File "/home/david/miniconda2/envs/pytorch-seq2seq/lib/python3.6/site-packages/seq2seq-0.1.6-py3.6.egg/seq2seq/trainer/supervised_trainer.py", line 55, in _train_batch
  File "/home/david/miniconda2/envs/pytorch-seq2seq/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/david/miniconda2/envs/pytorch-seq2seq/lib/python3.6/site-packages/seq2seq-0.1.6-py3.6.egg/seq2seq/models/seq2seq.py", line 48, in forward
  File "/home/david/miniconda2/envs/pytorch-seq2seq/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/david/miniconda2/envs/pytorch-seq2seq/lib/python3.6/site-packages/seq2seq-0.1.6-py3.6.egg/seq2seq/models/EncoderRNN.py", line 68, in forward
  File "/home/david/miniconda2/envs/pytorch-seq2seq/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/david/miniconda2/envs/pytorch-seq2seq/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 110, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/home/david/miniconda2/envs/pytorch-seq2seq/lib/python3.6/site-packages/torch/nn/functional.py", line 1110, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected object of type torch.cuda.LongTensor but found type torch.LongTensor for argument #3 'index'

I'm fine with this not being a full merge. Would an issue make more sense?

Hey @DavidLKing are you sure you have the latest changes? It seems to be working fine on my side:
psk@ubuntu:~/projects/pytorch-seq2seq$ python examples/sample.py $TRAIN_SRC $TRAIN_TGT $DEV_SRC $DEV_TGT
2018-11-12 02:19:04,671:root:INFO: train_source: data/toy_reverse/train/src.txt
2018-11-12 02:19:04,671:root:INFO: train_target: data/toy_reverse/train/tgt.txt
2018-11-12 02:19:04,671:root:INFO: dev_source: data/toy_reverse/dev/src.txt
2018-11-12 02:19:04,671:root:INFO: dev_target: data/toy_reverse/dev/tgt.txt
2018-11-12 02:19:04,671:root:INFO: experiment_directory: ./experiment
2018-11-12 02:19:06,860:seq2seq.trainer.supervised_trainer:INFO: Optimizer: Adam (
Parameter Group 0
    amsgrad: False
    betas: (0.9, 0.999)
    eps: 1e-08
    initial_lr: 0.001
    lr: 0.001
    weight_decay: 0
), Scheduler: <torch.optim.lr_scheduler.StepLR object at 0x7fd4f1c03c88>
Train Perplexity: 8.6506: 100%|██████████████████████████████████████| 157/157 [00:03<00:00, 45.63it/s]
2018-11-12 02:19:10,454:seq2seq.trainer.supervised_trainer:INFO: Finished epoch 1: Train Perplexity: 6.4669, Dev Perplexity: 4.7727, Accuracy: 0.6091
Train Perplexity: 1.1459: 100%|██████████████████████████████████████| 157/157 [00:03<00:00, 47.59it/s]
2018-11-12 02:19:13,900:seq2seq.trainer.supervised_trainer:INFO: Finished epoch 2: Train Perplexity: 2.4010, Dev Perplexity: 4.4814, Accuracy: 0.7656
Train Perplexity: 1.0013: 100%|██████████████████████████████████████| 157/157 [00:03<00:00, 46.45it/s]
2018-11-12 02:19:17,427:seq2seq.trainer.supervised_trainer:INFO: Finished epoch 3: Train Perplexity: 1.2013, Dev Perplexity: 1.0010, Accuracy: 1.0000
['3', '2', '1', '<eos>']

Okay, it seems there have been more drastic changes. I was using an older version of sample.py. The newer version runs fine with the develop branch. I'll need to update my own project, but I think we're good to go. Thanks for the time! When's the next release scheduled?


Default packaging without OCAML backend [brew-staging: remove-opam]

Fixes: #1600
This PR does:

Removes the OCAML backend dependency setup from the package installation process.
Removes the OCAML backend dependency teardown from the package removal process.
Updates the INSTALL and README instructions to let people know that OCAML dependencies are OPTIONAL and how to go about setting them up.

Base branch for homebrew-k repo: https://github.com/kframework/homebrew-k/compare/remove-opam

@dwightguth this is ready.

Evidence it works on Debian, Focal, Bionic, and MacOS: https://office.runtimeverification.com/jenkins/blue/organizations/jenkins/k/detail/PR-1618/24/pipeline
Evidence that it works on Arch: https://office.runtimeverification.com/jenkins/blue/organizations/jenkins/k/detail/PR-1618/25/pipeline


fmt: define what %p means (or doesn't mean) on interfaces

(split from https://github.com/golang/go/issues/16230#issuecomment-231563421)
What does %p mean in the fmt package for interface values?
It seems to do something, but the documentation says nothing.
/cc @josharian @robpike

It's well-defined. The fmt package docs say: "Regardless of the verb, if an operand is an interface value, the internal concrete value is used, not the interface itself." This is a bug in vet, and I already have a fix in my tree for it as part of fixing #11041.


Service registrar interface (#3545 part 1)

Overview

Add IMixedRealiyServiceRegistrar interface
Update MixedRealityToolkit component to implement the new interface
Improve consistency with regards to static vs instance methods/properties (ex: now need .Instance for both Register and Unregister calls)
Clean up some unused method args

Verified using Solver demo scene and unit tests
NOTE: You may need to delete the Library folder after taking this change.
[Added for clarity]
This is the first in a series of changes related to providing flexibility for customers to implement their own service locator object / service containers.

Long term i think we should remove this instance call.

Which instance call? MixedRealityToolkit.Instance? If so i agree and am in the process of making that change

Re: @wiwei comments (for some reason, i cannot reply). The goal of this PR was to leave the core behavior of the current service locator as unchanged as possible.
These suggetions should definitely be incorportated into future PRs in this series.

Overall lgtm, not having to do typeof so often looks much nicer :)

first part of #3545 completed.

Send kinda fast getting these PRs merged before the community can review them 😏


Don't call FixGenerics twice

FixGenerics is called by ITypeHelpers.GetTypeName.  ClrmdType shouldn't also call it.

Contributes to https://github.com/microsoft/clrmd/issues/627.  Just a workaround, need a real decision or fix on that issue.


Performance issue: much slower compared to "vanilla" opencv

Description of the problem
I notice running many different applications using opencv libraries are showing very significant performance difference when using the upstream python bindings vs. opencv-python. I verified this on Red Hat 7 and Fedora 30 / 31 Linux distributions. Fedora provides opencv python packages (3.4 series) which I also adapted to work on Red Hat Linux (modifying the RPM spec file to disable a couple of components not available in Red Hat).
When running the same code / applications slowdown is significant. Depending on the application it can take 3 to more than 10 longers. One of the tested application is: DeepLabCut. Switching only the opencv package and keeping the rest of the environment exactly the same, tripled the performance of our analysis time.
I tried to recompile the opencv-python locally using the setup.py and gcc-8 on CentOS 7, but it didn't make a big difference (there was a 10-20% improvement, but still very far away from the 300% observed with upstream build).
Looking at the cmake arguments in the RPM spec file and comparing it to the one in setup.py I cannot see anything that could explain the problem. I can see the Fedora RPM forces the use of OPENMP rather than using pthreads (and I understand opencv-python uses pthreads instead), not sure if this could be the primary cause of the issue. Sounds strange if it is though. I ran out of ideas and thought about opening this issue.
Expected behaviour
Same or close to same performance as upstream python bindings
Actual behaviour
Not entirely sure, but multiple applications are showing significant slowdowns when using this opencv-python rather than the upstream python binding
Steps to reproduce

https://github.com/AlexEMG/DeepLabCut / any 3D reconstruction demo from the opencv example code should also work
CentOS 7 / Fedora 30 - 31
x86_64
opencv-python both 29 and 30 (so opencv 3.4 and 4.2) seems to be affected

Thank your for any help.

I think something related that could help would be documentation on how opencv-python is compiled. Maybe a FAQ entry. What should users expect in terms of performance? I have little experience with OpenCV, but for example, to make the opencv-python wheels universally installable, are certain optimizations disabled? Should users expect CUDA support, SIMD, etc?
Maybe a link to the relevant build files/flags would help?

I didn't checked the details, but it doesn't look like generic compiler optimizations are disabled. While compiling opencv-python I see options such as -O2 and -O3 passed to the compiler. The packages I compiled are also generic x86_64 and not tuned to any CPU architecture.

You can check the compiler flags as well as other information about the build environment with cv2.getBuildInformation().
About the performance issue: it might be that pthreads vs. OpenMP is the culprit. Hard to say.
Keep in mind that these packages are not optimized for performance. The main target is to provide an easy way to install OpenCV on almost any OS. There has been made some tradeoffs especially regarding 3rd party dependencies. The current CentOS 5 environment is not the easiest to work with since it's quite old. This means that the maintenance burden becames overwhelming very fast if I enable every dependency or optimization OpenCV might have support for. Due to this a locally built OpenCV version will almost always have better performance than the one provided via this repository.

I understand, but this is getting extreme. My own code runs in a few seconds with Fedora packages and also on Red Hat 7 with my own build based on the Fedora one. With opencv-python after one full hours it didn't finished.
It might be time to consider dropping older platforms, the slowdown is getting unsustainable in my humble opinion.

Perhaps manylinux2010 could be the new baseline or alternative? It's based on CentOS 6.

I looked into the Fedora configs and also Lapack is on there while it's not enabled in opencv-python. There might be also other differences.
And yes, manylinux2010 is the next step.

And for completeness, since at this point CentOS 6 is already EOL, wanted to also reference manylinux2014

It might be good to skip manylinux2010 and migrate straight to manylinux2014.

It might be good to skip manylinux2010 and migrate straight to manylinux2014.

I would agree.

manylinux2010 support was added in pip 19.0
manylinux2014 support was added in pip 19.3

It's best practice to pip install --upgrade pip in a virtual environment these days, so I think this should fine, especially sense this would lower maintenance burden and (potentially) improve performance.

Could you check how the latest release perform?

Default configuration can have many differences from opencv-python, most obvious things are:

image reading - perhaps opencv-python uses libjpeg-turbo provided with OpenCV, it will be built with disabled vectorization, when you build OpenCV by yourself you most probably will link with system-provided library which should be optimized
video reading - default OpenCV can use optimized system FFmpeg and GStreamer for decoding (latter can have HW-accelerated plugins installed), opencv-python uses built-in FFmpeg (probably with safe optimization level)
GUI - default OpenCV uses GTK+3 on Linux, opencv-python uses Qt backend with more features
some math processing can be done with LAPACK/MKL/etc - can be auto-detected when building OpenCV, not used in opencv-python
opencv-python is built with older compiler, your system most probably have newer version (esp. Fedora)

Other features/dependencies in opencv-python are similar to the default build configuration, including CPU optimizations (SSSE3 + all dispatching) and parallel-processing backend (pthreads).
So it is important to localize performance drop: compare configs, measure execution time on different stages (input, processing, GUI), profile, etc..

Closing due to inactivity.


feat: add stop words for Hungarian language

Adding the Hungarian Language stop words to stop word filter.

Codecov Report

Merging #2069 (db7cd2b) into main (b325d56) will decrease coverage by 0.01%.
The diff coverage is 0.00%.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
@@            Coverage Diff             @@
##             main    #2069      +/-   ##
==========================================
- Coverage   94.38%   94.38%   -0.01%
==========================================
  Files         319      319
  Lines       60040    60041       +1
==========================================
  Hits        56670    56670
- Misses       3370     3371       +1

Impacted Files
Coverage Δ

src/tokenizer/stop_word_filter/mod.rs
73.19% <0.00%> (-0.77%)
:arrow_down:


Xd 3742 - Enable in line SSL properties as an alternative to external properties files.

This adds explicit in line properties for keyStore, keyStorePassphrase, trustStore, and trustStorePassphrase. These can be set enable storing properties in encrypted form enabled in https://jira.spring.io/browse/XD-3738.
This affects SSL settings for RabbitMessageBus,  Rabbit source, Rabbit sink, and Http source.
Manual testing performed:

Regression test each of these using the current mechanisms: Enable SSL, use rabbit SSL address, external properties files with valid and invalid values in servers.yml, modules.yml, and module config properties.
Test Rabbit MB with in line configuration in servers.yml
Test Rabbit modules inherit the default MB configuration
Test Streams with rabbit sink and source modules, configured with inline properties
Regression test http source with SSL configured from new config/modules/source/http/http.properties
http --https=true | log and curl -X post --cacert path/to/cecert https://localhost:9000 -d hello

Otherwise, LGTM; I can remove that on merge if you agree.

good catch

@dturanski: Thoughts on Gary's comment?

"good catch"  ^^^ wasn't clear enough I guess. Yes we should fix it on the merge.

Yup - merging now

Rebased, squashed, merged as e63e6c545f188cc388dfe2433003596be7c53a19 with a bit o' spit and polish.


Error while trying to change signal length in config.py

Hello BirdNET team,
I have been trying to change the prediction window length from 3 to 5 seconds for quite some time now. This is "SIG_LENGTH" in the config.py script. The sample rate stays at 48 kHz, and no prediction overlap. This gets down to 240,000 samples per chunk. For a 10-minute audio recording, about 120 chunks are produced. I verified the getRawAudioFromFile() function in the analyze.py script, and the 5-sec windows are produced correctly with the correct file offset.  However, the issue arises when I try to run any analyze.py command. I continuously get this error.
Error: /tensorflow/tensorflow/lite/util.cc BytesRequired number of elements overflowed. Node number [number] (CONV_2D) failed to prepare. The size_splits must sum to the dimension of value along the axis.
Chunk shape before prediction: (240000,1)
These errors seem to indicate that the shape of the input tensor does not match the expected dimensions of the model, and it may also suggest that the tensor is too large for the available memory. Also the FILE_SPLITTING_DURATION=600, could that be an issue with changing the value of SIG_LENGTH?
Can anyone with more technical expertise help me with what I am missing out? Additionally, could I get the idea for the expected model's input shape and any recommendations for adjusting the processing logic to ensure compatibility?

As far as I know, the model is trained to work with 3 seconds audio signal at 48 kHz and cannot make predictions for anything else. Changing the signal chunk length back to 3 seconds should make your error go away.
If you think your hardware cannot handle so many chunks at once (regarding memory), you could split your audio recordings before you feed them to the analyzer.

Thanks for the reply. I agree with you totally. 3 second window length works. But the bird that we are trying to study has quite a complex repertoire, making it difficult to get a number of vocalizations from long-term datasets. Hence an attempt to get varying detection window lengths.
I also realized something, if the expected input shape is (1,144000), i.e., 48000 * 3, the default. We can change our sample rate, say 28.8kHz, for detections like 5s (which gives the same 144000 samples). That worked for the general classifier but not for the custom-trained ones (still troubleshooting). But again, a lower sample rate has its own repercussions; it may not train on really high frequencies, but it should be enough to capture between 1 and 10kHz (target vocalizations).
So I wanted to know if the tensor model input shape, i.e. (1,144000), is alterable or if this is what the model stays fixed at.

This has been discussed before: https://github.com/kahst/BirdNET-Analyzer/issues/288#issuecomment-2063839251. The ideas of varying sample rate didn’t crop up, it’s an interesting one, but note that the model discards audio above 15KHz. If you fed it 5seconds @ 28.8KHz, it would discard sounds over 9KHz

With currently latest model v2.4 input length is fixed to 3 seconds, but this video metions that with variable input length is planned for v3.0 model https://youtu.be/Faavvmi9JZw?t=798

Understood, I had missed the earlier discussion. Thanks a lot for the help! Looking forward to v3.0 model.


BUG Navbar mismatch to story page #262

Title and Issue number

Title : BUG Navbar mismatch to story page #262
Issue No. : #262
Code Stack :
Close #<issue_no> #262

Video/Screenshots (mandatory)

Checklist:

[x] I have mentioned the issue number in my Pull Request.
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have gone through the  contributing.md file before contributing

Additional context (Mandatory )
Are you contributing under any Open-source programme?

JWOC

pls check this PR


[HUDI-1104] Adding support for UserDefinedPartitioners and SortModes to BulkInsert with Rows

What is the purpose of the pull request

Adding bulk_insert sort modes for row writer path

(for example:)

Adding bulk_insert sort modes for row writer path
Caching RowCreateHandles if applicable

Verify this pull request
This pull request is already covered by existing tests, such as TestBulkInsertInternalPartitionerForRows
This change added tests and can be verified as follows:

TestHoodieBulkInsertDataInternalWriter

Committer checklist

[ ] Has a corresponding JIRA in PR title & commit

[ ] Commit message is descriptive of the change

[ ] CI is green

[ ] Necessary doc changes done or have another open PR

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

Codecov Report

Merging #3149 (b12f427) into master (b9e28e5) will decrease coverage by 42.66%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##             master   #3149       +/-   ##
============================================
- Coverage     45.79%   3.13%   -42.67%
+ Complexity     5270      82     -5188
============================================
  Files           909     274      -635
  Lines         39390   10652    -28738
  Branches       4244    1090     -3154
============================================
- Hits          18039     334    -17705
+ Misses        19508   10292     -9216
+ Partials       1843      26     -1817

Flag
Coverage Δ

hudicli
?

hudiclient
0.00% <0.00%> (-30.40%)
:arrow_down:

hudicommon
?

hudiflink
?

hudihadoopmr
?

hudisparkdatasource
?

hudisync
6.72% <ø> (-45.01%)
:arrow_down:

huditimelineservice
?

hudiutilities
9.52% <ø> (-47.12%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b9e28e5...b12f427. Read the comment docs.


fixes 'Error: task completion callback called too many times'

Thanks!
How'd I miss that... :dizzy_face:

published to v1.3.1


Fix broken link to Orb Registry [semver:skip]

Should the references to the sandbox namespace orb be updated to the public one? This will give us an accurate version number :)

@gmemstr Ah yes.. good call..

Yup that did the trick 👍
Thanks @gmemstr 🙏


react-native-git-upgrade fatal: working tree '.' already exists.

Description
Trying to upgrade using cli react-native-git-upgrade 0.39.2
Reproduction
~/D/D/M/R/_/Smish ❯❯❯ react-native-git-upgrade 0.39.2           dc/upgrade_39 ✭
git-upgrade info Check for updates
git-upgrade info Using yarn 0.18.1
git-upgrade info Read package.json files
git-upgrade info Check declared version
git-upgrade info Check matching versions
git-upgrade info Check React peer dependency
git-upgrade info Check that Git is installed
git-upgrade info Get information from NPM registry
git-upgrade info Upgrading to React Native 0.39.2, React ~15.4.0-rc.4
git-upgrade info Setup temporary working directory
git-upgrade info Configure Git environment
git-upgrade info Init Git repository
git-upgrade info Add all files to commit
git-upgrade info Commit current project sources
git-upgrade info Create a tag before updating sources
git-upgrade info Generate old version template
git-upgrade info Add updated files to commit
git-upgrade info Commit old version template
git-upgrade info Install the new version
error Command failed.
Exit code: 128
Command: git
Arguments: clone https://github.com/hiaw/react-native-layout-tester /Users/danielchong/Library/Caches/Yarn/.tmp/c055b13a602c46e87e94ee25168d3eb7
Directory: /Users/danielchong/Documents/Development/Mobile/React-Native/_client/Smish
Output:
fatal: working tree '.' already exists.
git-upgrade ERR! An error occurred during upgrade:
git-upgrade ERR! Error: Command 'yarn add<EMAIL_ADDRESS><EMAIL_ADDRESS>exited with code 1:
stderr: undefinederror Command failed.
Exit code: 128
Command: git
Arguments: clone https://github.com/hiaw/react-native-layout-tester /Users/danielchong/Library/Caches/Yarn/.tmp/c055b13a602c46e87e94ee25168d3eb7
Directory: /Users/danielchong/Documents/Development/Mobile/React-Native/_client/Smish
Output:
fatal: working tree '.' already exists.

stdout: yarn add v0.18.1
[1/5] Resolving packages...
[2/5] Fetching packages...
info Visit https://yarnpkg.com/en/docs/cli/add for documentation about this command.

    at ChildProcess.child.on.code (/Users/danielchong/.node/lib/node_modules/react-native-git-upgrade/cliEntry.js:58:18)
    at emitTwo (events.js:106:13)
    at ChildProcess.emit (events.js:191:7)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:215:12)
git-upgrade ERR! Restore initial sources
Note: checking out 'project-snapshot'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 54f0025... Project snapshot

I suspect it has to do with github dependency for react-native-layout-tester?
"devDependencies": {
  "babel-jest": "17.0.2",
  "babel-preset-react-native": "1.9.0",
  "jest": "17.0.3",
  "jest-react-native": "17.0.3",
  "react-native-components-viewer": "git+https://github.com/hiaw/react-native-components-viewer",
  "react-native-layout-tester": "git+https://github.com/hiaw/react-native-layout-tester",
  "react-test-renderer": "~15.3.1"
},

Solution
upgrade should work flawlessly
Additional Information

React Native version: 0.37.0
Platform: both
Operating System: MacOS

+1

I can confirm this issue too.

React Native version: 0.39.0
Platform: both
Operating System: MacOS

+1

It has to do with the Git dependencies indeed. That should be fixed.
In the meantime, just remove them temporary from your package.jsonbefore running "react-native-git-upgrade".

I solved it. For me running yarn instead of npm worked. The steps:

clone a fresh copy of my repo (which is now at 0.40.0)
run yarn
run react-native-git-upgrade 0.41.0
everything is fine!
doing the same with npm didn't work.

Same here. I just did yarn then react-native-git-upgrade and it worked.

I tried cloning a copy of my old repo (0.42.0), running yarn, and then running react-native-git-upgrade 0.43.0 and still getting fatal: working tree '.' already exists. error. I saw something about removing a package from package.json temporarily? How do you even do that? Looks like react-native-camera may be causing the issue?
{
  "name": "Snapshelf",
  "version": "0.0.1",
  "private": true,
  "scripts": {
    "start": "node node_modules/react-native/local-cli/cli.js start",
    "test": "jest"
  },
  "dependencies": {
    "firebase": "^3.7.2",
    "native-base": "^2.1.1",
    "react": "~15.4.1",
    "react-native": "0.42.0",
    "react-native-animatable": "^1.1.1",
    "react-native-camera": "git+https://github.com/lwansbrough/react-native-camera.git",
    "react-native-elements": "^0.9.7",
    "react-native-fetch-blob": "^0.10.2",
    "react-native-vector-icons": "^4.0.0",
    "react-redux": "^5.0.3",
    "redux": "^3.6.0",
    "redux-thunk": "^2.2.0"
  },
  "devDependencies": {
    "babel-jest": "19.0.0",
    "babel-plugin-root-import": "^5.1.0",
    "babel-preset-react-native": "1.9.1",
    "jest": "19.0.2",
    "react-test-renderer": "~15.4.1",
    "remote-redux-devtools": "^0.5.7"
  },
  "jest": {
    "preset": "react-native"
  }
}

i just had to run yarn before git-upgrade and everything is fine :)

Hi there! This issue is being closed because it has been inactive for a while.
But don't worry, it will live on with Canny! Check out its new home: https://react-native.canny.io/feature-requests/p/react-native-git-upgrade-fatal-working-tree--already-exists

git-upgrade info Check for updates
git-upgrade info Using yarn 0.27.5
git-upgrade info Read package.json files
git-upgrade info Check declared version
git-upgrade info Check matching versions
git-upgrade info Check React peer dependency
git-upgrade info Check that Git is installed
git-upgrade info Get information from NPM registry
git-upgrade info Upgrading to React Native 0.47.2, React 16.0.0-alpha.12
git-upgrade info Setup temporary working directory
git-upgrade info Configure Git environment
git-upgrade info Init Git repository
git-upgrade info Add all files to commit
warning: CRLF will be replaced by LF in android/gradlew.bat.
The file will have its original line endings in your working directory.
git-upgrade info Commit current project sources
git-upgrade info Create a tag before updating sources
git-upgrade info Generate old version template
git-upgrade info Add updated files to commit
warning: CRLF will be replaced by LF in android/gradlew.bat.
The file will have its original line endings in your working directory.
git-upgrade info Commit old version template
git-upgrade info Install the new version
error Command failed.
Exit code: 1
Command: git
Arguments: pull
Directory: /Users/Will/Library/Caches/Yarn/v1/.tmp/356e7fb1c51407d4d88112a3cfb3564c
Output:
fatal: 'origin' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
git-upgrade ERR! An error occurred during upgrade:
git-upgrade ERR! Error: Command 'yarn add<EMAIL_ADDRESS>exited with code 1:
stderr: undefinederror Command failed.
Exit code: 1
Command: git
Arguments: pull
Directory: /Users/Will/Library/Caches/Yarn/v1/.tmp/356e7fb1c51407d4d88112a3cfb3564c
Output:
fatal: 'origin' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

stdout: yarn add v0.27.5
[1/4] Resolving packages...
[2/4] Fetching packages...
info Visit https://yarnpkg.com/en/docs/cli/add for documentation about this command.

    at ChildProcess.child.on.code (/usr/local/lib/node_modules/react-native-git-upgrade/cliEntry.js:58:18)
    at emitTwo (events.js:106:13)
    at ChildProcess.emit (events.js:194:7)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:215:12)

Upgrading to 0.47.2 displays the above notice. I've tried clearing the yarn cache without any luck.

I was having the same issue, followed these steps and issue is resolved now:
run command yarn in your project folder and after that run react-native-git-upgrade

I had to both use yarn to reinstall but also remove the following cache folder /Users/jun/Library/Caches/Yarn/v1/.tmp.

Tried using yarn then react-native-git-upgrade without avail (also cleared the cache). I get:

git-upgrade info Install the new version
error Command failed.
Exit code: 128
Command: git
Arguments: clone https://github.com/GeekyAnts/react-native-drawer C:\Users\Gabriel\AppData\Local\Yarn\cache\v1.tmp\356e7fb1c51407d4d88112a3cfb3564c
Directory: C:\Users\Gabriel\Dropbox\React\native\chaski-react
Output:
fatal: working tree '.' already exists.
Error: Command failed.
Exit code: 128
Command: git
Arguments: clone https://github.com/GeekyAnts/react-native-keyboard-aware-scroll-view C:\Users\Gabriel\AppData\Local\Yarn\cache\v1.tmp\a3450fedabcb67009ae55c8f322f57b0
Directory: C:\Users\Gabriel\Dropbox\React\native\chaski-react
Output:
fatal: working tree '.' already exists.
at MessageError (C:\Program Files (x86)\Yarn\lib\cli.js:139:5)
at ProcessTermError (C:\Program Files (x86)\Yarn\lib\cli.js:158:1)
at ChildProcess.proc.on (C:\Program Files (x86)\Yarn\lib\cli.js:29937:15)
at emitTwo (events.js:106:13)
at ChildProcess.emit (events.js:191:7)
at maybeClose (internal/child_process.js:877:16)
at Process.ChildProcess._handle.onexit (internal/child_process.js:226:5)
git-upgrade ERR! An error occurred during upgrade:
git-upgrade ERR! Error: Command 'yarn add<EMAIL_ADDRESS><EMAIL_ADDRESS>exited with code 1:

In my case it was giving an error because of yarn no being able to read private repositories like this:
   "react-native-elements": "git+https://github.com/*****/react-native-elements.git#Custom_Button",
Some open issue: issue1, issue2


[hmac,dv] Block-level tests with SHA-2 NIST test vectors for new digest sizes 384/512

This adds block-level testing with SHA-2 NIST test vectors for newer digest sizes 384/512, which addresses https://github.com/lowRISC/opentitan/issues/22932 in part.
Block-level testing for HMAC with different key lengths is not yet added. Once in, this would close out https://github.com/lowRISC/opentitan/issues/22932.

Thanks @martin-velay I've pushed again now but a single test run (when it's not smoke) is now very long since it runs through all test vectors for each digest size: SHA-2 256 then 384 then 512, and some time out. I think we should be splitting these up into 3 tests for each digest size and we shouldn't merge now.

I've pushed again to this PR and split the test into 3 tests for each of the digest sizes (I think some timeouts might happen still but we'll see and adjust accordingly), and also released the constraints to hit illegal digest size/key length. I've tested/implemented changes for this before to make sure they are casted correctly in the HW and predicted correctly in the scoreboard, but I'm still running a local block-level regression at my end to confirm. Once merged, this should also close https://github.com/lowRISC/opentitan/issues/23592.

@martin-velay, thanks for your feedback and comments, I've addressed them now in my last push after rebasing.
@vogelpi, can we go ahead and merge?

The failing CI tests are unrelated to the changes in this PR, can we go ahead and merge @rswarbrick?


Add CI (GitHub Actions)

For running checks on every PR/commit, you can check https://github.com/capacitor-community/bluetooth-le/blob/main/.github/workflows/main.yml
For automatically publishing to NPM on a Git tag push, see https://github.com/dennisameling/muuri-angular/blob/master/.github/workflows/npm-publish.yml

[ ] PR/Commit
[ ] Publish


network: rename SocketAddressProvider as ConnectionInfoProvider

Commit Message: network: rename SocketAddressProvider as ConnectionInfoProvider
Additional Description:
Since SocketAddressProvider isn't only include the socket address info, also include the information for connection, like connection id, SSL connection info and SNI name. So rename it as ConnectionInfoProvider.
Risk Level: low
Testing: all
Docs Changes: n/a
Release Notes: n/a
Part of #17168

/retest

/retest

I realise the changes in the PR are renames (only?), but I think this is too large to review. Perhaps consider renaming the classes but leave the variable names as they are? Those can be renamed in smaller chunks, or as drive-bys?
Should ConnectionInfoProvider and ConnectionInfoProviderSetter be moved into a dedicated header file?

I realise the changes in the PR are renames (only?), but I think this is too large to review. Perhaps consider renaming the classes but leave the variable names as they are? Those can be renamed in smaller chunks, or as drive-bys?

yes, it is only about rename. also agree it is huge...terrible for review. I can separate them. Thanks for review this terrible PR!

Should ConnectionInfoProvider and ConnectionInfoProviderSetter be moved into a dedicated header file?

Yes, it could be, do you want me do it in separate PR?

It is still huge, I'm trying to only rename the classname without the member name first.

Yes, it could be, do you want me do it in separate PR?

Don't think it would increase the number of changes that much, I think it's probably ok to do it here. It's up to you, I think either way is ok though.

I think either way is ok though.

Perhaps a dedicated PR would be better after all.


[Discussions] Why remove preloadPagesCount?

Content
It makes a lot to improve the quality of the gallery.
Can you tell me why it was removed?

It has an unable fixing issue, and it has anothoer workaround for it

It has an unable fixing issue, and it has anothoer workaround for it

Can you tell me what the solution is?

https://github.com/fluttercandies/extended_image/blob/7b444abd813b4a5d256f4196045327dd48840754/example/lib/pages/simple/simple_photo_view_demo.dart#L109


I have to scroll down after every step for EC2

v0.21.2
Results:
For every step I take for EC2 host setup I have to scroll down
Expected:
Auto scroll to section I need to work on next

Scrolls to the bottom after the step changes now

v0.22.0-rc1
Verified fixed


get request ID from S3 upload by AWS .net framework SDK

I have this code write in .net framwork 4.5 and need to get request ID for S3 upload
anyone have idea how to do it thank you .
Boolean uploadS3(string filePath, string fileName )
{
        // Get ket S3
         string existingBucketName = System.Configuration.ConfigurationManager.AppSettings["S3BucketName"]; ;
        string s3Accesskey = System.Configuration.ConfigurationManager.AppSettings["s3Accesskey"]; ;
        string s3Secret = System.Configuration.ConfigurationManager.AppSettings["s3Secret"]; ;

        try
        {

            // Set S3 file premission and path
            TransferUtilityUploadRequest fileTransferUtilityRequest = new TransferUtilityUploadRequest
            {
                BucketName = existingBucketName,
                FilePath = filePath,
                StorageClass = S3StorageClass.ReducedRedundancy,
                PartSize = 6291456, // 6 MB.
                Key = fileName,
                CannedACL = S3CannedACL.PublicRead,
                TagSet = new List<Tag>{new Tag { Key = "TEMP", Value = "Y"} }
            };

            // Set S3 Upload Class
            TransferUtility fileTransferUtility = new  TransferUtility(
                s3Accesskey, s3Secret, Amazon.RegionEndpoint.APSoutheast1);
            fileTransferUtility.Upload(fileTransferUtilityRequest);

            return true;
        }
        catch (AmazonS3Exception s3Exception)
        {
            Console.WriteLine(s3Exception.Message,
                              s3Exception.InnerException);
            return false;
        }
    }

What do you mean "request ID"?
Indy

I have problem in some server cannot put object to S3 and need request ID to sent to AWS support Team to see log on aws site to see what happen but I cannot get request ID for that.
I try on this for .net to get request ID but it not working
https://docs.aws.amazon.com/AmazonS3/latest/dev/troubleshooting.html
https://aws.amazon.com/blogs/developer/logging-with-the-aws-sdk-for-net/
can we print request ID by conolse log or anything else on code ?

AWSConfigs.LoggingConfig.LogResponses = ResponseLoggingOption.Always;
AWSConfigs.LoggingConfig.LogTo = LoggingOptions.Console;
AWSConfigs.LoggingConfig.LogMetrics = true;

That works for me, hope that helps!
Indy

When you catch and find an AamzonServiceException in the exception tree, you could grab the RequestId property.
If you are not hitting the catch block ,it's probably because the AmazonS3Exceptino is buried as an inner exception.

@sstevenkang
catch (AmazonServiceException s3Exception)
{
vError = "AmazonServiceException Error : " + s3Exception.Message + "#" + vStepError + "#" + s3Exception.RequestId;
Console.WriteLine("AmazonServiceException Error : " + s3Exception.Message + s3Exception.RequestId, s3Exception.InnerException);
return false;
}
vStepError = "Step3";
fileTransferUtility.Upload(fileTransferUtilityRequest);
vStepError = "Step4";
output for exception is "error s3 Wak AmazonServiceException Error : A WebException with status SendFailure was thrown.#Step3# " not get request ID
We have something missing or not ? Thank.

@koranit   have you tried using something like Fiddler to retrieve the raw HTTP traffic?

@koranit It's static configuration. So just put it before your method call.
Indy

@koranit , AmazonS3Exception.RequestId and AmazonS3Exception.AmazonId2 are the properties that will be populated when the SDK receives an error response from S3. These are the properties AWS Support will need to troubleshoot S3 related issues.
There are other cases where an AmazonServiceException can be thrown and either the request itself was not sent  (WebExceptionStatus.SendFailure), or a response was not received (e.g the request timed out), in these cases the request ID will not be populated because a response was not received.
How frequently do you see the AmazonServiceException with the message "A WebException with status SendFailure was thrown"? SendFailure is automatically retried by the SDK .

Resolving due to inactivity.  Feel free to open a new issue & linked this one.

I have been asked to provide S3 RequestID for AWS Support to troubleshoot intermittent exceptions when reading an object from S3 using the .NET SDK call GetObjectAsync. The exception is below:
System.Net.Sockets.SocketException: Connection reset by peer
How do I get Request ID consistently for such S3 failures that result in IO Exception? I understand that the Request ID is a 2 piece info. The catch block for AmazonS3Exception doesn't work since the exception thrown is of type SocketException. So I cannot use the RequestID or the ResponseHeaders property of the AmazonS3Exception class. Please note that I cannot reproduce these IO Exception on demand..its intermittent in production.


[WIP]Start experimenting with DI

Checklist

[ ] Appropriate tests were added
[ ] Any code changes were done in a way that does not break public API
[ ] All documentation related to code changes were updated
[ ] The new code follows the
contributor guidelines, in particular the SciML Style Guide and
COLPRAC.
[ ] Any new documentation only uses public API

Additional context
Add any other context about the problem here.

@Vaibhavdixit02 DifferentiationInterface v0.2 is out, and we got rid of the bang-bang in favor of always mutating the provided storage. This will make all of our lives easier

@Vaibhavdixit02 DI v0.3 is out, and it even removes the need for SparseDiffTools in the simplest cases. Take it for a spin!

You need a more recent version of DI, at least 0.5.1 for Enzyme to work in second order and 0.5.2 for it to be fast. The tests are still running with 0.5.0.

Come on, I run all tests against DI master locally depending on releases is for p******! 🤣

It seems the last tests were run 3 days ago, you closing and reopening has apparently not re-triggered CI

Because of the merge conflict

Indeed you pushed a merge conflict at least in the project file


[WIP] Feat: Clusteradm init --wait for progressive hub init

/hold
until https://github.com/open-cluster-management-io/clusteradm/pull/93, will rebase after the precedence merged

/hold cancel


fix: initialize sticky elements for dynamically loaded DOM, fixes #8016

We have to check for document.readyState === "complete" when the window and content is already loaded and more content is loaded afterwards that is needed for the magellan and sticky solution.

Working codepen: https://codepen.io/DanielRuf/pen/LQOWmE

I just found the same issue with Magellan. There is probably others issues like this across all components. :/

Right, we may have to add this to some more components.

I'll check the other components, prepare an issue and collect them there and link the PRs then (I would rather go step for step or component for component to have better and more atomar changes which is easier for bisecting and regression testing imo).

I was going to take care of this and open a PR but ok, have fun ;)

keeping everything in one PR is easier to manage.
Agreed. I'll prepare the other commits then. If you want you can also push to my branch. Or do we already have some PRs for the other components?

Or do we already have some PRs for the other components?

I don't think. There was PRs before that resolved some cases manually (and sometimes badly), but everything is merged by now.

Also see https://github.com/zurb/foundation-sites/issues/9047

Thanks.

Pushed a new commit. Did not find more occurences where we wait for the load event.

@DanielRuf Seen. LGTM but require some cleaning as said at https://github.com/zurb/foundation-sites/pull/10988#discussion_r171405136, maybe in an other PR (that or in that PR but with a squash after). Do you want to take care of this or should I ?

LGTM but require some cleaning as said at #10988 (comment), maybe in an other PR (that or in that PR but with a squash after). Do you want to take care of this or should I ?

Would be glad if I couls get some assistance here so feel free to prepare a PR with the needed changes to make them all consistent then.

Closing in favor of https://github.com/zurb/foundation-sites/pull/11077


Added all new thread relating routes

Pull Request Etiquette

[X] I have checked the PRs for upcoming features/bug fixes.
[X] I have read the contributing guidelines.

Changes

[X] Internal code
[ ] Library interface (affecting end-user code)
[ ] Documentation
[ ] Other: _____

Description
Added the new thread relating routes

START has been renamed to CREATE, anything else?

@Tais993 Can you rebase this against v5?
The automatic rebase included the changes in features/threads but we're abandoning those in favor of building them out in v5

Looks like I fucked something up

It's because it's originally based on feature/threads which is not compatible with the history of v5.
You will likely need to open a fresh PR with the changes freshly applied against v5 from scratch unfortunately.