fix: sort the nondeterministic arrays

I opened this PR to start the conversation with maintainers in modifying the program related to non-deterministic operations.
There are two modules junit4 and junit5 in the package, but this PR only modifies the junit4 module before starting the discussion to talk about any concerns in the proposed solution.
We could then modify junit5 module after we meet an agreement on the junit4 module.
What is the purpose of the change?
This PR aims to fix some nondeterministic behaviors in the program junit4/src/main/java/com/google/testing/junit/testparameterinjector/TestParameterAnnotationMethodProcessor.java
which is identified by the engine NonDex. The program uses implementation-dependent operations such as getDeclaredFields() and getDeclaredMethods, which would return arrays not in particular order given different Java version or JVM.
After changing the implementation-dependent operations in Java such as getDeclaredFields() or getDeclaredMethods with NonDex, the nondeterministic behaviors would cause the test failures in the following test classes:

com.google.testing.junit.testparameterinjector.TestParameterAnnotationMethodProcessorTest
com.google.testing.junit.testparameterinjector.TestParameterTest
com.google.testing.junit.testparameterinjector.TestParametersMethodProcessorTest
com.google.testing.junit.testparameterinjector.ParameterValueParsingTest

There are three categories of error/failures shown in NonDex logs:

Assertion fails because parameter values are not in particular order (Will not be fixed after discussion)

[ERROR] test[DuplicateIdenticalFieldAnnotationTestClass:SUCCESS_ALWAYS](com.google.testing.junit.testparameterinjector.TestParameterAnnotationMethodProcessorTest)  Time elapsed: 0.004 s  <<< FAILURE!
java.lang.AssertionError:

----------------------
ImmutableMap.<String, String>builder()
    .put("test[foo,foo]", "foo:foo")
    .put("test[foo,bar]", "foo:bar")
    .put("test[bar,foo]", "foo:bar")
    .put("test[bar,bar]", "bar:bar")
    .build()
----------------------

keys with wrong values
for key       : test[bar,foo]
expected value: bar:foo
but got value : foo:bar
---
expected      : {test[foo,foo]=foo:foo, test[foo,bar]=foo:bar, test[bar,foo]=bar:foo, test[bar,bar]=bar:bar}
but was       : {test[foo,foo]=foo:foo, test[foo,bar]=foo:bar, test[bar,foo]=foo:bar, test[bar,bar]=bar:bar}
        at com.google.testing.junit.testparameterinjector.TestParameterAnnotationMethodProcessorTest.test(TestParameterAnnotationMethodProcessorTest.java:833)
Caused by: com.google.common.truth.AssertionErrorWithFacts:

----------------------
ImmutableMap.<String, String>builder()
    .put("test[foo,foo]", "foo:foo")
    .put("test[foo,bar]", "foo:bar")
    .put("test[bar,foo]", "foo:bar")
    .put("test[bar,bar]", "bar:bar")
    .build()
----------------------

Assertion fails because parameter names are not in particular order(Will not be fixed after discussion)

[ERROR] test[IndependentAnnotation:SUCCESS_ALWAYS](com.google.testing.junit.testparameterinjector.TestParameterAnnotationMethodProcessorTest)  Time elapsed: 0.183 s  <<< FAILURE!
java.lang.AssertionError:

----------------------
ImmutableMap.<String, String>builder()
    .put("test[B1,C1,A1]", "A1:B1:C1")
    .put("test[B2,C2,A2]", "A2:B2:C2")
    .put("test[B2,C3,A2]", "A2:B2:C3")
    .build()
----------------------

missing keys
for key         : test[A1,B1,C1]
expected value  : A1:B1:C1
for key         : test[A2,B2,C2]
expected value  : A2:B2:C2
for key         : test[A2,B2,C3]
expected value  : A2:B2:C3
unexpected keys
for key         : test[B1,C1,A1]
unexpected value: A1:B1:C1
for key         : test[B2,C2,A2]
unexpected value: A2:B2:C2
for key         : test[B2,C3,A2]
unexpected value: A2:B2:C3
---
expected        : {test[A1,B1,C1]=A1:B1:C1, test[A2,B2,C2]=A2:B2:C2, test[A2,B2,C3]=A2:B2:C3}
but was         : {test[B1,C1,A1]=A1:B1:C1, test[B2,C2,A2]=A2:B2:C2, test[B2,C3,A2]=A2:B2:C3}
        at com.google.testing.junit.testparameterinjector.TestParameterAnnotationMethodProcessorTest.test(TestParameterAnnotationMethodProcessorTest.java:835)
Caused by: com.google.common.truth.AssertionErrorWithFacts:

----------------------

Errors because of Index out of bounds(Fixed)

<<stringTest[one,number1=2,stringParam=null](com.google.testing.junit.testparameterinjector.TestParameterTest$WithValuesProvider)>> java.lang.ArrayIndexOutOfBoundsException: Index 7 out of bounds for length 4
        at com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:77)
        at com.google.testing.junit.testparameterinjector.TestParameterAnnotationMethodProcessor.getParameterValuesForTest(TestParameterAnnotationMethodProcessor.java:785)
        at com.google.testing.junit.testparameterinjector.TestParameterAnnotationMethodProcessor.maybeGetTestMethodParameters(TestParameterAnnotationMethodProcessor.java:662)
        at com.google.testing.junit.testparameterinjector.TestMethodProcessorList.lambda$getTestMethodParameters$2(TestMethodProcessorList.java:109)
        at com.google.common.collect.Iterators$6.transform(Iterators.java:828)
        at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:52)
        at com.google.common.collect.Iterators$5.computeNext(Iterators.java:671)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:140)
        at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46)
        at com.google.common.collect.FluentIterable.first(FluentIterable.java:515)
        at com.google.testing.junit.testparameterinjector.TestMethodProcessorList.getTestMethodParameters(TestMethodProcessorList.java:112)
        at com.google.testing.junit.testparameterinjector.PluggableTestRunner.methodInvoker(PluggableTestRunner.java:245)
        at com.google.testing.junit.testparameterinjector.PluggableTestRunner.methodBlock(PluggableTestRunner.java:212)
        at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
        at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
...

How to fix them?
The operations getDeclaredFields() or getDeclaredMethods are implementation-dependent operations, and the returned elements in the array are not in particular order based on different Java versions.
To minimize the impact of modification, this PR will only sort the returned arrays of getDeclaredMethods to prevent nondeterministic behaviors caused by implementation-dependent operations.
The non-deterministic operation getDeclaredFields will be left untouched.
Fix for parameter names or parameter values not in particular order (Will not be fixed after discussion)
When setting remainingTestParameterValuesForFieldInjection in postProcessTestInstance , it will use the following function call chain: getParameterValuesForMethod -> getAnnotationValuesForUsedAnnotationTypes  -> getAnnotationFromParametersOrTestOrClass.
One cause of failure is in getAnnotationFromParametersOrTestOrClass, in which getDeclaredFields returns a values array and it is not in particular order.
https://github.com/google/TestParameterInjector/blob/5d67f8cb66481638bcd117227bd0103c32c84491/junit4/src/main/java/com/google/testing/junit/testparameterinjector/TestParameterAnnotationMethodProcessor.java#L927
In addition, when matching the parameter values, it only checks the type without parameter names.
https://github.com/google/TestParameterInjector/blob/5d67f8cb66481638bcd117227bd0103c32c84491/junit4/src/main/java/com/google/testing/junit/testparameterinjector/TestParameterAnnotationMethodProcessor.java#L1049-L1055
It would cause the assertion failures when two parameters are in the same type as in the test case com.google.testing.junit.testparameterinjector.TestParameterAnnotationMethodProcessorTest$DuplicateIdenticalFieldAnnotationTestClass.test()
https://github.com/google/TestParameterInjector/blob/5d67f8cb66481638bcd117227bd0103c32c84491/junit4/src/test/java/com/google/testing/junit/testparameterinjector/TestParameterAnnotationMethodProcessorTest.java#L326-L348
At last, when creating the list fieldAnnotations in calculateAnnotationTypeOrigins. There is an implementation-dependent operation found in the following code snippet, which will return a list of fields not in particular order
https://github.com/google/TestParameterInjector/blob/5d67f8cb66481638bcd117227bd0103c32c84491/junit4/src/main/java/com/google/testing/junit/testparameterinjector/TestParameterAnnotationMethodProcessor.java#L320-L325
The list of fields fieldAnnotations would then be used by the function call chain getParameterValuesForTest -> getParameterValuesForMethod -> getAnnotationValuesForUsedAnnotationTypes -> getAnnotationTypeOrigins. Among them, the function getParameterValuesForTest is extensively used in different functions such as getTestParameterValues, maybeGetConstructorParameters, maybeGetTestMethodParameters, and postProcessTestInstance, which would at last cause the parameter names non-deterministic.
Fix for index out of bound error
The error happened because the cache value returned from getParameterValuesForMethod(testMethod, testClass) is not as expected. When query the cache from getParameterValuesForMethod with the key testMethod, testMethod is nondeterministic.
https://github.com/google/TestParameterInjector/blob/5d67f8cb66481638bcd117227bd0103c32c84491/junit4/src/main/java/com/google/testing/junit/testparameterinjector/TestParameterAnnotationMethodProcessor.java#L783C2-L785
testMethod is nondeterministic because there is an implementation-dependent operation getDeclaredMethods in the following code snippet. It will returns the methods array which is not in particular order.
https://github.com/google/TestParameterInjector/blob/5d67f8cb66481638bcd117227bd0103c32c84491/junit4/src/main/java/com/google/testing/junit/testparameterinjector/TestParameterAnnotationMethodProcessor.java#L1275-L1278
How to verify the change?
Test environment:
openjdk version "<IP_ADDRESS>"
Apache Maven 3.8.8
Ubuntu 20.04.6 LTS
Linux version: 5.4.0-163-generic

Command:
mvn -pl junit4 edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=<TEST-CLASS> -Drat.skip=true -DnondexRuns=<NUM_RUNS>

replace the <NUM_RUNS> with an integer number of runs to change different implementation.
replace the <TEST-CLASS> with one of the following test classes:

com.google.testing.junit.testparameterinjector.TestParameterAnnotationMethodProcessorTest
com.google.testing.junit.testparameterinjector.TestParameterTest
com.google.testing.junit.testparameterinjector.TestParametersMethodProcessorTest
com.google.testing.junit.testparameterinjector.ParameterValueParsingTest

For example,
mvn -pl junit4 edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=com.google.testing.junit.testparameterinjector.TestParameterAnnotationMethodProcessorTest -Drat.skip=true -DnondexRuns=5

It will runs the test com.google.testing.junit.testparameterinjector.TestParameterAnnotationMethodProcessorTest for 5 times with different non-deterministic implementation.
Note that some aforementioned tests, such as Assertion fails because parameter values are not in particular order running with NonDex, will still fail, because, after discussion below, we decided to only fix some of non-deterministic operations instead of all of them.

I think your changes make sense generally, but I have a few worries:

Checking whether the field names match: Makes sense.
Sorting methods by name in getMethodsIncludingParents(): This is only used for indexed lookup, so it's fine.
Sorting fields by name:

If we do this, their values are no longer shown in the test names in the order that they were declared (which is the current behavior IIRC, even if it not specified). This makes for a worse test name for everyone to fix a rather hypothetical issue that only really affects those few clients that depend on the test name.
This PR changes the test names, which will break some clients (I know at least internally in Google, there are frameworks that depend on the test name, and they will need some manual work to fix)

By the way, you don't need to worry about junit4/junit5. The files are derived from the same master copy in our internal codebase, so whatever we change here, I'll have to port to the master copy and then the junit5 is automatically updated as well.

Sorting fields by name:

If we do this, their values are no longer shown in the test names in the order that they were declared (which is the current behavior IIRC, even if it not specified). This makes for a worse test name for everyone to fix a rather hypothetical issue that only really affects those few clients that depend on the test name.
This PR changes the test names, which will break some clients (I know at least internally in Google, there are frameworks that depend on the test name, and they will need some manual work to fix)

Hi @nymanjens ,
Thanks for your thoughtful reply!
Do you have any suggestions to keep the values in original order when they were declared?
Instead of sorting by their name, would it be a good idea to introduce another variable to keep the variables in original order?

Instead of sorting by their name, would it be a good idea to introduce another variable to keep the variables in original order?

Where would this other variable come from?

Where would this other variable come from?
It seems that the built-in Java library does not provide such function to preserve the order of declared variables.

I found that there are some ways to get the order of declared variables in source code.
Some libraries that are able to do this are java-parser and spoon.
However, I am thinking that it would introduce more complexity in order to solve this non-determinism problem.

So how about half-fixing this issue and only retain the changes to the field name matching and getMethodsIncludingParents()?

That will still improve the non-determinism.
We could do it.
Thanks!

hi @nymanjens ,
I've updated the PR to only fix half of non-deterministic behaviors.
Please feel free to contact me if you still have any concerns.
Thank you!

Thanks! I'll make sure this change also is reflected in the Google-internal and junit5 fork


Complete connection module for Novena + LMS7

This issue should start by implementing the connection registry module for the Novena with spi interface only, no streaming.
However, the work for Novena LMS6 driver with streaming should be applicable to this work. Possibly using the existing FPGA driver work from LMS6, Novena has a functional streaming RX and TX channel that can be used.

Which LMS7 board would be used with Novena?

I believe it's called MyriadRF7-Novena

Ah, I wasn't aware of this. I guess it might be nice if it were called Novena-RF7, or something else that seemed to fit with the naming of the LMS6 module, "Novena-RF". Otherwise it can get a bit confusing...

FYI - This is complete with the exception of UpdateExternalBandSelect which has the old implementation commented out. It needs to be reimplemented around the new API hook:
void ConnectionNovenaRF7::UpdateExternalBandSelect(const size_t channel, const int trfBand, const int rfePath)
{
    //TODO...
    printf("Fill in the UpdateExternalBandSelect() function for NovenaRF7!\n");
    /*
    //in case of Novena board, need to update GPIO
    if(controlPort->GetInfo().device == LMS_DEV_NOVENA)
    {
        uint16_t regValue = SPI_read(0x0706) & 0xFFF8;
        //lms_gpio2 - tx output selection:
        //		0 - TX1_A and TX1_B (Band 1),
        //		1 - TX2_A and TX2_B (Band 2)
        regValue |= Get_SPI_Reg_bits(LMS7param(SEL_BAND2_TRF)) << 2; //gpio2
        //RX active paths
        //lms_gpio0 | lms_gpio1      	RX_A		RX_B
        //  0 			0       =>  	no active path
        //  1   		0 		=>	LNAW_A  	LNAW_B
        //  0			1		=>	LNAH_A  	LNAH_B
        //  1			1		=>	LNAL_A 	 	LNAL_B
        switch(Get_SPI_Reg_bits(LMS7param(SEL_PATH_RFE)))
        {
            //set gpio1:gpio0
            case 0: regValue |= 0x0; break;
            case 1: regValue |= 0x2; break;
            case 2: regValue |= 0x3; break;
            case 3: regValue |= 0x1; break;
        }
        SPI_write(0x0706, regValue);
    }
    */
}

Novena support is discontinued in the current LimeSuite


CLOUDP-67741: Add a global --output flag

Proposed changes
Jira ticket: CLOUDP-67741
Checklist

[x] I have added tests that prove my fix is effective or that my feature works
[x] I have added any necessary documentation (if appropriate)
[x] I have run make fmt and formatted my code

Further comments
This adds a flag to control output format, it defaults to json (what we do) and there's not much to it currently just trying to do this big change as incremental as possible

General question, any idea why the e2e tests are failing at the moment on this?

@colm-quinn the only failing tests is a cloud manager IAM issue we are waiting for this to be fixed on cloud


installation not working on rails-api

Hello,
I'm not sure if this is a rspec-rails issue or if it should be addressed somewhere else. I've spent some time reading the code and haven't figured out what the problem may be.
Steps to reproduce
$ rails-api new my_app
$ cd my_app

# Add rspec-rails to Gemfile

$ bundle install
$ rails g rspec:install
# => Could not find generator 'rspec:install'

$ rails g --help
# => Rspec generators not listed

For reference, running the exact same steps (except for $ rails g devise:install) with the gem Devise doesn't generate an error, and the generator is listed when running rails g --help.
That's why I thought that it might be a rspec-rails issue.

This is weird. It works now in a new app but not on my previous one, with (apparently) exact same stuff in the gemfile.  Sorry for the noise!


AFAICT openssl's strategy for handling TLS 1.3 session tickets makes it impossible to reliably implement communication patterns where the server never sends application-level data

I maintain a Python networking library called Trio, and I've been struggling to get it working with Openssl v1.1.1/TLS 1.3. We use openssl with memory BIOs and have an extensive test suite that passes with earlier openssl versions, but hits a number of problems after upgrading to v1.1.1. The main issue seems to be the session tickets that openssl sends after the handshake in server mode, and how they affect connections where the client never calls SSL_read.
Due diligence: I found these previous issues/PRs that are all about the exact same issue that I'm facing, but having read them carefully I still can't figure out how to make this work: #6342, #6904, #6944. Also, for reference, this is the Trio bug: https://github.com/python-trio/trio/issues/819
TCP (as I understand it)
Let's ignore TLS for the moment and just talk about TCP. Suppose we have a client that connects, sends some data, and then disconnects, without ever calling recv. In pseudo-code:
# Socket client
sock = connect(...)
while ...:
    sock.send(...)
sock.close()

# Socket server (safe)
sock = accept()
while ...:
    sock.recv(...)
    if eof:
        sock.close()
        break

This isn't a terribly common pattern, but it's perfectly legal and reliable. Call this the safe pattern.
But, TCP has a gotcha: suppose we have the server send a bit of data, and change nothing else. In particular the client still never calls recv:
# Socket server (unsafe)
sock = accept()
sock.send(<one byte of data>)   # <--- This is the only line that's different
while ...:
    sock.recv(...)
    if eof:
        sock.close()
        break

Now, this is a little funny looking, because the server is sending some data that the client will never read. But, whatever, that shouldn't affect anything, right? This shouldn't affect the data being sent from the client→server, right?
Well, that would be logical, but it's wrong! In the unsafe pattern, an arbitrary amount of the data sent by the client can be lost, even though the client code didn't change at all.
This happens because of arcane details of how TCP works: in the safe pattern, when the client calls close, the client's kernel sends a FIN packet, and the server's kernel queues that up behind all the other data in the server's receive buffer, and everything proceeds in an orderly fashion. But in the unsafe pattern, the client has incoming data. And if there's incoming data before or after a close, then RFC 1122 says:

If such a host issues a CLOSE call while received data is still pending in TCP, or if new data is received after CLOSE is called, its TCP SHOULD send a RST to show that data was lost.

So in this case, the client's kernel might send a RST, instead of or in addition to the FIN. And then when the server's kernel sees an RST packet, it discards all buffered data. So, if there's any data that the client sent that's still sitting in the server kernel buffers, it disappears forever.
References:

https://blog.netherlabs.nl/articles/2009/01/18/the-ultimate-so_linger-page-or-why-is-my-tcp-not-reliable
http://blog.olivierlanglois.net/index.php/2010/02/06/tcp_rst_flag_subtleties
The websocket RFC cites this as motivation for that protocol's shutdown semantics: https://tools.ietf.org/html/rfc6455#section-1.4

What does this have to do with TLS?
Generally speaking, it should be possible to take any application that uses raw TCP, and switch it to use TLS-over-TCP instead, right? (With the notable exception of half-closed connections, but never mind.) So let's port our client/server to use TLS:
# TLS-over-TCP client
tlssock = connect(...)
tlssock.do_handshake()
while ...:
    tlssock.send(...)
tlssock.send_close_notify()  # Often skipped in practice, but let's be standards-compliant
tlssock.close_tcp()

# TLS-over-TCP server ("safe")
tlssock = accept()
tlssock.do_handshake()
while ...:
    tlssock.recv(...)
    if eof:
        tlssock.send_close_notify()
        tlssock.close_tcp()
        break

Now here's the issue: with TLS 1.2 and earlier, if we follow the "safe pattern" at the application layer, like this, then openssl will ultimately translate that into the "safe pattern" at the TCP layer. But with TLS 1.3, openssl's habit of sending session tickets after the handshake means that this exact same code now produces the unsafe pattern at the TCP layer. The server→client session tickets could cause the client→server application data to be lost.
#6944 changes how openssl reacts to getting notified of a client close while sending session tickets, so that the server can keep calling recv. But that doesn't help if the kernel has already discarded the buffer that recv is trying to read out of. The problem here is all inside the TCP stacks; there's nothing openssl can do about it, except avoid sending the session tickets in the first place.
There's also a secondary problem, but it's more theoretical: if the server→client buffer is small enough, then this code could deadlock at the handshake – the server's call to SSL_do_handshake won't return until the client calls SSL_read, but the client is calling SSL_send, which will eventually block until the server calls SSL_read, but the server can't because it's waiting for the client to call SSL_read... I'm not sure if there are any realistic cases where people use buffers that are small enough to trigger this though. (We have some torture tests with small buffers to flush out problems like this, which of course did catch it.)
What to do?
Of course we can disable session tickets, but this is difficult in our case because Python's openssl bindings don't expose SSL_set_num_tickets and SSL_CTX_set_session_ticket_cb. Also, as a generic networking library, we wouldn't want to disable session tickets in general. But, we also really don't want to have to explain to our users that enabling TLS is just a matter of switching from a SocketStream to a SSLStream, the APIs are identical except that if you happen to know that your client might not ever read data, then on the server side you have to call this special API before the handshake. That's a super leaky abstraction.
We could do full bidirectional-shutdown, as suggested here. This seems problematic, though – the RFCs explicitly say that bidi shutdown is never required, bidi shutdown has always been useless before, and I've never heard of any existing software that does it. Trio doesn't have any way to force its peers to use it. If OpenSSL is going to say that all TLS 1.3 implementations that want to interoperate with OpenSSL have to switch to bidi shutdown, then that seems like a huge ask. Also, isn't TLS 1.3 supposed to reduce the number of round-trips we make?
The best idea I have is:

don't sent tickets automatically after the TLS 1.3 handshake
automatically send tickets the first time the server calls SSL_write
also provide an explicit SSL_write_tickets function to send tickets immediately

Due diligence: I found these previous issues/PRs that are all about the exact same issue that I'm facing, but having read them carefully I still can't figure out how to make this work: #6342, #6904, #6944. Also, for reference, this is the Trio bug: python-trio/trio#819

Have you read the shutdown documentation after #7188 has been merged?
Did you read the wiki about TLS 1.3, in particular the section about session?
https://wiki.openssl.org/index.php/TLS1.3#Sessions

Generally speaking, it should be possible to take any application that uses raw TCP, and switch it to use TLS-over-TCP instead, right?

Yes. But generally speaking, any encapsulation of a protocol by an other protocol causes problems. Generally, it's hard to abstract things away. And you point it out yourself above that there is an unsafe pattern in TCP.

So let's port our client/server to use TLS:
# TLS-over-TCP client
tlssock = connect(...)
tlssock.do_handshake()
while ...:
    tlssock.send(...)
tlssock.send_close_notify()  # Often skipped in practice, but let's be standards-compliant
tlssock.close_tcp()

You really can't skip sending the close notify. If you skip that you're vulnerable to a truncation attack. The other side should react to that with a protocol error.
The standards compliant case would be to receive the close notify before closing the tcp socket. The other side could send back a close notify alert, closing the connection without waiting for it can cause your unsafe TCP pattern. So this really is the unsafe TLS example.
That you need to close the TLS layer before you close the TCP layer is one of the many ways in which TLS leaks things to the application.

There's also a secondary problem, but it's more theoretical: if the server→client buffer is small enough, then this code could deadlock at the handshake – the server's call to SSL_do_handshake won't return until the client calls SSL_read, but the client is calling SSL_send, which will eventually block until the server calls SSL_read, but the server can't because it's waiting for the client to call SSL_read... I'm not sure if there are any realistic cases where people use buffers that are small enough to trigger this though. (We have some torture tests with small buffers to flush out problems like this, which of course did catch it.)

I'm sure that you can cause any TCP connection to deadlock.

We could do full bidirectional-shutdown, as suggested here. This seems problematic, though – the RFCs explicitly say that bidi shutdown is never required, bidi shutdown has always been useless before, and I've never heard of any existing software that does it. Trio doesn't have any way to force its peers to use it. If OpenSSL is going to say that all TLS 1.3 implementations that want to interoperate with OpenSSL have to switch to bidi shutdown, then that seems like a huge ask.

The only case where the bidirectional shutdown is required is when you want to resume the session. I currently don't see when only a one way shutdown shouldn't work, as long as you disable tickets.
Any TLS 1.3 implementation that wants to support resumption really is going to have the same problem. And you really want to support resumption.
So for clients that only send data and never receive any, TLS 1.3 really forces you to either disable resumption or do a bidirectional shutdown.

Also, isn't TLS 1.3 supposed to reduce the number of round-trips we make?

Only at the start of the connection, to get the data faster.

The best idea I have is:
* don't sent tickets automatically after the TLS 1.3 handshake
* automatically send tickets the first time the server calls `SSL_write`

You really also want a client that only writes to support resumption. In that case if SSL_write() is never called, you can never resume the session.

Have you read the shutdown documentation after #7188 has been merged?
Did you read the wiki about TLS 1.3, in particular the section about session?
https://wiki.openssl.org/index.php/TLS1.3#Sessions

Yes :-)

generally speaking, any encapsulation of a protocol by an other protocol causes problems. Generally, it's hard to abstract things away.

It's true that this kind of encapsulation is difficult, but previous versions of openssl managed it, and current openssl manages it except when using TLS 1.3 with session tickets.

And you point it out yourself above that there is an unsafe pattern in TCP.

For sure. But it used to be that if you used the "safe pattern" at the application level, that would map onto the "safe pattern" at the TCP level, and vice-versa – that's a non-leaky abstraction. The regression is that now if you use the safe pattern at the application level, openssl will silently convert that into the unsafe pattern at the TCP level. That's a leaky abstraction.

You really can't skip sending the close notify. If you skip that you're vulnerable to a truncation attack. The other side should react to that with a protocol error.

Yeah, Trio supports two modes: by default it does unidirectional close_notify and expects unidirectional close_notify. Or, if you set https_compatible=True, then it neither sends nor expects close_notify, since this is required for interoperability with most real HTTPS servers. This is mostly orthogonal to the current issue though; the session tickets break both modes in the same way.

The standards compliant case would be to receive the close notify before closing the tcp socket. The other side could send back a close notify alert, closing the connection without waiting for it can cause your unsafe TCP pattern. So this really is the unsafe TLS example.

Actually, no! It's not at all obvious, but AFAICT it really is true that safe patterns used to map to safe patterns and vice-versa, even if you have a mix of peers using unidirectional and bidi shutdown.
In your example, say that our program implements bidi shutdown, and we're talking to a peer that does unidirectional shutdown. It sends us a close_notify, and we send one back. You're right, our close_notify may provoke them to send a RST... but the fact that we've already received their close_notify means that we've already emptied our receive buffer, so their RST can't cause any harm. Disaster is narrowly averted.
Another tricky case is when our peer performs a unidirectional shutdown while we're in the middle of sending data. In this case, the data we're sending might provoke them to send a RST back, and that might cause their close_notify to be lost, and now we don't know where the end of their data is. But, this still doesn't break the abstraction, because in this case plain TCP suffers from exactly the same problem: if they close their socket when we're in the middle of sending data, then their FIN may be lost, and we don't know where the end of their data is. So the semantics are annoying, but they're predictable and consistent regardless of whether you're using TLS.

That you need to close the TLS layer before you close the TCP layer is one of the many ways in which TLS leaks things to the application.

Not sure what you mean. Empirically, this is something that we abstracted away and it used to work fine. Trio's SSLStream and SocketStream classes both have close methods with the same semantics as far as the user's concerned; one of them sends close_notify (at least in standards compliant mode) and the other doesn't, but they don't have to care about that.

I'm sure that you can cause any TCP connection to deadlock.

??? "Writing reliable network protocols is impossible, so there's no point in trying" is not the attitude I was hoping to hear from an openssl dev :-(.
Empirically, we have plenty of protocol implementations that survive our deadlock torture test just fine, including previous versions of openssl.
And actually, now that I think about it, this deadlock thing is actually more of a practical problem than I realized, because it means that with openssl 1.1.1 we can no longer use the deadlock torture test to test protocols that run on top of TLS :-(

The only case where the bidirectional shutdown is required is when you want to resume the session. I currently don't see when only a one way shutdown shouldn't work, as long as you disable tickets.

Right, but here I'm talking about the case where you don't disable tickets. If tickets aren't disabled, then @mattcaswell pointed out in #6904 that the example client/server can be made to work again by turning on bidi shutdown on the client, so it's a possible workaround.

Any TLS 1.3 implementation that wants to support resumption really is going to have the same problem. And you really want to support resumption.

So for clients that only send data and never receive any, TLS 1.3 really forces you to either disable resumption or do a bidirectional shutdown.

My understanding is that bidi shutdown on the client is sufficient to prevent data loss in cases like my example client/server, but it isn't sufficient to support resumption, because openssl doesn't process session tickets that it receives after sending a close_notify.
In any case, sure, it would be nice if our example client/server could support resumption. It's too bad that with TLS 1.3, they can't, without further changes. But that doesn't mean it's OK to stop transmitting their data reliably! Our client/server here are only relying on the traditional guarantees made by TCP and all previous versions of openssl.

You really also want a client that only writes to support resumption. In that case if SSL_write() is never called, you can never resume the session.

In your quote you dropped my third bullet point, which partially addresses this?
Besides... current openssl has exactly the same flaw in practice: if you have a protocol where the server never calls SSL_write, then presumably the client never calls SSL_read, and so it doesn't matter whether the server sends the tickets during the handshake, the client's not going to read them anyway.

And you point it out yourself above that there is an unsafe pattern in TCP.

For sure. But it used to be that if you used the "safe pattern" at the application level, that would map onto the "safe pattern" at the TCP level, and vice-versa – that's a non-leaky abstraction. The regression is that now if you use the safe pattern at the application level, openssl will silently convert that into the unsafe pattern at the TCP level. That's a leaky abstraction.

The safe pattern for both TCP and TLS is to make sure that both ends agree all data has been sent before closing the connection. TLS has a mechanism for that, and it's to send the close notify in both directions.

You really can't skip sending the close notify. If you skip that you're vulnerable to a truncation attack. The other side should react to that with a protocol error.

Yeah, Trio supports two modes: by default it does unidirectional close_notify and expects unidirectional close_notify. Or, if you set https_compatible=True, then it neither sends nor expects close_notify, since this is required for interoperability with most real HTTPS servers. This is mostly orthogonal to the current issue though; the session tickets break both modes in the same way.

I don't know enough about http(s), but is it always clear from the protocol when it's finished sending the data? Is a truncation attack possible?
In https tests I've done I've seen any combination of close notify that's possible, including no and bidirectional.

The standards compliant case would be to receive the close notify before closing the tcp socket. The other side could send back a close notify alert, closing the connection without waiting for it can cause your unsafe TCP pattern. So this really is the unsafe TLS example.

Actually, no! It's not at all obvious, but AFAICT it really is true that safe patterns used to map to safe patterns and vice-versa, even if you have a mix of peers using unidirectional and bidi shutdown.
In your example, say that our program implements bidi shutdown, and we're talking to a peer that does unidirectional shutdown. It sends us a close_notify, and we send one back. You're right, our close_notify may provoke them to send a RST... but the fact that we've already received their close_notify means that we've already emptied our receive buffer, so their RST can't cause any harm. Disaster is narrowly averted.

TLS 1.3 makes it very explicit that sending a close notify only closes the write direction. The other peer is still allowed to send application data back. If it still wants to send application data, and not yet the close notify, you do have a problem. In TCP you can also do a shutdown(SHUT_WR), the connection isn't fully closed.
OpenSSL has supported that for a very long time, even when the older TLS standards didn't say that was supported.

I'm sure that you can cause any TCP connection to deadlock.

??? "Writing reliable network protocols is impossible, so there's no point in trying" is not the attitude I was hoping to hear from an openssl dev :-(.

That's not at all what I'm saying. I'm saying that if both sides of a connection are doing the wrong thing, it's very easy to cause a deadlock. If you think there is a bug in OpenSSL that makes it deadlock while talking to itself, file a bug. Example code would be nice.

My understanding is that bidi shutdown on the client is sufficient to prevent data loss in cases like my example client/server, but it isn't sufficient to support resumption, because openssl doesn't process session tickets that it receives after sending a close_notify.

We've fixed processing session tickets after sending close notify in #7114.

You really also want a client that only writes to support resumption. In that case if SSL_write() is never called, you can never resume the session.

In your quote you dropped my third bullet point, which partially addresses this?

Oh, you want a server that never calls SSL_write() to call some other function instead if it wants to be able to resume? That suddenly changes servers not to support resumption any more, and if the server changes to do it, the client still has the same problem that it needs to read the session.

TLS 1.3 makes it very explicit that sending a close notify only closes the write direction

Oh wow, I'd missed this, and it's super exciting, thank you! That fixes the one place where it used to be impossible to abstract away the difference between TLS and other byte-stream transports. That's really my main concern here... I want to be able to write protocol code in a generic way, that works the same on TCP, TLS, or whatever other transport makes sense. The openssl 1.1.1 session ticket handling breaks that guarantee.

The safe pattern for both TCP and TLS is to make sure that both ends agree all data has been sent before closing the connection. TLS has a mechanism for that, and it's to send the close notify in both directions.

In the past it's always been safe to take any program that worked correctly using plain TCP, and switch it to use TLS with unidirectional close_notify. I guess you can argue that unidirectional close_notify was always "wrong" in some vague existential sense, but it was explicitly allowed by the RFCs and it worked in practice, which seems more important to me.

That's not at all what I'm saying. I'm saying that if both sides of a connection are doing the wrong thing, it's very easy to cause a deadlock. If you think there is a bug in OpenSSL that makes it deadlock while talking to itself, file a bug. Example code would be nice.

This is the bug you are asking me to file :-). My test suite literally started deadlocking when I upgraded to 1.1.1. It's a test where the client does SSL_do_handshake → SSL_write, while the server does SSL_do_handshake → SSL_read, and the transport layer between them has minimal buffering. This is the same communication pattern used by e.g. HTTP/1.1, and it worked fine with previous versions of openssl.
The deadlock happens because the server's SSL_do_handshake doesn't return until it finishes writing out the session tickets, but by the time the server starts trying to write the tickets, the client's SSL_do_handshake has already completed and the client has switched to SSL_write. Neither write can complete because neither side is reading → deadlock.
I could make a standalone reproducer (in Python, say), if that would be helpful, but the problem is very straightforward. As long as the server's SSL_do_handshake sends session tickets but the client's SSL_do_handshake doesn't read them, you have a deadlock hazard here.

We've fixed processing session tickets after sending close notify in #7114.

Oh, excellent, thanks for the update.

Oh, you want a server that never calls SSL_write() to call some other function instead if it wants to be able to resume? That suddenly changes servers not to support resumption any more, and if the server changes to do it, the client still has the same problem that it needs to read the session.

I think this is confusing and we need to break it down by cases :-)
Case 1: client/server where the server sends application data to the client:

On TLS 1.2 and earlier: works correctly and supports resumption
On TLS 1.3 with openssl's current session ticket heuristic: works correctly and supports resumption
On TLS 1.3 with the alternative session ticket heuristic I suggested: works correctly and supports resumption

Case 2: client/server where the server never sends application data to client, with non-bidirectional close:

On TLS 1.2 and earlier: works correctly and supports resumption
On TLS 1.3 with openssl's current session ticket heuristic: can lose data and doesn't support resumption. Can be fixed by modifying the client to use bidi shutdown.
On TLS 1.3 with the alternative session ticket heuristic I suggested: works correctly, but doesn't support resumption. Can be fixed by modifying the client to use bidi shutdown + the server to call SSL_write_tickets.

Case 3: client/server where the server never sends application data to the client, with bidi close:

On TLS 1.2 and earlier: works correctly and supports resumption
On TLS 1.3 with openssl's current session ticket heuristic: works correctly and supports resumption
On TLS 1.3 with the alternative session ticket heuristic I suggested: works correctly, but doesn't support resumption. Can be fixed by modifying the server to call SSL_write_tickets.

So both strategies have some downsides compared to the old TLS 1.2 way of doing things, but make slightly different trade-offs about who suffers if they don't update their programs, and what kind of suffering they experience. It seems like openssl is saying that it's OK if some "case 2" users lose data, because in exchange, some "case 3" users will get slightly lower latency.
My impression is that for existing apps, "case 1" is far more common than "case 2", and "case 2" is far more common than "case 3".
I don't think it's a good trade-off to sacrifice correctness in the relatively common case, in order to speed up the relatively uncommon case.
To be clear though, I'm not like, wedded to that particular proposal or anything. It's just one idea, and I totally get that these are tricky issues and TLS 1.3's new session ticket design forces implementors to make awkward trade-offs. But I think for something as fundamental and widely-used as openssl, it's worth getting the edge cases right.

At least one concern I have is what happens now in the case where
the server never sends data, the client tries to do a
bidirectional shutdown (that it didn't used to do) and the server
just closes the connection instead. The client should get an error
in that case.
I think the above is also broken behaviour. The server should not
assume that the client doesn't want the bidirectional shutdown. I
think the only correct behaviour is to send the close notify alert
and not wait for the other side to send it back before closing the
TCP connection. And that works only in the case where it's clear
the other side is never going to send data back.
I don't know if this actually happens now, but I assume it does.
Which means that fixing the client to support resumption with TLS
1.3 might instead result in the client getting an error.

Yes, that's the scenario. The problem if that happens is that the
client can't tell which part was succesful. But if you're not
worried about it, that's ok for me.

TLS 1.3 will force a client that only sends to change behaviour in
case it wants session resumption. And I'm just worrried that that
might break something.
Note that you safe example of TLS with TLS 1.2 also produces the
unsafe TCP pattern, the server will try to send back a close
notify. TLS 1.3 just sends more.

Note that you safe example of TLS with TLS 1.2 also produces the unsafe TCP pattern, the server will try to send back a close notify.

I think I already explained why this isn't true up above? I'll try explaining again in the hopes that I was just unclear, but if there's a deeper disagreement lmk.
The "unsafe TCP pattern" requires a very specific combination of things all happen together:

Peer A sends data,
...and then closes the connection.
Peer B sends data,
...and hasn't yet read all of the data that Peer A sent

With TLS 1.2, I think you're talking about the case where the client (peer A) sends a unidirectional close_notify and then closes its socket, and then the server (peer B) does what the spec says it should do: after it receives the client's close_notify, it sends back its own close_notify. This means that conditions 1, 2, and 3 are all satisfied... but condition 4 is not. If the server has received the client's close_notify, then by definition the server has already read all the client's data, and can't possibly lose any of it, no matter whether it sends its own close_notify back or not.

You're right that in the example you give as safe, the server will
always properly process the close notify of the client in TLS 1.2.
But at the same time, the TCP connection is not closed properly,
because the server will attempt to write a close notify back, and
there will be some error. I think this is an unsafe pattern that
just happens to work.
Using the same pattern in TLS 1.3, the server will also get an
error like in the case of TLS 1.2, but it might get it earlier
than in the case of TLS 1.2. It might now get that error before
it processed the application data.
So I think that even when the RFC suggest it's possible, that it
was never a good idea to do it. Since there might be applications
that do this, something needs to change. I'm currently not sure if
that something is openssl, or the client application. And I
currently don't know of any real affected client other than some
test suite.

But at the same time, the TCP connection is not closed properly, because the server will attempt to write a close notify back, and there will be some error. I think this is an unsafe pattern that just happens to work.

Yeah, I guess it's again a matter of definitions... I don't know if anyone intentionally designed this behavior, or if it's an accidental outcome of several different features in TCP and its common implementations. But either way, the behavior has been standard and documented for decades, and common protocols like websockets are designed around it.

Since there might be applications that do this, something needs to change. I'm currently not sure if that something is openssl, or the client application. And I currently don't know of any real affected client other than some test suite.

Yeah, I don't know of any real affected applications either, and I totally sympathize with your reluctance to change things based on a weird corner case like this. The reason I think this is important, though, is the way it breaks abstractions.
I'm not writing an application; I'm writing a generic networking library. I guess most applications that use TLS don't use the openssl C API directly, but go through some layer like Trio (my library), or the node.js tls module, or similar. With TLS 1.2, it was certainly difficult for networking library authors to understand all the details of the openssl API and to abstract away the differences between TCP vs. TLS-over-TCP, but it was possible, and that was a cost paid once by the networking libraries, instead of over and over by every applications developer.
Like you, I have no idea whether any of my users are writing applications that would be broken by the new session ticket behavior, so I have to assume the worst. If I can't abstract away the differences, then I have to take on the burden of figuring out exactly what openssl does and doesn't guarantee, how that translates into the higher-level API that my library provides, and then teach all my users about these rare corner cases, just case they happen to ever write an app that would run into them.
There's been a ton of work in recent years to make TLS more universal and accessible. Right now in Trio it's extremely simple... you write trio.open_tcp_stream(host, port) to make a regular TCP connection, and trio.open_ssl_over_tcp_stream(host, port) to make a TLS-protected TCP connection; they act identically after that.  When people implement new protocols, it's very common to have bugs that sneak through testing because they never happen on loopback, and only rarely on real networks; in Trio you can flip a switch to test your protocol against simulated nasty network conditions to shake out these kinds of bugs.
But this all relies on being able to abstract away the difference between TCP and TLS. If that goes away, then it doesn't really matter that it's only in an obscure edge condition; we still risk losing all these features.

Only thing that comes to mind would be to always turn of all tickets...just so that the tickets in TLS 1.3 don't confound the system... :(

@rdp Disabling tickets is quite tricky; you need to use both SSL_CTX_set_num_tickets and define a custom session ticket callback. And of course then you lose session tickets.
We did find and deploy a workaround for this that I think makes TLS 1.3 work correctly on all openssl versions. Instead of letting openssl do I/O directly, we use MemoryBIO, and manually copy data to/from the socket. After SSL_do_handshake completes, we check whether we're running in server mode, and whether we negotiated TLS v1.3. If so, then we capture the last volley of data that openssl asked us to send, and we don't send it – instead we stash it in an internal buffer, and send it the first time we have application data to write.
This is convoluted and gross, but I believe it is a correct and full workaround. And unless the openssl devs decide to fix this bug and backport it to v1.1.1, I guess this is the only way to get correct TLS 1.3 support from openssl on RHEL 8.
Analysis of why this workaround is correct: https://github.com/python-trio/trio/issues/819#issuecomment-517529763
Example implementation: https://github.com/python-trio/trio/pull/1171

I sense some dissatisfaction with how this issue has been addressed up to now. @njsmith and @kroeckx have different views about it and I am not in the position to judge who’s arguments are more conclusive. Maybe some other member of @openssl/committers can chime in and help resolve the ‘deadlock’ in this discussion?

I'm not sure why we stopped discussing this.
So I understand that what you is to have the same effect as only
writing the session data on the first SSL_write() or
SSL_shutdown(). I don't think doing it at SSL_shutdown() was ever
mentioned, and doing it at the first SSL_write() or SSL_shutdown()
might actually be better that what we do now.

In summary, the problem is that the client might have send all
it's data, send a close_notify, and closed the connection before
the server tries send the session ticket. The server tries to
write the tickets before it reads the data from the client. When
the server writes the tickets after the client closed the
connection, the TCP connection will be reset, and the server can't
actually read what the client send.
The solution is not to write any ticket as part of the handshake,
but at a later time when we would send something anyway. We might
still get reset, but we would have received all the data. It also
doesn't change anything for a client that does a bidirectional
shutdown.

This can also come up in non-closing scenarios. Consider an HTTP/1.1 client. HTTP/1.1 never reads and writes at the same time. Typically, the client sends the entire HTTP request and only tries to read once it's finished with that. This means OpenSSL will deadlock if neither the ticket (which scales with client cert size) nor the HTTP request (which is not under OpenSSL's control) fit in transport receive buffers. BoringSSL defers the ticket write to the first SSL_write to avoid this problem.

The solution is not to write any ticket as part of the handshake, but at a later time when we would send something anyway. We might still get reset, but we would have received all the data.

If you get reset, you won't (reliably) receive the data. TCP treats resets as error conditions and will drop buffers, truncate sequence numbers, stop trying to retransmit, etc., when it happens. The first half of the bug report discusses this. We ran into a similar situation with trying to make client certificate alerts (which are now post-handshake from the client's perspective in TLS 1.3) reliably delivered, though I think that one may be unfixable.
However, deferring the ticket write to the first SSL_write should avoid the reset. It means TLS never (outside of some error conditions like client cert errors) triggers transport writes without application writes. That means TCP resets from an unexpected ticket if and only if it would have done so from unexpected application data anyway.
This does mean the reset is avoided by saying a server which never writes never sends tickets, but I don't see another option under this I/O pattern. Also protocols without server writes likely have no client reads, so the client won't pick up the ticket anyway.
In general, the TLS library should avoid writing "out of turn" or it will confuse application-level handling of transport backpressure and resets. (See also mentions of surprising I/O patterns in https://github.com/openssl/openssl/issues/8677. I think OpenSSL only defers if the write buffer is busy. This may still cause TCP resets in this I/O pattern if the peer's TLS library sends KeyUpdates transparently. BoringSSL unconditionally defers to SSL_write.)

The solution is not to write any ticket as part of the handshake, but at a later time when we would send something anyway. We might still get reset, but we would have received all the data.

If you get reset, you won't (reliably) receive the data [...]

Oh, sorry, I just realized by "send something anyway", you might have meant the close_notify. I think that is fine as long as the server send close_notify until after it's done reading whatever it would have tried to read. Then the reset won't drop anything because it's all been read. But this is also super weird. IMO TCP is being overaggressive in classifying out-of-turn writes as an error condition. Or perhaps it needs a notion of droppable "FIN data". But it's a bit late to fix that. :-/
On the topic of close_notify, some comments to the earlier discussion:

Or, if you set https_compatible=True, then it neither sends nor expects close_notify, since this is required for interoperability with most real HTTPS servers.

It's true that expecting close_notify is, sadly, required for interop with the HTTPS ecosystem because a lot of HTTPS servers fail to send close_notify. (As recently as a few years ago, I accidentally made Chrome enforce close_notify during a refactor. This broke loads of sites and had to get quickly reverted.)
However, there is no interop requirement to not send close_notify. It is good to send it (or, for HTTP, exclusively use one of the self-delimited options, see below) so you don't contribute to the interop problem.

I don't know enough about http(s), but is it always clear from the protocol when it's finished sending the data? Is a truncation attack possible?

HTTP/2 frames are self-delimited and streams have an explicit close, so there isn't a truncation problem there. Transport EOF should not be considered a clean stream closure, whether or not it's authenticated.
HTTP/1.1 is a complex mess of a text protocol and has potential truncation problems. First, the header block is delimited by two newlines. It is important that an HTTPS parser either enforce close_notify (which alas is not interoperable) or enforce the two newlines. Otherwise it is vulnerable to a truncation attack.
Second, there are three ways to delimit message bodies:

If you get a Content-Length header, measure up to Content-Length bytes.
If you get a Transfer-Encoding: chunked header, each chunk is self-delimited and there is an explicit end chunk.
If you get neither, read until transport EOF.

(1) and (2) are self-delimited. You don't need close_notify, provided you do not treat transport EOF as a substitute for the existing clean termination signal.
(3) has truncation problems and needs close_notify. Sadly, that is also exactly the case where clients hit interop issues if they enforce close_notify. :-( Thus servers should never use (3). Use (1) or (2) or HTTP/2. This is necessary for connection reuse too.
Text protocols are the worst.

@mattcaswell: Any update on this?

@mattcaswell: Any update on this?

It's not made it to the top of my priority list yet (it's battling for bandwidth with a long list of 3.0 tasks). But thanks for the reminder. I will try to find some time to look at it, but not quite sure when that will be.

This seems like it will be operating in a similar space as #10069 that I have an as-yet-unpublished WIP for

Having just pushed #11416 for review I can say that doing this on top of that would be pretty elegant and result in some code reduction if non-optional.  Alas, I'm inclined to agree with the concerns that this would break things and is not appropriate for 1.1.1, and maybe not even for 3.0

For what it's worth, this issue is not just hypothetical.  The scenario described by https://github.com/openssl/openssl/issues/7948#issuecomment-562946766 is exactly what happens during an FTPS upload when the data connection uses TLSv1.3, where the server only ever reads, and the client only ever writes, resulting in an ECONNRESET on the server end.  FTP has no other framing for indicating the end of the data stream, other than closing the data transfer TCP connection, for uploads.
The workaround I used was to accept TLSv1.3 session tickets for the data connection TLS handshake, but not to renew them, so that the server does not do any TCP writes on the data connection for uploads using TLSv1.3.

I don't think anyone was claiming it was purely theoretical, just that the people with the expertise to write the patch have a lot of conflicting demands on their time.
FWIW, I think that if you pull in #11416 you can get the desired effect at the application layer by configuring the SSL_CTX to send zero tickets and using SSL_new_session_ticket() before the server's first write.

How can the client know that the upload was actually completly
received by the server, if the client doesn't wait for the server to
stut down the connection?

@kroeckx it can't; and it's the same case for plain FTP uploads as well.  FTP defers that verification to the application.  Many FTP clients follow an upload by a directory list request (or server-side checksum) of the just-uploaded file.


Talk title should be stripped before insertion in DB

I am getting some errors processing the talks for the booklet.
I think the best way to solve this is to process the title before insertion:

strip
have double space "  " replaced by " "
if the title starts and ends with quotes: remove them

On 03.06.2015 08:40, Alexandre M. S. wrote:

I am getting some errors processing the talks for the booklet.
I think the best way to solve this is to process the title before insertion:

strip
have double space "  " replaced by " "
if the title starts and ends with quotes: remove them

Right. That's what I'm already doing in the script to generate the
talk list:
https://github.com/EuroPython/epcon/blob/develop/p3/management/commands/accepted_talks.py#L48
(except for the folding of additional spaces)
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Source  (#1, Jun 03 2015)

Python Projects, Coaching and Consulting ...  http://www.egenix.com/
mxODBC Plone/Zope Database Adapter ...       http://zope.egenix.com/
mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::
eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/

Nice!
I'm using the talk list on the website scrapped on text files.
Maybe you find useful for showing the abstracts:
http://ipython.org/ipython-doc/2/api/generated/IPython.nbconvert.filters.markdown.html

This issue is still open, is there a reason ?


typeahead use $interval instead $timeout for protractor

Based on https://github.com/angular-ui/bootstrap/issues/2454

Closing as per https://github.com/angular-ui/bootstrap/pull/2798#issuecomment-148554704


Picture proword missing from picture call

Picture call should be prefaced by "Picture" ... blablabla" rather then just immediately starting with the groups.
i.e. should be "BOXCAR, PICTURE, 2 groups...."
2:27AM INF extended next PICTURE broadcast time callsign="viper 1 4" deadline=2024-11-08T02:37:20Z type=*brevity.PictureRequest
2:27AM INF composing brevity call params={"Count":2,"Groups":[{},{}]} type=brevity.PictureResponse
2:27AM INF composed brevity call params={"Count":2,"Groups":[{},{}]} speech="BOXCAR, 2 groups. Group bullseye 0 5 1, 64, 1000, track west, hostile, 2 contacts, Frogfoot. Group bullseye 0 4 4, 134, 31000, track south, hostile, Mainstay." subtitle="BOXCAR, 2 groups. Group bullseye 051/64, 1000, track west, hostile, 2 contacts, Frogfoot. Group bullseye 044/134, 31000, track south, hostile, Mainstay." type=brevity.PictureResponse
2:27AM INF synthesizing speech text="BOXCAR, 2 groups. Group bullseye 0 5 1, 64, 1000, track west, hostile, 2 contacts, Frogfoot. Group bullseye 0 4 4, 134, 31000, track south, hostile, Mainstay."
2:27AM INF synthesized audio clockTime=1.3430837s
2:27AM INF transmitting audio traceID=9KUVZGeTyAefmCeKwQHfDn

Hi, please reference the examples in MULTI-SERVICE
TACTICS, TECHNIQUES, AND PROCEDURES FOR Air Control Communication (ATP 3-52.4), Chapter IV, Sections 9 through 17. I counted at least 19 examples of PICTURE calls which do not include the "PICTURE" prefix and zero which did.


[ALLUXIO-2611]Test2611

https://alluxio.atlassian.net/browse/ALLUXIO-2611
replace  throw Throwables.propagate(e); with   throw new RuntimeException(e);

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/14389/
Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/14392/
Test FAILed.

Build finished. Test FAILed.

also, please resolve merge conflicts


EAA pod will run on every node or one in complete cluster?

Hi ,
Please help with one information, EAA pod will run on each node (like a daemon set) or it  will be running as one instance for one K8S Cluster ?
Thanks  and Regards
Abhijit

running as one instance for one K8S Cluster。
daemonset default one instance。


docs: fix appsflyerExternalId in appsflyer streaming destination docs

What do these changes do?
This PR fixes misspell in appsFlyerExternalId to appsflyerExternalId in the Appsflyer streaming destination docs.
See https://github.com/rudderlabs/rudder-sdk-react-native/issues/381#issuecomment-2399431279 for confirmation that appsflyerExternalId is the correct format.
Type of documentation

[ ] New documentation
[x] Documentation update
[ ] Hotfix

Thanks @petetnt! This has been fixed in RudderStack docs. If there is nothing else, we can close this PR.


Support validation of fully qualified task names (Especially for tasks installed as extensions).

To allow for better discrimination of the source of tasks, it can be handy to include the fully qualified list of task names, eg:
- task: qetza.replacetokens.replacetokens-task.replacetokens@3
  displayName: Replace Tokens
  inputs:
    rootDirectory: $(TerraformDirectory)
    targetFiles: |
      variables.tf
    tokenPrefix: "#{"
    tokenSuffix: "}"
    actionOnMissing: fail
    escapeType: json
    enableTransforms: true

Unfortunately this currently fails to validate:

Part of this is probably potentially due to the autogenerated (assumedly - I don't control it for my org) yaml schema, which looks like this:
// most properties elided
{
// ...
                        {
                            "description": "Replace tokens in files",
                            "doNotSuggest": false,
                            "ignoreCase": "value",
                            "enum": [
                                "replacetokens@3"
                            ]
                        },
// .....
                    "properties": {
                        "task": {
                            "description": "Replace Tokens\n\nReplace tokens in files",
                            "ignoreCase": "value",
                            "pattern": "^replacetokens@5$"
                        },
                        // ......
                    },
// .....

So it has no concept that there could be a fully qualified name....

Yes, I agree this is much needed :). Unfortunately not something this extension can handle - that knowledge comes from ADO itself, and you're correct in assuming that it does all the work necessary to generate that schema file.

I got this working 😃
The Azure DevOps schema would need to change:

Update Task Property List

{
    "description": "Replace tokens in text based files",
    "doNotSuggest": false,
    "ignoreCase": "value",
    "enum": [
        "qetza.replacetokens.replacetokens-task.replacetokens@6",
        "replacetokens@6"
    ]
},

Update pattern matching

"task": {
    "description": "Replace Tokens\n\nReplace tokens in text based files",
    "ignoreCase": "value",
    "pattern": "^(qetza.replacetokens.replacetokens-task.)?replacetokens@6$"
},

I downloaded my org schema file, saved it locally, updated it and pointed the VSCode extension to my local file:
"azure-pipelines.customSchemaFile": "C:\\your-custom-schema.json"
The whole process (download + edit) could be scripted and ran regulary.
But of course it would be nice if the Azure DevOps team could expand their schema.


[Request for External] Page 490 "Acc. Schedule Overview" - protected variables

Hello, we need the following currently global variable/-s to be set as protected. Can you please check this? Thank you!
protected var
   TempFinancialReport: Record "Financial Report" temporary;

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.
Build ID: 10652.


Can't run

In the the Simulation Server:
(aiworld) PS E:\AIworld\generative_agents\reverie\backend_server> python reverie.py
Enter the name of the forked simulation: base_the_ville_isabella_maria_klaus
Enter the name of the new simulation: test
Note: The agents in this simulation package are computational
constructs powered by generative agents architecture and LLM. We
clarify that these agents lack human-like agency, consciousness,
and independent decision-making.
Enter option: run 100
Error.
Enter option:
And in the Environment Server, only reply:
"POST /update_environment/ HTTP/1.1" 200 14

same problem

I also meet the same problem.
`python reverie.py
Enter the name of the forked simulation: base_the_ville_isabella_maria_klaus
Enter the name of the new simulation: test-simulation
Note: The agents in this simulation package are computational
constructs powered by generative agents architecture and LLM. We
clarify that these agents lack human-like agency, consciousness,
and independent decision-making.
Enter option: run 100
Error.
Enter option: run 1000
Error.
Enter option: `

in reverie.py , import traceback , then add traceback.print_exc() before print ("Error."),save file and restart python reverie.py
you will see more detail info about your error
my is  "openai.error.RateLimitError: You exceeded your current quota, please check your plan and billing details."

in reverie.py , import traceback , then add traceback.print_exc() before print ("Error."),save file and restart python reverie.py
you will see more detail info about your error my is "openai.error.RateLimitError: You exceeded your current quota, please check your plan and billing details."

I have the same error. Is this need GPT-4 API? How can I use GPT-3.5 API to replace?

在 reverie.py 中，导入 traceback ，然后在 print ("Error.") 之前添加traceback.print_exc()，保存文件并启动重新 python reverie.py
您将看到有关您的错误的更多详细信息，我的错误是“openai.error.RateLimitError：您超出了当前消耗，请检查您的计划和账单详细信息。”

我也有同样的错误。这需要GPT-4 API吗？如何使用GPT-3.5 API进行替换？

I have replaced the old API with a new one (GPT3.5) and it seems to be working properly.
But the new problem is that there is only output in Termin , I can't see the Simulation on my browser

in reverie.py , import traceback , then add traceback.print_exc() before print ("Error."),save file and restart python reverie.py
you will see more detail info about your error my is "openai.error.RateLimitError: You exceeded your current quota, please check your plan and billing details."

I added the code and got the error.
openai.error.RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-1SwtoMrxxx on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method.
Can the call frequency be modified?

in reverie.py , import traceback , then add traceback.print_exc() before print ("Error."),save file and restart python reverie.py
you will see more detail info about your error my is "openai.error.RateLimitError: You exceeded your current quota, please check your plan and billing details."

I added the code and got the error. openai.error.RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-1SwtoMrxxx on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method. I input run 1, but it still throws this error. Can the call frequency be modified?

You can through insert a breakpoint in "generative_agents/reverie/backend_server/persona/prompt_template/gpt_structure.py, line 278" to control the call frequency. It's a simple way.

in reverie.py , import traceback , then add traceback.print_exc() before print ("Error."),save file and restart python reverie.py
you will see more detail info about your error my is "openai.error.RateLimitError: You exceeded your current quota, please check your plan and billing details."

I added the code and got the error. openai.error.RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-1SwtoMrxxx on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method. I input run 1, but it still throws this error. Can the call frequency be modified?

run 1, it still sent many message to chatgpt. For example, every character will sent a message to chatgpt.

maybe the issue from running out of the quota😂

I'm the same. How can I solve it

maybe the issue from running out of the quota😂

I'm the same. How can I solve it

maybe the issue from running out of the quota😂

I think your account balance has been used up, and you need to subscribe to OpenAI or replace the API key.

in reverie.py , import traceback , then add traceback.print_exc() before print ("Error."),save file and restart python reverie.py
you will see more detail info about your error my is "openai.error.RateLimitError: You exceeded your current quota, please check your plan and billing details."

I added the code and got the error. openai.error.RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-1SwtoMrxxx on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method. I input run 1, but it still throws this error. Can the call frequency be modified?

same error


HBASE-25627: [Backport]HBase replication should have a metric to represent if the source is stuck getting initialized

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 11s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ branch-2.3 Compile Tests _

+0 :ok:
mvndep
0m 18s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
3m 33s
branch-2.3 passed

+1 :green_heart:
compile
3m 57s
branch-2.3 passed

+1 :green_heart:
checkstyle
1m 33s
branch-2.3 passed

+1 :green_heart:
spotbugs
2m 55s
branch-2.3 passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 14s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
3m 11s
the patch passed

+1 :green_heart:
compile
3m 56s
the patch passed

+1 :green_heart:
javac
3m 56s
the patch passed

+1 :green_heart:
checkstyle
0m 13s
The patch passed checkstyle in hbase-hadoop-compat

+1 :green_heart:
checkstyle
0m 12s
The patch passed checkstyle in hbase-hadoop2-compat

+1 :green_heart:
checkstyle
1m  5s
hbase-server: The patch generated 0 new + 0 unchanged - 11 fixed = 0 total (was 11)

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
hadoopcheck
16m 52s
Patch does not cause any errors with Hadoop 2.10.0 or 3.1.2 3.2.1.

+1 :green_heart:
spotbugs
3m 25s
the patch passed

_ Other Tests _

+1 :green_heart:
asflicense
0m 38s
The patch does not generate ASF License warnings.

51m 24s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3076/1/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/3076

JIRA Issue
HBASE-25627

Optional Tests
dupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile

uname
Linux 7da4b6845bc7 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
branch-2.3 / b6d4c9e114

Default Java
AdoptOpenJDK-1.8.0_282-b08

Max. process+thread count
96 (vs. ulimit of 12500)

modules
C: hbase-hadoop-compat hbase-hadoop2-compat hbase-server U: .

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3076/1/console

versions
git=2.17.1 maven=3.6.3 spotbugs=4.2.2

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 35s
Docker mode activated.

-0 :warning:
yetus
0m  7s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ branch-2.3 Compile Tests _

+0 :ok:
mvndep
0m 20s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
4m 25s
branch-2.3 passed

+1 :green_heart:
compile
1m 49s
branch-2.3 passed

+1 :green_heart:
shadedjars
5m 56s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m 20s
branch-2.3 passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 21s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
3m 53s
the patch passed

+1 :green_heart:
compile
1m 42s
the patch passed

+1 :green_heart:
javac
1m 42s
the patch passed

+1 :green_heart:
shadedjars
5m 45s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m 18s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
0m 28s
hbase-hadoop-compat in the patch passed.

+1 :green_heart:
unit
0m 30s
hbase-hadoop2-compat in the patch passed.

+1 :green_heart:
unit
128m 31s
hbase-server in the patch passed.

159m 47s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3076/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/3076

JIRA Issue
HBASE-25627

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux 09b777af3999 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
branch-2.3 / b6d4c9e114

Default Java
AdoptOpenJDK-11.0.10+9

Test Results
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3076/1/testReport/

Max. process+thread count
4021 (vs. ulimit of 12500)

modules
C: hbase-hadoop-compat hbase-hadoop2-compat hbase-server U: .

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3076/1/console

versions
git=2.17.1 maven=3.6.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 16s
Docker mode activated.

-0 :warning:
yetus
0m  6s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ branch-2.3 Compile Tests _

+0 :ok:
mvndep
0m 14s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
3m 58s
branch-2.3 passed

+1 :green_heart:
compile
1m 29s
branch-2.3 passed

+1 :green_heart:
shadedjars
5m 29s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m  7s
branch-2.3 passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 14s
Maven dependency ordering for patch

+1 :green_heart:
mvninstall
3m 36s
the patch passed

+1 :green_heart:
compile
1m 28s
the patch passed

+1 :green_heart:
javac
1m 28s
the patch passed

+1 :green_heart:
shadedjars
5m 28s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
1m  2s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
0m 25s
hbase-hadoop-compat in the patch passed.

+1 :green_heart:
unit
0m 27s
hbase-hadoop2-compat in the patch passed.

-1 :x:
unit
204m 12s
hbase-server in the patch failed.

232m 46s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3076/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/3076

JIRA Issue
HBASE-25627

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux 4cd37a33b0cd 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
branch-2.3 / b6d4c9e114

Default Java
AdoptOpenJDK-1.8.0_282-b08

unit
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3076/1/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-server.txt

Test Results
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3076/1/testReport/

Max. process+thread count
2899 (vs. ulimit of 12500)

modules
C: hbase-hadoop-compat hbase-hadoop2-compat hbase-server U: .

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3076/1/console

versions
git=2.17.1 maven=3.6.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

FYI @bharathv
failing test is unrelated

Closing this PR, it looks like the branches diverged quite a bit and this is not critical fix, so perhaps not worth the effort.


[REQUEST] A7s 120fps HDMI out

Would be cool to be able to record in A7s 100fps or 120fps externally. At least 720p :)

please for a7s2 too


Add low level APIs to help with tying together arrays, combing data and buffering data. This could be built on Span, or something new

Helpers for combining or adding to arrays with or without copying the original array would probably be extremely helpful. Span does not current support increasing its length, only splicing. This could be done through pseudo pointers, or child spans that are switched between depending on the index

API should be similar to IList, with options to add to the internal arrays.
Consider a single jagged array for managing data. Each additional array should be double the size of the previous array to reduce the number of times that growth is needed.

Add some way to set capacity from the start if the size of the collection is known. This should set the size of the first entry in the jagged arrays

byte[] myArr = new byte[5];
ArraySegments as = new ArraySegments(myArr);
// Or
ArraySegments as = new ArraySegments(myArr, 3, 1);
// Or
ReadOnlyArraySegments as = new ArraySegments(myArr, 3, 1);

// Points to original array
byte first = as[0];
as[0] = first;

Expanding ArraySegments is confusing - how to understand that ArraySegments[0] and ArraySegments[1] will copy to completely different segments?
A common usage scenario might also be building up ArraySegments from a buffer, not realising that overwriting position [0] in my buffer before appending to ArraySegments again has overwritten multiple sections within ArraySegments

A better name might be JoinedArraySegments

This could easily use a kind of span-like class until Span<T> is brought to .NET Standard and supported by enough platforms

Abandoned spec in favor of new extension method in Collections,ArrayExtensions for joinng arrays together into a single array


Tips for generating good results.

@andabi can you tell me what you did exactly using the tools? A normalization is required for the train and target audio? I used 1 script to see the min and max db, but I'm getting the noise that causes sounds like robots in movies. How do you know the exact values for n_mels and n_mfcc, for spanish I'm using a default TTS of windows, the net 1 is reaching to 95% acc.
Loss for the net2

Needs more training?
results.zip

Hi @carlfm01 , I just hear your spanish case. If it's trained with dataset which has a phoneme label?

Hi @sailor88128  I built it by my own using a Windows existing TTS in C#, Windows speech api has an event that raises when a phoeneme is reached by the TTS, so I wrote a code that writes the phn files using the windows api. I any case if you want label the data or generate it from an existing TTS  remember to multiply the current phoneme time per the sampling date (16000) in my case. Remember this aproach only will work with the voice of the TTS used for the dataset.
I can share the code and the dataset if you want.

Sounds interesting! @carlfm01 , I would like to learn and try your method, pls share with we. Email<EMAIL_ADDRESS>Thanks a lot~

Sent, let me know any questions. I'll make a repo later.

I have a question about the generated data, @carlfm01 .
For the Net1 training, TIMIT dataset used. TIMIT contains 630 speakers' utterances and corresponding phones that speaks similar sentences. The TTS tool only contains the voice of 'Helena' or less than ten. Are these enough to train the phoneme classifier?

Hi @sailor88128, If you want many to one then the TTS is not an option, to make a many to one requires a TIMIT kind dataset, even more depending on the languague. The TTS aproach only allows to convert that TTS voice into the target style voice. I was really curious to see if the style transfer can convert the horrible sound of the TTS into more natural voice, and to be honest the result is interesting. Now im trying to minimize the roboting sound, but in features of the languague is really amazing.

@carlfm01 I agree the results sound like a person or robot talking into a fan.
https://www.youtube.com/watch?v=vNBajseh6sE
Id love to figure out how to de-noise these results into clearer audio.

Before having hard time with Linux and cuda versions here's other result, I increased the win lenght and hop, for Spanish looks like the vocals are litle longer, algo changed the lr from lr = 0.0003 to lr = 0.0001
mresults.zip

Sounds much better, great! @carlfm01

Thanks @sailor88128, still not time to create your dataset, I got access to a P100 with Linux on Azure but is much slower than Windows with a k80, same version and I don't know where is the bottleneck. There's considerable amount of operations on the CPU, I'm trying to allocate operations manually on the GPU manually then if it crash look for options or newer tensorflow operations that are allowed in the GPU.
@sailor88128 can you add log_device_placement=True to tf.ConfigProto in train1  example:
session_conf = tf.ConfigProto(
        log_device_placement=True,
        gpu_options=tf.GPUOptions(
            allow_growth=False,
        ), allow_soft_placement=False)#allow_soft_placement=False

to see where the ops alocated, also your version of py and tf, please.
For python 3.6 and tf 1.11.0
op_alloc.txt

Hi @carlfm01 , python 2.7 and tf 1.8.0. The log file is as below.
Thanks for your help of data. Maybe I can try to generate Chinese data for Net1 myself firstly, could you please provide a more detailed introduction?
op_alloc-py2.txt

@carlfm01 Did you ever figure out how to move more operations to the P100 GPU instead of the CPU on Linux?

Hi @robd003, no. I tried but failed, the problem is that the net 2 is using types that only the CPU can execute.

Hi @carlfm01  now, I still have two questions  to disturb you.
One is that I could not find the data like TIMIT with my lang for the net1，How should I do to use this code for CV ?
The other is that when I observed the results, I found that the output voice was all two seconds, and some words seemed to be truncated, not completely finished. For example, the source speaker has 3 seconds of sound, and the result only contains 2 seconds, which means that the conversion is incomplete. How should I modify the code in order that the speech can be completely converted?
Look forward to your reply！

Hi @Huishou yes, is hard to find TIMIT like dataset for other langs, I don't know what do you mean with CV, common voice from Mozilla?
To increase the time you have to change the value of duration here https://github.com/carlfm01/deep-voice-conversion/blob/5de50b955c8dd37d0948c0cc1bc965fa515aed88/params.py#L25

Hi @carlfm01, I am very grateful for your help. I mistakenly wrote VC (voice convertion) as CV. Now, what should I do to use this code for other langs convertion ? Just one to one is ok ! I remember that you convert Microsoft helena TTS into more natural voice. I pray for you to give me some help and advice, thank you！

@Huishou yes, using the Helena TTS I've created a dataset with the phonemes and speech aligned. If you want to do the same use the windows tts for your lang and see if it generates the phonemes with its time. Then use the target voice that you like to train net2. take a look to librivox

@carlfm01，thank for your reply，but  I could not to understand how to use the  TTS system to create a dataset with the phonemes and speech aligned. Can you share the code to me？ I am a fresh in speech synthesis field！ Thanks much!

Hi @Huishou, the code is C#, it is ok? Where I can send it to you?

hi，@carlfm01, thank you very much , you can send it to my email<EMAIL_ADDRESS>thank you!

hi，@carlfm01，my Email is
<EMAIL_ADDRESS>. I think my mailbox should not have problem. But I haven't received your email yet. I don’t know what 's happening now.

Hi, just sent the email, sorry for the delay :p


Not able to see linked fields

I have linked a fileClass to a note, but it doesn't reveal the linked fields.
This is the video: https://www.loom.com/share/e51cd93ccaf94b6f8b15495781b41910

Hi thanks for the video
since you haven't defined any type and options for your Person fileClass' fields, it's considered empty and therefore fall-back to the settings-defined fields.
To add a proper field definition in a fileClass you can:

right click on the fileClass tabheader > manage fields > add new field
from the fileClass menu, command palette > Metadata Menu: insert a new fileClass attribute.
or add a fileClass field from the note fields "+" button that you have clicked at the end of your demo and fill the settings

Thanks. I had assumed from this image that syntax wasn't necessary for text input fields:
music.md
genre
difficulty
masterization
artist
tone
tab available
But I've added :: {"type":"Input","options":{}} to each and it works fine now.
Much appreciated.

oh i see,
this was a legacy way of hand-writing fileclasses
i'll change the documentation accordingly


OAI-PMH harvester

Hello, this is the PR for OAI-PMH
This PR has:

Code to download XML record using OAI-PMH protocol
Support to download the records in multiple XML schemas
Support for checkpoints, even incremental
Support for parallel execution, to download multiple repositories at the same time.
Support Rate-Limit to avoid DDoS and 429 errors, this is supported asynchronous in the parallel execution, which means that every repo can have a different rate limit
Allows saving the XML without losing information thanks to the package xmltodict, allowing at the same time, to save the records in MongoDB
It has a command line tool for the execution
Support python packaging using setup-tools

Coming soon:

Github actions for CI/CD
Documentation

this package is almost ready, I will merge it and move it to colav organization.


fix(eslint-plugin): false positive for factory functions

fixes https://github.com/TanStack/query/issues/4476
Honestly, I'm unsure how I feel about this fix. Since we need to support @babel/eslint-parser, we need to find a different way to get a function's "return type".
I'm not sure how I feel about my approach - The linter will check all possible return nodes, and if at least one node is not an ObjectExpression, the linter will report that node.
I'm open to suggestions.

Codecov Report
Base: 96.36% // Head: 92.43% // Decreases project coverage by -3.92% :warning:

Coverage data is based on head (9ede1ca) compared to base (eab6e2c).
 Consider uploading reports for the commit 83c2d40 to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4497      +/-   ##
==========================================
- Coverage   96.36%   92.43%   -3.93%
==========================================
  Files          45       88      +43
  Lines        2281     3649    +1368
  Branches      640      961     +321
==========================================
+ Hits         2198     3373    +1175
- Misses         80      260     +180
- Partials        3       16      +13

Impacted Files
Coverage Δ

src/core/infiniteQueryObserver.ts

src/core/utils.ts

src/devtools/useLocalStorage.ts

src/react/QueryErrorResetBoundary.tsx

src/devtools/useMediaQuery.ts

src/core/infiniteQueryBehavior.ts

src/react/useQuery.ts

src/core/queriesObserver.ts

src/core/mutationObserver.ts

src/core/queryCache.ts

... and 123 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

I think it's good 👍 . The only other thing would be to depend on typescript types, not sure if you think this would be generally better. But if, we'll do this for v5 only I guess...

It would require much less code and probably a more precise check, but it will hurt the plugin's adoption, so I guess we'll stick with the current implementation for now.


vimPlugins.mini-nvim: init at 2021-12-28

Since newer version of mini.nvim was merged into master, I think we can close this:
https://github.com/NixOS/nixpkgs/pull/154246


Ported visual_tracking controller to python 3

Partial fix for #520.

👍

:eyes:

I was about to do the same comment :-)

OK, I will try to fix that.


add tssc concept to build

decide on the tooling. Not sure what goes here, but there will be a need for an evidence/attestation wharehouse and a policy enforcer. I'd like to use sigstore and kiverno if they fit the bill.
deploy the tooling. using gitops
enhance the pipeline to use the tooling
implement one policy to prove the concept, possibly an image signature verification.

We can start here -> https://github.com/ploigos-automated-governance/1-demo/blob/main/README.md

I can walk through how I think the implementation should look like.

as a first step, can we discuss what tools we need and how they are going to interact? Let's say for now that we want to sign images. and allow only signed images to be ran.


ensure counter is increased by at least once to avoid getting stuck

Fixed here with complex gate:
https://github.com/arabold/aws-to-slack/commit/3078547fb24c055e5da299a9d9fcc1b8f35ada39
Thank you for finding, @lehneres !