Correct Elasticsearch Errors not getting propagated via NEST

NEST/Elasticsearch.Net version: 2.4.4
Elasticsearch version: 2.4.1
Description of the problem including expected versus actual behavior: We send bulk requests to ES that are quite big in size. In that case, sometimes ES throws OutOfMemory error or ContentLength exceeded errors that are treated differently inside our error handling code. But surprisingly, NEST only notifies about "Connection Closed" error by not propagating the actual error thrown by ES. Please refer to below error stack trace for the actual error reported by NEST and ES.
Our requirement is to correctly get the actual error thrown by ES, so that we could error handle it differently. But at this stage it's not possible.
Steps to reproduce:

Java heap space - OutOfMemoryError Create a batch of documents having large files and send the bulk to elasticsearch
HTTP content length exceeded : Send a 200 MB file to Elasticsearch to exceed it's default 100 MB content length.

NEST ERROR

FailureReason: BadResponse while attempting POST http://<IP_ADDRESS>:9200/_bulk?refresh=false
Audit trail of this API call:

[1] BadResponse: Node: http://<IP_ADDRESS>:9200/ Took: 00:00:03.5589871
[2] SniffOnFail: Took: 00:00:00.0039999
[3] AllNodesDead: Took: 00:00:00
[4] Resurrection: Node: http://<IP_ADDRESS>:9200/ Took: 00:00:00
[5] SniffSuccess: Node: http://<IP_ADDRESS>:9200/ Took: 00:00:00.0035002

OriginalException: System.Net.WebException: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.
at System.Net.HttpWebRequest.GetResponse()
at Elasticsearch.Net.HttpConnection.Request[TReturn](RequestData requestData) in C:\code\elasticsearch-net\src\Elasticsearch.Net\Connection\HttpConnection.cs:line 141
Request:
<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>
Response:
<Response stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>
Inner Exception: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.
System.Net.WebException: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.
at System.Net.HttpWebRequest.GetResponse()
at Elasticsearch.Net.HttpConnection.Request[TReturn](RequestData requestData) in C:\code\elasticsearch-net\src\Elasticsearch.Net\Connection\HttpConnection.cs:line 141
Exception:
Elasticsearch.Net.ElasticsearchClientException: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server. ---> System.Net.WebException: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.
at System.Net.HttpWebRequest.GetResponse()
at Elasticsearch.Net.HttpConnection.Request[TReturn](RequestData requestData) in C:\code\elasticsearch-net\src\Elasticsearch.Net\Connection\HttpConnection.cs:line 141
--- End of inner exception stack trace ---
at Elasticsearch.Net.RequestPipeline.BadResponse[TReturn](ElasticsearchResponse1& response, RequestData data, List1 pipelineExceptions) in C:\code\elasticsearch-net\src\Elasticsearch.Net\Transport\Pipeline\RequestPipeline.cs:line 482
at Elasticsearch.Net.Transport1.Request[TReturn](HttpMethod method, String path, PostData1 data, IRequestParameters requestParameters) in C:\code\elasticsearch-net\src\Elasticsearch.Net\Transport\Transport.cs:line 103
at Elasticsearch.Net.ElasticLowLevelClient.Bulk[T](PostData1 body, Func2 requestParameters) in C:\code\elasticsearch-net\src\Elasticsearch.Net\ElasticLowLevelClient.Generated.cs:line 32
at Nest.LowLevelDispatch.BulkDispatch[T](IRequest1 p, PostData1 body) in C:\code\elasticsearch-net\src\Nest_Generated_LowLevelDispatch.generated.cs:line 0
at Nest.ElasticClient.Nest.IHighLevelToLowLevelDispatcher.Dispatch[TRequest,TQueryString,TResponse](TRequest request, Func3 responseGenerator, Func3 dispatch) in C:\code\elasticsearch-net\src\Nest\ElasticClient.cs:line 56
at Fti.Infinity.Data.ElasticRepository1.BulkIndex(String toIndex, IEnumerable1 model, Boolean refresh, Func4 bulkDescriptorDelegate, Func5 bulkDescriptorDelegateForItemsWithErrors, Action`2 onItemsFailed) in C:\TFS_DATA\Infinity\Dev\Main\source\Fti.Infinity\Data\ElasticRepository.cs:line 216

ES Error for Java heap space - OutOfMemoryError

RemoteTransportException[[Juggernaut][<IP_ADDRESS>:9300][indices:data/read/search[phase/fetch/id]]]; nested: ElasticsearchException[Java heap space]; nested: OutOfMemoryError[Java heap space];
Caused by: ElasticsearchException[Java heap space]; nested: OutOfMemoryError[Java heap space];
at org.elasticsearch.ExceptionsHelper.convertToRuntime(ExceptionsHelper.java:50)
at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:604)
at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:408)
at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:405)
at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.readField(CompressingStoredFieldsReader.java:214)
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:595)
at org.apache.lucene.index.CodecReader.document(CodecReader.java:81)
at org.apache.lucene.index.FilterLeafReader.document(FilterLeafReader.java:405)
at org.elasticsearch.search.fetch.FetchPhase.loadStoredFields(FetchPhase.java:413)
at org.elasticsearch.search.fetch.FetchPhase.createSearchHit(FetchPhase.java:213)
at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:178)
at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:592)

ES error for HTTP content length exceeded

[2016-11-16 10:26:29,377][WARN ][http.netty               ] [Hercules] Caught exception while handling client http traffic, closing connection [id: 0x1c1c5ee8, /<IP_ADDRESS>:49614 => /<IP_ADDRESS>:9200]
org.jboss.netty.handler.codec.frame.TooLongFrameException: HTTP content length exceeded 104857600 bytes.
at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:169)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:135)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

NEST can only handle errors that are returned in the REST layer.  In the case of an OOM exception where the node becomes unresponsive, there's not much we can really do.  The exception in this case accurate since we can't communicate with the node.  There's no way for us to propagate JVM errors in the log file up to the client.
As for the content-length exception.  What behavior are you expecting?  The exception is pretty clear that the content-length has been exceeded.

@sumithub closing this as I am not seeing anything actionable on our end, please let us know if you feel there are ways we could propagate this.

@gmarz @Mpdreamz apologies for replying late.
In case of content-length exception we get "System.Net.WebException: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server." error which is not expected.
The correct behavior would be to propagate  "HTTP content length exceeded 104857600 bytes." thrown by elastic-search to the REST layer which is not happening at the moment.

Have you found a solution for this?
I am facing the same issue with regard to content length exceeded exception. NEST throws a ElasticsearchClientException with a message "The underlying connection was closed", but ES console shows the correct error.

@sumithub sorry for the delayed response.  I'm trying to reproduce this but I observing the correct behavior in the client.  Purposely setting http.max_content_length to 1 byte and sending any request (specifically _bulk) bubbles the expected exception:
Elasticsearch.Net.ElasticsearchClientException
The remote server returned an error: (413) Request Entity Too Large.
   at Elasticsearch.Net.RequestPipeline.BadResponse[TReturn](ElasticsearchResponse`1& response, RequestData data, List`1 pipelineExceptions) in C:\code\elasticsearch-net\src\Elasticsearch.Net\Transport\Pipeline\RequestPipeline.cs:line 482
   at Elasticsearch.Net.Transport`1.Request[TReturn](HttpMethod method, String path, PostData`1 data, IRequestParameters requestParameters) in C:\code\elasticsearch-net\src\Elasticsearch.Net\Transport\Transport.cs:line 101
   at Elasticsearch.Net.ElasticLowLevelClient.DoRequest[T](HttpMethod method, String path, PostData`1 data, IRequestParameters requestParameters) in C:\code\elasticsearch-net\src\Elasticsearch.Net\ElasticLowLevelClient.cs:line 61
   at Elasticsearch.Net.ElasticLowLevelClient.Bulk[T](PostData`1 body, Func`2 requestParameters) in C:\code\elasticsearch-net\src\Elasticsearch.Net\ElasticLowLevelClient.Generated.cs:line 33
   at Nest.LowLevelDispatch.BulkDispatch[T](IRequest`1 p, PostData`1 body) in C:\code\elasticsearch-net\src\Nest\_Generated\_LowLevelDispatch.generated.cs:line 27
   at Nest.ElasticClient.Nest.IHighLevelToLowLevelDispatcher.Dispatch[TRequest,TQueryString,TResponse](TRequest request, Func`3 responseGenerator, Func`3 dispatch) in C:\code\elasticsearch-net\src\Nest\ElasticClient.cs:line 57
   at Nest.ElasticClient.Nest.IHighLevelToLowLevelDispatcher.Dispatch[TRequest,TQueryString,TResponse](TRequest request, Func`3 dispatch) in C:\code\elasticsearch-net\src\Nest\ElasticClient.cs:line 47
   at Nest.ElasticClient.Bulk(IBulkRequest request) in C:\code\elasticsearch-net\src\Nest\Document\Multiple\Bulk\ElasticClient-Bulk.cs:line 33
   at Nest.IndexManyExtensions.IndexMany[T](IElasticClient client, IEnumerable`1 objects, IndexName index, TypeName type) in C:\code\elasticsearch-net\src\Nest\Document\Multiple\Bulk\ElasticClient-IndexMany.cs:line 25
   at Tests.Reproduce.ExceedingContentLength.ExceedContentLength() in C:\code\elasticsearch-net\src\Tests\Reproduce\ExceedingContentLength.cs:line 16

System.Net.WebException
The remote server returned an error: (413) Request Entity Too Large.
   at System.Net.HttpWebRequest.GetResponse()
   at Elasticsearch.Net.HttpConnection.Request[TReturn](RequestData requestData) in C:\code\elasticsearch-net\src\Elasticsearch.Net\Connection\HttpConnection.cs:line 143

Can you maybe share your connection settings, or perhaps try and create a reproducible test?

Thanks @gmarz. Did you try this with Nest 5.0? We're in a process to upgrade to NEST 5.0 and try to reproduce the same problem again.


The seller links in search query redirect to products not seller

There aren't 51 products, but the ID of the seller is 51.

True


vm service: Allow injecting objects into scope during eval

It's often useful to do some operation that combines several objects retrieved from the service protocol. The eval rpc should accept a set of names and corresponding object ids to be in scope for the expression.

This feature will be extremely helpful for flutter debugger tools that visually display widget, element, and render trees.


Added my portfolio

I added my portfolio

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).
View this failed invocation of the CLA check for more information.
For the most up to date status, view the checks section at the bottom of the pull request.

I checked all requirements and signed the CLA but it's the still the same. Please look into it


🐛 Bug Report: App URL not working

👟 Reproduction steps
I changed the App URL inside of the configuration to something like "http://example.com" but most of the Copy Link dialogs show localhost:3000 instead of http://example.com
👍 Expected behavior
It should use the App URL
👎 Actual Behavior
It uses the current window URL
📜 Logs
No response

This behavior is expected. Pingvin Share uses the URL you are currently accessing. The App URL is only utilized in scenarios where the application cannot determine the current URL.

That's unfortunate, cause i am using pingvin share inside of a iFrame, and was able to use the app url to go to a entirely different site than the one actually hosting pingvin share

Could that be fixed by using window.top.location.origin instead of window.location.origin?

@Nos4893 Could you share a bit more about why you’re running Pingvin Share in an iframe?

Sorry for my late response.

window.top.location.origin

This isn't possible because of the Same-Origin Policy.
Since this appears to be a very specific use case, I would recommend forking the repository and modifying the logic to use the app URL instead of window.location.origin. Adding configuration options for such niche scenarios would overcrowd the settings page.

Let me know if you need any help to do these changes.


[1.1.1] Added multiply delete profiles

Added the ability to delete multiple profiles

@da-the-dev Do you think I should add a confirmation of deletion?


Actions doesn't work

I have used the action as follows:
    - name: CINotify Email
    # You may pin to the exact commit or the version.
    # uses: cinotify/github-action@d99e3149178b27572a58ace227621c333884e3fb
      uses<EMAIL_ADDRESS>      with:
        # Recipient email address
        to<EMAIL_ADDRESS>        # Email subject
        subject: Your PR has been merged
        # Email body
        body: Enjoy your day off!
   - run: echo ${{steps.serve.outputs.status }}

Can you tell me why am I unable to receive a mail? I checked in spam and trash folders as well.

sorry about that @jayayy - I've been fighting some spam that exceeded all of my limits - preventing further emails from sending. Do you mind trying again when you get a chance? Thank you!

Hey thanks for the prompt response. Yes I did try right away but to no avail. I don't see any mail with the words "Enjoy your day off".
Can you tell me the email id I should expect the mail from?

Any updates for the same? We are eager to add this to our workflow, would love it if this can be enabled.

Just recieved the email, thanks a lot!!! Love this!

I just get

Error: Request failed with status code 500

@antlu sorry about that - Heroku destroyed my db :(((((( I've reprovisioned and am curious if that solves your problem? thanks!

It worked. Thank you.


Fixes #13 and small changes

Fixes issue #13
Re-added the original MIT license
Re-added the original Visual Studio solution file
Added a .gitignore file
Added a section to the readme where I point to the original source code
Changed the .NET framework to 4.5.2 (which is default on Windows 10)
Increased the release number

Thanx for PR!
Why switch to NET 4.5? This app is really simple and can work on win7 by default (NET 3.5).

@kostapc Okay, you are right. .NET 3.5 is better for non-Windows 10 users. Within Visual Studio, the switch back to 3.5 is no issue. Should I switch the framework with a new PR or do you want to switch?

@SommerEngineering i'll swith by myself, without additional PR. For keep history clear.

done. And old .sln, .csproj files removed.


Fix typo in tutorial

Remove duplicated word "move".

Looks good; merging the typo fix. Thanks @GitHubPang.


Suggestion: Add $FlowIgnore to the standardized suppression comments

I'm writing this as a sort of preliminary issue, as the feature that this suggestion is about hasn't been released yet, but I happened to notice commit 99fa5bf37c5630b44d134df0f1e5e0b6cb0ffe16 that will apparently remove the custom suppress comments–feature.
Proposal
I suggest that $FlowIgnore is added to the list of possible suppressors.

Use case
$FlowIgnore is already used a lot in the wild. While refactoring a codebase away from it would probably be easy regex replace job, it's still a lot of changed lines in a large number of repositories.
Also, in my opinion, $FlowIgnore carries a semantic meaning that is different from the other mentioned suppressors ($FlowFixMe, $FlowIssue or $FlowExpectedError). To me, $FlowIgnore implies that the next line does something that Flow can't know to be correct, nor is it supposed to know (so it's not a "FlowIssue"), but the next line is clear enough for the programmer to reason about and verify for correctness, so it's not necessarily something to be fixed either. The most common example of these for me have been down-casts and casts from exact type to another exact type that is a subset of the first, where I would very explicitly write something like
const _dog: Animal = animal
// $FlowIgnore We know this is a Dog because its type was checked earlier.
const dog: Dog = _dog

Edit: A lot of the code shearch hits for FlowIgnore seem to come from the same dependency, maybe this search result is better: grep.app.

I'd happily accept a PR adding this feature.

@dsainati1 Neat, I opened #8397


PKI.NCALayer v1.3

[x] Have you signed the Contributor License Agreement?
[x] Have you checked that there aren't other open pull requests for the same manifest update/change?

[x] Does your manifest conform to the 1.0 schema?

Note: <path> is the name of the directory containing the manifest you're submitting.

Microsoft Reviewers: Open in CodeFlow

Hello @Kesantielu, When processing this package, the URL to the installer returned a 404 error or was determined to be invalid.
Please update the URL and hash at your convenience and resubmit your PR.

I don't know why the autocheck fails.
curl -I https://ncl.pki.gov.kz/images/NCALayer/NCALayerInstall_x86.exe
HTTP/1.1 200 OK
Server: nginx
Date: Fri, 04 Mar 2022 07:48:50 GMT
Content-Type: application/octet-stream
Content-Length: 54828392
Last-Modified: Fri, 21 Jan 2022 06:57:04 GMT
Connection: keep-alive

curl -I https://ncl.pki.gov.kz/images/NCALayer/NCALayerInstall_x64.exe
HTTP/1.1 200 OK
Server: nginx
Date: Fri, 04 Mar 2022 07:48:55 GMT
Content-Type: application/octet-stream
Content-Length: 58833312
Last-Modified: Fri, 21 Jan 2022 06:57:03 GMT
Connection: keep-alive

/AzurePipelines run


flash_range_program will not work if flash is not erased

flash_range_erase must be called before calling flash_range_program. Otherwise flash_range_program will not work.
To test this behavior , just comment line 40 in flash_program example.
Could you please add a comment in the header file rp2_common/hardware_flash/include/hardware/flash.h to indicate this.
This is not obvious and it may save a lot of time to a lot of people.
Thanks

Are you commenting on the behaviour of flash memory itself ?
Or is there some different behaviour / error code sent back by the SDK ?
For regular flash chips, the behaviour is:

when erased, all bits are '1'
when programmed, various bits are set to '0'
reprogamming can not bring any '0' values back to '1', but may bring additional '1' bits down to a '0'.
if the flash memory has an auto-verification method, it may error out before programming the full block (specific chip-defined behaviour).
The only way to return a '0' value back to '1' is with erase (which is usually sector/block/chip in scope)

@dshadoff Thank you for your response.
Yes i was asking about the behavior of flash memory itself.
I was not aware of this behavior and when i discovered it , i thought we may add a comment to the header file to highlight this.
But then this is not necessarily as we can also program the flash without erasing if that sector bits are already ones.  So no need to change the header.
Thank you again.

just stumbled over the same unexpected behavior.
a hint in the header of fash.h would be nice for people not familiar with how flash cells work

Jup. Spent half a day because it's not documented. It would be very nice if it's not just a note in the header but contained in the online documentation.

re-opened to add documentation to header

One option here would be to assert in debug mode like
for (int i = 0; i < write_size; ++i) {
  assert(!(writebuf[i] & ~existingbuf[i]));
}

for all bytes to be programmed. It could easily prevent accidental misuse and serve as a hint to users. I'm not sure how much the overhead y'all tolerate on debug-mode vs. release-mode builds.

Fixed by #1772


Getting first and last hidden outputs for BiLSTM with variable-length sequences

Hi,
I have a zero-masked bidirectional RNN for mini-batched data with variable lengths. I want to extract the last hidden output of the forward pass and the first hidden output of the backward pass as the representation of the sequence.
Since each sequence in the batch has different lengths, I can think of two possible ways to implement this:
i) Prepare two versions of the data, one with zeros padded on the left while the other with zeros padded on the right. The forward pass uses the left-padded data and use SelectTable(-1). Similarly, the backward pass uses the right-padded data and use SelectTable(1).
ii) Alternatively, I can implement a custom MaskZeroSelectTable(-1) to select the last hidden output that is not 0.
I guess this is not an uncommon usage, but I could not find an existing solution. Am I missing anything?
If there is no over-the-shelf implementation, what is the best way to implement this?

Hi @ccsasuke, I'm facing the same issue. Have you found any good solution ?

Hi @aron-bordin, it seems this didn't catch the attention of the devs.. I switched to some other framework for this function. Maybe you can try the workaround i) I mentioned?


Laravel Support

Until the release 1.12

This package works on Laravel 5.4?

Yes @jilvanx,
PHPJasper Can be used regardless of your PHP Framework

Sorry @jilvanx, excludes your message by accident

how do return arrays for jrxml


[GLUTEN-2557][VL][FOLLOWUP] Compatible with CELEBORN-798

What changes were proposed in this pull request?
The previous change in #2557 is not sufficient, making a follow-up to address it.
How was this patch tested?
Pass GA.

@kerwin-zk @ulysses-you


Problems to starting

I'm having trouble running this template first time.
The steps I followed:

I downloaded the files and copied them to an isolated folder
Accessing the folder through the terminal I executed npm install
Then I executed react-native run-ios

And then he returned all these errors to me:
/Users/viniciuslara/Downloads/react-native-template-master/node_modules/@babel/core/lib/config/validation/options.js:113
    throw new ReferenceError(unknownOptErr);
    ^
ReferenceError: Unknown option: .caller. Check out http://babeljs.io/docs/usage/options/ for more information about options.
    at buildUnknownError (/Users/viniciuslara/Downloads/react-native-template-master/node_modules/@babel/core/lib/config/validation/options.js:113:11)
    at Object.keys.forEach.key (/Users/viniciuslara/Downloads/react-native-template-master/node_modules/@babel/core/lib/config/validation/options.js:99:57)
    at Array.forEach (<anonymous>)
    at validate (/Users/viniciuslara/Downloads/react-native-template-master/node_modules/@babel/core/lib/config/validation/options.js:69:21)
    at loadPrivatePartialConfig (/Users/viniciuslara/Downloads/react-native-template-master/node_modules/@babel/core/lib/config/partial.js:38:50)
    at loadFullConfig (/Users/viniciuslara/Downloads/react-native-template-master/node_modules/@babel/core/lib/config/full.js:43:39)
    at loadOptions (/Users/viniciuslara/Downloads/react-native-template-master/node_modules/@babel/core/lib/config/index.js:28:36)
    at OptionManager.init (/Users/viniciuslara/Downloads/react-native-template-master/node_modules/@babel/core/lib/config/index.js:34:12)
    at compile (/Users/viniciuslara/Downloads/react-native-template-master/node_modules/@babel/register/lib/node.js:61:42)
    at compileHook (/Users/viniciuslara/Downloads/react-native-template-master/node_modules/@babel/register/lib/node.js:102:12)
MacBook-Air-de-Vinicius:react-native-template-master viniciuslara$

I then opened the project on xCode and tried to start manually but the following error appears:
'React/RCTEventEmitter.h' file not found
I do not know what's happening, it's not the first time this happens with a project that I try to start.

Hello!
I think we found the issue, do you mind testing this branch fix/build-0.56 to see if it builds propperly?
Thanks!

Tested it myself and it's working OK, it's already merged on master, so use that and if fails, re open the issue.

I do not know why but the mistakes continue, but now the second error has varied:
'React/RCTInvalidating.h' file not found
Sorry for the delay, I've been busy but now I'm free.

heyy @mcousillas6


3.0-m2 - maxdepth seems not working

OrientDB Version: 3.0-m2
On demodb, try:
SELECT COUNT(*)
FROM (
TRAVERSE * FROM (
SELECT FROM Profiles WHERE Name='Santo' and Surname='OrientDB'
) MAXDEPTH 1
)
SELECT COUNT(*)
FROM (
TRAVERSE * FROM (
SELECT FROM Profiles WHERE Name='Santo' and Surname='OrientDB'
) MAXDEPTH 2
)
SELECT COUNT(*)
FROM (
TRAVERSE * FROM (
SELECT FROM Profiles WHERE Name='Santo' and Surname='OrientDB'
) MAXDEPTH 3
)
The queries above all return 21303 as result. This is a regression
Thanks,

Hi @santo-it
Reproduced, working to a fix
Thanks!
Luigi

Fixed on develop branch, closing
Thank you @santo-it !
Luigi


gzip and deflate encoding and decoding support in middleware handlers

Expected behavior:

Requests with gzip as content-encoding should be decoded before the are processed by any request handler.

Current behavior:

Request body is gzipped in the handler and can not be handled properly

Proposed steps to fix this:

Add gzip decoding middleware to handle this transparently

 Encode and decode middleware handlers


mobile ui improvements

@mzampetakis The .DS_Store files have been removed from the repo. Could you pull the latest changes and fix the conflict?


Add comment on GetWorkflow

and fix typos

Pull Request Test Coverage Report for Build 4510100b-cec5-448b-a499-6598c6dae176

0 of 0   changed or added relevant lines in 0 files are covered.
2 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-0.02%) to 74.716%

Files with Coverage Reduction
New Missed Lines
%

internal/internal_task_pollers.go
2
73.04%

Totals

Change from base Build 7aae5ba8-ed20-4bfd-ade3-3199b9f4307d:
-0.02%

Covered Lines:
8933

Relevant Lines:
11956

💛  - Coveralls

Why would it timeout if workflow is not found? It should fail fast in this case.

Why would it timeout if workflow is not found? It should fail fast in this case.

I agree it should fail fast in this case.
Dig into the then found and fixed an issue on server side that caused this timeout.
https://github.com/uber/cadence/pull/3188
Now it immediately return EntityNotFound.


Add Shadow of the Beast trainers back

In early MegaAGS revisions Shadow of the Beast had trainers like infinite energy, a jump button using cd32 controller mapping, etc...  but recent revisions trainers for this game is completely gone. Its a pretty difficult game, please add them back.

We reverted to an older version since the newer version were showing bugs and artifacts, but we can certainly have a look again, and file upstream bugs if it hasn’t been addressed.

Oh, and looks like there is a built-in infinite energy toggle with F1 in that version, since infinite energy always can get you stuck in certain areas. Doesn’t fix the CD32 button-to-jump support, but you can do that with a joystick remap in MiSTer (and Analogue Pocket too, I believe). In case you want to play it with those two capabilities right now. :)


Fix Android export

_should_compress_asset() now takes the header data from the buffer instead of trying to re-read the actual files, which is why it failed for temp files (atlas, etc.).
Fixes #5208.

Both this and #5179 should be merged now or kept for 2.1. Don't hey?

They should be merged now ideally. The current master branch is what will become the stable 2.1, so when we tag issues and PRs for 2.1, it means that it should be fixed before going stable (and ideally as soon as possible).

Good to know! Since I plan to contrib as much as I can.


Bump kernel version to 4.9.19

$Subject

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/7006343643

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/7007995929
Status: cancelled

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/7013470521

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/7016837847

Known failure - Therefore merged the PR
product-is BUILD FAILURE
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] org.wso2.identity.integration.test.openid.OpenIDSSOTestCase.testOpenId(org.wso2.identity.integration.test.openid.OpenIDSSOTestCase)
[ERROR] Run 1: OpenIDSSOTestCase.testOpenId:179->executePhaseAfterApproval:231->assertClaims:393 expected: but was:
[ERROR] Run 2: OpenIDSSOTestCase.testOpenId:179->executePhaseAfterApproval:231->assertClaims:393 expected: but was:
[ERROR] Run 3: OpenIDSSOTestCase.testOpenId:179->executePhaseAfterApproval:231->assertClaims:393 expected: but was:
[ERROR] Run 4: OpenIDSSOTestCase.testOpenId:179->executePhaseAfterApproval:231->assertClaims:393 expected: but was:
[ERROR] Run 5: OpenIDSSOTestCase.testOpenId:179->executePhaseAfterApproval:231->assertClaims:393 expected: but was:
[ERROR] Run 6: OpenIDSSOTestCase.testOpenId:179->executePhaseAfterApproval:231->assertClaims:393 expected: but was:
[ERROR] Run 7: OpenIDSSOTestCase.testOpenId:179->executePhaseAfterApproval:231->assertClaims:393 expected: but was:
[ERROR] Run 8: OpenIDSSOTestCase.testOpenId:179->executePhaseAfterApproval:231->assertClaims:393 expected: but was:
[ERROR] Run 9: OpenIDSSOTestCase.testOpenId:179->executePhaseAfterApproval:231->assertClaims:393 expected: but was:
[ERROR] Run 10: OpenIDSSOTestCase.testOpenId:179->executePhaseAfterApproval:231->assertClaims:393 expected: but was:
[ERROR] Run 11: OpenIDSSOTestCase.testOpenId:179->executePhaseAfterApproval:231->assertClaims:393 expected: but was:
[ERROR] Run 12: OpenIDSSOTestCase.testOpenId:179->executePhaseAfterApproval:231->assertClaims:393 expected: but was:
[INFO]
[INFO]
[ERROR] Tests run: 2088, Failures: 1, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/7016837847
Status: failure


Can MGEfinder detect MGEs in the presence of strain diversity?

Hi all,
I was wondering if MGEfinder can be used in samples that are not fully clonal? e.g. have you tested whether MGEfinder can detect MGEs with different levels of "purity" of the sample?
Specifically, my use case would be data from experimental evolution carried in a natural environment, where populations of a given strain are sequenced after a given time of evolution. To be more specific, the design is similar to this, where mice (that contain a microbiota) are colonized with a bacterial clone, which is allowed to evolve within different mice for a specific period. After this period, faecal samples are plated in selective media for the clone of interest and all colonies growing on a plate are scrapped and sequenced, which is where it differs from the standard MGEfinder use case.
Multiple experiments have shown that bacteria readily adapt and we can detect SNPs/ISs, etc. Though, as this environment contains multiple species, HGT is a possibility. So the data contains substantial microvariation, but the genomes constituting these populations are not as different as if we were to sample different clones from different people.
One thing I worry is if the SPADES assembly step will basically lead to a consensus sequence that ignores the microvariation present in the population. Therefore, I was wondering if it is possible to use metaSPADES for the assembly process (assuming this would allow us to keep more of that microvariation)?
PS. sorry for the rambling post

Thanks for the the question!
MGEfinder was designed to work with fully clonal samples, and its performance was evaluated under that assumption.
I would not use MGEfinder in a highly diverse metagenomic sample. But what you have is different, and I think that MGEfinder could be helpful. The concerns you have with an assembly removing the low-frequency variation is certainly a valid one.
I think you should give it a try and see what you find. I would carefully curate any putative MGEs that MGEfinder identifies. You may want to then try playing with some of the parameters to make it more sensitive toward detecting low-frequency variants (although this could increase false positives).
I think this is an interesting question and experimental design, and I would be happy to help you further if you run into other issues along the way.


adds line_plot

Adds the function line_plot to the utilities.display to easily plot line profiles.
The API is similar to show2D. A line like the following produces the plot:

line_plot( [gt, fbp_recon, algo1.solution * A1.norm()],
           label=['Ground Truth', 'FBP', 'PDHG + TV + nn'],
           line_coords=(('horizontal_x',64), ('vertical',64)),
           title=f'Comparison alpha {alpha}',
           color=('cyan', 'purple', 'orange'),
           size=(15,9))

data  and line_coords are required parameters, the others are optional and described in the docstring.

In the future I would like to merge this functionality in show2D but the interface needs to be discussed.

Doesn't seem to work for 2D images

Closing this PR until ready. Development will be in https://github.com/paskino/CIL/tree/add_lineplot


Labels get escaped inconsistently across Dashboard

Bug Report
When defining a label with "special" characters (e.g. /) the label will be escaped inconsistently across the Dashboard (see images). I feel like they shouldn't be escaped at all, since they are defined within code but I would at least expect it to be consistent.
Steps to Reproduce

Define a label with a "special" character (e.g. Example/Plural)
It will be escaped in some places (Example/Plural => Example/Plural)
but not in others (Example/Plural => Example/Plural)

Collection Example:
const Examples: CollectionConfig = {
  slug: "examples",
  labels: {
    singular: "Example/Singular",
    plural: "Example/Plural",
  },
  ...
};

Global Example:
const Home: GlobalConfig = {
  slug: "home",
  label: "Page/Home",
  ...
};

Other Details
Payload version 1.5.4
I had a quick look into the code and I am pretty sure that this behaviour is due to i18next, which escapes values by default.
https://www.i18next.com/translation-function/interpolation#unescape

I have face the problem too, however I found the solution to add as in the https://www.i18next.com/translation-function/interpolation#additional-options
i18n: {
    interpolation: {
      escapeValue: false,
    },
  },

to the payload-config.ts file, then it will solve the problem.
Question: Should this be added to the default options of the config in https://github.com/payloadcms/payload/blob/master/src/translations/defaultOptions.ts, since the purpose of escaping the special character of i18next is to avoid XSS injection for a security reason. If so, I can open PR to fix that.

since the purpose of escaping the special character of i18next is to avoid XSS injection for a security reason.

@iamprompt this was exactly my first thought, but React escapes all JSX already (described here so this is not an issue.


CLOUDSTACK-8304: disable snapshots from non-snapshot repositories

Signed-off-by: Laszlo Hornyak<EMAIL_ADDRESS>

LGTM. @K0zka please merge it on both master and 4.5!

I dont think this fixes the issue. we are specifying a version number. maven will not check for snapshots.

Not a maven expert, leaving for someone to review/merge who is.

@karuturi So the problem that it adresses is this:  We have SNAPSHOT dependencies, since snapshots are not disabled on those repositories, they will be checked. Now the problem is that if one of those servers is not available, that is breaking the build.

understood. This will definitely save build time. (will do the merge in sometime)
But, the build will fail irrespective of whether snapshot is enabled right? if the server goes offline, we will not get the dependency jar as well(in this case libvirt).

@karuturi Correct, if you do not have the libvirt jar and libvirt.org goes down, the build will still fail. It would be nice if we could use only the central repository, but that will need some work with the ceph and libvirt team.


Find a bug in load_masks

Issue: When training an instance segmentation model with setting poly2mask=False，the mask map is compared to poly2mask=True, it drops by 5-6 points.
Reason: When load_masks (polygons format) is not forced to convert to np.float64, in crop_and_resize, multiple operations are performed to take integers, resulting in a large deviation of mask_target.

polygon is [x1, y1, x2, y2, .... y_n, y_n], and during crop_and_resize https://github.com/open-mmlab/mmdetection/blob/3b72b12fe9b14de906d1363982b9fba05e7d47c1/mmdet/core/mask/structures.py#L724-L763
I don't think np.float64 or np.float32 have different. Have you set seed to 0 during training?

It's not the problem here, it's that if the polygons in your annotation file are int64, then the loading polygons is np.int64, which causes the related operations in crop_and_resize to be np.int64.
File: mmdetection/mmdet/datasets/pipelines/loading.py
Modifying this line of code is normal: polygons = [np.array(p, dtype=np.float64) for p in polygons]

@BIGWangYuDong

Seems bboxes and masks are all np.float64?  I think it is not common that they are both np.int64?

Also, I think bboxes are float32(or float64) so masks will be float when they are calculated with bboxes?

Also, I think bboxes are float32(or float64) so masks will be float when they are calculated with bboxes?

Since the polygons in the coco annotation are float, your load polygons are np.float64. When you encounter other new datasets where the polygons are labeled as int64, the polygons you load will be int64, there will be problems here.

It's OK, would you please create a PR to fix this @chenzyhust

Also, can you give me an example that masks maybe int64? (lol)

@BIGWangYuDong

So if   p[0::2] = p[0::2] - bbox[0] it will be float32?

@BIGWangYuDong No, will be int64, same with p[0::2] = p[0::2] * w_scale

Sorry for the late reply, would you mind creating a PR to fix this potential error?


Fix browser.site so that it supports file:/ url scheme

if I run:
browser.site = "file:///some/file/path/";
browser.visit("index.html");
!== It fails, because it resolves the path to be "http://file:///some/file/path/index.html"
Zombie should support file:// urls as first-class citizens, no?

Thanks.


NoClassDefFoundError: Failed resolution of: Lcom/google/android/exoplayer2/trackselection/MappingTrackSelector$SelectionOverride

Current behavior
version react-native-video @3.2.1;   app crash when video load completed;  it looks for a class that not available on latest exoplayer2;
java.lang.NoClassDefFoundError: Failed resolution of: Lcom/google/android/exoplayer2/trackselection/MappingTrackSelector$SelectionOverride;
at com.brentvatne.exoplayer.ReactExoplayerView.setSelectedTrack(ReactExoplayerView.java:823)
at com.brentvatne.exoplayer.ReactExoplayerView.setSelectedAudioTrack(ReactExoplayerView.java:851)
at com.brentvatne.exoplayer.ReactExoplayerView.videoLoaded(ReactExoplayerView.java:514)
at com.brentvatne.exoplayer.ReactExoplayerView.onPlayerStateChanged(ReactExoplayerView.java:493)
at com.google.android.exoplayer2.ExoPlayerImpl$PlaybackInfoUpdate.notifyListeners(ExoPlayerImpl.java:769)
at com.google.android.exoplayer2.ExoPlayerImpl.updatePlaybackInfo(ExoPlayerImpl.java:681)
at com.google.android.exoplayer2.ExoPlayerImpl.handlePlaybackInfo(ExoPlayerImpl.java:622)
at com.google.android.exoplayer2.ExoPlayerImpl.handleEvent(ExoPlayerImpl.java:567)
at com.google.android.exoplayer2.ExoPlayerImpl$1.handleMessage(ExoPlayerImpl.java:109)
at android.os.Handler.dispatchMessage(Handler.java:105)
at android.os.Looper.loop(Looper.java:164)
at android.app.ActivityThread.main(ActivityThread.java:6938)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.Zygote$MethodAndArgsCaller.run(Zygote.java:327)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1374)
from the auto-generate iml file, i could see it refer to
com.google.android.exoplayer:exoplayer-hls-2.8.4
com.google.android.exoplayer:exoplayer-core-2.8.4
shouldn't it refer to 2.7.3?   any idea ?
Reproduction steps
always
Expected behavior
no crash
Platform

Android ExoPlayer

Video sample
If possible, include a link to the video that has the problem that can be streamed or downloaded from.

my fault,  another module introduce that, just close

@yoyo001 Can you explain how you solved this?


Backwards find motion should not include initial character

F (Find) and T (Till) backwards motions do not behave like in vim when combined to operators. Indeed, the character at the initial position is included.
For example, if the cursor is on cake's c in
Can I lie to the freaking cake?

pressing dFf will produce
Can I lie to the ake?

The c of cake gets eaten, and it shouldn't :( we would expect instead
Can I lie to the cake?

I've already fixed the bug and I'll link a PR to this issue in a minute.

As stated in the README, this package is no longer maintained and is deprecated. We recommend that people use the vim-mode-plus package instead. Because of this, we are archiving this repository and closing all issues and pull requests. Thanks very much for your support and contributions!


Error in loading tokenizer for 'facebook/esm2_t36_3B_UR50D'

I'm running an ordinary task,
chai fold input.fasta output_dir
Here is the error:
OSError: Can't load tokenizer for 'facebook/esm2_t36_3B_UR50D'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same
name. Otherwise, make sure 'facebook/esm2_t36_3B_UR50D' is the correct path to a directory containing all relevant files for a EsmTokenizer tokenizer.

There is no local directory with the same name, and huggingface is accessible.

I have solved this problem by downloading esm2 .bin file manually from huggingface and change the parameter file directory ( line 57 in chai_lab/data/dataset/embeddings/esm.py)

hello，i do it in your way,but it has same probelm,could you please say detail

hello，i do it in your way,but it has same probelm,could you please say detail

Which files did you copy?

thanks，I have already solved this .


Add images based on Debian 11 (Bullseye)

The stable release of Debian 11 (Bullseye) is expected on 2021-08-14, see https://lists.debian.org/debian-devel-announce/2021/07/msg00003.html
The base images buildpack-deps:bullseye and debian:bullseye-slim are available on Docker Hub.

after the release is officially released

That's what I expected :)

drop all stretch variants

Done

#636 Since the update I've been unable to install the drivers for MSSQL I thought I had broken something, but I'm guessing it could be these changes?

I'm guessing it could be these changes

Only if you're using python:lastest.  But I hope you're not doing that.

I'm guessing it could be these changes

Only if you're using python:latest or any other tag that doesn't mention the OS version (eg. python:3.9 instead of python:3.9-buster or python:3.9-bullseye). But I hope you're not doing that.

Thanks @michael-k
Learnt something new today. Appreciate the help.


MINOR: Clarify how to fix conversion issues when plain JSON data is used with schemas.enable=true

@hachikuji @gwenshap @ijuma Trivial message cleanup to help reduce frequency of a FAQ.

LGTM


permutate function sometimes causes cudaErrorIllegalAddress

permutate function sometimes causes cudaErrorIllegalAddress on backward process when cupy.take is called. We need to investigate its detailed condition.

Unno-san, please reconfirm this issue still exists.

I can't reproduce this error. I'll close it. When I find the situation, I'll reopen it.


Possible issues in haul_dur, num and cpue related columns

I was having a quick look at the CPUE-related variables & I noticed likely issues in multiple columns, unless I am doing something silly:

haul_dur ranges from < 0.01 to > 24. Is this right? I wonder if there has been a mix up with units here which propagates to CPUE-related columns.
num includes negative values and decimals.
num_cpue and wgt_cpue columns do not match num/haul_dur or wgt/haul_dur . There are also num_cpue values in the hundreds of millions.

I haven't checked the _cpua columns, but it seems likely that these issues will propagate & affect those too.
It might be worth including tests for negative numbers, whole numbers/decimals, numerical ranges/unique values & additional mathematical checks (e.g., num/haul_dur, num/haul_dur/area_swept etc.) into your quality checking routines for relevant columns (or double checking your existing tests).
## Load data
load(url("https://github.com/AquaAuma/FishGlob_data/blob/main/outputs/Compiled_data/FishGlob_public_clean.RData?raw=true"))
d <- as.data.frame(data)

## Select relative abundance metric (e.g., num{_cpue} or wgt{_cpue})
metric <- c("num", "wgt")
metric <- metric[1] # metric[2]

## Check example columns
# Check haul duration, which varies from 0.01 to > 24 hours
# ... is there an issue with units here?
range(d$haul_dur, na.rm = TRUE)
# Check selected metric column:
# ... For num, we have many negative numbers (but not weights)
range(d[, metric], na.rm = TRUE)
# Check the first few unique values
# ... For num, we have decimals - is this problematic?
head(sort(unique(d[, metric])))

## Calculate CPUE by number or weight and compare to dataset column
# NB: These calculations may be affected by issues in both num and haul_dur
# ... columns identified above.
d$ra   <- d[, paste0(metric, "_cpue")]
d$ra_2 <- d[, metric]/d$haul_dur

## Compare calculated and provided values: they should be identical.
isTRUE(all.equal(d$ra, d$ra_2))

## Isolate problematic data
tol <- 1e-6
cols <- c("survey", "haul_id", "accepted_name", metric, "haul_dur", "ra", "ra_2")
issues <- d[which(abs(d$ra - d$ra_2) > tol), cols]
issues$delta <- issues$ra_2 - issues$ra

## Examine problematic data
head(issues)
tail(issues)

## Visualise problematic data & get ranges
pp <- par(mfrow = c(2, 2))
hist(issues$num); range(issues$num, na.rm = TRUE)
hist(issues$ra); range(issues$ra)
hist(issues$ra_2); range(issues$ra_2)
hist(issues$delta); range(issues$delta)
par(pp)

## Pull out select examples with very high CPUE
# We have num_cpue in the hundreds of millions...
head(issues[order(issues$ra, decreasing = TRUE), ])

Hope this helps!

Relatedly, there are sometimes substantial discrepancies between num and wgt.
This is easiest to see by pulling out the largest num values for each species:
library(dplyr)
load(url("https://github.com/AquaAuma/FishGlob_data/blob/main/outputs/Compiled_data/FishGlob_public_clean.RData?raw=true"))
d <- as.data.frame(data)
d |>
  group_by(accepted_name) |>
  # Pull out largest numbers
  filter(num == max(num)) |>
  ungroup() |>
  select(accepted_name, max_num = num, wgt = wgt) |>
  # Calculate implied individual weight in grams
  mutate(wgt_per_id_in_grams = wgt/max_num * 1000) |>
  arrange(desc(max_num))

E.g., 223,298 Prionotus paralatus individuals apparently only weigh 4.87 kg (or 0.0218 g each).

First I just want to say how much I appreciate this data product that is going to prove extremely useful for all kinds of applications!
But just picking up on @edwardlavender's point about negative abundance values, we have also spotted these. I think they ultimately result from ICES using -9 as a missing value code (I'm trying to get confirmation of this, but it is something I have come across for other variables in ICES data before). This will then propagate into a range of negative values when converted to CPUE or CPUA, and most likely when aggregating to species level too (i.e. combining size classes). To give an example - after loading the compiled data FishGlob_public_std_clean.RData, the object data contains the full database, then:
data %>% filter(num < 0) %>% nrow()

Shows 7561 negative abundances, from the following surveys:
> data %>% filter(num_cpua < 0) %>% count(survey)
# A tibble: 6 × 2
  survey       n
  <chr>    <int>
1 BITS      1841
2 EVHOE       14
3 FR-CGFS   2041
4 NS-IBTS   3433
5 PT-IBTS    229
6 SWC-IBTS     3

We can cross-check some of these using the ICES data directly - here using the icesDatras package, and extracting data for one survey where I can see there are negative values:
# check using ICES data directly
library(icesDatras)
library(icesVocab)

bits_2001_1 <- getHLdata(survey = "BITS", year = 2001, quarter = 1) %>%
  as_tibble()

bits_2001_1 %>%
  filter(TotalNo < 0) %>%
  select(Survey, Quarter, Ship, HaulNo, SpecCode, TotalNo)

Gives
   Survey Quarter Ship  HaulNo SpecCode TotalNo
   <chr>    <int> <chr>  <int>    <int>   <dbl>
 1 BITS         1 AA36      20   127123      -9
 2 BITS         1 AA36      20   127203      -9
 3 BITS         1 AA36      20   126450      -9
 4 BITS         1 AA36      20   126736      -9
 5 BITS         1 AA36       2   127141      -9
 6 BITS         1 AA36       2   127123      -9
 7 BITS         1 AA36       2   126417      -9
 8 BITS         1 AA36      20   127214      -9
 9 BITS         1 AA36      20   127141      -9
10 BITS         1 AA36      20   127149      -9
11 BITS         1 AA36      20   126436      -9
12 BITS         1 AA36      20   126417      -9
13 BITS         1 AA36       2   126436      -9
14 BITS         1 AA36       2   127214      -9

Summarising to species level (by haul) in this case does not cause any problems (total species-level abundances are either -9 or positive) but I guess it may in some cases lead to combining -9 with positive abundances - which would make the simple solution of just filtering out negative abundances possibly problematic.

Thank you both for looking into this! Getting back to this very soon, and I will fix and update the dataset/code

@tomjwebb do we know whether it might happen that for one species in one haul, we might find positive abundances and "-9" values? Wondering whether we might have aggregated positive and negative abundances, or if "-9" is only used for empty values of species that have no other record in that specific haul.

@AquaAuma I'm not sure, there is minimal documentation for what causes a missing value to be entered. As you say it becomes a more complicated issue if positive abundances risk being aggregated with -9. I will have a look and see if I can find any examples, but I'm unsure how that would be done systematically. It might be worth looking at previous data products built on the ICES data (e.g. https://data.marine.gov.scot/dataset/manual-version-3-groundfish-survey-monitoring-and-assessment-data-product and updates such as https://data.cefas.co.uk/view/21421) to see if this issue has previously been addressed.

I tried yesterday, and I did not encounter hauls in which taxa are assigned both positive and -9 values. It seems to happen when individual lengths are not measured, but total number of individuals/weights are still reported. I had kept this in the data originally, but never assigned the abundance/weight reported at the taxa level because I only focused on recalculated weights from abundance at length information. This seems to concern specific taxa in the surveys which might be important (or not), and in some cases it's very minor. I am not sure what the best way to handle this in the dataset is. In any case, this will need to be flagged as something for users to be aware of.
The number of hauls for which this happens are:
13 out of 3283 in EVHOE
168 out of 1179 in PT-IBTS
649 out of 10423 in BITS
1129 out of 2071 in FR-CGFS (consistently the same few taxa)
1229 out of 25217 in NS-IBTS
3 out of 3850 in SWC-IBTS

Yes I think you're correct - if you use the TotalNo for haul-level species abundances, this should be either a positive value or -9, so there will be no issues with aggregating negative and positive values. The issue would arise potentially if you were deriving total numbers in a haul by summing the haul number-at-length, as missing values for a single length class would result in summing -9 with positive counts. Maybe the best thing is to add a QC flag identifying records with negative abundances? This would mean they could still be used to record presence of a species, but could easily be filtered out for users wanting to analyse abundance?

re-including taxa with SpecCode == to 5, 6, 8 to include all pres-abs records in the DATRAS dataset

I've fixed the -9 codes throughout the entire DATRAS R code, so no negative values should appear (only positive or NAs). Keeping the issue open because I haven't double checked the ranges/correspondance between columns yet

@mpinsky @zoekitchel do you know why there might be mismatches between num and wgt in the North American surveys? Ed showed an example of Prionotus paralatus (GMEX species it seems). seems like an ongoing issue we might keep, but I don't have much time to look into this

These aren't necessarily mismatches, and it would require asking the data providers to sort down what happened and why. The trawls sometimes catch large aggregations of small individuals, so it is not necessarily impossible. I would suggest making sure the numbers were in the raw data files we got from the data providers, and if yes, keep it in the dataset.

I can look into the raw data for GMEX!
On Fri, Nov 10, 2023 at 2:18 PM Malin Pinsky @.***>
wrote:

These aren't necessarily mismatches, and it would require asking the data
providers to sort down what happened and why. The trawls sometimes catch
large aggregations of small individuals, so it is not necessarily
impossible. I would suggest making sure the numbers were in the raw data
files we got from the data providers, and if yes, keep it in the dataset.
—
Reply to this email directly, view it on GitHub
https://github.com/AquaAuma/FishGlob_data/issues/24#issuecomment-1806510693,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AHQNR7UTPBMVSYJP6ELOJBLYD2R3VAVCNFSM6AAAAAAYVS2DFGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBWGUYTANRZGM
.
You are receiving this because you were mentioned.Message ID:
@.***>

--
Zoë Kitchel
207-807-4426

Just traced Ed's observation for Prionotus paralatus in haul ID<PHONE_NUMBER>32793 back to original files we have from Summer SEAMAP Groundfish Survey.

I agree, no need to change the data.
We could always ask Ed for more information and which surveys seem
problematic overall, but I don't have more time to spend on this issue.
On Mon, Nov 13, 2023 at 7:33 PM Malin Pinsky @.***>
wrote:

I think that means we keep it, barring other information from the data
providers?
—
Reply to this email directly, view it on GitHub
https://github.com/AquaAuma/FishGlob_data/issues/24#issuecomment-1808771687,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ALGBQUKYVOCPXFBJRHHXBNDYEJRXBAVCNFSM6AAAAAAYVS2DFGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBYG43TCNRYG4
.
You are receiving this because you were mentioned.Message ID:
@.***>


Update toc.yml

Added a link to Knowledge Base.

@jerrykindall, Would you review the proposed changes? Thanks.

@v-shils The link goes to the right place, but I don't see any articles, even though the sidebar says there are two. This isn't an issue with this commit, though.
#sign-off


export/import feature

would be nice export data in case changing browser or pc. export can be locally or g-drive

I had been planning to add that, i'll add it soon.

working on this

thanks @dantehemerson.
i'll release an updated extension once the PR is merged

I added import/export local feature here: https://github.com/atharvakadlag/excalisave/pull/15.
Tell me what do you think! @atharvakadlag @keremcankabadayi

This has not been release yet, right?

Is this feature already running? Or have I done something wrong when trying?
What I did:

opened a excalidraw drawing in my old Chrome browser
changed to "settings" within the excalisave addon
pressed `export'
the export file excalisave-backup-2024-06-12...zip was created within my profile's download folder

now, change to another computer

installed excalisave on another computer
opened a new excalidraw drawing
changed to "settings" within the excalisave addon
pressed import
navigated to the (previously transferred) backup file
clicked on open in the file dialog

but nothing seems to change. there is still no drawing within the excalisave pane.

Hi @nnako, I tested the import/export functionality and seems working.
If your data is not sensitive, could you please share the exported file with me? You can send it to my email at<EMAIL_ADDRESS>Otherwise, we can try to get the error logs, follow the next steps:

Open Import/export page
Right click and click on Inspect
Click on Console Tab
Click in the settings icon on the top right and check the Preserve Logs option

Try import
Share a screenshot of the logs with me.

Thanks in advance!

Thanks. I sent you a PM as requested.


append_id will remove the node when the new node is the last child

When I tried to sort in the tree, I used the code below which caused a panic in detach function:
use ego_tree::{tree, Tree};

fn sort<T>(tree: &mut Tree<T>)
where
    T: Ord + Clone + std::fmt::Display,
{
    let binding = tree.clone();
    let children = binding.root().children();
    let mut node_vec = children.collect::<Vec<_>>();
    node_vec.sort_by_key(|node| node.value());

    let mut root_mut = tree.root_mut();
    node_vec.iter().for_each(|node| {
        println!(
            "{:?}: {}; tree: {}",
            node.id(),
            node.value(),
            root_mut.tree()
        );
        // match root_mut.last_child() {
        //     Some(last_child) if last_child.id() != node.id() => {
        root_mut.append_id(node.id());
        //     }
        //     _ => (),
        // }
    });
}

fn main() {
    let mut tree = tree!('a' => { 'd', 'c', 'b' });
    sort(&mut tree);
    println!("{}", tree);
}

I found that every time I append, the last node disappears. After reviewing the code, it seems that when a node detaches, the previous node's next_sibling is set to None, but the parent's children still hold the original value. When only one node remains, it enters the condition first_child_id == self.id.
The simplest way to fix this is, as I commented in the match, to directly return the Node if last_child_id == new_child_id. The same logic applies to prepend_id.
BTW, Is it possible to implement a sort or sort_by_key method? By operating directly within NodeMut, it would eliminate the need for operations like clone to_vec that I'm currently using.

Thank you for reporting this and I agree with your assessment. This might also be the root cause of #35 (prepending an ID that already is the first child).
Let me try to come up with a regression test and then implement your fix...

I found a way to implement sorting. However, since sorting wasn't implemented before, I'm not sure if it can be merged.
use std::cmp::Ordering;

impl<'a, T: 'a> NodeMut<'a, T> {
    pub fn sort(&mut self)
    where
        T: Ord,
    {
        if self.has_children() {
            let (unsorted, sorted) = self.sort_handler(|nodes| {
                nodes.sort_by(|(_, a), (_, b)| a.cmp(b));
            });

            self.swap(unsorted, sorted);
        }
    }

    pub fn sort_by<F>(&mut self, mut compare: F)
    where
        F: FnMut(&T, &T) -> Ordering,
    {
        if self.has_children() {
            let (unsorted, sorted) = self.sort_handler(|nodes| {
                nodes.sort_by(|(_, a), (_, b)| compare(a, b));
            });

            self.swap(unsorted, sorted);
        }
    }

    pub fn sort_by_key<K, F>(&mut self, mut f: F)
    where
        F: FnMut(&T) -> K,
        K: Ord,
    {
        if self.has_children() {
            let (unsorted, sorted) = self.sort_handler(|nodes| {
                nodes.sort_by_key(|(_, value)| f(value));
            });

            self.swap(unsorted, sorted);
        }
    }

    fn sort_handler<F>(&mut self, mut f: F) -> (Vec<NodeId>, Vec<NodeId>)
    where
        F: FnMut(&mut Vec<(NodeId, &T)>),
    {
        let children = unsafe { self.tree.get_unchecked(self.id()).children() };
        let (unsorted, mut nodes): (Vec<_>, Vec<_>) =
            children.map(|n| (n.id(), (n.id(), n.value()))).unzip();
        f(&mut nodes);
        let sorted = nodes.into_iter().map(|(id, _)| id).collect::<Vec<_>>();
        (unsorted, sorted)
    }

    fn swap(&mut self, unsorted: Vec<NodeId>, sorted: Vec<NodeId>) {
        let mut swap = |sorted_id: NodeId, unsorted_id: NodeId| {
            let mut node = unsafe { self.tree.get_unchecked_mut(unsorted_id) };
            node.insert_id_before(sorted_id);
        };

        let mut cache = None;
        let mut unsorted = unsorted.into_iter();
        for id in sorted {
            match cache {
                Some(cache_id) if cache_id != id => swap(id, cache_id),
                Some(_) => cache = None,
                None => match unsorted.next() {
                    Some(unsorted_id) if unsorted_id != id => {
                        swap(id, unsorted_id);
                        cache = Some(unsorted_id);
                    }
                    _ => (),
                },
            }
        }
    }
}


Display checkbox answers as a comma-separated list

A nicer way of displaying checkbox answers:

What are people doing?
Playing games, Enjoying nature, Biking.

Currently looks like:

Playing games
yes
Enjoying nature
yes

Thoughts on using line breaks rather than commas? I feel like I've seen some checkbox options that are long and possibly even sentence-like. And then the presentation mimics the actual checklist nature of the form.

:+1:


Mac Capitan: only work git plugin

I installed oh-my-zsh in Mac Capitan, but it only works git plugin. Example: I see in mi repository with Ruby on Rails: web-app git:(master) ✗
Features

Mac Capitan Version 10.11.4
zsh 5.0.8 (x86_64-apple-darwin15.0)
zshrc file  by default (It includes default plugins git, ruby and rails)
Default terminal

Apparently not load the configuration file, because only work the default theme.

When you talk about "work", you mean that only git info shows up in your prompt?
Also, can I see your plugins=() definition? Sometimes users put commas: plugins=(git, ruby, ...) instead of only using spaces: plugins=(git ruby ...).

@mcornella in my zshrc i have plugins=(git brew osx ruby) but for example, in my promp I can only see git:(master)

Most of the plugins don't have a prompt function. Have a look at the Plugins page of the wiki to know what each plugin does.
Apart from that, the theme you use has to include the extra prompt info, so your theme probably doesn't. Take a look at the existing themes to choose one that also includes ruby info.

@mcornella yes, but ruby or rails plugins if they work in the promp.  In the previous version of Mac it works without problem, now in the capitan version does not work. It's like not charge settings.
The default theme is ZSH_THEME="robbyrussell"

It can't be that it worked before because the robbyrussell theme never had ruby info. You must've been using another theme. For example, the bira theme.
BTW the Ruby prompt info is set up by plugins rvm, chruby or rbenv. The ruby plugin only defines a couple of aliases.

Look, I do not mean not work for that. What I mean is that not carrying the information I have on the zshrc. For example if I change the subject, still using the default theme.

Please post your zshrc file so I can take a look.

# Path to your oh-my-zsh installation.
export ZSH=/Users/andres/.oh-my-zsh

# Set name of the theme to load.
# Look in ~/.oh-my-zsh/themes/
# Optionally, if you set this to "random", it'll load a random theme each
# time that oh-my-zsh is loaded.
ZSH_THEME="amuse"

# Uncomment the following line to use case-sensitive completion.
# CASE_SENSITIVE="true"

# Uncomment the following line to use hyphen-insensitive completion. Case
# sensitive completion must be off. _ and - will be interchangeable.
# HYPHEN_INSENSITIVE="true"

# Uncomment the following line to disable bi-weekly auto-update checks.
# DISABLE_AUTO_UPDATE="true"

# Uncomment the following line to change how often to auto-update (in days).
# export UPDATE_ZSH_DAYS=13

# Uncomment the following line to disable colors in ls.
# DISABLE_LS_COLORS="true"

# Uncomment the following line to disable auto-setting terminal title.
# DISABLE_AUTO_TITLE="true"

# Uncomment the following line to enable command auto-correction.
# ENABLE_CORRECTION="true"

# Uncomment the following line to display red dots whilst waiting for completion.
# COMPLETION_WAITING_DOTS="true"

# Uncomment the following line if you want to disable marking untracked files
# under VCS as dirty. This makes repository status check for large repositories
# much, much faster.
# DISABLE_UNTRACKED_FILES_DIRTY="true"

# Uncomment the following line if you want to change the command execution time
# stamp shown in the history command output.
# The optional three formats: "mm/dd/yyyy"|"dd.mm.yyyy"|"yyyy-mm-dd"
# HIST_STAMPS="mm/dd/yyyy"

# Would you like to use another custom folder than $ZSH/custom?
# ZSH_CUSTOM=/path/to/new-custom-folder

# Which plugins would you like to load? (plugins can be found in ~/.oh-my-zsh/plugins/*)
# Custom plugins may be added to ~/.oh-my-zsh/custom/plugins/
# Example format: plugins=(rails git textmate ruby lighthouse)
# Add wisely, as too many plugins slow down shell startup.
plugins=(git brew osx ruby rails)

# User configuration

export PATH="/Users/andres/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/go/bin"
# export MANPATH="/usr/local/man:$MANPATH"

source $ZSH/oh-my-zsh.sh

# You may need to manually set your language environment
# export LANG=en_US.UTF-8

# Preferred editor for local and remote sessions
# if [[ -n $SSH_CONNECTION ]]; then
#   export EDITOR='vim'
# else
#   export EDITOR='mvim'
# fi

# Compilation flags
# export ARCHFLAGS="-arch x86_64"

# ssh
# export SSH_KEY_PATH="~/.ssh/dsa_id"

# Set personal aliases, overriding those provided by oh-my-zsh libs,
# plugins, and themes. Aliases can be placed here, though oh-my-zsh
# users are encouraged to define aliases within the ZSH_CUSTOM folder.
# For a full list of active aliases, run `alias`.
#
# Example aliases
# alias zshconfig="mate ~/.zshrc"
alias vim="vim -O"
alias json="python -mjson.tool"

if which rbenv > /dev/null; then eval "$(rbenv init -)"; fi

I don't see any error, that should work without a problem. Have you restarted the terminal?

Yes. I think that for some reason does not load the configuration file, as if there is another file in another location.

Ok, this might help you. Have a look at the zsh startup files and the order in which they are loaded. $ZDOTDIR is your home directory.

Have you been able to solve this? What's the status?

My $ZSOTDIR path is empty.

$ZDOTDIR refers to your home directory. https://github.com/robbyrussell/oh-my-zsh/issues/5067#issuecomment-222390625

Try changing export PATH="..." for `export PATH="$PATH:...".

I had a similar problem on ubuntu
I removed .zcompdump-<my_machine_name> file and restart terminal
after other plugins started to work


preserve dtype for numpy arrays

Fixes #625. This lets the user preserve the dtype of numpy array to pyarrow array which was getting lost due to conversion of numpy array -> list -> pyarrow array.

Hi @lhoestq,
It turns out that pyarrow ListArray are not recognized as list-like when we get output from numpy_to_pyarrow_listarray. This might cause tests to fail. If possible can we convert that ListArray output to list inorder for tests to pass? Under the hood it'll maintain the dtype as that of numpy array passed during input only

Brought down the failing tests from 7 to 4. Let me know if that part looks good. Failing tests are looking quite similar. In test_map_torch https://github.com/huggingface/datasets/blob/3d46bc384f811435e59e3916faa3aa20a1cf87bc/tests/test_arrow_dataset.py#L1039 and test_map_tfhttps://github.com/huggingface/datasets/blob/3d46bc384f811435e59e3916faa3aa20a1cf87bc/tests/test_arrow_dataset.py#L1056
they're expecting float64. Shouldn't that be float32?

It's normal: pytorch and tensorflow use float64 by default, unlike numpy which uses float32.
I think that we should always keep the precision of the original tensor (torch/tf/numpy).
It means that as it is in this PR it's fine (the precision is conserved when doing the torch/tf -> numpy conversion).
This is a breaking change but in my opinion the fact that we had Value("float64") for torch.float32 tensors was an issue already.
Let me know what you think. Cc @albertvillanova if you have an opinion on this
If we agree on doing this breaking change, we can just change the test.

Hi @lhoestq,
Merged master into this branch. Only changing the test is left for now (mentioned below) after which all tests should pass.

Brought down the failing tests from 7 to 4. Let me know if that part looks good. Failing tests are looking quite similar. In test_map_torch
https://github.com/huggingface/datasets/blob/3d46bc384f811435e59e3916faa3aa20a1cf87bc/tests/test_arrow_dataset.py#L1039
and test_map_tf
https://github.com/huggingface/datasets/blob/3d46bc384f811435e59e3916faa3aa20a1cf87bc/tests/test_arrow_dataset.py#L1056
they're expecting float64. Shouldn't that be float32 now?

they're expecting float64. Shouldn't that be float32 now?

Yes feel free to update those tests :)
It would be nice to have the same test for JAX as well

Added same test for for JAX too. Also, I saw that I missed changing test_cast_to_python_objects_jax like I did for TF and PyTorch. Finished that as well


Read object for fixext broken by design

Hello.
Currently, I try to read fixext like a MessagePackObject, but it's not possible.

fixext read as Binary (MessagePackObject<byte[]>) instead MessagePackObject<MessagePackExtendedTypeObject>. Some extra proves I found here https://github.com/msgpack/msgpack-cli/wiki/New-message-pack-spec .
I can't convert to Bytes[] by AsBinary because you check string here:

An exception of type 'System.ArgumentException' occurred in MsgPack.dll but was not handled in user code: 'Do not convert System.Byte[] MessagePackObject to MsgPack.MessagePackString.'
   at MsgPack.MessagePackObject.VerifyUnderlyingType[T](MessagePackObject instance, String parameterName)

What should I do for reading objects from a stream and just check the object for fixext? (in my stream I also have other types)

Thank you for feedback.

Why do you use it so? Could you tell me a your use case or scenario?
It is bug for type check. The message should be like `Do not convert MsgPack.MessagePackExtendedTypeObject MessagePackObject to Byte[].'

Thanks for answers.

I should implement custom serializer over "msgpack-cli"  because I have already specific lib/service in another language.
Ok, it's logical but how I can recognize that is current object MessagePackExtendedTypeObject?

You can use MessagePackObject.IsTypeOf<MessagePackExtendedTypeObject>(). Do you use this method to implement your custom serializer?

You can use MessagePackObject.IsTypeOf()

Ohh thanks! I just skip it because I expected what
myMessagePackObject.UnderlyingType == typeof(MessagePackExtendedTypeObject)

should be same as
MessagePackObject.IsTypeOf<MessagePackExtendedTypeObject>()

I confirm IsTypeOf for MessagePackExtendedTypeObject is work correctly.


Added quadTree (Part 1) in Processing and renamed quadTree.js

Added quadTree in Processing (only Part 1) as (CC_98_QuadTree) and renamed quadTree for js (CC_98_QuadTree) to (CC_98_QuadTree.js).
I request you to kindly accept it.
Do we have to make it part wise like Circle Morphing which has 4 parts on GITHUB or one single file?
And please kindly give me advice on the issue #539.
Thanks a lot.

As I have also added another folder so creating a fresh pull request.


Messed Gui

can u read the other issues please i have. seen this posted like at least 15 times. run on cmd prompt: “path to javaw.exe” -jar obfuscator-1.9.3.jar

Running it with Java 8 worked for me

can u read the other issues please i have. seen this posted like at least 15 times. run on cmd prompt: “path to javaw.exe” -jar obfuscator-1.9.3.jar

how do i do that exactly? i did the path to javaw.exe and -jar obfuscator-1.9.3.jar and it said its not recognized as a command


Voxel light-emission rendering

Commit 761b4b11a898d8939ecf9a0e63ced994215dee8f revised the world model so that light emission is a property of individual Atoms rather than BlockAttributes, so recursive blocks can be partially lit. However, this is currently only interpreted by light propagation (and that in a non-directional way), not for renderers. Tasks:

[X] Implement in raytracer. (Code is pushed in commit 39f8db0dc8d741661434e99da9f79b22994768e7, but disabled for now.)
[ ] Implement in wgpu renderer. Preferably without allocating twice the texture space, but that would require being able to split meshes by material (which would be useful, but hairy).
[ ] Implement in glTF export; ditto regarding texturing.
[ ] Update example content as needed so it looks right under these new conditions.

Preferably without allocating twice the texture space, but that would require being able to split meshes by material (which would be useful, but hairy).

From discussion with Ralith on the Rust GameDev Discord server: Note that we don't have to entirely split meshes and use separate draw calls. We can just have multiple bound textures, and store in the vertices an index of which texture set, if any, to use, like so:

0 = use vertex color
1 = use (reflectance atlas 0, constant emission zero)
2 = use (reflectance atlas 1, emission atlas 1)

My other idea prior to this was to indirect through a table of texture coordinate offsets, so that the emissive data could be written to the same texture in a different location; that would have the advantage of fewer textures and supporting things like defragmenting the texture atlas, but is likely to cost more.

As of commit 2e9f6c6c4638992b0839deec5fe6922b65aa991a we have a plan for how to allocate textures as above, by having the mesher tell the allocator what Channels are needed.
Next steps:

Have the allocators actually allocate separate textures (and designate them in the vertex coordinates)
Implement emission in the shaders
Have the mesh generation switch to textures even for simple blocks if there is an emission value (this way, we don't need two vertex colors).

As of commit a2fd653ab13a375c55fd95bcb8aae73d9a0f7422, emission rendering is active. It is not yet implemented in the glTF export, but since texturing is still generally a work in progress there, I'm going to call this done.


Change the copyright notice

*Acceptance criteria
The copyright notiice reads: "Copyright 2018 © mgm. All rights reversed."

tested successfully on https://radio-stagging.herokuapp.com/


Licensing: Please add a license to this repository

Hello!
I would like to use this repository as a dependence in my project. As github suggests, an unlicensed repository is protected by the default copyright laws, so I can't use it, just view.
Please add a license to this repository, or I'll be forced to look for alternatives for your code.

Hi there,
No problem. License has been added.
Hope it helps.
On Tue, Dec 12, 2023 at 10:12 AM Tarasus @.***> wrote:

Hello!
I would like to use this repository as a dependence in my project. As github
suggests
https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/licensing-a-repository#choosing-the-right-license,
an unlicensed repository is protected by the default copyright laws, so I
can't use it, just view.
Please add a license to this repository, or I'll be forced to look for
alternatives for your code.
—
Reply to this email directly, view it on GitHub
https://github.com/CipherBitCorp/VolumeButtonHandler/issues/1, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAFHEGGWGRLASIT4CBBGF43YJBX5NAVCNFSM6AAAAABARULZNGVHI2DSMVQWIX3LMV43ASLTON2WKOZSGAZTOOJXGU3DSOA
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

Woah, that was FAST. Thank you!


changes

fix gcc warning warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
with 2 lines splitup:

     Concept *cpt = reinterpret_cast<Concept*>(memory);
     cpt->~Concept();

use std::size_t instead of unsigned
align according to alignof(std::max_align_t)
use mutable F to get the following functor compiled:

    struct Func1
    {
        int operator()(int val) // const
        {
           return val * 2;
        }

    int a[10];
    };

(used here: https://github.com/user706/CxxFunctionBenchmark).
I knew what to do here, thanks to @arobenko - ref

use static_assert instead of enable_if
add:  SmallFun& operator=(F const&f)

You can see it in action in the following benchmark:
https://github.com/user706/CxxFunctionBenchmark
(used in various.cpp)
PS: what commands do I have to use, to get this buckaroo thing working?
Thanks!

I'm compiling with gcc under Linux. Which compiler do you use?
Does it compile with your compiler?
Are my changes with static_asserts working in your setup?

Thanks for your efforts!

Are my changes with static_asserts working in your setup?

The enable_if vs static_assert is always an interesting discussion.
Although static_assert is easier to read and understand, it has one big disadvantage:
You cannot use it in a SFINAE context.
Hence we cannot use it here and have to patiently wait for the Concepts-TS to be adopted before we can make it more sane and readable.

use mutable F to get the following functor compiled:

It works for your use case but it would break other cases.
Imagine a functor that has different implementations of operator().
Using mutable will always prefer the non-const variant even if in a const context.
Checkout the godbolt example and see how the return value of T::operator() changes based on the presence of the mutable keyword.
As of the alignment issue, good spot! Would love to get that merged.

The enable_if vs static_assert is always an interesting discussion.
Although static_assert is easier to read and understand, it has one big disadvantage:
You cannot use it in a SFINAE context.

?
Which SFINAE use-case do you mean exactly?
Please can you give an example where you enable_if is useful for a user using the smallfun library.

Imagine a functor that has different implementations of operator().
Using mutable will always prefer the non-const variant even if in a const context.
Checkout the godbolt example and see how the return value of T::operator() changes based on the presence of the mutable keyword.

Your example needs fixing ;)
    {
        T t;
        smallfun::SmallFun<int()> f(T{});
        std::cout << f() << "\t should get " << t() << std::endl;
    }
    {
        const T t;
        smallfun::SmallFun<int()> f(T{});
        std::cout << f() << "\t should get " << t() << std::endl;
    }

See it here on wandbox, to see how your original code (on the master branch) also does not return what you want.

I should also mention that the following

add: SmallFun& operator=(F const&f)

is important!
It's one way to fix a bug in your original, where a destructor is not called.
See it here -- prints dtor only once, but should come twice.
The fix is to uncomment line 3
#include "smallfun_changed.hpp" /* mine */

-- it should then print dtor twice.

I've got a branch called my_experiments: https://github.com/user706/smallfunction/tree/my_experiments
Still working on this (when I find time). (Still need to experiment some more, and fix some stuff that's still open... such as fixing the rhs after move operations, etc. )


Integration with Lighthouse breaks only on localhost

Error:
11:47:38 [Temp Tests] Unhandled Rejection at: Promise Promise {
11:47:38 [Temp Tests]   <rejected> Error: You probably have multiple tabs open to the same origin.
11:47:38 [Temp Tests]     at versions.forEach.ver (/home/jenkins/workspace/or_Pipeline_ray9656_jenkins-K4TELRXZ2GGXTCMJMLMVULSDKD4IGNTMJLVG7MI2QWX3S5ZZL4BQ/builder/node_modules/lighthouse/lighthouse-core/gather/driver.js:354:21)
11:47:38 [Temp Tests]     at Array.forEach (<anonymous>)
11:47:38 [Temp Tests]     at registrations.filter.forEach.reg (/home/jenkins/workspace/or_Pipeline_ray9656_jenkins-K4TELRXZ2GGXTCMJMLMVULSDKD4IGNTMJLVG7MI2QWX3S5ZZL4BQ/builder/node_modules/lighthouse/lighthouse-core/gather/driver.js:346:20)
11:47:38 [Temp Tests]     at Array.forEach (<anonymous>)
11:47:38 [Temp Tests]     at getServiceWorkerRegistrations.then.then.then.then._ (/home/jenkins/workspace/or_Pipeline_ray9656_jenkins-K4TELRXZ2GGXTCMJMLMVULSDKD4IGNTMJLVG7MI2QWX3S5ZZL4BQ/builder/node_modules/lighthouse/lighthouse-core/gather/driver.js:345:10)
11:47:38 [Temp Tests]     at <anonymous>
11:47:38 [Temp Tests]     at process._tickCallback (internal/process/next_tick.js:188:7) } reason:  Error: You probably have multiple tabs open to the same origin.
11:47:38 [Temp Tests]     at versions.forEach.ver (/home/jenkins/workspace/or_Pipeline_ray9656_jenkins-K4TELRXZ2GGXTCMJMLMVULSDKD4IGNTMJLVG7MI2QWX3S5ZZL4BQ/builder/node_modules/lighthouse/lighthouse-core/gather/driver.js:354:21)
11:47:38 [Temp Tests]     at Array.forEach (<anonymous>)
11:47:38 [Temp Tests]     at registrations.filter.forEach.reg (/home/jenkins/workspace/or_Pipeline_ray9656_jenkins-K4TELRXZ2GGXTCMJMLMVULSDKD4IGNTMJLVG7MI2QWX3S5ZZL4BQ/builder/node_modules/lighthouse/lighthouse-core/gather/driver.js:346:20)
11:47:38 [Temp Tests]     at Array.forEach (<anonymous>)
11:47:38 [Temp Tests]     at getServiceWorkerRegistrations.then.then.then.then._ (/home/jenkins/workspace/or_Pipeline_ray9656_jenkins-K4TELRXZ2GGXTCMJMLMVULSDKD4IGNTMJLVG7MI2QWX3S5ZZL4BQ/builder/node_modules/lighthouse/lighthouse-core/gather/driver.js:345:10)
11:47:38 [Temp Tests]     at <anonymous>
11:47:38 [Temp Tests]     at process._tickCallback (internal/process/next_tick.js:188:7)
11:47:38 [Temp Tests] events.js:183
11:47:38 [Temp Tests]       throw er; // Unhandled 'error' event
11:47:38 [Temp Tests]       ^
11:47:38 [Temp Tests]
11:47:38 [Temp Tests] Error: connect ECONNREFUSED <IP_ADDRESS>:37142
11:47:38 [Temp Tests]     at Object._errnoException (util.js:992:11)
11:47:38 [Temp Tests]     at _exceptionWithHostPort (util.js:1014:20)
11:47:38 [Temp Tests]     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1186:14)
11:47:38 [Temp Tests] npm ERR! code ELIFECYCLE
11:47:38 [Temp Tests] npm ERR! errno 1
11:47:38 [Temp Tests] npm ERR<EMAIL_ADDRESS>lighthouse-jenkins-test: `node puplighthouse.js`
11:47:38 [Temp Tests] npm ERR! Exit status 1
11:47:38 [Temp Tests] npm ERR!
11:47:38 [Temp Tests] npm ERR! Failed at the<EMAIL_ADDRESS>lighthouse-jenkins-test script.
11:47:38 [Temp Tests] npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
11:47:39 [Temp Tests]
11:47:39 [Temp Tests] npm ERR! A complete log of this run can be found in:
11:47:39 [Temp Tests] npm ERR!     /var/lib/jenkins/.npm/_logs/2018-06-19T18_47_38_974Z-debug.log
11:47:39 [Temp Tests]
11:47:39 [Temp Tests] INFO: Gracefully shutting down. Please wait...
11:47:39 [Temp Tests] { Error: Command failed: npm run lighthouse-jenkins-test
11:47:39 [Temp Tests]     at makeError (/home/jenkins/workspace/or_Pipeline_ray9656_jenkins-K4TELRXZ2GGXTCMJMLMVULSDKD4IGNTMJLVG7MI2QWX3S5ZZL4BQ/builder/node_modules/start-server-and-test/node_modules/execa/index.js:172:9)
11:47:39 [Temp Tests]     at Promise.all.then.arr (/home/jenkins/workspace/or_Pipeline_ray9656_jenkins-K4TELRXZ2GGXTCMJMLMVULSDKD4IGNTMJLVG7MI2QWX3S5ZZL4BQ/builder/node_modules/start-server-and-test/node_modules/execa/index.js:277:16)
11:47:39 [Temp Tests]     at <anonymous>
11:47:39 [Temp Tests]     at process._tickCallback (internal/process/next_tick.js:188:7)
11:47:39 [Temp Tests]   code: 1,
11:47:39 [Temp Tests]   stdout: null,
11:47:39 [Temp Tests]   stderr: null,
11:47:39 [Temp Tests]   failed: true,
11:47:39 [Temp Tests]   signal: null,
11:47:39 [Temp Tests]   cmd: 'npm run lighthouse-jenkins-test',
11:47:39 [Temp Tests]   timedOut: false,
11:47:39 [Temp Tests]   killed: false }
11:47:39 [Temp Tests] error Command failed with exit code 1.

const puppeteer = require('puppeteer');
const lighthouse = require('lighthouse');
const {URL} = require('url');
const ReportGenerator = require('lighthouse/lighthouse-core/report/report-generator');
const fs = require('fs');

const HOME_PAGE = 'http://localhost:9000/';

const opts = {
  chromeFlags: ['--headless', '--no-sandbox', '--disable-gpu', '--disable-setuid-sandbox', '--disable-dev-shm-usage'],
}

async function createNewBrowser() {
  const browser = await puppeteer.launch({
    args: opts.chromeFlags
  })
  return browser;
}

async function createNewPage(browser) {
  const page = await browser.newPage();
  return page;
}

async function run() {
  const browser = await createNewBrowser();
  const page = await createNewPage(browser);

  process.on("unhandledRejection", async (reason, p) => {
    console.error("Unhandled Rejection at: Promise", p, "reason: ", reason);
    browser.close();
  });

  await page.goto(HOME_PAGE);

  await home_lighthouse(browser)
  browser.close();
}

function generateReport(lhr){
  const htmlReportAssets = require('lighthouse/lighthouse-core/report/html/html-report-assets');

  const sanitizedJson = JSON.stringify(lhr)
    .replace(/</g, '\\u003c') // replaces opening script tags
    .replace(/\u2028/g, '\\u2028') // replaces line separators ()
    .replace(/\u2029/g, '\\u2029'); // replaces paragraph separators
  const sanitizedJavascript = htmlReportAssets.REPORT_JAVASCRIPT.replace(/<\//g, '\\u003c/');

  return ReportGenerator.replaceStrings(htmlReportAssets.REPORT_TEMPLATE, [
    {search: '%%LIGHTHOUSE_JSON%%', replacement: sanitizedJson},
    {search: '%%LIGHTHOUSE_JAVASCRIPT%%', replacement: sanitizedJavascript},
    {search: '/*%%LIGHTHOUSE_CSS%%*/', replacement: htmlReportAssets.REPORT_CSS},
    {search: '%%LIGHTHOUSE_TEMPLATES%%', replacement: htmlReportAssets.REPORT_TEMPLATES},
  ]);
}

async function home_lighthouse(browser) {
  // BREAKS HERE
  const {lhr} = await lighthouse(HOME_PAGE, {
    port: (new URL(browser.wsEndpoint())).port,
    output: 'json',
    logLevel: 'info'
  });

  const lighthouse_results = generateReport(lhr);

  fs.writeFile('lighthouse-reports/lighthouse-home-report.html', lighthouse_results, 'utf8', function(err){
    if (err) return console.log(err);
    console.log('home-lighthouse file was saved');
  });
}

run();

lighthouse: 3.0.0-alpha.2
"puppeteer": "^1.5.0",

So in the jenkins I run the lp:ci command:
"lp:ci": "start-server-and-test l:start http://localhost:3000 lighthouse-jenkins-test",
"l:start": "serve -s -l 3000 build",
"lighthouse-jenkins-test": "node puplighthouse.js",

Then I launch Puppeteer and when it tries to generate the first Lighthouse Report on localhost:9000 (with the code above), it breaks with the above error.
It's able to generate the lighthouse reports on public URLs like example.com twitter.com it works flawlessly.
But what I don't get is why it won't work on the PC for localhost (I've tried changing the port to like 7000, and it still doesn't work while it works on the VM with the same specs and locally on my Mac).
I've also made sure that port 3000 isn't in use prior to me starting the server.
What is the expected result?
Expect to generate a lighthouse report successfully for page opened by Puppeteer on localhost:9000
What happens instead?
Does not work on PC (RHLS 7.4) while it works on VM (RHLS 7.4) and locally on MacOS
I have read: link but it doesn't seem to take care of the error...
I'm out of ideas as I have nearly tried everything I could think of..

Weird. It throws some 'Error: You probably have multiple tabs open to the same origin.' - it has nothing to do with puppeteer.
I'd try asking lighthouse folks, they might have an idea of what's going on here.


Deleted ImagePolicies and ImageRepositories gotk_reconcile_condition values are incoherent

👋
I have noticed that deleted ImagePolicies and ImageRepositories conditions metrics are misreported by the controller.
Steps to reproduce

Create a valid ImageRepository and an associated ImagePolicy, e.g. with:

flux create image repository podinfo \
    --image ghcr.io/stefanprodan/podinfo \
&& \
flux create image policy podinfo \
    --image-ref=podinfo \
    --select-semver="6.x"

Delete the 2 previously created resources with kubectl delete imagerepository podinfo && kubectl delete imagepolicy podinfo (order does not matter much, although it slightly impacts the experienced behavior)

Expected behavior

gotk_reconcile_condition{kind="imagerepository" name="podinfo", type="Ready", status="Deleted"} and gotk_reconcile_condition{kind="imagepolicy" name="podinfo", type="Ready", status="Deleted"} equal 1 for, metrics for other status values equal 0

Experienced behavior

gotk_reconcile_condition{kind="imagerepository" name="podinfo", type="Ready", status="True"} equals 1, gotk_reconcile_condition{kind="imagepolicy" name="podinfo", type="Ready", status="False"} equals 1 too (as the referenced ImageRepository was deleted before the ImagePolicy itself I guess),
metrics for other status values equal 0

Can confirm, also affected by this.


Moving between avoided fields doesn't update the view offset

I'm using this for a series of text inputs for which it works well. But if the view has been moved and I tap an input in view that could be too high the view does not move to a location completely within view.
Does that make sense?

So you have 2 UITextFields (for example), and you tap uiTextField1 which triggers the avoiding UIView to move, then you tap uiTextField2 and the avoiding UIView isnt moved to the correct place? Is uiTextField2 higher or lower than uiTextField1?

Lower. There's also a nav bar in play, I realise.


WIP: use hedge middleware

The hedge middleware implements hedged requests, as described in The Tail At Scale.  The idea is that we auto-tune our retry logic according to the actual network conditions, pre-emptively retrying requests that exceed some latency percentile.  This would hopefully solve the problem where our timeouts are too long on mainnet and too slow on testnet.
Currently this doesn't work well, since for some reason as soon as the hedge middleware kicks in, something blocks requests from actually going to the peer set.  This shows up as big delays in pushing requests through the stack of middleware but many ready peer connections in the peer set.
I'm not totally sure why this happens.  At the moment digging into it seems like a low priority, so this PR can sit open for a while.

Normal operation, showing spikes when we start downloading new blocks, followed by dead periods where we're waiting for failed or high-latency requests to time out before we can keep going:

Here you can see that the peer set is being fully utilized in the peak periods and goes unused in the dead periods.
With these changes, after the hedge middleware kicks in, downloads slow to a crawl:

Here you can see that the peer set is not being well-utilized.

Using https://github.com/hdevalence/tower/commit/486360e5597c5e1580a8549aa713dd1cf8aa5954 fixes the performance regression, and seems to do better.  These charts show higher sustained download speeds and periods where there's less waiting for a single block to arrive, although the problem is not completely gone:

@hdevalence are we going to pursue hedge or?

Could this PR fix the sync hang in #1181?

This was blocked on tower-rs/tower#472 referenced above but can go ahead now.  I'd prefer to combine this PR with #1041, because that way there's just one set of sync changes to keep track of and review.

Sounds good.
Did you want a review, even though the tests are failing?
(I think the failure is spurious - the retry timeout test is obsoleted by the download set changes.)

This is now obsoleted by #1041.

Oops, sorry, we're missing the final commit here in #1041.

I'd rather just close this and wait until being finished with rebasing #1041.


[stable/rocketchat] bump app version to 1.1.1

What this PR does / why we need it:
Which issue this PR fixes
(optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged)

fixes #

Special notes for your reviewer:
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[x] DCO signed
[x] Chart Version bumped
[x] Variables are documented in the README.md
[x] Title of the PR starts with chart name (e.g. [stable/chart])

/assign srueg

➔ +-------------------------------------------------------+
➔ |                     SERVER RUNNING                    |
➔ +-------------------------------------------------------+
➔ |                                                       |
➔ |  Rocket.Chat Version: 1.1.1                           |
➔ |       NodeJS Version: 8.11.4 - x64                    |
➔ |      MongoDB Version: 4.0.9                           |
➔ |       MongoDB Engine: unknown                         |
➔ |             Platform: linux                           |
➔ |         Process Port: 3000                            |
➔ |             Site URL: https://rocketchat.xxx.xxx  |
➔ |     ReplicaSet OpLog: Enabled                         |
➔ |          Commit Hash: 80341ed12f                      |
➔ |        Commit Branch: HEAD                            |
➔ |                                                       |
➔ +-------------------------------------------------------+

/lgtm


Default aliasing bug fix

Fixes #4827

@DanielRosenwasser you wanna check this over before merging it?

:+1:

:+1:


Add back the message that the initial test run can take a couple of minutes

Add the message back, removed in the epic/mutation-switching branch as part of the refactoring.

Why the 😕, @kmdrGroch ? 😅


showWithMaskType:

Previously, there was a showWithMaskType: method and this was removed.  Our app uses this method and we preferred not setting a default mask type.
Was there a technical limitation or something else that caused this method to be removed?  If not, is it possible to deprecate the method if that's no longer the preferred way?
Thanks for this amazing library.  Several of my apps depend on it.

+1

I use this library for a project as well. I agree with @vinnybad; deprecating this method would be a preferred solution.
+1

Within the scope of #384 We decided to simplify 'SVProgressHud'. From @samvermette and mine perspective the 'showWithMaskType:' appendix was kind of unnecessary ballast for many methods. This is especially true because, from the feedback we got, many people always use the same mask. I decided to make a "clean" cut and not deprecate the methods. Explanation of this is, that at the moment I am the only maintainer and won't to keep the cognitive load as small as possible.

BTW: This is again work in progress, the Cocoapods release is not scheduled yet.

@honkmaster ok this makes sense.  Thanks for the update on this...
I was just a little surprised to see it go away entirely and not deprecated, that's all.

Same problem here. We use a wrapper over SVProgressHUD and it's easy to change it in only one place, but IMHO, a __deprecated macro should be better to this kind of disruptive changes, isn't it?