cuda error caused by negative tensor value

Hi, thanks for your nice code again! But I got a wired error when run your code, error info as below:
THCudaCheck FAIL file=/opt/conda/conTHC/generic/THCTensorCopy.c line=20 error=59 : device-side assert triggered Traceback (most recent call last):   File "train.py", line 229, in <module>     main()   File "/data1/jayzjwang/opt/anaconda3/envs/deeplab/lib/python3.5/site-packages/click/core.py", line      return self.main(*args, **kwargs)   File "/data1/jayzjwang/opt/anaconda3/envs/deeplab/lib/python3.5/site-packages/click/core.py", line      rv = self.invoke(ctx)   File "/data1/jayzjwang/opt/anaconda3/envs/deeplab/lib/python3.5/site-packages/click/core.py", line      return ctx.invoke(self.callback, **ctx.params)   File "/data1/jayzjwang/opt/anaconda3/envs/deeplab/lib/python3.5/site-packages/click/core.py", line      return callback(*args, **kwargs)   File "train.py", line 183, in main     target_ = target_.to(device) RuntimeError: cuda runtime error (59) : device-side assert triggered at /opt/conda/conda-bld/pytorch_TensorCopy.c:20 Exception ignored in: <bound method _DataLoaderIter.__del__ of <torch.utils.data.dataloader._DataLoad Traceback (most recent call last):   File "/data1/jayzjwang/opt/anaconda3/envs/deeplab/lib/python3.5/site-packages/torch/utils/data/data     self._shutdown_workers()   File "/data1/jayzjwang/opt/anaconda3/envs/deeplab/lib/python3.5/site-packages/torch/utils/data/data     self.worker_result_queue.get()   File "/data1/jayzjwang/opt/anaconda3/envs/deeplab/lib/python3.5/multiprocessing/queues.py", line 33     return ForkingPickler.loads(res)   File "/data1/jayzjwang/opt/anaconda3/envs/deeplab/lib/python3.5/site-packages/torch/multiprocessinge_fd     fd = df.detach()   File "/data1/jayzjwang/opt/anaconda3/envs/deeplab/lib/python3.5/multiprocessing/resource_sharer.py"     with _resource_sharer.get_connection(self._id) as conn:   File "/data1/jayzjwang/opt/anaconda3/envs/deeplab/lib/python3.5/multiprocessing/resource_sharer.py"     c = Client(address, authkey=process.current_process().authkey)   File "/data1/jayzjwang/opt/anaconda3/envs/deeplab/lib/python3.5/multiprocessing/connection.py", lin     c = SocketClient(address)   File "/data1/jayzjwang/opt/anaconda3/envs/deeplab/lib/python3.5/multiprocessing/connection.py", lin     s.connect(address) ConnectionRefusedError: [Errno 111] Connection refused
it may be caused by negative tensor value when set ignore_label to -1 in preprocessing label map according to this issue torch/cutorch#708, after I set the ignore label to 255 (I make minor change to your codes to run it on voc12), it can work fine

Let me confirm your modification.

it may be caused by negative tensor value when set ignore_label to -1 in preprocessing label map according to this issue torch/cutorch#708, after I set the ignore label to 255 (I make minor change to your codes to run it on voc12), it can work fine

Are you referring to config/voc12.yaml? Or your dataloader for voc?
In cocostuff case, the negative value has never caused the runtime error in target_.to(device)

I have no idea why the target_ = target_.to(device) cased the runtime error.
Did the target tensor contain a negative value?
My guess is that it is possible that the label 255 in "target_" would be denied by the loss criterion (the next line) which assumes indices 0 to 20 for classes and another index -1 for "ignore label".
And I think it can be solved by changing the "ignore label" from -1 to 255 as you said.
Haven't you modified your code during training? (doubt for the stack trace, sorry)

I refered voc12.yaml in your config folder and I just add voc12.py in dataset folder as this file voc12.py(L33-L34, L54, L147-L149, I don't know how to highlight multi lines in github, sorry for that) to load voc12 dataset, for train.py, I only change the dataset class to voc.
Btw, after training 20000 iters, I can get these miou results on voc12:
eval on single scale:  75.455%
eval on multi scale as original paper (0.5, 0.75, 1.0): 74.291%
eval on multi scale (0.75, 1.0, 1.25): 75.809%
The best score is still about 0.5% difference with the paper's result (76.35%), which is acceptable and I think the difference may caused by the reason that pytorch doesn't support synchronous batch normalization which can bring 0.5-0.6 difference on the final result in multi-gpu training process (my friend told me this information), and I found a sync bn implementation here, but after I simply changed the bn layer in your model, the results was only about 74%, maybe I didn't make it on the right way, how do you think about it?
ps: I found another great segmentation repo semantic-segmentation-pytorch which used the sync bn in pytorch, maybe it will be helpful
Thanks for your codes again!

here are some other files:
train.py
eval.py

Anyway, I fixed config/voc12.yaml. Thank you for the information about synchronous batch normalization! I didn't know that. I think that's crucial but, to be honest, have no idea how to solve your problem now. Let me close this issue because the bn problem is out of scope.
I will try to fix the problem, any PRs are welcome.

I have no idea why the target_ = target_.to(device) cased the runtime error.
Did the target tensor contain a negative value?
My guess is that it is possible that the label 255 in "target_" would be denied by the loss criterion (the next line) which assumes indices 0 to 20 for classes and another index -1 for "ignore label".
And I think it can be solved by changing the "ignore label" from -1 to 255 as you said.
Haven't you modified your code during training? (doubt for the stack trace, sorry)

Why can the tensors not contain any negative values?


add some conventions for sorting files

see README in this PR.
"." is shown last becasue it is a common convention to hide such files.
readme is an important file, so show it first.

IMHO the way this is done by NYT library is probably better. https://nyt-library-demo.herokuapp.com/how-to-use-library/how-do-i-use-tags

Thanks a lot!


Slow autocomplete

Are you requesting a feature, reporting a bug or asking a question?
Asking a question
What is the current behavior?
I have around 5000 items, and I am using the Tagbox question: https://surveyjs.io/Examples/Library/?id=custom-widget-select2-tagbox&platform=jQuery&theme=default. The autocomplete functionality is very slow.
What is the expected behavior?
I would like the view that autocomplete options quicker, as it is too slow to be useable at the moment. A solution could be to add a minLength parameter to only initialise the Select2 plugin after the user has typed in 3 characters, but I'm not sure where to set this is a configuration parameter. Another option could be to only start the search after 1 second since the users last key stroke.
How would you reproduce the current behaviour (if this is a bug)?
Change the list in the example to a list that has 5000 items. https://surveyjs.io/Examples/Library/?id=custom-widget-select2-tagbox&platform=jQuery&theme=default

surveyjs platform Vue
surveyjs version: 1.0.59

I just found select2Config! This helps.

Here is the sample - https://next.plnkr.co/edit/SnvX05j0opC1yPKX

Perfect! Thank you.

When I add this example to the builder - I get an error:
Error: Expected 'a' instead of 'u'

It seems that I can't put javascript directly into the survey editor JSON definition.

@willgdjones could you please provide a small example to reproduce the problem?

To reproduce, add:
{
    "elements": [
        {
            "type": "tagbox",
            "isRequired": true,
            "select2Config": {
                minimumInputLength: 3,
                ajax: {
                    delay: 250,
                    url: 'https://api.stackexchange.com/2.2/questions',
                    data: function (params) {
                        var query = {
                            tagged: params.term,
                            site: 'stackoverflow',
                            order: 'desc',
                            page: params.page || 1
                        }
                        return query;
                    },
                    processResults: function (data) {
                        return {
                            results: data.items.map(function(dataItem) {
                                return {
                                    id: dataItem.link,
                                    text: dataItem.title
                                }
                            })
                        };
                    }
                },
            },
            "name": "countries",
            "title": "Please select all countries you have been for the last 3 years."
        }
    ]
};

Into the Survey Editor. Thanks for your help!


test against, and upgrade to, libxml-2.9.10

libxml 2.9.10-rc1 has been released, and the team is targetting a final release later this week, according to this email thread.
this issue is to:

run the CI pipeline against this version of libxml2
cut a release depending on the final libxml 2.9.10 when it ships

branch pushed: libxml-2.9.10-rc1
pipeline running: https://ci.nokogiri.org/teams/nokogiri-core/pipelines/nokogiri-libxml-2.9.10

v1.10.5 has been released with these dependency updates.


Expand the inbound external connectivity documentation

The docs section talking about accessing containers from outside of the calico managed nodes is pretty high level and it would be nice to document some of the other options available and have an example or 2.
Here are some of the options that have came up for inbound external connectivity:

Configuring calico to peer with the networking fabric so that the other hosts' L3 hop already knows how to route to the containers. This is philiosphically the "best" solution but not always one people can implement if they're not running in private cloud and realistically not unless they're building production infrastructure.
Run a virtual bgp service on the host directly (such as BIRD). The cheating way to do this is just run calico/node on that host too, even if it's not going to run containers.
Set up a loadbalancer / proxy service to forward requests to the Calico IP. This is often orchestrator dependent. For Mesos, use Marathon-LoadBalancer. For Kubernetes, there's service ports that can be used
BGP peer to the next router up, and ensure that the non-Calico host's route to the calico IPs goes via that router.
Enable proxy arp on the eth0 interfaces of the compute hosts, so they will claim that the Calico pods are directly on the network, and use IPs for the Calico containers that are on subnet.
Add a static route to the entire Calico IP range via one of the Calico hosts (but this is a suboptimal traffic path).
Thanks @matthewdupre and @djosborne for these options

Is there any documentation for adding a static route to the calico ClusterIP range?

Any updates?
Interesting in the cheating way to run calico/node on nodes out of calico cluster.

@cbluth
What is calico cluster ip range? Pod ip range or kube-proxy cluster ip range?
Could this be what you want?
https://docs.projectcalico.org/v3.4/usage/service-advertisement
Probably this feature is also not meant for production though. Using some Loadbalancer is the only way to export the service.
But yeah, clear documentation of what's recommended for exporting the k8s ip range would be helpful. For example, even though calico exports pod ip when bgp peering is enabled, it's never meant to accessed from outside other than testing purpose. I believe this service-advertisement falls into the same category as well.

Yeah, the cluster IP advertisement is definitely a part of this, as would be something like MetalLB integration.


Update add new TextStyle attributes

add the new attributes:

double-underlined
antiqua
fracturswitch

Antiqua stands for the change of the main font of the text from Fraktur to Antiqua. This can involve passages, individual words or parts of words. Antiqua does not stand for a specific font.
Fracturswitch stands for the change of the main font of the text from Fraktur to Fraktur. A fracturswitch is a special type of highlighting. Since, for example, there is no bold cut in the Fraktur fonts, a different Fraktur font (e.g. Schwabacher) was used to make a bold typeface. This can concern passages, single words as well as word parts.

Or replace antiqua and fracturSwitch with some mechanism compatible with https://github.com/seuretm/ocrd_typegroups_classifier
Essentially: instead of boolean attributes for switch to antiqua/blackletter fonts use an enumeration of values based on a small classification of typefaces that subsumes the classes defines by https://github.com/seuretm/ocrd_typegroups_classifier
Off the top of my head:
Blackletter
  Fraktur
  Schwabacher
  Bastarda
  [...]
Antiqua
  Serif
  Sans-Serif
  Monospace

and then typefaceFamily="Antiqua" instead of antiqua="true" usw.
@seuretm


Avoids changing the type of the osmid

Avoids changing the type of the osmid.
In addition, allow other types than int as osmid.

I went thru the package and fixed all the omsid type alterations.
Test are finally passing again and the example notebooks work as well.
The changes came along with some small speedups.
Let me know if this is fine as is?

@JonasKlasen thanks, added a couple comments for you.

Did I miss something, where did you add the comments?

@JonasKlasen

Did I miss something, where did you add the comments?

No -- sorry I forgot to actually submit the code review! Whoops. It's there now.


Adds springdoc-openapi-ui and removes old swagger-ui

Changes

Adds springdoc-openapi-ui, see https://springdoc.org/ and http://localhost:8080/ehrbase/swagger-ui.html when running EHRbase
Removes old swagger-ui, incl. all the technological dept from the REST spec descriptions we had in the code.
Deprecated EHRscape API from code

Related issue

closes https://github.com/ehrbase/project_management/issues/380
Additional information and checks

[ ] Pull request linked in changelog

This PR needs a team discussion, as this only started as proof of concept for our general idea to revamp our auto-generated endpoint documentation. Specifically, we need decisions on both changes listed above, as well on how to continue with the self/auto/manual documentation of our endpoints (e.g. I think the official REST API doc does that already - or should get updates from us if there is room for improvement).


Can't add liveness and readiness health checks using inline XML

Description
When generating the Kubernetes configuration, the liveness and readiness health checks are not being added to deployment.
The following XML snippet, in theory, should add a liveness health check to the deployment.
<resources>
    <liveness>
        <initialDelaySeconds>30</initialDelaySeconds>
        <timeoutSeconds>3</timeoutSeconds>
        <getUrl>HTTP://:8500/v1/status/peers</getUrl>
    </liveness>
</resources>

Running mvn fabric8:resource failes to produce the appropriate livenessProbe configuration.
I had a look through the code, while it looks like the configuration is being parsed there doesn't seem to be a way of adding the configuration to the appropriate deployment/petset/pod.
I think the solution is a health check enricher, which takes the configured liveness and readiness resources and attaches the configuration as required.
Info

f-m-p version :
Maven version (mvn -v) :

Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00)
Maven home: /usr/local/Cellar/maven/3.3.9/libexec
Java version: 1.8.0_45, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.10.5", arch: "x86_64", family: "mac"

Kubernetes / OpenShift setup and version :
If it's a bug, how to reproduce :
If it's a feature request, what is your use case :
Sample project : https://github.com/ucosty/kubernetes-consul-test.git

I feel I should have dug through the existing issues a bit before raising this, as this can likely be covered by a combination of Issue #600 and Issue #601. Perhaps the work done in Pull Request #624 can be extended to support these scenarios?


Documentation

A. Overview
Project name:
Purpose: Provide a high-level overview of the project, including its goals and objectives.
Brief description:
Main features:
Technologies used (e.g., Dart/Flutter, FlutterFlow, Firestore, Firebase):
B. Architecture
Purpose: Detail the structure and components of the app, both frontend and backend.
Architecture Diagram. Visual representation of the system architecture, showing the interaction between different components.
Frontend Architecture. Overview of the frontend structure, major components, and libraries used.
Backend Architecture.  Overview of the backend structure, including the server, database, and third-party services.
C. UX Flow
Purpose: Describe the user experience flow, including the navigation and interaction within the app.
Wireframes: Visual sketches of each app screen.
App Screen Flow: Flowchart showing the sequence of user interactions and transitions between screens.
D. Database Schema
Purpose: Detail the structure of the Firestore database used in the app.
Database Diagram. Visual representation of the collections and documents.
Collections and Documents. Description of each collection and the structure of its documents.
Fields and Data Types. Details of the fields within each document, including data types and any relationships between collections.
JSON Schema. 305Soft standard JSON schema
E. Metrics
Metrics description - HEART framework
Metrics collection (formerly METRICS.md)
F. API Documentation
Purpose: Document the Web APIs and your own cloud functions used by the app, including endpoints, request methods, and data formats.
API Endpoints: List of API endpoints with descriptions of their purpose.
Request and Response Formats: Details of the expected request and response formats, including headers, parameters, and body content. If JSON is used, then provide the JSON schema.
Authentication: Description of any authentication mechanisms used (e.g. API keys and how they are managed).
G. Backend Services
Purpose: Detail the configuration and usage of various services integrated into the app.
Authentication. Configuration and usage of Authentication, including the authentication method(s) used (e.g. email/password, Google, Facebook, etc) any custom rules or integrations
Cloud Functions.  Overview of Cloud Functions used, their triggers, code, code documentation, and deployment
Database Security. Describe database security rules and how/where they are set.
Metrics/Analytics.  Describe how all metrics are collected referring to specific services on the backend (e.g. Firebase Analytics).
Remote Configuration. If Remote Configuration is used with your app, then describe the configurations controlled by it and the typical instances in which a component would be updated.


fuss, goldeney

This pull request was generated by the 'mq' tool
[test]
flake rate: 0.4
logical conflict every: 100
sleep for: 300s
[pullrequest]
requests per hour: 30
deps=[f,g]

/trunk merge


Smoke test on macOS hits socket file path limit issue

I swear we seem to hit every single exoteric issue with this smoke test.
The smoke test creates brand new user-data-dir folders in TEMP. In macOS, TEMP is usually a huge path. The communication sockets live inside user-data-dir. UNIX sockets have a file path length limit. So the sockets are simply cut. This surfaces as incredibly random and unconnected issues within the smoke test.

This took me way too long to find out.

It seems we're not alone.

cc @sandy081 @isidorn

Marking verified after review.


feat: token list

closes FRO-370

ETH text should be black/white
button text should have the icon color

button text color wasn't changed...

loading state was not added (similar to what we have in "Add Network" screen in the wallet)

after I click in "Add Token", this is what I see. the action was successful but I see an error.

address in input should be cleaned.

loading state not added again
input is not cleared, I still see the error message
usdc text is not centered compared to avatar


Adding more debug logging for runners.py

It's almost impossible to investigate issues like https://github.com/locustio/locust/issues/1535 and some of my previous issues without making custom local changes to add more logging, which is annoying and time consuming.
So here I'm adding some more debug log messages and add a function for updating the state, with a debug log message.

lgtm.


onagre doesn't start under XFCE

Describe the bug
The onagre application (from the same-named package) doesn't start under XFCE, complaining that it can't find libX11.so.6 or libX11.so. I suspect it won't start under any X Window desktop environment.
Steps To Reproduce
Steps to reproduce the behavior:

services.xserver.desktopManager.xfce.enable = true;
environment.systemPackages = [ pkgs.onagre ];
Log into XFCE, run xfce4-terminal.
Try to run onagre from the terminal.

Expected behavior
The application starts and show some kind of GUI.
Screenshots
paul@nixos:~/ > onagre
Failed to parse theme:
Failed to open config file
Failing back to default theme
thread 'main' panicked at 'Failed to initialize any backend! Wayland status: NoCompositorListening X11 status: LibraryOpenError(OpenError { kind: Library, detail: "opening library failed (libX11.so.6: cannot open shared object file: No such file or directory); opening library failed (libX11.so: cannot open shared object file: No such file or directory)" })', /build/cargo-vendor-dir/winit-0.26.1/src/platform_impl/linux/mod.rs:619:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Additional context
Looks like the application does dynamic discovery of a graphics system, and fails to find any because X libraries are not at their usual location in /usr/lib.
Notify maintainers
ping @jfvillablanca
Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.
[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.59, NixOS, 23.05 (Stoat), 23.05.4448.5550a85a087c`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.13.6`
 - channels(paul): `""`
 - channels(root): `"nixos-23.05"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

we're waiting for a fix to be merged https://github.com/NixOS/nixpkgs/pull/235072

Thank you, I tried a fix, and it resolved the problem with starting the application. (Even though it still remains unusable, as I mentioned in my comment in https://github.com/NixOS/nixpkgs/pull/235072.)

in the meantime, you can create an override in your config by using a-kenji's fix while waiting for the PR to be updated and merged

in the meantime, you can create an override in your config by using a-kenji's fix while waiting for the PR to be updated and merged

Do you mean adding the fixed package file somewhere in my filesystem and importing it from my /etc/nixos/configuration.nix using callPackage? Or there is some other, less verbose way?


Aerisweather forecast

I wonder how hard it would be to make Aerisweather forecasts work with this theme?
As https://darksky.net does not allow any new registrations and stops providing the free forecast at the end of this year so that would not be an option. If you provide weather data to https://www.pwsweather.com you can use AerisWeather
forecasts. There are couple of templates which have done this. Like Saratoga template https://saratoga-weather.org/index.php and Belchertown skin https://github.com/poblabs/weewx-belchertown .
There is api to use with AerisWeather.
https://www.aerisweather.com/support/docs/api/

Hi! Creator of open-meteo here.
I created a non-commrecial no-api-key weahter api. I looks like it covers most data in NeoWX. Maybe it could be of use to you. Let me know if I can help.
Best, Jeff

Hi! Creator of open-meteo here.
I created a non-commrecial no-api-key weahter api. I looks like it covers most data in NeoWX. Maybe it could be of use to you. Let me know if I can help.
Best, Jeff

Hi Jeff,
Can you deploy installation instructions and I can test it?


As a user I want to display my notice boards on small screens

Description:
Not every user is working on a desktop PC or a laptop. And sometimes users just want to check their notice boards for new posts while they are walking around the company. Therefore, it is important the application can be displayed on small devices, such as mobile phones and tablets. The layout of the notice boards shall be responsive to different sized screens.
Acceptance criteria:

[ ] Notice boards have a responsive web design
[ ] Tested on a smartphone

Test case (Web browser):

[ ] Open midnightboard application on Apple smartphone in Safari browser and check if it's displayed correctly
[ ] Open midnightboard application on Android smartphone in Google browser and check if it's displayed correctly
[ ] Open midnightboard application on Apple tablet in Safari browser and check if it's displayed correctly
[ ] Open midnightboard application on Android tablet in Google browser and check if it's displayed correctly

Additional notes:
Examples:  mobile devices, tablets, etc.

@MistereoSC Please add your knowledge from testing on an android smartphone as comment.


[Feature Request] Upload Progress

Showing percent/amount uploaded, upload speed etc. would be useful.

It's doable but I don't have time to work on that right know. PR is welcome

Same here! Will do when I got the time.
Also, bit off-topic.. Its not working with large files. streamToPromise reading the entire file into memory, and the process gets terminated due to out-of-memory. You should read the file in chunks as well :)

I guess something with fs.read and just read a part of the buffer will be better. But I don't use this library anymore just maintain it. So I will not invest too much time in it.


FSharp highlighting incorrect

Description
FSharp syntax highlighting is on two different servers (Codeberg and private Gitea instance) rendered incorrectly.
The Chroma playground renderes it beautifully.
Screenshots

Gitea Version
Current
Can you reproduce the bug on the Gitea demo site?
Yes
Operating System
No response
Browser Version
Firefox 108.0.2

https://github.com/alecthomas/chroma/issues/727


update to have sdk key use a different cdn path in the datafile config

Coverage remained the same at 97.261% when pulling 0273c545024d90bee42d8b578b3d6118012a29a2 on sdkKey/url/change into 01769db055b503783826b3a37281d7bf36632d7c on master.


use SwiftLang's license header check

Now that https://github.com/swiftlang/github-workflows/pull/36 is merged, we don't need a custom script to check the license header

The CI will be fixed when https://github.com/swiftlang/github-workflows/pull/38 will be merged


Trials & Errors

Let's collect some problems with rustfix encountered in real life here.

Just a few to get started: https://gist.github.com/killercup/7142b37c1dbd1e5266fc9b0c2ee3ebb8
As this might also interest the good folks who did all the work (i.e., wrote the clippy lints), I'll start with: cc @mcarton

I'll look into it, but is the one about new_without_default still broken? It should be fixed in 0.0.79.

Yup, I installed the new version before testing. Let me make a diff and open a new issue.

By the way the problem you describe with needless_range_loop is actually the feature I described in https://github.com/killercup/rustfix/issues/14#issuecomment-231754724.


Created @medplum/fhirtypes

Before:  @medplum/core contained a lot, including utilities, FHIR type definitions, FHIRPath, etc
Now:  @medplum/core still contains a lot, but FHIR type definitions are pulled out into the new @medplum/fhirtypes
Why?  This is part of an ongoing effort to slim up @medplum/core and to make the FHIRPath implementation a separate project.

Screenshot for readme:

Another screenshot:


Support for Hyperlink Templates on Dimensions

Added support for hyperlink templates in dimensions which work with the reacttable widget

Coverage increased (+0.2%) to 90.036% when pulling cb09711da6a4af5c8dd65376098189ebcde093dd on hyperlink_template_dimensions into 692c029421e8b962e1b20e6a74c910460b5f8f09 on master.


Out of memory error when reading from large files.

It appears that the latest version of elasticdump (1.0.0) has a problem reading from large input files.
$  du -m input.json
166 input.json
$ wc -l input.json
   20000 input.json
$ ./node_modules/elasticdump/bin/elasticdump --input input.json --output output.json
Fri, 05 Feb 2016 17:22:43 GMT | starting dump
Fri, 05 Feb 2016 17:22:43 GMT | got 100 objects from source file (offset: 0)
Fri, 05 Feb 2016 17:22:43 GMT | sent 100 objects to destination file, wrote 100
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
Abort trap: 6

I tested with smaller files, and it appears that elastcdump reads the first 100 lines from the input file, writes them to the output, then on the next read from the input, it tries to read the entire remainder of the input.
For example:
$ head -n 250 input.json | ./node_modules/elasticdump/bin/elasticdump --input $ --output output.json
Fri, 05 Feb 2016 17:27:19 GMT | starting dump
Fri, 05 Feb 2016 17:27:19 GMT | got 100 objects from source file (offset: 0)
Fri, 05 Feb 2016 17:27:19 GMT | sent 100 objects to destination file, wrote 100
Fri, 05 Feb 2016 17:27:19 GMT | got 150 objects from source file (offset: 100)
Fri, 05 Feb 2016 17:27:19 GMT | sent 150 objects to destination file, wrote 150
Fri, 05 Feb 2016 17:27:19 GMT | got 0 objects from source file (offset: 250)
Fri, 05 Feb 2016 17:27:19 GMT | Total Writes: 250
Fri, 05 Feb 2016 17:27:19 GMT | dump complete

and
$ head -n 623 input.json | ./node_modules/elasticdump/bin/elasticdump --input $ --output output.json
Fri, 05 Feb 2016 17:27:42 GMT | starting dump
Fri, 05 Feb 2016 17:27:42 GMT | got 100 objects from source file (offset: 0)
Fri, 05 Feb 2016 17:27:42 GMT | sent 100 objects to destination file, wrote 100
Fri, 05 Feb 2016 17:27:42 GMT | got 523 objects from source file (offset: 100)
Fri, 05 Feb 2016 17:27:43 GMT | sent 523 objects to destination file, wrote 523
Fri, 05 Feb 2016 17:27:43 GMT | got 0 objects from source file (offset: 623)
Fri, 05 Feb 2016 17:27:43 GMT | Total Writes: 623
Fri, 05 Feb 2016 17:27:43 GMT | dump complete

and
$ head -n 2623 input.json | ./node_modules/elasticdump/bin/elasticdump --input $ --output output.json
Fri, 05 Feb 2016 17:28:04 GMT | starting dump
Fri, 05 Feb 2016 17:28:05 GMT | got 100 objects from source file (offset: 0)
Fri, 05 Feb 2016 17:28:05 GMT | sent 100 objects to destination file, wrote 100
Fri, 05 Feb 2016 17:28:08 GMT | got 2523 objects from source file (offset: 100)
Fri, 05 Feb 2016 17:28:09 GMT | sent 2523 objects to destination file, wrote 2523
Fri, 05 Feb 2016 17:28:09 GMT | got 0 objects from source file (offset: 2623)
Fri, 05 Feb 2016 17:28:09 GMT | Total Writes: 2623
Fri, 05 Feb 2016 17:28:09 GMT | dump complete

as I keep increasing the size of the input file, the pattern persists: the second read/write iteration attempts to read the entire remainder of the input.  Once the input file gets large enough, elasticdump will run out of memory trying to read it all in.

A more formal stack trace from a newer node version:
> ./bin/elasticdump --input ~/Desktop/data.json --output http://localhost:9200/data
Fri, 05 Feb 2016 19:21:02 GMT | starting dump
Fri, 05 Feb 2016 19:21:02 GMT | got 100 objects from source file (offset: 0)
Fri, 05 Feb 2016 19:21:02 GMT | sent 100 objects to destination elasticsearch, wrote 100

<--- Last few GCs --->

   11090 ms: Scavenge 1408.5 (1457.0) -> 1408.5 (1457.0) MB, 3.9 / 0 ms (+ 2.5 ms in 1 steps since last GC) [allocation failure] [incremental marking delaying mark-sweep].
   11858 ms: Mark-sweep 1408.5 (1457.0) -> 1408.5 (1457.0) MB, 768.1 / 0 ms (+ 3.5 ms in 2 steps since start of marking, biggest step 2.5 ms) [last resort gc].
   12715 ms: Mark-sweep 1408.5 (1457.0) -> 1408.5 (1457.0) MB, 856.4 / 0 ms [last resort gc].

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0xb140b44a49 <JS Object>
    2: write [/Users/evantahler/Dropbox/Projects/taskrabbit/elasticsearch-dump/node_modules/jsonparse/jsonparse.js:~87] [pc=0x18a6a3b4303c] (this=0xae6cabc4839 <a Parser with map 0x3da79179d869>,buffer=0xb140bc3231 <an Uint8Array with map 0x3da7917203a9>)
    3: /* anonymous */ [/Users/evantahler/Dropbox/Projects/taskrabbit/elasticsearch-dump/node_modules/JSONStream/index.js:~18] [pc=0x18a6a389dc...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
Abort trap: 6

The problem seems isolated to the file reader stream not being paused properly...

will be solved via https://github.com/taskrabbit/elasticsearch-dump/pull/161

SFO1502579255M:~ 502579255$ head -n 92006 /Users/502579255/ELK/kibana-4.4.1-darwin-x64/Kibana_Essentials-master/tweet.json | /usr/local/bin/elasticdump --bulk=true --input $  --output=http://localhost:9200/
Fri, 11 Mar 2016 01:13:55 GMT | starting dump
<--- Last few GCs --->
15098 ms: Scavenge 1405.7 (1458.1) -> 1405.7 (1458.1) MB, 19.0 / 0 ms (+ 54.7 ms in 1 steps since last GC) [allocation failure] [incremental marking delaying mark-sweep].
16596 ms: Mark-sweep 1405.7 (1458.1) -> 1405.2 (1457.1) MB, 1498.1 / 0 ms (+ 1147.8 ms in 2587 steps since start of marking, biggest step 54.7 ms) [last resort gc].
18066 ms: Mark-sweep 1405.2 (1457.1) -> 1405.1 (1458.1) MB, 1469.7 / 0 ms [last resort gc].
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x314dfe6e3ac1
2: write [/usr/local/lib/node_modules/elasticdump/node_modules/jsonparse/jsonparse.js:~87] [pc=0x1a8070b2eb36] (this=0xf08f9dccf41 <a Parser with map 0x320cf4783be9>,buffer=0xcea3db43cc9 <an Uint8Array with map 0x320cf4705911>)
3: /* anonymous */ [/usr/local/lib/node_modules/elasticdump/node_modules/JSONStream/index.js:~18] [pc=0x1a8070b26c0f] (this=0xf08f9dcd0b9 <a Stream with map 0x32...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
Abort trap: 6
SFO1502579255M:~ 502579255$ head -n 90000 /Users/502579255/ELK/kibana-4.4.1-darwin-x64/Kibana_Essentials-master/tweet.json | /usr/local/bin/elasticdump --bulk=true --input $  --output=http://localhost:9200/
Fri, 11 Mar 2016 01:14:20 GMT | starting dump
Fri, 11 Mar 2016 01:14:34 GMT | got 0 objects from source file (offset: 0)
Fri, 11 Mar 2016 01:14:34 GMT | Total Writes: 0
Fri, 11 Mar 2016 01:14:34 GMT | dump complete
SFO1502579255M:~ 502579255$

@craighawki what version of node and elasticdump are you using?

@craighawki it might also be the case that you have a very large document (over 1GB?)

i get the same issue, may i know how do you solved it?
<--- Last few GCs --->
1145104 ms: Mark-sweep 1389.2 (1434.0) -> 1389.1 (1434.0) MB, 975.4 / 0.0 ms [allocation failure] [scavenge might not succeed].
1146280 ms: Mark-sweep 1389.2 (1434.0) -> 1389.1 (1434.0) MB, 1175.0 / 0.0 ms (+ 0.4 ms in 1 steps since start of marking, biggest step 0.4 ms) [allocation failure] [scavenge might not succeed].
1147268 ms: Mark-sweep 1389.2 (1434.0) -> 1389.1 (1434.0) MB, 987.4 / 0.0 ms [allocation failure] [scavenge might not succeed].
<--- JS stacktrace --->
Cannot get stack trace in GC.
FATAL ERROR: MarkCompactCollector: semi-space copy, fallback in old gen Allocation failed - JavaScript heap out of memory
1: node::Abort() [node]
2: 0xdecf4c [node]
3: v8::Utils::ReportApiFailure(char const*, char const*) [node]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
5: v8::internal::MarkCompactCollector::EvacuateNewSpaceVisitor::Visit(v8::internal::HeapObject*) [node]

Your node process has run our of RAM... perhaps you are trying to import many large documents (all of which have to be parsed).  Reduce how many documents you import per batch via --limit

Your node process has run our of RAM... perhaps you are trying to import many large documents (all of which have to be parsed).  Reduce how many documents you import per batch via --limit

I got the same Issue while testing, Need some help:
<--- Last few GCs --->>
91873 ms: Mark-sweep 1391.5 (1437.1) -> 1391.5 (1437.1) MB, 1263.2 / 0.0 ms [allocation failure] [scavenge might not succeed].
93098 ms: Mark-sweep 1391.5 (1437.1) -> 1391.5 (1437.1) MB, 1224.4 / 0.0 ms [allocation failure] [scavenge might not succeed].
94300 ms: Mark-sweep 1391.5 (1437.1) -> 1391.5 (1437.1) MB, 1202.0 / 0.0 ms [allocation failure] [scavenge might not succeed].
2: /* anonymous /(aka / anonymous */) [C:\Users\nbiradar\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver.js:188] [pc=000003FA2F0C603B] (this=00000368019
04381 ,value=000000D6ADD95629 <a Promise with map 000003ED7BE169F9>,key=0000024A87472C31 <String[9]: _idleNext>)
<--- JS stacktrace --->orEachKey) [C:\Users\nbiradar\AppData\Roaming\np...
Cannot get stack trace in GC.
FATAL ERROR: MarkCompactCollector: semi-space copy, fallback in old gen Allocation failed - JavaScript heap out of memory

I got the same issue and need help: I ran the protractor test and got following error
<--- Last few GCs --->>
91873 ms: Mark-sweep 1391.5 (1437.1) -> 1391.5 (1437.1) MB, 1263.2 / 0.0 ms [allocation failure] [scavenge might not succeed].
93098 ms: Mark-sweep 1391.5 (1437.1) -> 1391.5 (1437.1) MB, 1224.4 / 0.0 ms [allocation failure] [scavenge might not succeed].
94300 ms: Mark-sweep 1391.5 (1437.1) -> 1391.5 (1437.1) MB, 1202.0 / 0.0 ms [allocation failure] [scavenge might not succeed].
2: /* anonymous /(aka / anonymous */) [C:\Users\nbiradar\AppData\Roaming\npm\node_modules\protractor\node_modules\selenium-webdriver\lib\webdriver.js:188] [pc=000003FA2F0C603B] (this=00000368019
04381 ,value=000000D6ADD95629 <a Promise with map 000003ED7BE169F9>,key=0000024A87472C31 <String[9]: _idleNext>)
<--- JS stacktrace --->orEachKey) [C:\Users\nbiradar\AppData\Roaming\np...
Cannot get stack trace in GC.
FATAL ERROR: MarkCompactCollector: semi-space copy, fallback in old gen Allocation failed - JavaScript heap out of memory


Frends.Zip.ExtractArchive initial implementation

Create initial implementation for ExtractArchive task, using the community ZIP ExtractArchive task as baseline (https://github.com/CommunityHiQ/Frends.Community.Zip).
In addition:

The task should have its own nuget (as opposed to having a generic zip package)
The task has to have good unit test coverage, as zipping is something that can be automated
No integration tests required besides just testing zipping in unit tests
It would be nice to have some performance indication in the performance test section of the readme.
Add SonarCube
Add workflows

Task has been implemented. CI workflows created for the task.

Please add a task descriptions, otherwise it shows up as "Task Description" when trying to install it in FRENDS


ToDo: Create Visual Indicator For Note Creation

Objective
Create a new header that notifies what current action the user is doing.
Tasks

Create two new header elements

One for adding new tasks
One for editing existing tasks

Write the JavaScript logic that toggles between both headers

This change will happen within public/assets/js/index.js

Potentially change the color of the outline to emphasize a users editing task.

Additionally, when viewing existing tasks, have a header indicating when a users viewing an existing task.

The task of this issue has been met.


Add Firebase CLI

Tool information

Tool name: Firebase CLI (firebase-tools)
Tool license: MIT
Add or update? Add
Desired version: whatever is latest in NPM (https://www.npmjs.com/package/firebase-tools)
Approximate size: 1.79MB
If this is an add request:

Brief description of tool: The Firebase Command Line Interface (CLI) Tools can be used to test, manage, and deploy Firebase projects from the command line.
URL for tool's homepage: https://github.com/firebase/firebase-tools
Provide a basic test case to validate the tool's functionality: firebase help

Virtual environments affected

[ ] Ubuntu 16.04
[ ] Ubuntu 18.04
[x] Ubuntu 20.04
[ ] macOS 10.15
[ ] macOS 11
[ ] Windows Server 2016 R2
[ ] Windows Server 2019

Can this tool be installed during the build?
Yes
Tool installation time in runtime
~40sec
Are you willing to submit a PR?
Yes

Hi @alexleonov-tactiq, there are plenty of actions to setup and use Firebase CLI, that one has a lot of stars github-action-for-firebase, will it fit your needs?
Using actions is recommended way to work with the images https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#using-preinstalled-software

@AlenaSviridenko I'll check it close this week, thanks.
Still, I feel uncomfortable with using an action that only wraps a call to an executable to a security-critical piece of infrastructure without providing any additional value/logic. Firebase CLI is there along with google-cloud-sdk, azure cli and others. The action above just becomes another risk in the supply chain, and adds to the workload - it's a set of scripts plus a docker image that will get access to our production environment.

I am closing this issue, and suggest using action for Firebase CLI.
Please, let me know if you have any concerns. Thanks.

Hi There,
basically I disagree in the result/solution given: Use GitHub-Actions.
if so, why even make these platforms like Ubuntu, Mac, etc.?
My take on these platforms (Like Ubuntu, Mac et.c) at there were super swiss army knives and therefor had many tools preinstalled.
In my case I have a flutter barley. Who will build an android, ios and a Docker / web images. all in all it takes 20-30 just to install tools. That is, before anything is built at all.   So if I have to build / test my app 10-20 times a day, then the use of github actions ends up being very expensive, probably WAY too expensive.
Please reopen this case, or my issue.
And btw. If I can participate, help make these platforms better then I would like to participate, just reach out. :-)
BR,
Martin


Emit binary coded descriptors from dart protoc_plugin

Port of an internal change
These descriptors contain extensions, and thus can be used to reflect over the options given to the descriptor
Fixes: https://github.com/dart-lang/protobuf/issues/341

Please review only the latest commit, as the first 3 commits correspond to other pending PRs

to make analyzer happy, I've also added writing '// ignore_for_file: deprecated_member_use_from_same_package' from file_generator.dart for pbjson files

leading_underscores_test shows that this one needs more work both externally and internally: name generation in writeBinaryDescriptor is not efficient enough, also deprecation warning is confusing

Comparing to an internal version, there are a few extra improvements

generate deprecation warning for message and service descriptors
use exact descriptor field names in deprecation annotation descriptors
disambiguate names and handle leading underscores correctly

Some non-visible changes:

extract common function lowerCaseFirstLetter
more fine-grained control for ignored  lints in generated files

I'll also prepare the same change for a null safety branch and then update internal version as well

Comparing to an internal version, there are a few extra improvements

generate deprecation warning for message and service descriptors
use exact descriptor field names in deprecation annotation descriptors
disambiguate names and handle leading underscores correctly

Some non-visible changes:

extract common function lowerCaseFirstLetter
more fine-grained control for ignored  lints in generated files

I'll also prepare the same change for a null safety branch and then update internal version as well

No - not expected - not sure what is going on.
Trying one more restart.

No - not expected - not sure what is going on.
Trying one more restart.

Rebased on top of enabling github actions, will wait for results and merge when passes

Rebased on top of enabling github actions, will wait for results and merge when passes

Is there a command-line parameter to enable this?  This merge is published in plug-in version 19.3.1.  I am using the latest plug-in version (20.0.1).  The generated Dart code does not contain the custom options in descriptors.  However in generated Python code, I can see the binary encoded descriptors.


Incorrect results for some calls to Moon.time

I am getting some incorrect results when trying to fetch the next full moon time when using Moon.time - public func time inside Moon.swift.
Here is some code
            print ("loop begin")
            for index in 1...20 {
                calculationDate = Calendar.current.date(byAdding: .day, value: index, to: Date())!
                print("calcdate, next fullmoon",calculationDate, Moon(julianDay:JulianDay(calculationDate)).time(of: .fullMoon, forward: true, mean: false).date)
            }
            print ("end loop")

and here is the result
loop begin
calcdate, next fullmoon 2019-02-09 09:01:26 +0000 2019-02-19 15:54:38 +0000
calcdate, next fullmoon 2019-02-10 09:01:26 +0000 2019-02-19 15:54:38 +0000
calcdate, next fullmoon 2019-02-11 09:01:26 +0000 2019-02-19 15:54:38 +0000
calcdate, next fullmoon 2019-02-12 09:01:26 +0000 2019-02-19 15:54:38 +0000
calcdate, next fullmoon 2019-02-13 09:01:26 +0000 2019-02-19 15:54:38 +0000
calcdate, next fullmoon 2019-02-14 09:01:26 +0000 2019-03-21 01:44:00 +0000 <<incorrect
calcdate, next fullmoon 2019-02-15 09:01:26 +0000 2019-03-21 01:44:00 +0000 <<incorrect
calcdate, next fullmoon 2019-02-16 09:01:26 +0000 2019-03-21 01:44:00 +0000 <<incorrect
calcdate, next fullmoon 2019-02-17 09:01:26 +0000 2019-03-21 01:44:00 +0000 <<incorrect
calcdate, next fullmoon 2019-02-18 09:01:26 +0000 2019-03-21 01:44:00 +0000 <<incorrect
calcdate, next fullmoon 2019-02-19 09:01:26 +0000 2019-03-21 01:44:00 +0000 <<incorrect
calcdate, next fullmoon 2019-02-20 09:01:26 +0000 2019-03-21 01:44:00 +0000
calcdate, next fullmoon 2019-02-21 09:01:26 +0000 2019-03-21 01:44:00 +0000
calcdate, next fullmoon 2019-02-22 09:01:26 +0000 2019-03-21 01:44:00 +0000
calcdate, next fullmoon 2019-02-23 09:01:26 +0000 2019-03-21 01:44:00 +0000
calcdate, next fullmoon 2019-02-24 09:01:26 +0000 2019-03-21 01:44:00 +0000
calcdate, next fullmoon 2019-02-25 09:01:26 +0000 2019-03-21 01:44:00 +0000
calcdate, next fullmoon 2019-02-26 09:01:26 +0000 2019-03-21 01:44:00 +0000
calcdate, next fullmoon 2019-02-27 09:01:26 +0000 2019-03-21 01:44:00 +0000
calcdate, next fullmoon 2019-02-28 09:01:26 +0000 2019-03-21 01:44:00 +0000
end loop

I have marked some lines as incorrect above, they show the next full moon as being on the 21st March, whereas the next full moon is actually still to come on 19th February.
I haven't worked out the root cause of the issue yet, whether it is in SwiftAA or AA+, but I thought I would report it here. Cheers.

Hi. Thanks a lot for the bug report. It looks like the algorithms don't switch from one full moon to the next at the right moment. Will look into it.

As far as I can tell, this is now fixed (see 9a0937654a02e0e4ce42bffc00d549a9de7a63b5). And a Unit Test covers it. That was a simple floor/round error. Thanks again for your bug report!
The Pod is also updated.

Thank you to the generous folk who have fixed this. It is indeed resolved for all cases I have tested. Thanks again.


suffix option / multiple colors

It's currently not possible to initialize multiple instances on the same context when having equal terms in multiple instances (removeMark needs to be called first). Demonstration why it's not possible:

Instance 1 search term "lorem ipsum"
Instance 2 search term "lore"

Initially this will work. However, when removing the "m" at instance 1 and reinserting it, jquery.mark is unable to find the term "lorem ipsum" because instance 2 already splitted the text node into a marked element with "lore". jquery.mark only searches inside one text node, not across multiple.
As I don't see any possible solution to workaround this behavior an option "suffix" should be implemented.
This option will add a number at the end of the defined className to allow multiple colors, e.g. myclass-1 and myclass-2.
Possible values:

-1 (default) no suffix will be added
0 a unique suffix number will be added to each marked element, 1–unlimited
[any number > 1] tells jquery.mark until what number a unique suffix should be added until it starts repeating. Example when setting 2:

marked element will receive class myclass-1
marked element will receive class myclass-2
marked element will receive class myclass-1
marked element will receive class myclass-2
...

This would solve #8

After thinking about this again and again, I came to the conclusion that this alone is no satisfactory solution. Reason: When setting separateWordSearch to true it is not clear what should happen. Should the suffix counter increment for each word or just for the entire term?
In my opinion another option in combination to suffix should be available. It should define if the counter should increment for each word or just for the entire term.
So something like that:
var options = {
    suffix: {
        thresold: -1,
        countSingleWords: false
    }
};

I'm closing this for now until someone involved in this discussion, e.g. with a use case.


Partition selector strategy does not use the configured partition count for multiple outbounds

Framework Version:
spring-boot: 3.0.2
spring-cloud: 2022.0.1
Issue:
We have a reactive function, which consumes messages from one topic and sends them to N topics based on some conditions. Additionally, we also have a custom key extractor and partition selector configured for each of the output bindings. However, the number of partitions of the outbound topics is different which does not seem to be reflected in the custom partition selector.
I did some digging and it seems only the binding properties of the first outbound are being used when creating the PartitionAwareFunctionWrapper, which are then passed to the PartitionHandler invoking the overriden selectPartition(Object key, int partitionCount) method with incorrect partition count.
Sending the message to the incorrect partition results in TimeoutException.
Code snippet from org.springframework.cloud.stream.function.FunctionConfiguration
if (!CollectionUtils.isEmpty(outputBindingNames)) {
	BindingProperties bindingProperties = this.serviceProperties.getBindings().get(outputBindingNames.iterator().next());
	ProducerProperties producerProperties = bindingProperties == null ? null : bindingProperties.getProducer();
	if (producerProperties != null) {
		function.setSkipOutputConversion(producerProperties.isUseNativeEncoding());
	}
	functionToInvoke = new PartitionAwareFunctionWrapper(function, this.applicationContext, producerProperties);
}

Expected behavior:
I would expect the framework to honor the configured partition count for each output binding.

How do you send them to N topics? Could you provide us with a small sample application where we can see the issue?

Hi @sobychacko
I have created a sample application showing the issue: https://github.com/angelbeshirov/spring-cloud-partition-selector-issue
The reactive function reads random numbers from one topic test-input and sends the even numbers to output-even and the odd ones to output-odd. The output-even topic has 10 partitions and output-odd has 5 partitions.
Publishing a message to output-odd fails with:
Exception thrown when sending a message with key='null' and payload='byte[30]' to topic output-odd and partition 9:
org.apache.kafka.common.errors.TimeoutException: Topic output-odd not present in metadata after 60000 ms.

@angelbeshirov The issue is now resolved via the commit mentioned ^^. You can try the fix on the latest snapshots.

Thanks, @sobychacko, I can confirm the issue is resolved in 4.1.0-SNAPSHOT.

Thanks @angelbeshirov for confirming.


Reduce VRAM usage

VRAM usage has went up a lot since version 0.7, because of new tilesets. It was 0.8GB in version 0.7 and now it's 1.4GB in version 0.8. Should work on reducing it where possible.
For example, high VRAM usage causes crashes for some players: #273
Texture compression

None of the compression options are of any help.

Lossless - this is the option we currently use for tilesheets.
Lossy - doesn't reduce VRAM usage, only affects the size of the executable/pckfile.
VRAM Compressed - would reduce VRAM usage by a lot BUT, it's not usable for textures used by tilesets. You will see Godot output errors like this "Can't set_pixel() on compressed image, sorry".
VRAM Uncompressed - same VRAM usage as Lossless.
Basis Universal - same as VRAM Compressed, not usable for tilesets.

Online tools like TinyPNG is same thing as Lossy compression - only reduces size on disk, doesn't reduce VRAM usage.
Empty space in tilesheets
We currently have a lot of unused space in tilesheets.

The first thing we should do is remove equal amounts of space above all tile, using a script. This way, the tilesizes will still be equal to each other and setting up tilesets will still be easy.
Later we can look into packing tilesheets tightly - but that will make setting up tilesets more complicated.
Also some tilesheets have empty tiles - should remove them.

Packing tilesheets
Decoration tilesheets can be packed like creep spritesheets:

Use PackSpriteSheet.gd script for that. Note that you need to fix the tileset after doing this.
Tileset duplication
We have two tilemap nodes - "Map" and "Decorations". We need two separate tilemaps because "Map" has tile size set to 256x128 and "Decorations" has tile size set to 64x32. This is so that tiles can be placed on the grid in "Map" but you can also do more flexible positioning of decoration tiles in "Decorations" tilemap.
When two different tilesets reference the same png, the png will be stored in VRAM twice.
For example, the  "Floor.png" has two tilesets and it's memory usage is doubled.

Should avoid situations like this where possible.

Disabling texture padding reduced VRAM usage by 50%: ddea2fd3eb449ee57b14db2827a4986489664510
Before: 1.5GB
After: 0.75GB
Godot documentation says:

Disabling this setting might lead a small performance improvement
https://docs.godotengine.org/en/stable/classes/class_tilesetatlassource.html#class-tilesetatlassource-property-use-texture-padding
But for some reason, in our case it leads to huge performance improvement.

Tried out prerendering the background here:
https://github.com/Praytic/youtd2/pull/276
This reduced VRAM usage: 0.75MB->0.4MB
Breakdown of current VRAM usage (using prerendered background):

30MB prerendered background
30MB foreground tilemap
10MB buildable area
20MB bottom menu bar
10MB upper bar
40MB horadric menu theme
40MB horadric menu + unit menu
10MB shaders
85MB creep sprites

10MB air

25MB mass

25MB champion

30MB normal

55MB boss

50MB preloaded scenes in Globals.gd
30MB unaccounted, I didn't spend time on figuring out what uses this portion

Enabled VRAM compression for creep atlases:
9199a8867debbbc8a0face2115d8146c9fe34b5c
234bc2692f83c03e61d0381e7c7d99c5a0bf07cf
This reduced VRAM usage by creep sprites from 145MB to 45MB. Almost 4 times reduction, as expected.

Had to disable VRAM compression for creep atlases - does not work on Web.
Tried to fix it but couldn't figure it out.
With "High Quality" enabled you get "failed to load resource" errors in console and sprites don't draw at all.
With "High Quality" disabled resources appear to load but creep sprites draw only one frame and then disappear.

Closing this because the biggest sources of VRAM usage were fixed and the game is back to VRAM usage levels from 0.7.
Can tackle specific VRAM usage sources separately later, as needed.

I've looked into Basis Universal again and it seems usable.
These errors appear to be harmless:

Basis universal cannot unpack level 1.

There is one caveat.
When using Basis Universal for textures used in tilesets, you need to disable "Use Texture Padding" option. Otherwise, the texture will appear empty and you will get spammed with console errors.


Quartz Scheduler Executes Multiple Times Despite Fixed Interval and Date Configuration

We have encountered an issue with the Quartz Scheduler where jobs are executing multiple times even after being configured for a fixed time interval and specific date. This behavior is causing duplicate job executions and potential data inconsistencies.
Environment:

Quartz Scheduler Version: 2.3.0
Database: MongoDB

Scheduler Configuration:
Here are the properties used to instantiate the SchedulerFactoryBean:
properties.setProperty("org.quartz.threadPool.threadsInheritContextClassLoaderOfInitializingThread", "true");
properties.setProperty("org.quartz.jobStore.class", "com.novemberain.quartz.mongodb.MongoDBJobStore");
properties.setProperty("org.quartz.jobStore.mongoUri", "mongoUri");
properties.setProperty("org.quartz.jobStore.dbName", "mongoDatabase");
properties.setProperty("org.quartz.jobStore.collectionPrefix", "quartz-entry");
properties.setProperty("org.quartz.threadPool.threadCount", "5");

Trigger Configuration:
The trigger is configured as follows:
Date startDate = new SimpleDateFormat("yyyy-MM-dd").parse(req.getSchedule().getStartDate());
Date endDate = new SimpleDateFormat("yyyy-MM-dd").parse(req.getSchedule().getEndDate());

SimpleTrigger cronTrigger = TriggerBuilder.newTrigger()
        .withIdentity("trigger", req.getId() + "_" + req.getName())
        .startAt(startDate)
        .endAt(endDate)
        .withSchedule(simpleSchedule()
                .withIntervalInMinutes(req.getSchedule().getIntervalMinutes())
                .withMisfireHandlingInstructionFireNow()
                .repeatForever())
        .build();

Expected Behavior: The job should execute only once according to the specified schedule, without any duplicate executions.
Actual Behavior: Jobs are executing multiple times.
The issue occurs infrequently and has not been observed in the local environment.
Any assistance in identifying the root cause of this issue or advice on debugging and preventing these rare occurrences of multiple executions would be greatly appreciated. If anyone has encountered similar behavior with Quartz Scheduler or can suggest troubleshooting steps, please share your insights.

Hi!
Not aware of any similar problem reports, and thinking what could cause the issue makes me think of transaction isolation issues as a likely culprit.  Then looking at your config, you have this:  properties.setProperty("org.quartz.jobStore.class", "com.novemberain.quartz.mongodb.MongoDBJobStore");    -- which means the thing you have doing the work around transactions is not something that is part of Quartz.


Kerberos client error

Kerberos authentication fails with error:
{"level":"info","msg":"Kerberos client error: [Root cause: KRBMessage_Handling_Error] KRBMessage_Handling_Error: AS Exchange Error: AS_REP is not valid or client password/keytab incorrect \u003c KRBMessage_Handling_Error: clock skew with KDC too large. Greater than 300 seconds","source":"sarama","time":"2020-03-21T14:03:55Z"}
Environment variables used:
ENV VERSION 1.0.0
ENV KAFKA_BROKERS xx144eza:6668,xx144ey9:6668
ENV KAFKA_SASL_ENABLED true
ENV KAFKA_SASL_GSSAPI_AUTH_TYPE KEYTAB_AUTH
ENV KAFKA_SASL_GSSAPI_KEY_TAB_PATH /app/kerberos/testuser.keytab
ENV KAFKA_SASL_MECHANISM GSSAPI
ENV KAFKA_SASL_GSSAPI_SERVICE_NAME=kafka
ENV KAFKA_SASL_GSSAPI_REALM=DTA.KLM.COM
ENV KAFKA_SASL_GSSAPI_KERBEROS_CONFIG_PATH=/app/kerberos/krb5.conf
ENV KAFKA_SASL_GSSAPI_USERNAME testuser
ENV LOG_LEVEL debug
client and its password are for sure correct:
keytab and krb5.conf do work on system where they are used by kinit and/or from java application using JAAS config.
clock skew?? : From the same container I can authentiate using kinit
Also tried to use USER_AUTH: instead of keytab and this gave the same error
Could this be an issue in the underlying sarama library in kerberos GSSAPI handling?
Any help would be very much appreciated.

Hi @alexanderdehes ,
unfortunately I don't have a kerberized Kafka cluster to test against, hence that was merged untested. I am not well versed with Kerberos either, but it looked easy to implement because I just needed to pass all options to sarama as shown here: https://github.com/cloudworkz/kafka-minion/blob/master/kafka/connection_helper.go#L74-L87
Do you see any obvious issues here?
Other than that we could probably submit an issue at sarama and ask them for help. I found one who mentioned the same root cause KRBMessage_Handling_Error: https://github.com/Shopify/sarama/issues/1519

Hi @weeco I have not seen any obvious issues in the kafka minion part so I also suspect that the problem is either in sarama or maybe even in jcmturner/gokrb5. In gokrb5 the actual error is raised (complaining about clock skew).

Hello !
I get same error if try kerberos:
"Kerberos client error: [Root cause: Decrypting_Error] KRBMessage_Handling_Error: AS Exchange Error: AS_REP is not valid or client password/keytab incorrect \u003c Decrypting_Error: error decrypting EncPart of AS_REP \u003c Decrypting_Error: error decrypting AS_REP encrypted part

@alexanderdehes @Sergeyemcev Have you guys figured it out or created an issue in the upstream repository (sarama) ?

I just pushed the code for v2.0.0 (still to be tagged/released). This release use a new kafka client which also has Kerberos support. Do you mind trying it with this library? I'll close this issue for now, but I'm happy to reopen if the issue remains


Each Player’s current total bet amount should be visible to all other players.

closed commit: a225372a9476abe4271285d78539b37dd05fd345


Write unit tests for BeginSkill

Fixes #3879
Description
This PR adds two BeginSkill action tests using .dialog files and required middleware, increasing code coverage from 0% to 78%.
One case tests connecting to a mocked skill via BeginDialogAsync while the other one tests the ContinueDialogAsync and EndDialogAsync methods specifically.
Specific Changes

Adds middleware classes SetSkillBotFrameworkClientMiddleware, MockSkillBotFrameworkClient, and SetSkillConversationIdFactoryBaseMiddleware and their references in TestScript
Adds two .dialog files and their corresponding ActionTests references.

Testing
The following image shows the coverage increments achieved:

@ceciliaavila  thank you for adding these tests. Can you add information about what the coverage rate was before this change?

Hi @EricDahlvang we are reviewing the possibilities that you commented and also reviewing the failing pipelines in order to solve it. Thanks!

Hi @EricDahlvang , we reviewed your requested changes.
We located these middleware classes in libraries/Microsoft.Bot.Builder.Dialogs.Adaptive.Testing following the current pattern.
We agree that having all testing exclusive files within tests/Microsoft.Bot.Builder.Dialogs.Adaptive.Tests makes sense, but to relocate them there we would have to refactor TestScripts.cs first (as you suggested).
Refactoring TestScript's middleware assignment would affect references preexisting middleware, which would make it out of scope for the issue we are addressing in this PR.
Would you like us to do it anyways? If that so, it'd be a good idea to create a new issue to keep track of this change and do it on a separate PR (we'd need to re-estimate this work to add it to our iteration backlog).


Make Void mean Nil, but keep Void* as such

This is yet another way to solve #2673, different from #2687
The changes are:

When Void is used as a return type, either in a proc, a method or a lib fun, it is replaced by Nil
Void* (or Pointer(Void)) is still allowed with the same meaning as C: a pointer to some bytes without an unknown meaning.

What matters most for this PR is to get rid of the Void type, and use only Nil to mean "absence of a value". In Crystal land one should use Nil, never Void, and once (and if) we merge this, after the next release we can replace all usages of Void with Nil.
However, when writing C bindings, it's common to use Void*: it's easier to copy from a C header and it's also how the type is called in C. But it should only be used for interfacing with C, in Crystal one can use UInt8* to mean "a pointer to raw bytes" (though one usually uses slices instead).
When writing C bindings it's also common to use Void as a return type. One can do that, but that Void gets automatically translated to Nil, because that's the way to represent that concept in Crystal.
In this way, a Void value can never be produced, with one exception: reading from a Pointer(Void). But one never reads from such pointer because it has no meaning, so this shouldn't happen.
In a way, this solution is closer to the original intention of #2673, which was to replace Void with Nil. One can simply program thinking that Void is an alias of Nil, but one can still use Void* in C bindings, and that's different from Nil*. A later minor change that we plan to do is to allow any pointer to match an argument of type Void* (there are a few redundant .as(Void*) casts in the code right now), but that shouldn't apply for Nil*, so that's why Void isn't exactly an alias of Nil.
In any case, for a user that doesn't have to deal with C bindings, Void will never appear or bother.
Another thing this PR does is to allow marking a method as returning Nil, and then the compiler won't check the return type:
def foo : Nil
  1 + 2
end

foo # => nil

One can of course just return nil at the end, but in this way one is telling the compiler, and the user reading the docs or the method signature, "this method doesn't return anything useful".
This change affects issue #2672, it now behaves like this:
class Foo
  # This is Proc(Nil)
  @action : ->

   def initialize
    # OK, this is Proc(Nil)
    @action = ->{ }

    # Error, this is Proc(Int32)
    @action = ->{ 1 }

    # OK, this is also Proc(Nil)
    @action = ->{ 1; nil }
   end
end

But the main problem in #2672 was that one can't easily produce a value of type Void, so would have to use Proc(Void).new { ... }. If one needs a proc to return nil one can simply add nil at the end of it.
I personally like this solution more than #2687, because it doesn't introduce a new type/concept (void), and it makes nil be used in more places where previously void was used/returned.

Love this, thank you!

I don't really like the idea that Void doesn't exist, but actually does.
Would this be acceptable in Crystal code?
def foo : Void
  1 + 2
end

foo # => nil

If the answer is "no", then it is odd that Void is mostly restricted to lib land, but there is Void* available. Perhaps there could be Byte*, where Byte behave just like an UInt8, but isn't implicitly convertible to it. This is not really an alias, as it means another concept. to_s could use hex encoding for instance. Not sure thought.

It's really confusing to let Void behave in such an ambiguous way - on one hand it is an alias of Nil, on other hand Pointer(Void) is not Pointer(Nil). For those who decide to intentionally use Void as return type in their API there might be surprising to see nil-related errors instead of void-related.
Moreover, this code looks a bit weird because the coder's idea is unclear
def foo : Nil # <- What on earth does it mean?!
  1 + 2
end

foo # => nil

Given Void, it'd have been more sensible
def foo : Void # <- Coder doesn't want this value be used.
  1 + 2
end

foo # => void

I have an alternative proposal:

We remove Void: the Void type doesn't exist anymore
We do allow Void to be used as a return type in lib functions, and this means Nil. Void can't be used outside a lib
We do allow Void* to be used inside lib, and it means UInt8*

Points 2 and 3 are there because:

Void is a concept in C, so it's natural to have it in lib declarations
if we don't keep it, all code out there immediately breaks

In short, only inside a lib, Void is an alias of Nil, and Void* is an alias of UInt8*. Outside lib, Void doesn't exist.
Because the language is bootstrapped we can't just remove Void outside lib because all code stops compiling. So, we define a top-level alias Void = UInt8 for one version, and in the next version we remove it, after replacing all Void with UInt8.
I started implementing this, but stopped to comment here. I'm still not sure about this either. For example, consider this function:
lib LibFoo
  fun some_fun(string : UInt8*, custom_data : Void*)
end

If we eventually remove Void from the language, or make Void be an alias of UInt8 inside libs, that will be the same as:
lib LibFoo
  fun some_fun(string : UInt8*, custom_data : UInt8*)
end

This is confusing. UInt8* almost always means "a buffer to some bytes, usually ascii or utf-8 or something that's a string", while Void* means "a pointer to something, not necessarily text". I believe this is the reason void* exists in C and one just doesn't use char*. Another reason is that any pointer should be implicitly convertible to Void*, while the same is not true for UInt8*.
Now, to answer some specific concerns:
@lbguilherme Yes, we could introduce a different type Byte, that would behave like UInt8, so we'd use Byte* instead of Void*. However, since in C this concept already exists and it's called void, I don't see why we need to rename Void (Byte* is basically the same a Void*)
@pkorotkov Void belongs to C. If you do:
def foo : Void
  1 + 2
end

my reply is: Void belongs to C. Don't use Void in Crystal code (in fact, I will make such code not compile, and suggest using Nil instead). The concept for "nothing" in Crystal is Nil. So this:
def foo : Nil
  1 + 2
end

makes sense. Maybe not in this silly example, because such code won't really exist as it doesn't have side effects, but this:
def print(*objects : _) : Nil
  STDOUT.print *objects
  STDOUT.flush
end

makes more sense. Basically I'm saying "I don't care about the type of the last expression in the method, just make it return nil".
I'd say that using Void as a return type makes more sense if you come from other languages. If you, for example, only know Ruby, Void doesn't exist for you and you'd choose Nil for this.
One more thing to take into account is that it doesn't really matter that Void can still be used in regular Crystal code, outside lib. It's there so we can interface with C more easily. Yes, it makes the type system a bit weird, that Void is Nil, and Void* is more like UInt8*, but it doesn't matter, that's because that's how it behaves in C. For example, consider a program like sidekiq. I searched "Void" and couldn't find a single reference to it. That is, Void doesn't happen in high-level code, only when interfacing with C.

@asterite on the alternative proposal
This is the most apt compromise between C friendliness and Crystal way of programming, as I see it. I'd vote up.
A tiny question just to make sure I didn't lose the thread: What about the idea of merging Nil with other types? Is this snippet supposed to work as written?
def a : Int32
  10
end

def b : Nil
  a
end

b # => nil

@pkorotkov That last snippet works, because when you specify Nil as a return type it forces the method to return nil. However, merging nil with other types gives nilable types, not just nil.

I'll soon update this PR with something that travis likes :-)
Another thing I just realized is that there's actually no way to produce a void value. One can do:
ptr = Pointer(Void).malloc(1)
ptr.value # => nil

I originally thought that would return a Void, because the method is marked as returning T, and T = Void, but since a method that returns Void is considered as a method returning Nil by the compiler, it effectively returns nil. This is really good, as reading from a pointer value has no useful value, and nil means exactly that :-)

@asterite yeah, right. Well, this all sounds great now.

I believe this is the reason void* exists in C and one just doesn't use char*.

Not really. void* was introduced more as the idea of a generic pointer that can be implicitly cast to anything. For instance:
int* m = malloc(sizeof(int));

If malloc returned char*, then the compiler would likely warn about the implicit conversion, since in most other cases, it wouldn't be safe according to strict aliasing rules, and C memory management sucks. However, since malloc returns void*, the compiler doesn't give out a warning, as casting around void* isn't an issue according to the standard.
In a way, it's really just what happens when your only memory allocation function returns a bunch of bytes and you don't have generics.

Updated this PR once more, this time to allow implicit conversion of any pointer to Void* (only for lib functions and struct fields). This means one needs to type .as(Void*) less, so it agains makes explicit usages of Void* less frequent. And this rule is the same as in C, which allows implicit casting between void*, as @kirbyfan64 says.

A final thing I forgot to say before merging is that this also changes the codegen: now functions that return nil are generated as returning void, which should maybe make things easier for LLVM, or at least generate cleaner code when debugging.

Just à single case : we sometimes have to cast to Void* in crystal code to pass data to C. I suppose we'll just have to cast to LibC::Void*? A bit less elegant, but acceptable, we don't usually call C anyway.

@ysbaddaden One can still do value.as(Void*), Void is a top-level type just like before. However, this change makes it so that a single Void type (not a Void*) is the same as Nil. The single Void type was only used for function return types and proc returns types, so now it means they return nil.

Oh. I misread. I understood that you removed the Void type altogether and only kept an alias for lib. Good, then.

First off: Hurray for def x : Nil ignoring all return paths, now I can ditch "nil-filling" in Onyx B-)
Some thoughts:
Void also should not allow pointer arithmetic, but I guess that is moot now.
Nil does have a value: nil, the way I see it. Void is often defined "almost like Unit, but not exactly" (in C(++)), which is often described as an empty tuple. Nothing, is a completely value-less type (whih would be reasonable for side-effects only "functions").
Some thoughts at low-level: If I define def puts(x) : Nil now, does it shuffle a 0 on the stack/regs at all, or is it a Nop/Black Hole when the return value is not used, and/or is the value nil passed to assignee if used with a Nil-ret func? I mean:
puts "foo"  -- is this "basically Nothing" in low-level?
a = puts "foo"  -- is this simply a = nil; puts... -- again with Nil being Nothing in practise?

The pro of a true Nothing, or Void, would be that it errors if it's ever used as an expression, since it has no value, only a type (well, Void has the empty tuple really). Maybe #2687 does have some points?
I hope this issue can be discussed a little more before nailing for eternity?

Is there any way now to declare a proc that returns any type (when we don't care about the return type). Just like mentioned in #2672.
If no, would it be possible to have something like this:
class Foo
  @action: (-> _)

  def initialize
   # OK, this is Proc(Nil)
   @action = ->{ }

   # Ok, this is Proc(Int32)
   @action = ->{ 1 }
  end
end

The following compiles now, which it previously didn't:
def foo(&block : Int32 ->)
  p yield 1
end

def bar(&block : Int32 -> Nil)
  p yield 2
end

def baz(&block : Int32 -> Void)
  p yield 3
end

foo do |i|
  p i
  true
end

foo do |i|
  p i
  :blub
end

bar do |i|
  p i
  true
end

bar do |i|
  p i
  :blub
end

baz do |i|
  p i
  true
end

baz do |i|
  p i
  :blub
end

The output is
1
true
1
:blub
2
true
2
:blub
3
true
3
:blub

The following too:
def foo(&block : Int32 ->)
  p block.call 1
end

def bar(&block : Int32 -> Nil)
  p block.call 2
end

def baz(&block : Int32 -> Void)
  p block.call 3
end

foo do |i|
  p i
  true
end

foo do |i|
  p i
  :blub
end

bar do |i|
  p i
  true
end

bar do |i|
  p i
  :blub
end

baz do |i|
  p i
  true
end

baz do |i|
  p i
  :blub
end

With the following output:
1
nil
1
nil
2
nil
2
nil
3
nil
3
nil

The first case is perhaps a bit unexpected and should always return nil too, WDYT @asterite ?

@jhass Yes, probably. When you specify a block restriction but don't invoke it via block.call the logic is a bit different right now, the block body isn't forced to be nil in those cases.

@veelenga what you want is basically Object: https://github.com/crystal-lang/crystal/issues/2733

@asterite looks promising. Would love to have such possibility!
At the moment i solved it in the following way:
class Foo
  @action: (-> Nil)

  def initialize(action : (-> _))
    @action = -> { action.call; nil }
  end
end

Foo.new -> {} # ok
Foo.new -> { 1 } # ok


feat(meta): support inject meta info from env

What type of PR is this?

feat
What this PR does / why we need it (en: English/zh: Chinese):

en:
zh: 支持根据环境变量自定义 xds node 的 meta 信息
Which issue(s) this PR fixes:

lgtm


picker cropped text

hello i have big data (9000 Item) and  using  native base picker for select , when i open picker my inner text was cropped (android)
can anyone help me ?
see this pic
how i can break text at new line  if text doesn't fit?
this is picker code
`
                    <Picker iosHeader="Select one"
                            mode="dropdown"
                            selectedValue={this.state.query[input.model]}
                            onValueChange={ e => this.query(input.model, e) }>
                        {array.map((val, index) => (
                            <Text key={index}
                                  label={ (val.name) ? val.name : val}
                                  value={(val.id) ? val.id : val}/>
                        ))}
                    </Picker>
                </View>`

Seem to work fine.
Check the latest of NativeBase(v2.1.5)

Closing this due to inactivity.
Also the above solution should work

I have the same problem, I try with your example but it dont work for me, could you add your style.container or full code example?

I have a same issue on android, picker text gets cropped but works fine on iOS. The above solution fails on android. Any solution for this issue?

yes same issue on Android... @ReactFan23 reported the issue on Android and @shivrajkumar attached iOS screenshot and said it seems to work fine! 😄

Any updates on this fix for android?


dice game

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
dic egame which decides which player won
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

Is your feature request related to a problem? Please describe.
I can complete this project,plz assign it to me
Describe the solution you'd like
By using HTML,CSS,JS
Additional context

Game projects not accepted now.


Added resync button when sync state is already successful

This is a temporary solution

Changes added with d7838e4 commit, just written differently.
Thank you for the contribution,
Matej, Neuralab


Restricts cpu yield instructions a little.

adding clobber for ARM and preventing older 32 bits chips not supporting this instruction.

Thanks so much David! I added you to the mimalloc readme -- let me know if that is ok :-)


[GLUTEN-6202][VL] No extra SortExec needed before WindowExec if window type is sort

What changes were proposed in this pull request?
When spark.gluten.sql.columnar.backend.velox.window.type is set to sort, there should be no extra SortExec before WindowExec in plan.
(Fixes: #6202)
How was this patch tested?
Mannual Test and UT

cc @JkSelf

And CH CI failed https://opencicd.kyligence.com/job/gluten/job/gluten-ci/10302/, please also take a look. Thanks.


Raw-rs: run tests in parallel

Runs the tests in parallel by default in the CI which will be useful when more images are added.
If the older sequential way is required (especially while print based debugging), enabling RAW_RS_TEST_RUN_SEQUENTIALLY env variable will make it sequential again.

Are the tests even executed in the ci? Don't see them in the logs

The code itself looks fine but I am somewhat doubtful this currently improve ci speeds but as a future proofing measure that should be fine

Are the tests even executed in the ci? Don't see them in the logs

running 2 tests
test extract_data_from_dng_images ... ignored
test test_images_match_with_libraw ... ok

test result: ok. 1 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 25.29s

Here are the logs from #1954 when no optimization were present
running 2 tests
test extract_data_from_dng_images ... ignored
test test_images_match_with_libraw has been running for over 60 seconds
test test_images_match_with_libraw ... ok

test result: ok. 1 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 140.02s

So the time went from 140s to 25s in the CI.

@TrueDoctor I have modified the code in compile_demo_art.rs wherein I used cfg_attr instead of attr. It does compile correctly and there will be no need for the drop function too.


Docker version show blank page

@robinwhittleton It seems that launching the website build via docker stopped working properly; it had worked flawlessly before for me. Does the dockerfile need updating?

I’ve just reinstalled it from scratch and it seems to be working fine. Are you getting any error messages_

Let me try rebuilding the entire containing one more time.

I had the same problem lately, after moving to a new laptop and switched to serving the site locally with Apache.

Did it report any errors? It’s working fine for me, so it’s difficult to debug without knowing what the problem is.

I saw a PR_END_OF_FILE_ERROR in Firefox before reloading it a bit and coming up with nothing. It worked fine on my last laptop.

I've just cloned the latest version, run docker build and when I docker run (instructions from the repo) I get a PR_END_OF_FILE_ERROR in Firefox, and the following in the Docker logs:

Click for trace

Generating a RSA private key
..++++
..............................................................++++
writing new private key to '/standardebooks.org/web/config/ssl/standardebooks.test.key'
-----
Loading composer repositories with package information
Warning from https://repo.packagist.org: Support for Composer 1 is deprecated and some packages will not be available. You should upgrade to Composer 2. See https://blog.packagist.com/deprecating-composer-1-support/
Info from https://repo.packagist.org: #StandWithUkraine
Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.
Problem 1
- Installation request for php-webdriver/webdriver 1.12.1 -> satisfiable by php-webdriver/webdriver[1.12.1].
- php-webdriver/webdriver 1.12.1 requires ext-curl * -> the requested PHP extension curl is missing from your system.
To enable extensions, verify that they are enabled in your .ini files:
- /etc/php/7.4/cli/php.ini
- /etc/php/7.4/cli/conf.d/10-opcache.ini
- /etc/php/7.4/cli/conf.d/10-pdo.ini
- /etc/php/7.4/cli/conf.d/15-xml.ini
- /etc/php/7.4/cli/conf.d/20-apcu.ini
- /etc/php/7.4/cli/conf.d/20-calendar.ini
- /etc/php/7.4/cli/conf.d/20-ctype.ini
- /etc/php/7.4/cli/conf.d/20-dom.ini
- /etc/php/7.4/cli/conf.d/20-exif.ini
- /etc/php/7.4/cli/conf.d/20-ffi.ini
- /etc/php/7.4/cli/conf.d/20-fileinfo.ini
- /etc/php/7.4/cli/conf.d/20-ftp.ini
- /etc/php/7.4/cli/conf.d/20-gd.ini
- /etc/php/7.4/cli/conf.d/20-gettext.ini
- /etc/php/7.4/cli/conf.d/20-iconv.ini
- /etc/php/7.4/cli/conf.d/20-intl.ini
- /etc/php/7.4/cli/conf.d/20-json.ini
- /etc/php/7.4/cli/conf.d/20-mbstring.ini
- /etc/php/7.4/cli/conf.d/20-phar.ini
- /etc/php/7.4/cli/conf.d/20-posix.ini
- /etc/php/7.4/cli/conf.d/20-readline.ini
- /etc/php/7.4/cli/conf.d/20-shmop.ini
- /etc/php/7.4/cli/conf.d/20-simplexml.ini
- /etc/php/7.4/cli/conf.d/20-sockets.ini
- /etc/php/7.4/cli/conf.d/20-sysvmsg.ini
- /etc/php/7.4/cli/conf.d/20-sysvsem.ini
- /etc/php/7.4/cli/conf.d/20-sysvshm.ini
- /etc/php/7.4/cli/conf.d/20-tokenizer.ini
- /etc/php/7.4/cli/conf.d/20-xmlreader.ini
- /etc/php/7.4/cli/conf.d/20-xmlwriter.ini
- /etc/php/7.4/cli/conf.d/20-xsl.ini
- /etc/php/7.4/cli/conf.d/25-apcu_bc.ini
You can also run php --ini inside terminal to see which files are used by PHP in CLI mode.
Enabling site standardebooks.test.
To activate the new configuration, you need to run:
service apache2 reload

Starting Apache httpd web server apache2                                                            *
The apache2 configtest failed.
Output of config test was:
AH00526: Syntax error on line 278 of /etc/apache2/sites-enabled/standardebooks.test.conf:
Invalid command 'XSendFile', perhaps misspelled or defined by a module not included in the server configuration
Action 'configtest' failed.
The Apache error log may have more information.
Restarting PHP 7.4 FastCGI Process Manager php-fpm7.4                                       [ OK ]

I just added a line to the dockerfile in 42b92bab286fc2096c5db427a9451df5f35309a9 that should fix that.

That's a bit better, now I get the certificate warning and then an empty page after I click through. Nothing interesting in the console except warnings about Strict-Transport-Security and Quirks mode.

OK, now I’m seeing the same issue, with a new build based on colima. Lets me debug it at least 🙂

Could people try the Docker instructions again? Alex has merged a patch that should allow composer to download the required packages to get the site running.
(I’d previously managed to install them into the site through macOS, so I hadn’t clocked that the composer install inside the Docker VM was failing.)

Closed due to inactivity


Enforce expression body for single-line methods

In the Kotlin style guide, it says to prefer expression body style for single-line methods:

So thought it would be nice to have a rule for it.  It does say "prefer", so perhaps this was omitted because there are legitimate cases to avoid it, but wanted to ask about it.  Was a rule for this left out intentionally?

Would definitely fit experimental ruleset I think

I don't know if it was left out intentionally, but the Android style guide has the same recommendation: https://developer.android.com/kotlin/style-guide#expression_functions
Feel free to open a PR.

Since this is a suggestion rather than a rule, I'm going to close this.


[fix][build] Client modules should be built with Java 8

This fixes #19982.
Motivation
Client modules are promised to be compatible with Java 8.
Modifications
Add --releaase 8 for these two remaining client modules.
Verifying this change
Locally tested.
Before:
file target/classes/org/apache/pulsar/client/impl/auth/AuthenticationAthenz.class
target/classes/org/apache/pulsar/client/impl/auth/AuthenticationAthenz.class: compiled Java class data, version 61.0

After:
file target/classes/org/apache/pulsar/client/impl/auth/AuthenticationAthenz.class
target/classes/org/apache/pulsar/client/impl/auth/AuthenticationAthenz.class: compiled Java class data, version 52.0 (Java 1.8)

I don't know if we can add a CI step to ensure it.
Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

[ ] Dependencies (add or upgrade a dependency)
[ ] The public API
[ ] The schema
[ ] The default values of configurations
[ ] The threading model
[ ] The binary protocol
[ ] The REST endpoints
[ ] The admin CLI options
[ ] The metrics
[ ] Anything that affects deployment

Documentation

[ ] doc
[ ] doc-required
[x] doc-not-needed
[ ] doc-complete

Matching PR in forked repository
PR in forked repository:

Hi @lhotari & @eolivelli!
The changeset for updating the broker compiler version is for profile integration-test-java8. But I agree I can revert the change as it won't hurt.

Updated. Please give another look.

Codecov Report

Merging #19991 (29f99a8) into master (68c10ee) will decrease coverage by 0.08%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master   #19991      +/-   ##
============================================
- Coverage     72.89%   72.82%   -0.08%
+ Complexity    31619    31592      -27
============================================
  Files          1861     1861
  Lines        137356   137362       +6
  Branches      15117    15118       +1
============================================
- Hits         100131   100039      -92
- Misses        29271    29366      +95
- Partials       7954     7957       +3

Flag
Coverage Δ

inttests
24.44% <ø> (+<0.01%)
:arrow_up:

systests
25.00% <ø> (-0.04%)
:arrow_down:

unittests
72.11% <ø> (-0.07%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.
see 76 files with indirect coverage changes

Hi @Technoboy-! This can be a release blocker to 2.11.1. Since you're the RM for this release, I'd like to let you know this patch and ask if you can cherry-pick it to branch-2.11 and include it in 2.11.1.

Hi @Technoboy-! This can be a release blocker to 2.11.1. Since you're the RM for this release, I'd like to let you know this patch and ask if you can cherry-pick it to branch-2.11 and include it in 2.11.1.

Thanks, cherry-picked to branch-2.11.


[BUG] Wrong URL for downloading chromedriver

Description
There seems to be an issue with fetching chromedriver from the following URL:
https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/122.0.6261.94/win32/chromedriver-win32.zip
The result is:
urllib.error.HTTPError: HTTP Error 404: Not Found
Additional Information

OS: Windows 10
Driver: chromedriver
Version: 2024-02-28 08:16:45,997 INFO Latest version of chromedriver: 122.0.6261.94

Additional context
Found out today when updating the drivers on my machine.
I think they moved the location drivers are stored.

Update via pip: Works perfectly!
Update of selenium driver: Works perfectly!
Works like a charm! Tank you for your quick fix!


Translate concurrent mode adoption

Geri bildirimlere müteşekkirim :)

Thanks for the translation. I posted some comments. Please check them! :)

Thanks, @alioguzhan . I've changed the revision points.


Update README.md

Add iterm to terminals collection

no, thanks for iTerm
reactjs is great but this list has a different purpose


nixpkgs-24.05-darwin blocked due to capacity issues in previous staging cycle

The nixpkgs-24.05-darwin channel is 3 weeks behind today because it was excluded from the last staging-next-24.05 cycle due to capacity issues around the darwin infrastructure.
It is planned to catch up with the next staging-next-24.05 cycle, which should start sometime this weekend.
The communication around this was bad, and we need to do better in the future. I'm sorry.

If I look right, the oldest missing changes were staged on November 7, so that's still not too bad IMHO at this point.

Oh, I was looking wrong somehow.  c23b03c781d65 staged on September 14.  😱

The stuff is being built now, but thousands of builds fail newly (transitively) and even the channel gets blocked by a failure.

It's not perfect, but I'd call it solved.  People are welcome to fix builds...


[I/O] Add capability to read/write fixed signed integers (16bit, 32bit and 64bit)

Closing this issue as this is not needed.


Broken pipe (write failed) for JVM metrics

We're using the 0.10.0 release to send JVM metrics to SignalFx, but our logs are filled with this log error every 10s (our submit rate).
[opentelemetry.auto.trace 2021-04-16 12:55:50:821 +0000] [signalfx-metrics-publisher] INFO com.signalfx.shaded.apache.http.impl.execchain.RetryExec - I/O exception (java.net.SocketException) caught when processing request to {}->http://<IP_ADDRESS>:9080:  Broken pipe (Write failed)
[opentelemetry.auto.trace 2021-04-16 12:55:50:822 +0000] [signalfx-metrics-publisher] INFO com.signalfx.shaded.apache.http.impl.execchain.RetryExec - Retrying request to {}->http://<IP_ADDRESS>:9080

This is happening both from containers running on OpenShift 3.11 with a Smart Agent on the underlying node and on OpenStack VMs with a Smart Agent locally.
This message does NOT appear if we send directly to ingest.<region>.signalfx.com
splunk-0.10.0-otel-1.1.0
signalfx-agent-5.7.1-1.x86_64
The metrics themselves seem to make it to SignalFx so perhaps it has something to do with how the connection is cleaned up?
I've attached the debug log with scrubbed details but a non-scrubbed one can be provided privately if needed.
otel-jvm-metrics-broken-pipe-stack.log
Should we try with the generic OTEL agent?
Any help trying to debug this would be appreciated!

Meant to add even at debug level the signalfx-agent doesn't show any errors sending these and the internal metrics show sfxagent.datapoints_failed=0.

I don't know much about the SmartAgent, nor the signalfx metrics library, but this looks to be harmless logging of reset connections and the resulting retries. Since everything seems to be working fine, would it be ok to just change the log level to WARN, and ignore the INFO logging?

Hmm, it is somewhat concerning that it's happening every 10 seconds. And judging by the fact that our backend does not seem to have this problem it seems to be something in the signalfx-agent.
@hseljenes it would be great if you could try opentelemetry-collector-contrib or splunk-otel-collector - it's the first time I'm seeing this kind of error, but there may be something unique to your setup that's causing it. Or maybe it's just a weird bug in the agent.