[SR GeekCamp] Create table statement support for List partition

issue #4035
What type of PR is this：

[ ] bug
[X] feature
[ ] enhancement
[ ] others

Create Table for single column partition
create database example_db  ;
use example_db ;

CREATE TABLE t_recharge_detail_single(
    id bigint  ,
    user_id  bigint  ,
    recharge_money decimal(32,2) ,
    province varchar ,
    dt varchar
) ENGINE=OLAP
DUPLICATE KEY(id)
PARTITION BY LIST (province) (
   PARTITION p1 VALUES IN ("beijing","chongqing"),
   PARTITION p2 VALUES IN ("shanghai","tianjing")
)
DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES(
    "replication_num" = "1"
)

Create Table for multi column partition
create database example_db  ;
use example_db ;

CREATE TABLE t_recharge_detail_multi(
    id bigint  ,
    user_id  bigint  ,
    recharge_money decimal(32,2) ,
    province varchar ,
    dt varchar
) ENGINE=OLAP
DUPLICATE KEY(id)
PARTITION BY LIST (dt,province) (
   PARTITION p1 VALUES IN (("2022-04-01", "beijing"),("2022-04-01", "chongqing")),
   PARTITION p2 VALUES IN (("2022-04-01", "shanghai"))
)
DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES(
    "replication_num" = "1"
);

Show create table for partition
show create table t_recharge_detail_single ;
show create table t_recharge_detail_multi ;

[FE PR Coverage check]
:disappointed: fail : 212 / 319 (66.46%)
file detail

path
covered line
new line
coverage

:large_blue_circle:
com/starrocks/analysis/PartitionDesc.java
1
10
10.00%

:large_blue_circle:
com/starrocks/catalog/ListPartitionInfo.java
12
83
14.46%

:large_blue_circle:
com/starrocks/catalog/OlapTable.java
1
3
33.33%

:large_blue_circle:
com/starrocks/analysis/CreateTableStmt.java
2
3
66.67%

:large_blue_circle:
com/starrocks/analysis/SingleListPartitionDesc.java
23
30
76.67%

:large_blue_circle:
com/starrocks/analysis/MultiListPartitionDesc.java
29
37
78.38%

:large_blue_circle:
com/starrocks/catalog/Catalog.java
11
12
91.67%

:large_blue_circle:
com/starrocks/analysis/ListPartitionDesc.java
105
112
93.75%

:large_blue_circle:
com/starrocks/catalog/PartitionProperties.java
26
27
96.30%

:large_blue_circle:
com/starrocks/catalog/PartitionType.java
2
2
100.00%

[FE PR Coverage check]
:disappointed: fail : 224 / 321 (69.78%)
file detail

path
covered line
new line
coverage

:large_blue_circle:
com/starrocks/catalog/OlapTable.java
0
2
00.00%

:large_blue_circle:
com/starrocks/analysis/PartitionDesc.java
1
10
10.00%

:large_blue_circle:
com/starrocks/catalog/ListPartitionInfo.java
12
83
14.46%

:large_blue_circle:
com/starrocks/analysis/CreateTableStmt.java
2
3
66.67%

:large_blue_circle:
com/starrocks/catalog/Catalog.java
11
12
91.67%

:large_blue_circle:
com/starrocks/analysis/MultiListPartitionDesc.java
35
38
92.11%

:large_blue_circle:
com/starrocks/analysis/ListPartitionDesc.java
105
112
93.75%

:large_blue_circle:
com/starrocks/analysis/SingleListPartitionDesc.java
29
31
93.55%

:large_blue_circle:
com/starrocks/catalog/PartitionProperties.java
27
28
96.43%

:large_blue_circle:
com/starrocks/catalog/PartitionType.java
2
2
100.00%

[FE PR Coverage check]
:heart_eyes: pass : 310 / 343 (90.38%)
file detail

path
covered line
new line
coverage

:large_blue_circle:
com/starrocks/catalog/OlapTable.java
0
2
00.00%

:large_blue_circle:
com/starrocks/analysis/CreateTableStmt.java
2
3
66.67%

:large_blue_circle:
com/starrocks/catalog/ListPartitionInfo.java
74
89
83.15%

:large_blue_circle:
com/starrocks/catalog/Catalog.java
11
12
91.67%

:large_blue_circle:
com/starrocks/analysis/SingleListPartitionDesc.java
50
54
92.59%

:large_blue_circle:
com/starrocks/analysis/ListPartitionDesc.java
104
111
93.69%

:large_blue_circle:
com/starrocks/analysis/MultiListPartitionDesc.java
66
69
95.65%

:large_blue_circle:
com/starrocks/catalog/PartitionType.java
2
2
100.00%

:large_blue_circle:
com/starrocks/analysis/PartitionDesc.java
1
1
100.00%

run starrocks_fe_unittest

[FE PR Coverage check]
:heart_eyes: pass : 301 / 338 (89.05%)
file detail

path
covered line
new line
coverage

:large_blue_circle:
com/starrocks/catalog/OlapTable.java
0
2
00.00%

:large_blue_circle:
com/starrocks/catalog/PartitionInfo.java
0
1
00.00%

:large_blue_circle:
com/starrocks/analysis/CreateTableStmt.java
2
3
66.67%

:large_blue_circle:
com/starrocks/catalog/ListPartitionInfo.java
65
83
78.31%

:large_blue_circle:
com/starrocks/catalog/Catalog.java
11
12
91.67%

:large_blue_circle:
com/starrocks/analysis/SingleListPartitionDesc.java
50
54
92.59%

:large_blue_circle:
com/starrocks/analysis/ListPartitionDesc.java
104
111
93.69%

:large_blue_circle:
com/starrocks/analysis/MultiListPartitionDesc.java
66
69
95.65%

:large_blue_circle:
com/starrocks/catalog/PartitionType.java
2
2
100.00%

:large_blue_circle:
com/starrocks/analysis/PartitionDesc.java
1
1
100.00%

[FE PR Coverage check]
:heart_eyes: pass : 328 / 366 (89.62%)
file detail

path
covered line
new line
coverage

:large_blue_circle:
com/starrocks/catalog/OlapTable.java
0
2
00.00%

:large_blue_circle:
com/starrocks/catalog/PartitionInfo.java
0
1
00.00%

:large_blue_circle:
com/starrocks/analysis/CreateTableStmt.java
2
3
66.67%

:large_blue_circle:
com/starrocks/catalog/ListPartitionInfo.java
87
110
79.09%

:large_blue_circle:
com/starrocks/analysis/SingleItemListPartitionDesc.java
45
49
91.84%

:large_blue_circle:
com/starrocks/catalog/Catalog.java
11
12
91.67%

:large_blue_circle:
com/starrocks/analysis/MultiItemListPartitionDesc.java
54
57
94.74%

:large_blue_circle:
com/starrocks/analysis/ListPartitionDesc.java
126
129
97.67%

:large_blue_circle:
com/starrocks/catalog/PartitionType.java
2
2
100.00%

:large_blue_circle:
com/starrocks/analysis/PartitionDesc.java
1
1
100.00%

[FE PR Coverage check]
:heart_eyes: pass : 340 / 391 (86.96%)
file detail

path
covered line
new line
coverage

:large_blue_circle:
com/starrocks/catalog/OlapTable.java
0
2
00.00%

:large_blue_circle:
com/starrocks/catalog/PartitionInfo.java
0
1
00.00%

:large_blue_circle:
com/starrocks/analysis/CreateTableStmt.java
2
3
66.67%

:large_blue_circle:
com/starrocks/catalog/ListPartitionInfo.java
71
105
67.62%

:large_blue_circle:
com/starrocks/catalog/Catalog.java
11
12
91.67%

:large_blue_circle:
com/starrocks/analysis/SingleItemListPartitionDesc.java
53
57
92.98%

:large_blue_circle:
com/starrocks/analysis/MultiItemListPartitionDesc.java
66
69
95.65%

:large_blue_circle:
com/starrocks/analysis/ListPartitionDesc.java
134
139
96.40%

:large_blue_circle:
com/starrocks/catalog/PartitionType.java
2
2
100.00%

:large_blue_circle:
com/starrocks/analysis/PartitionDesc.java
1
1
100.00%

[FE PR Coverage check]
:heart_eyes: pass : 348 / 366 (95.08%)
file detail

path
covered line
new line
coverage

:large_blue_circle:
com/starrocks/catalog/OlapTable.java
0
2
00.00%

:large_blue_circle:
com/starrocks/catalog/PartitionInfo.java
0
1
00.00%

:large_blue_circle:
com/starrocks/analysis/CreateTableStmt.java
2
3
66.67%

:large_blue_circle:
com/starrocks/catalog/Catalog.java
11
12
91.67%

:large_blue_circle:
com/starrocks/analysis/SingleItemListPartitionDesc.java
53
57
92.98%

:large_blue_circle:
com/starrocks/analysis/MultiItemListPartitionDesc.java
66
69
95.65%

:large_blue_circle:
com/starrocks/catalog/ListPartitionInfo.java
83
86
96.51%

:large_blue_circle:
com/starrocks/analysis/ListPartitionDesc.java
130
133
97.74%

:large_blue_circle:
com/starrocks/catalog/PartitionType.java
2
2
100.00%

:large_blue_circle:
com/starrocks/analysis/PartitionDesc.java
1
1
100.00%

[FE PR Coverage check]
:heart_eyes: pass : 348 / 366 (95.08%)
file detail

path
covered line
new line
coverage

:large_blue_circle:
com/starrocks/catalog/OlapTable.java
0
2
00.00%

:large_blue_circle:
com/starrocks/catalog/PartitionInfo.java
0
1
00.00%

:large_blue_circle:
com/starrocks/analysis/CreateTableStmt.java
2
3
66.67%

:large_blue_circle:
com/starrocks/catalog/Catalog.java
11
12
91.67%

:large_blue_circle:
com/starrocks/analysis/SingleItemListPartitionDesc.java
53
57
92.98%

:large_blue_circle:
com/starrocks/analysis/MultiItemListPartitionDesc.java
66
69
95.65%

:large_blue_circle:
com/starrocks/catalog/ListPartitionInfo.java
83
86
96.51%

:large_blue_circle:
com/starrocks/analysis/ListPartitionDesc.java
130
133
97.74%

:large_blue_circle:
com/starrocks/catalog/PartitionType.java
2
2
100.00%

:large_blue_circle:
com/starrocks/analysis/PartitionDesc.java
1
1
100.00%


[WIP] LINK-1379 enable external user events variable

Description :sparkles:
Disable external user features with environment variable ENABLE_EXTERNAL_USER_EVENTS
Issues :bug:
Closes :no_good_woman:
LINK-1379:
Related :handshake:
Testing :alembic:
Automated tests :gear:
Manual testing :construction_worker_man:
Screenshots :camera_flash:
Additional notes :spiral_notepad:

Codecov Report

Merging #193 (b9402f5) into develop (bd9d3d0) will decrease coverage by 0.06%.
The diff coverage is 82.14%.

@@             Coverage Diff             @@
##           develop     #193      +/-   ##
===========================================
- Coverage    99.66%   99.61%   -0.06%
===========================================
  Files          501      501
  Lines         9333     9354      +21
  Branches      2049     2063      +14
===========================================
+ Hits          9302     9318      +16
- Misses          31       35       +4
- Partials         0        1       +1

Impacted Files
Coverage Δ

...omain/event/eventHierarchy/subEvents/SubEvents.tsx
100.00% <ø> (ø)

src/domain/event/utils.ts
98.83% <73.33%> (-0.77%)
:arrow_down:

...onNotification/EventAuthenticationNotification.tsx
88.23% <75.00%> (-4.08%)
:arrow_down:

...main/event/createButtonPanel/CreateButtonPanel.tsx
95.45% <100.00%> (ø)

src/domain/event/eventForm/EventForm.tsx
84.25% <100.00%> (+0.38%)
:arrow_up:

src/domain/user/hooks/useUser.ts
100.00% <100.00%> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


[stress testing] Improve scenario matrix edge case handling

Handle case where the matrix produces a single length array. Powershell by default will return the array item type (object) instead of an array, which messes things up the generated values file.
Handle case where there is no values.yaml file defined.

The following pipelines have been queued for testing:java - templatejava - template - testsjs - templatenet - templatenet - template - testspython - templatepython - template - tests You can sign off on the approval gate to test the release stage of each pipeline. See eng/common workflow


MedicationRequest.doNotPerform is not included in Highlighting and Coverage Calculations

Summary
MADiE team received a helpdesk ticket where user is having issues achieving 100% for their measure that includes MedicationRequest.doNotPerform and the highlighting is also showing that it is not covered.
External Tracking Ticket
BONNIEMAT-1835
Expected Behavior
The attached test case should highlight DementiaMedication.doNotPerform in
"define Has Dementia Medications in Year Before or During Measurement Period". Also provided a List of all test cases which is expected to give 100% coverage.
Version or Commit
Inputs (e.g. Measure Bundle, Patient Bundle, CQL Library)
MeasureBundle.zip
PatientBundle.txt
CMS165FHIR-v0.1.000-FHIR4-TestCases.zip
Relevant Calculation Options (e.g. Measurement Period, meta.profile Validation)
MP : 1/1/2025 - 12/31/2025
cql-to-elm Version Used for Measure Logic Translation (if known)
3.3.2
Any Additional Info

Uploading CMS165FHIR-v0.1.000-FHIR.txt…

Hi @RohitKandimalla,
Could you please upload the measure bundle to this ticket or the BONNIEMAT ticket? It looks like it didn't sucessfully upload here.
What version of fqm-execution is being used with this?
-Chris

I have issues uploading some files on Github this morning, compressed it as a work around.
and here is the version we are currently using fqm-execution v1.3.3

This is expected behavior of the highlighting. doNotPerform needs a "truthy" value to be considered covered. The user just needs to add a case similar to the PatientBundle.txt that you uploaded but with doNotPerform being set to true on the MedicationRequest resource to cover that clause. This also is a good test case for the situation that the medication was specifically requested to NOT be given.

Clause highlighting and coverage is not based on just "executed or not" it is based on the logic being relevant, executed AND having a result that is either true, a non-empty list, or filled with a value. If we allowed for boolean false to count for clause coverage then logic such as exists "Some Expression" would result in the exists always being highlighted green/covered.
We reviewed the BONNIEMAT ticket linked to this and noticed it is using the QICore MedicationRequest profile which makes it so doNotPerform is fixed to false. The validity of the resources being executed will have been checked by the data provider before hitting this logic. So there is no need to actually check for doNotPerform to be true in the CQL as it will have been done in the profile validity checks.


Correction for raw affiliation Kyiv National University of Architecture and Construction [Kiev, Ukraine]; Laboratoire de Physique de la Matière Condensée - UR UPJV 2081

Correction needed for raw affiliation Kyiv National University of Architecture and Construction [Kiev, Ukraine]; Laboratoire de Physique de la Matière Condensée - UR UPJV 2081
raw_affiliation_name: Kyiv National University of Architecture and Construction [Kiev, Ukraine]; Laboratoire de Physique de la Matière Condensée - UR UPJV 2081
new_rors: 01gyxrk03;02qp15436
previous_rors: 04zaaa143;02qp15436
works_examples: W4226239902;W4296129634;W4296586047
contact: 69511f8a8679119618bfe583cf6ec840:0c6bbe7908dd8e7c3caadc72fd4d255d @ u-picardie.fr

This issue was accepted and ingested by the OpenAlex team on 2024-10-10. The new affiliations should be visible within the next 7 days.


Pass down refreshControl prop to ScrollView

This fixes some issue with RefreshControl usage on the Tabs element as per #1371 and #1168

After some further use I've noticed the refresh control prevents scrolling up in the tabs. I couldn't find a way to avoid that. I also noticed in the react docs:

Only works for vertical ScrollViews (horizontal prop must be false).
horizontal prop is true in this case, so I'm not sure why it worked at all, but that means it's hacky and likely a bad idea to use it at all (not to mention it doesn't work correctly.)
Thus I'm closing the pull request.


Pilot: Support filtering svc/pod through label

Is your feature request related to a problem? Please describe.
I donnot want pilot to fill envoy with all the svc/endpoint in k8s.
And I am sure my pods with istio-proxy will never access certain services.
Describe the solution you'd like
Current Pilot only supports restriction of  watching one ns through appNamesapce param.
Can it be enhanced to support filtering svc/endpoint/pod to be watched by label？

@rokii Do you have a particular scale, or have you noticed problems? Or is it more of a theoretical proposal?
We have plans to add similar functionality in the future, but not before 1.0.

@andraxylia  We have lots of applications deployed in k8s. Now some teams want to adopt Istio. And many applications like data mining or ML jobs  are not services and will never join the mesh. So it is better to have Istio just manage and list limited endpoints.  In the long run , we may have dedicated k8s cluster for Istio service mesh. But in the initial phase , operation could be cost effective if we reuse current cluster.
I agree that it is neither blocking issue nor urgent.

@rokii thanks for the clarification.

Given Istio is 1.0 now, I do vote for @rokii point of not auto-populating all service & endpoints into Envoy Config.
It only make sense to have a istio listen on a specific set of namespaces.
If not, should support ignore-namespaces


Talessya: Fix minor bugs

Changes / Comments
Fixing small bugs with advantage and whisper queries. In order to do that, we defaulted some players config attributes, but this won´t be a problem for the user experience and it would be transparent for them.
Roll20 Requests
Comments are very helpful for reviewing the code changes. Please answer the relevant questions below in your comment.

[x] Does the pull request title have the sheet name(s)? Include each sheet name.
[x] Is this a bug fix?
[ ] Does this add functional enhancements (new features or extending existing features) ?
[ ] Does this add or change functional aesthetics (such as layout or color scheme) ?
[x] If changing or removing attributes, what steps have you taken, if any, to preserve player data ?
[ ] If this is a new sheet, did you follow Building Character Sheets standards ?

If you do not know English. Please leave a comment in your native language.

Character Sheet Info Roll20 Internal Use only.


Char Counter Customization Hooks

Made it possible to customize the char counter in more details.

Introduced the concept of SLKCounterLayoutStyle for customizing the layout of the char counter. Implemented SLKCounterLayoutStylePinnedToRightButton which pins the char counter to the top of the right button but made SLKCounterLayoutStylePinnedToTop be the default behavior.
Made it possible to change the colors used for the normal and limit exceeded state of the char counter.
Exposed the charCounterLabel to make it easier to use custom fonts.
Implemented a new SLKCounterStyle: SLKCounterStyleCountdownReversed which starts at the maxChars value and counts down to 0.

@dzenbot have you had time to look at this?


Support build.rs git-lfs fetching for proto_descriptor

Is your feature request related to a problem? Please describe.
When trying to build a Rust project that uses the Penumbra crates as dependencies, cargo build fails with an error on the penumbra-proto crate, because the proto_descriptor.bin file isn't found.
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A workaround is to set a path dependency in the Cargo.toml file, rather than e.g. a git dependency. Provided you have the git-lfs assets checked out locally, that'll work.
Additional context
We've encountered this problem before, e.g. in Galileo https://github.com/penumbra-zone/galileo/issues/29 , and our "fix" was to use path dependencies as a workaround. However there's also the dope https://github.com/penumbra-zone/penumbra/pull/2201 which resolves the lfs problem specifically for the proof params.
Let's port that build.rs logic to the proto crate so that external projects can use git dependencies.
Unsure what the implications of the build.rs strategy are for future hosting on crates.io; I suggest we figure that out later.

In the meantime, we can disable LFS for the proto descriptor. (We're already checking in hex-formatted proto descriptors into the repo in the form of the generated Go code).

Closing with #3044 as "good enough" for now.


Asynchronous

How does isValidated() handle asynchronous field validation? For example: I want to check if the user's email is already registered and display an error message appropriately. I do this by firing a fetch() to my back-end. If ever it would return anything (ergo: the async part) would decide if the component should go to the next step. The next button triggers isValidated() which contains fetch(). The issue I'm having now is, because isValidated() has to wait for the result of fetch() first, it can't determine whether it should go to the next step or not at the point of click. Please help. Not sure if I explained that well hahaha.
Also is it normal that I can't go to the next step when I use isValidated() as a callback function?

isValidated() can also return a Promise. So I would do:
isValidated() {
     return new Promise((resolve, reject) => {
         fetch()
              .then(result => {
                    resolve()
              })
              .catch(error => {
                    // what ever
              }}
     })
}

This example of course only works if your fetch() method also returns a Promise. If not you could implement a callback argument and have fetch() run the callback when the request is completed. Then resolve() or reject() in the callback. I.e.
isValidated() {
     return new Promise((resolve, reject) => {
         fetch((result) => {
                 if(result.someVar === true) resolve()
                 else reject()
         })
     })
}

@danielsetreus thanks for your very clear explanation and appreciate your support in maintaining this project.
@RichardVitaliz you can have a look at this example code as well (@danielsetreus's code is also good)
https://github.com/newbreedofgeek/react-stepzilla/blob/master/src/examples/Step5.js
Closing this, but reopen if you need more help.


[Feature]: pre release or nightly builds

🚀 The feature, motivation and pitch
A lot of pr's get merged very often. And building it locally often takes a long time. Especially on databricks. Is it possible to get a --pre tag to install prereleases or nightly versions?
Alternatives
No response
Additional context
No response

Nightly builds would be useful on:
https://hub.docker.com/r/vllm/vllm-openai/tags

The release cadence of vLLM is quite fast, are the bi-weekly releases not fast enough?
One thing you can do to speed up your own builds is to make sure that the CCACHE is persistent between your builds. That way you don't have to recompile custom ops that weren't changed.


Fix a invalid failpoint random_receiver_async_msg_push_failure_failpoint

What problem does this PR solve?
Issue Number: close https://github.com/pingcap/tiflash/issues/7844
Problem Summary:
When grpc_recv_queue::pushWithTag returns Full, it must not do anything because this tag may be executed in other threads.
However, the failpoint random_receiver_async_msg_push_failure_failpoint changes the return value to CANCELLED, which leads to a panic bug.
What is changed and how it works?
Move random_receiver_async_msg_push_failure_failpoint before grpc_recv_queue::pushWithTag.
Check List
Tests

[ ] Unit test
[ ] Integration test
[ ] Manual test (add detailed scripts or steps below)
[x] No code

Side effects

[ ] Performance regression: Consumes more CPU
[ ] Performance regression: Consumes more Memory
[ ] Breaking backward compatibility

Documentation

[ ] Affects user behaviors
[ ] Contains syntax changes
[ ] Contains variable changes
[ ] Contains experimental features
[ ] Changes MySQL compatibility

Release note

None

/hold

/unhold


neo build tools has no method to control neoxp batch --reset option

NeoExpressBatch task has a Checkpoint property which can be used in conjunction with the Reset property.


is 'old-heap' growing by memory-leak?

OrientDB Version: 2.2.18
Java Version: 1.8.0_92
OS: Ubuntu-16.04
Expected behavior
When we have 'not-so-heavy' data loading active on our standalone server, I notice that the Old-generation size is growing with each Full Garbage Collection that is initiated:
% jstat -gc -h10 17079 10s
...
14848.0 15872.0 14356.0  0.0   4162560.0 3986224.8 5716480.0  5633496.7  37592.0 36577.7 4096.0 3739.3   3666  182.964   5      5.399  188.363
 S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     GCT
14848.0 15360.0  0.0   8140.3 4163072.0 238585.4 5716480.0  5642829.5  37592.0 36577.7 4096.0 3739.3   3669  183.129   5      5.399  188.528
14848.0 41984.0  0.0    0.0   4110336.0 3888676.8 5973504.0  2838417.4  37592.0 36577.7 4096.0 3739.3   3670  183.316   6      7.345  190.661
37376.0 38400.0  0.0   10415.9 4111872.0 1463619.6 5973504.0  2845162.4  37592.0 36577.7 4096.0 3739.3   3673  183.452   6      7.345  190.797
...

Our server is started with heap size parameters "java -server -Xms4G -Xmx12G ..."
However when we start an additional, heavy data-loading program, that old-generation capacity very quickly becomes exhausted until the server dies/halts with Out-Of-Memory.
-> is this growing Old-generation Capacity ("OC" column) figure normal ?
-> a Full-GC is not reducing the Old-generation Usage ("OU" column), so the "OC" is further increasing at each FGC

@rdelangh , as usual, we need heap dump. the best, of course, to take it after each full GC. I do not remember is it possible for you to send heap dump for us, if not could you send us screenshots of heap dumps ?

hi @laa,
"to take it after each full GC" -> how can I control when a FGC is happening? Because you suggest that I would then initiate (manually) a heap dump.
And indeed, the heap dump will be around Xmx12G size, so 12GB, and impossible to send it to you.
Can you suggest which tool I shall use to open such a big heap dump, and command options that will allow to open the big (>= 12GB) file, and which screenshots you desire?

Is this usefull?
jmap -heap <pid>
Attaching to process ID 17079, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.92-b14

using thread-local object allocation.
Parallel GC with 33 thread(s)

Heap Configuration:
   MinHeapFreeRatio         = 0
   MaxHeapFreeRatio         = 100
   MaxHeapSize              =<PHONE_NUMBER>8 (12288.0MB)
   NewSize                  =<PHONE_NUMBER> (1365.0MB)
   MaxNewSize               =<PHONE_NUMBER> (4096.0MB)
   OldSize                  =<PHONE_NUMBER> (2731.0MB)
   NewRatio                 = 2
   SurvivorRatio            = 8
   MetaspaceSize            = 21807104 (20.796875MB)
   CompressedClassSpaceSize =<PHONE_NUMBER> (1024.0MB)
   MaxMetaspaceSize         =<PHONE_NUMBER>4415 MB
   G1HeapRegionSize         = 0 (0.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity =<PHONE_NUMBER> (3282.5MB)
   used     =<PHONE_NUMBER> (2547.160430908203MB)
   free     = 771059424<PHONE_NUMBER>917969MB)
   77.59818525234435% used
From Space:
   capacity = 5767168 (5.5MB)
   used     = 5406736 (5.1562652587890625MB)
   free     = 360432 (0.3437347412109375MB)
   93.7502774325284% used
To Space:
   capacity = 18350080 (17.5MB)
   used     = 0 (0.0MB)
   free     = 18350080 (17.5MB)
   0.0% used
PS Old Generation
   capacity =<PHONE_NUMBER> (6228.0MB)
   used     =<PHONE_NUMBER> (4163.227363586426MB)
   free     =<PHONE_NUMBER> (2064.772636413574MB)
   66.84693904281352% used

13855 interned Strings occupying 1276864 bytes.

attached
jmap-histo.txt
is the output of "jmap -histo " , hope it helps

To see how objects are growing, take a look at a sequence of ascii output from jmap -histo:live  (this will force a fullgc and then collect histogram): https://docs.oracle.com/javase/8/docs/technotes/tools/unix/jmap.html .

@cmassi , @robfrank , @laa
I had to increase the Xmx parameter, and apparently got a stable 'high-water mark' for the old-generation heap capacity:
from repetitive 'jstat' outputs, I saw a continuous increase of the OC at each fullgc, until finally an increase after fullgc 11 :
...

11776.0 28672.0 11769.2  0.0   1577472.0 201597.2 7113728.0  7099833.0  37592.0 36241.2 4096.0 3699.7   5104  499.054  10     22.870  521.924
 S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     GCT
24064.0 22016.0  0.0   21759.1 1553920.0 669679.9 7113728.0  7101665.7  37592.0 36241.9 4096.0 3699.7   5105  499.090  10     22.870  521.960
17408.0 28160.0  0.0    0.0   1530880.0 860371.9 6729216.0  3238519.1  37592.0 36197.3 4096.0 3693.1   5106  499.158  11     24.892  524.050
7168.0 28160.0 6982.6  0.0   1485312.0 49609.7  6729216.0  3241704.6  37592.0 36197.3 4096.0 3693.1   5108  499.230  11     24.892  524.121
...

same after fullgc 12:
...

16384.0 24064.0 16376.4  0.0   2226176.0 151432.1 6729216.0  6717509.2  37592.0 36205.2 4096.0 3693.1   6128  539.809  11     24.892  564.701
14336.0 22528.0 14112.0  0.0   2141184.0 1034595.4 6729216.0  6721386.1  37592.0 36205.2 4096.0 3693.1   6130  539.867  11     24.892  564.759
18432.0 24064.0 18430.5  0.0   2060800.0 1666781.4 6564352.0  3185646.1  37592.0 36205.2 4096.0 3693.1   6132  539.928  12     26.108  566.036
...

Will keep an eye on it...

We see from latest jstat output that with Full GC number 12, the Old Usage has decreased, together with Old+Eden Capacity, but Eden was nearly full. There is no leak, so far. If you are interested in reducing the time spent in Full GC, which also does a resize of generations (26 secs in fullgc), you should try to understand which is the best size of each generation under heavy load (verify min/medium/max occupation and capacity), give some more space to each generation to be able to handle any peak (and so avoid OOM), and remove adaptive size policy.  e.g (for ParallelGC):  -Xms12g -Xmn3g -Xmx12g -XX:MetaspaceSize=512m -XX:MaxMetaspaceSize=512m -XX:CompressedClassSpaceSize=512m -XX:-UseAdaptiveSizePolicy . To study gc behaviour you need a detailed log  e.g: -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintGCID -XX:+PrintGCDetails -Xloggc:$ORIENTDB_HOME/log/gc_%p_%t.log   . Optionally you can study promotion of objects in survivors too survivors, with -XX:+PrintTenuringDistribution

@cmassi
thank you for the feedback, only I do not know how I can set the space/capacity for each generation; the only known parameters that I can find in ODB documentation are about the Xms and Xmx parameters.
Now, with Xms=4g and Xmx=12g, I hit again OOM under heavy load, the ODB engine has become inaccessible:
 S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     GCT
740864.0 1397760.0 149862.5  0.0   1398784.0 1398784.0 8388608.0  8388201.0  39808.0 38609.6 4224.0 3864.6  38452 1638.937  76<PHONE_NUMBER>.917
740864.0 1397760.0 149862.5  0.0   1398784.0 1398784.0 8388608.0  8388201.0  39808.0 38609.6 4224.0 3864.6  38452 1638.937  76<PHONE_NUMBER>.917
740864.0 1397760.0 149862.5  0.0   1398784.0 1398784.0 8388608.0  8388201.0  39808.0 38609.6 4224.0 3864.6  38452 1638.937  77<PHONE_NUMBER>.084
740864.0 1397760.0 149862.5  0.0   1398784.0 1398784.0 8388608.0  8388201.0  39808.0 38609.6 4224.0 3864.6  38452 1638.937  77<PHONE_NUMBER>.084
740864.0 1397760.0 149862.5  0.0   1398784.0 1398784.0 8388608.0  8388201.0  39808.0 38609.6 4224.0 3864.6  38452 1638.937  77<PHONE_NUMBER>.084
740864.0 1397760.0 149862.5  0.0   1398784.0 1398784.0 8388608.0  8388201.0  39808.0 38609.6 4224.0 3864.6  38452 1638.937  77<PHONE_NUMBER>.084
740864.0 1397760.0 149862.5  0.0   1398784.0 1398784.0 8388608.0  8388201.0  39808.0 38609.6 4224.0 3864.6  38452 1638.937  78<PHONE_NUMBER>.160
...

Shall I set those parameters that you mentioned? I.e.
-Xms12g -Xmn3g -Xmx12g -XX:MetaspaceSize=512m -XX:MaxMetaspaceSize=512m -XX:CompressedClassSpaceSize=512m -XX:-UseAdaptiveSizePolicy

The documentation of XX flags for Hotspot jvm is available on Oracle web site, not in OrientDB documentation.
The OOM are not always the same, they can be thrown because it fills one of the spaces (old/meta/class) or even due to native allocation issue.
We cannot see the timestamps in jstat output but it seems that it takes 9 secs for each FullGC (708/78), and probably most of the time has been spent to increase the size from initial 4g to current 11gb (S0Cx2 + EC + OC = 11269120 ), but it seems it cannot grow after 3 FGC   (please note that S1C seems to be wrong, because the 2 survivors are always identical in capacity, because objects are copied from 1 to 2 and from 2 to 1) .
Meta/Class spaces are quite small, so can use small sizes, e.g: -XX:MetaspaceSize=64m -XX:MaxMetaspaceSize=64m -XX:CompressedClassSpaceSize=12m
According to this jstat it seems OOM was due to Old, nearly full, which cannot accept promotion of new objects from survivor+eden
You can try to further increase old , raising max to 14gb
The New generation is setup with XX:NewSize XX:MaxNewSize or simply with Xmn . It is made of 2 survivors and eden, so nearly 3gb
The OrientDB java process uses also direct memory and does some other native allocations, so check process size to avoid swapping of java process (should be: process size < available ram)

Try  -Xms14g -Xmn3g -Xmx14g -XX:MetaspaceSize=64m -XX:MaxMetaspaceSize=64m -XX:CompressedClassSpaceSize=12m -XX:-UseAdaptiveSizePolicy

Pay attention to OOM generated for wrong generation sizes when removing adaptive size policy ( -XX:-UseAdaptiveSizePolicy ) so remove -XX:+HeapDumpOnOutOfMemoryError until you have setup the correct sizes for each space (generation)

Anyway, to really study the gc behaviour you have to understand how objects move (promotion, cleanup ) reading the detailed log

Did that, the server is running quite fine with reasonable (but not at all maximum required) load, the outputs of "jstat" are:
...
393216.0 393216.0  0.0   30283.2 2359296.0 1652231.5 11534336.0 9263875.0  39808.0 38511.7 4224.0 3828.8<PHONE_NUMBER>9.812  12     49.291 10969.103
 S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     GCT
393216.0 393216.0  0.0   28114.0 2359296.0 1221989.3 11534336.0 9268787.2  39808.0 38511.7 4224.0 3828.8<PHONE_NUMBER>1.168  12     49.291 10970.459
393216.0 393216.0  0.0   28590.5 2359296.0 1185331.2 11534336.0 9275613.7  39808.0 38511.7 4224.0 3828.8<PHONE_NUMBER>2.560  12     49.291 10971.850
393216.0 393216.0  0.0   30852.8 2359296.0 2240718.9 11534336.0 9282221.0  39808.0 38511.7 4224.0 3828.8<PHONE_NUMBER>3.790  12     49.291 10973.081
...

The server process is running with the following command-line arguments:
F S UID        PID  PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD
0 S orientdb 46946     1 99  80   0 - 90887861 futex_ Mar23 ?     18-02:03:45 java -server -Xms14G -Xmn3G -Xmx18G -XX:MetaspaceSize=64m -XX:MaxMetaspaceSize=64m -XX:CompressedClassSpaceSize=12m -XX:-UseAdaptiveSizePolicy -XX:+UseCompressedOops -Djna.nosys=true -XX:MaxDirectMemorySize=512g -Djava.awt.headless=true -Dfile.encoding=UTF8 -Drhino.opt.level=9 -Dstorage.openFiles.limit=1024 -Denvironment.dumpCfgAtStartup=true -Dquery.parallelAuto=true -Dquery.parallelMinimumRecords=10000 -Dquery.parallelResultQueueSize=200000 -Dstorage.wal.maxSize=51200 -Dstorage.diskCache.bufferSize=54000 -Dmemory.chunk.size=33554432 -Djava.util.logging.config.file=/opt/orientdb/orientdb-community-2.2.18-SNAPSHOT/config/orientdb-server-log.properties -Dorientdb.config.file=/opt/orientdb/orientdb-community-2.2.18-SNAPSHOT/config/orientdb-server-config.xml -Dorientdb.www.path=/opt/orientdb/orientdb-community-2.2.18-SNAPSHOT/www -Dorientdb.build.number=UNKNOWN@rf31f2e10de758cbdef4cee27451b4065b94d9ce2; 2017-03-04 00:50:53+0000 -cp /opt/orientdb/orientdb-community-2.2.18-SNAPSHOT/lib/orientdb-server-2.2.18-SNAPSHOT.jar:/opt/orientdb/orientdb-community-2.2.18-SNAPSHOT/lib/*:/opt/orientdb/orientdb-community-2.2.18-SNAPSHOT/plugins/* com.orientechnologies.orient.server.OServerMain

Loading programs can still send their transaction requests succesfully to the server, data gets loaded. However, after some hours, there is still a new OOM situation because when trying to launch a REST connection, I get the following error:
Error occurred during initialization of VMjava.lang.OutOfMemoryError: unable to create new native thread

Which memory setting is blocking to create new sessions?

The server is running with the following command-line args:
UID        PID  PPID  C STIME TTY          TIME CMD
orientdb  9879     1 20 Mar27 ?        03:37:12 java -server -Xms14G -Xmn3G -Xmx18G -XX:MetaspaceSize=64m -XX:MaxMetaspaceSize=64m -XX:CompressedClassSpaceSize=12m -XX:-UseAdaptiveSizePolicy -XX:+UseCompressedOops -Djna.nosys=true -XX:MaxDirectMemorySize=512g -Djava.awt.headless=true -Dfile.encoding=UTF8 -Drhino.opt.level=9 -Dstorage.openFiles.limit=1024 -Denvironment.dumpCfgAtStartup=true -Dquery.parallelAuto=true -Dquery.parallelMinimumRecords=10000 -Dquery.parallelResultQueueSize=200000 -Dstorage.wal.maxSize=51200 -Dstorage.diskCache.bufferSize=54000 -Dmemory.chunk.size=33554432 -Djava.util.logging.config.file=/opt/orientdb/orientdb-community-2.2.18-SNAPSHOT/config/orientdb-server-log.properties -Dorientdb.config.file=/opt/orientdb/orientdb-community-2.2.18-SNAPSHOT/config/orientdb-server-config.xml -Dorientdb.www.path=/opt/orientdb/orientdb-community-2.2.18-SNAPSHOT/www -Dorientdb.build.number=UNKNOWN@rf31f2e10de758cbdef4cee27451b4065b94d9ce2; 2017-03-04 00:50:53+0000 -cp /opt/orientdb/orientdb-community-2.2.18-SNAPSHOT/lib/orientdb-server-2.2.18-SNAPSHOT.jar:/opt/orientdb/orientdb-community-2.2.18-SNAPSHOT/lib/*:/opt/orientdb/orientdb-community-2.2.18-SNAPSHOT/plugins/* com.orientechnologies.orient.server.OServerMain

"jstat" outputs are:
...

The server itself still has quite some unused RAM (installed RAM=128GB):
$ vmstat 4
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 5  0<PHONE_NUMBER>2<PHONE_NUMBER>4260    0    0    56   105    0    0  3  0 96  1  0
 2  0<PHONE_NUMBER>0<PHONE_NUMBER>4760    0    0  1391 15428 9154 10217  8  1 90  1  0
 4  1<PHONE_NUMBER>6<PHONE_NUMBER>1652    0    0  1648 18393 12993 9999  9  1 89  1  0
...

However, sometimes I can not start a new ODB client program "console.sh":
$ /opt/orientdb/orientdb-community-2.2.18-SNAPSHOT/bin/console.sh "CONNECT ... ; desc myclass; quit"
Error occurred during initialization of VMjava.lang.OutOfMemoryError: unable to create new native thread

-> which of the JVM settings should be further adjusted? Is this connection from the client program trying to consume additional 'new' generation space?

Hi @rdelangh , what is output of ulimit -a ?

Could you also execute ps -o nlwp <pid> and provide output ?

@laa
$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 514444
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 10000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 514444
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

$ ps -o nlwp 9879
NLWP
 342

@rdelangh did you take this values at the moment after you have seen a problem with the last OOM?

trying again to launch an extra JVM program ("jstat"), fails with
orientdb@orient2:~$ /usr/lib/jvm/jdk1.8.0_92/bin/jstat -gc -h10 9879 10s
Error occurred during initialization of VM
Cannot create VM thread. Out of system resources.

At that very moment, only 2 Java processes are active on this machine:

the ODB server
one "console.sh" client program

$ ps -aef|grep java
orientdb  9879     1 41 Mar27 ?        08:27:38 java -server -Xms14G -Xmn3G -Xmx18G -XX:MetaspaceSize=64m -XX:MaxMetaspaceSize=64m -XX:CompressedClassSpaceSize=12m -XX:-UseAdaptiveSizePolicy -XX:+UseCompressedOops -Djna.nosys=true -XX:MaxDirectMemorySize=512g -Djava.awt.headless=true -Dfile.encoding=UTF8 -Drhino.opt.level=9 -Dstorage.openFiles.limit=1024 -Denvironment.dumpCfgAtStartup=true -Dquery.parallelAuto=true -Dquery.parallelMinimumRecords=10000 -Dquery.parallelResultQueueSize=200000 -Dstorage.wal.maxSize=51200 -Dstorage.diskCache.bufferSize=54000 -Dmemory.chunk.size=33554432 -Djava.util.logging.config.file=/opt/orientdb/orientdb-community-2.2.18-SNAPSHOT/config/orientdb-server-log.properties -Dorientdb.config.file=/opt/orientdb/orientdb-community-2.2.18-SNAPSHOT/config/orientdb-server-config.xml -Dorientdb.www.path=/opt/orientdb/orientdb-community-2.2.18-SNAPSHOT/www -Dorientdb.build.number=UNKNOWN@rf31f2e10de758cbdef4cee27451b4065b94d9ce2; 2017-03-04 00:50:53+0000 -cp /opt/orientdb/orientdb-community-2.2.18-SNAPSHOT/lib/orientdb-server-2.2.18-SNAPSHOT.jar:/opt/orientdb/orientdb-community-2.2.18-SNAPSHOT/lib/*:/opt/orientdb/orientdb-community-2.2.18-SNAPSHOT/plugins/* com.orientechnologies.orient.server.OServerMain
qcontrol 38625     1  0 12:56 ?        00:00:07 java -client -Xmx1024m -XX:MaxDirectMemorySize=512g -Djava.util.logging.config.file="/opt/orientdb/orientdb-community-2.2.18-SNAPSHOT/config/orientdb-client-log.properties" -Djava.awt.headless=true -Dclient.ssl.enabled=false -Dfile.encoding=utf-8 -Dorientdb.build.number=UNKNOWN@rf31f2e10de758cbdef4cee27451b4065b94d9ce2; 2017-03-04 00:50:53+0000 -cp /opt/orientdb/orientdb-community-2.2.18-SNAPSHOT/lib/orientdb-tools-2.2.18-SNAPSHOT.jar:/opt/orientdb/orientdb-community-2.2.18-SNAPSHOT/lib/*:/opt/orientdb/orientdb-community-2.2.18-SNAPSHOT/plugins/* -Djavax.net.ssl.keyStore=/opt/orientdb/orientdb-community-2.2.18-SNAPSHOT/config/cert/orientdb-console.ks -Djavax.net.ssl.keyStorePassword=password -Djavax.net.ssl.trustStore=/opt/orientdb/orientdb-community-2.2.18-SNAPSHOT/config/cert/orientdb-console.ts -Djavax.net.ssl.trustStorePassword=password com.orientechnologies.orient.graph.console.OGremlinConsole

Only when I "kill" that Java client program, the "jstat" command can be launched again:
orientdb@orient2:~$ kill 38625
orientdb@orient2:~$ /usr/lib/jvm/jdk1.8.0_92/bin/jstat -gc -h10 9879 10s
 S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     GCT
393216.0 393216.0 27548.1  0.0   2359296.0 2073754.0 11534336.0 5994826.1  39040.0 37640.2 4224.0 3794.0   7322  279.558   1      1.629  281.187
393216.0 393216.0  0.0   23546.8 2359296.0 1878604.6 11534336.0 5995314.1  39040.0 37640.2 4224.0 3794.0   7323  279.605   1      1.629  281.233
...

but from the output of "jstat", it looks like there is plenty of heap space available, no?

Hi @rdelangh OS limits an amount of threads but not amount of processes. Also threads are created outside of heap memory.  The method which you used to run jstat instance suggests me that you hit this limit. Lets check total amount of threads running in your system currently. Could you execute ps -eo nlwp | tail -n +2 | awk '{ num_threads += $1 } END { print num_threads }'

@laa
At this moment, that number of threads is more or less stable around 370
Meanwhile, the system-wide max nr of threads is about 1M :
$ cat /proc/sys/kernel/threads-max
1028888

Hm, @rdelangh very strange.
Let's try another way.
Could you print output of ````ulimit -u(limit of processes for current user) andps -eLf | grep 'my user' | wc -l``` total amount of processes for the user. If you have enough of memory then the allocation of new heap should not be a problem, so we need to look closer to OS limits.
Could you also do it once you have got the error mentioned above?

hi @laa
Number of processes surely will not be any issue, because we do not launch any client programs on this machine, all client programs run remotely on other servers.
So the number of processes remains static, independent of which clients are connecting, and VERY small:
$ ps -aef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 Jan31 ?        00:00:11 /sbin/init
root        16     1  0 Jan31 ?        00:00:14 /lib/systemd/systemd-journald
syslog      55     1  0 Jan31 ?        00:00:05 /usr/sbin/rsyslogd -n
root        61     1  0 Jan31 ?        00:00:06 /usr/sbin/cron -f
root       129     1  0 Jan31 ?        00:00:03 /usr/sbin/sshd -D
root       131     1  0 Jan31 pts/1    00:00:00 /sbin/agetty --noclear --keep-baud console<PHONE_NUMBER>0 9600 vt220
root       132     1  0 Jan31 pts/0    00:00:00 /sbin/agetty --noclear --keep-baud pts/0<PHONE_NUMBER>0 9600 vt220
root       133     1  0 Jan31 pts/1    00:00:00 /sbin/agetty --noclear --keep-baud pts/1<PHONE_NUMBER>0 9600 vt220
root       134     1  0 Jan31 pts/3    00:00:00 /sbin/agetty --noclear --keep-baud pts/3<PHONE_NUMBER>0 9600 vt220
root       135     1  0 Jan31 pts/2    00:00:00 /sbin/agetty --noclear --keep-baud pts/2<PHONE_NUMBER>0 9600 vt220
root      3327   129  0 10:56 ?        00:00:00 sshd: myuser  [priv]
myuser  3492  3327  0 10:56 ?        00:00:00 sshd: myuser @pts/4
myuser  3493  3492  0 10:56 pts/4    00:00:00 -bash
myuser  3573  3493  0 10:57 pts/4    00:00:00 ps -aef
orientdb  9879     1 99 Mar27 ?        8-19:49:58 java -server -Xms14G -Xmn3G -Xmx18G -XX:MetaspaceSize=64m -XX:MaxMetaspaceSize=64m -XX:CompressedClassSpa

The only additional program that I sometimes run on this server host, is a "jstat" command, or a "console.sh" (which in turns launches "java")

@rdelangh I have already written you before. It is called some processes, but in Linux terminology, it is a number of lightweight processes aka threads.  If you noticed in command which I provided there is flag ```L`` according to ps documentation it will make command to add NLWP (which is acronym for number of lightweight processes). For example snippet of execution of this command on currently running benchmark.

hi @laa, sorry for my ignorance, I know very well what the LWP are, but I was reading your comment too quickly.
So here are the outputs you asked for:
$ ulimit -u
514444
$ ps -eLf | grep 'my user' | wc -l
390

hello,
any update on this?
I run each time against an OOM situation, increasing the heap size params, but this seems really like a memory leakage :

currently already with the following settings: -Xms14G -Xmn3G -Xmx20G -XX:MetaspaceSize=64m -XX:MaxMetaspaceSize=64m -XX:CompressedClassSpaceSize=12m -XX:-UseAdaptiveSizePolicy

outputs from "jstat" at this moment:

$ /usr/lib/jvm/jdk1.8.0_92/bin/jstat -gcutil 41883
  S0     S1     E      O      M     CCS    YGC     YGCT    FGC    FGCT     GCT
  3.81   0.00  84.00  99.15  96.80  90.82<PHONE_NUMBER>2.863    38<PHONE_NUMBER>4.848

$ /usr/lib/jvm/jdk1.8.0_92/bin/jstat -gc -h10 41883 10s
 S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     GC                      T
393216.0 393216.0 10329.7  0.0   2359296.0 2359296.0 12031488.0 11928461.2 39808.0 38532.6 4224.0 3836.2<PHONE_NUMBER>2.730  38<PHONE_NUMBER>4.715
393216.0 393216.0  0.0   10553.7 2359296.0 104145.5 12031488.0 11928929.9 39808.0 38532.6 4224.0 3836.2<PHONE_NUMBER>2.804  38<PHONE_NUMBER>4.789
393216.0 393216.0  0.0   10553.7 2359296.0 229703.2 12031488.0 11928929.9 39808.0 38532.6 4224.0 3836.2<PHONE_NUMBER>2.804  38<PHONE_NUMBER>4.789
393216.0 393216.0  0.0   10553.7 2359296.0 329738.2 12031488.0 11928929.9 39808.0 38532.6 4224.0 3836.2<PHONE_NUMBER>2.804  38<PHONE_NUMBER>4.789
...

output from "ps" shows that the resident size of the process (= RAM usage) is now 65GB !

$ ps -o pid,vsz,rss,args 41883
  PID    VSZ   RSS COMMAND
41883 435728852 68394480 java -server -Xms14G -Xmn3G -Xmx20G -XX:MetaspaceSize=64m -XX:MaxMetaspaceSize=64m -XX:CompressedClassSpaceSize=12m ...

symptom: dbase can run fine for 1 week or so, until no more space seems to be reclaimed by garbage collections; the effect on the client programs is that their REST calls do not complete, but timeout. Other effect is that any modification via the console, such as creation of an index, do not even start.

The only solution possible is a "shutdown", then a restart.
I raise now the heap setting further to "-Xmx24G"
-> sure that this is not a case of memory leakage??

Your jvm has executed 38 fullgc and old is nearly full again. The last 3 lines do not show any gc (238564) or fullgc (38) increased number, and so nothing is cleaned (OU is stable because they are not fullgc), but Eden usage increases each time of 100m. If you enable detailed gclog and you see something like "java.lang.OutOfMemoryError: GC overhead limit exceeded" , then it could be you do not have hw resources to handle this huge heap cleanup (need more cpu), or there is nothing else to clean (so you need more heap or there is a leak). From jstat I can only see that average time, which seems to be fine (111.985 secs/38=2.94 secs). You can check and try to reduce promotion of objects (or keep it) to a small value, tuning New generation, with the best size of Survivors and Eden (add PrintGCDetails and PrintTenuringDistribution). The OU gc shows promoted objects (I see only 500k promoted in first line, but you should look at jstat log for a long time).  If your jvm do not survive with 24 gb, it could be a leak, and you can collect histogram (jmap -histo:live) to see which objects continously increase (note: the live collection of histogram requires an additional fullgc).

hi @cmassi , thanks for your extensive answer. I am not an experience Java programmer/administrator, but I will study your answer and try to analyze what's going on.
After 1 day of running the server process, I tried to launch the "jmap" command for the histogram, but this caused a sudden death of the server process. The very last logs, after I launched "jmap":
Error occurred during initialization of VM
java.lang.OutOfMemoryError: unable to create new native thread

Then the server process was gone. Poof. RIP.
Very frustrating, this Java memory management (or rather: lack of).
So, nothing else can be done apart from restarting the server. For sure, this sudden death has (again!!!) caused a lot of impact on the Lucene indexes, which go silently corrupted...
This server should help us to solve ICT matters, but so far it has only consumed zillions of man-days to get it stable...

by the way, hardware resources are not the cause of the problems: the server is running nearly idle (plenty of unused cores), and has still some 10's of GB unused RAM.
User limits are not the cause either:
$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 514444
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 514444
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

@rdelangh if the root cause of the jvm eating heap is a leak, it is an issue of the orientdb product (or jvm).   But the lack of resources is a configuration issue of the system, if you cannot start another java tool on the same server (unable to create new native thread). Without collection of data you have nothing to analyze to tune the server.  The jvm auto-tuning (AdaptiveSizePolicy) is always limited to generic installations without peaks (it react slowly); it is often needed human activity, to setup the best configuration.  Furthermore you can trace/limit the resources used by the tool you are using, with -J followed by any jvm option (this works on HotSpot jvm and tools).  e.g: on a server with 16gb ram, running orient 2.2.18 with  -Xms12g -Xmn3g -Xmx12g -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=128m -XX:+UseCompressedOops -XX:CompressedClassSpaceSize=128m -XX:-UseAdaptiveSizePolicy  , the following jmap collected successfully the histogram :  jmap  -J-Xmx32m -J-XX:+PrintGCDetails  -J-XX:ParallelGCThreads=2 -histo:live pid_orient

Instead of using ulimit, please use /proc/$pid/limits , for a running process

as far as the limits are concerned:
$ cat /proc/6287/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             514444               514444               processes
Max open files            10000                10000                files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       514444               514444               signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

The server process is started with the following options (among others):
java -server -Xms14G -Xmn3G -Xmx24G -XX:MetaspaceSize=64m -XX:MaxMetaspaceSize=64m -XX:CompressedClassSpaceSize=12m -XX:-UseAdaptiveSizePolicy -XX:+UseCompressedOops ...

My system has 128GB RAM.
The server process has been restarted, because an index was corrupted, and could not even be dropped to recreate it. After the restart, this all worked correct again.
When I run now the "jmap", the output is like this:
$ jmap -J-Xmx32m -J-XX:+PrintGCDetails  -histo:live pid_orient
[GC (Allocation Failure) [PSYoungGen: 1536K->512K(2048K)] 1536K->664K(7680K), 0.0287216 secs] [Times: user=0.52 sys=0.03, real=0.04 secs]

 num     #instances         #bytes  class name
----------------------------------------------
   1:         26102      419395424  [B
   2:         43235      179189624  [J
   3:       2299344      147158016  java.nio.DirectByteBuffer
   4:       1548045       74306160  java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync
   5:       1501788       48057216  java.util.concurrent.ConcurrentHashMap$Node
   6:        713884       45688576  com.orientechnologies.orient.core.storage.cache.OCachePointer
   7:       1548046       37153104  java.util.concurrent.locks.ReentrantReadWriteLock
   8:       2289152       36626432  java.util.concurrent.atomic.AtomicReference
   9:        760444       30417760  com.orientechnologies.orient.core.storage.cache.OCacheEntry
  10:        714096       28563840  java.lang.ref.Finalizer
...
2010:             1             16  sun.util.locale.provider.TimeZoneNameUtility$TimeZoneNameGetter
2011:             1             16  sun.util.resources.LocaleData
2012:             1             16  sun.util.resources.LocaleData$LocaleDataResourceBundleControl
Total      23777330<PHONE_NUMBER>
Heap
 PSYoungGen      total 2048K, used 1954K [0x00000000ff580000, 0x00000000ff980000, 0x0000000100000000)
  eden space 1536K, 93% used [0x00000000ff580000,0x00000000ff6e8a40,0x00000000ff700000)
  from space 512K, 100% used [0x00000000ff700000,0x00000000ff780000,0x00000000ff780000)
  to   space 512K, 0% used [0x00000000ff900000,0x00000000ff900000,0x00000000ff980000)
 ParOldGen       total 5632K, used 152K [0x00000000fe000000, 0x00000000fe580000, 0x00000000ff580000)
  object space 5632K, 2% used [0x00000000fe000000,0x00000000fe026020,0x00000000fe580000)
 Metaspace       used 3466K, capacity 4728K, committed 4864K, reserved 1056768K
  class space    used 380K, capacity 424K, committed 512K, reserved 1048576K

I will now startup my loading programs again, and stop them after an hour or so to collect a "jmap" again...

ok, have loaded a few 100's of files of data, current situation is now:
System resources:
$ vmstat 4
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0   2128 32551412<PHONE_NUMBER>3952    0    0    60   128    0    0  4  1 95  1  0
 0  0   2128 32550512<PHONE_NUMBER>3952    0    0     0     0  213  433  0  0 100  0  0
 0  0   2128 32550480<PHONE_NUMBER>3984    0    0     0   156  366  809  0  0 100  0  0
...

Process size:
$ ps -aefl|egrep -e 'PID|server'
F S UID        PID  PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD
0 S orientdb  6287     1 99  80   0 - 111418480 -  12:07 pts/5    02:52:35 java -server -Xms14G -Xmn3G -Xmx24G -XX:MetaspaceSize=64m -XX:MaxMetaspaceSize=64m -XX:CompressedClassSpaceSize=12m -XX:-UseAdaptiveSizePolicy -XX:+UseCompressedOops -Djna.nosys=true -XX:MaxDirectMemorySize=512g -Djava.awt.headless=true -Dfile.encoding=UTF8 -Drhino.opt.level=9 -Dstorage.openFiles.limit=1024 -Denvironment.dumpCfgAtStartup=true -Dquery.parallelAuto=true -Dquery.parallelMinimumRecords=10000 -Dquery.parallelResultQueueSize=200000 -Dstorage.wal.maxSize=51200 -Dstorage.diskCache.bufferSize=54000 -Dmemory.chunk.size=33554432 -Djava.util.logging.config.file=/opt/orientdb/orientdb-community-2.2.18-SNAPSHOT/config/orientdb-server-log.properties -Dorientdb.config.file=/opt/orientdb/orientdb-community-2.2.18-SNAPSHOT/config/orientdb-server-config.xml -Dorientdb.www.path=/opt/orientdb/orientdb-community-2.2.18-SNAPSHOT/www -Dorientdb.build.number=UNKNOWN@rf31f2e10de758cbdef4cee27451b4065b94d9ce2; 2017-03-04 00:50:53+0000 -cp /opt/orientdb/orientdb-community-2.2.18-SNAPSHOT/lib/orientdb-server-2.2.18-SNAPSHOT.jar:/opt/orientdb/orientdb-community-2.2.18-SNAPSHOT/lib/*:/opt/orientdb/orientdb-community-2.2.18-SNAPSHOT/plugins/* com.orientechnologies.orient.server.OServerMain

JMAP output:
$ jmap -J-Xmx32m -J-XX:+PrintGCDetails -histo:live 6287
[GC (Allocation Failure) [PSYoungGen: 1536K->512K(2048K)] 1536K->772K(7680K), 0.0162176 secs] [Times: user=0.31 sys=0.00, real=0.01 secs]

 num     #instances         #bytes  class name
----------------------------------------------
   1:        162136<PHONE_NUMBER>  [B
   2:        150574      469525112  [J
   3:       2133510      136544640  java.nio.DirectByteBuffer
   4:       2426339      116464272  java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync
   5:       2160832       69146624  java.util.concurrent.ConcurrentHashMap$Node
   6:       1076383       68888512  com.orientechnologies.orient.core.storage.cache.OCachePointer
   7:       2426340       58232160  java.util.concurrent.locks.ReentrantReadWriteLock
   8:       3308780       52940480  java.util.concurrent.atomic.AtomicReference
   9:       1100892       44035680  com.orientechnologies.orient.core.storage.cache.OCacheEntry
  10:       1076678       43067120  java.lang.ref.Finalizer
...
2170:             1             16  sun.util.locale.provider.SPILocaleProviderAdapter
2171:             1             16  sun.util.locale.provider.TimeZoneNameUtility$TimeZoneNameGetter
2172:             1             16  sun.util.resources.LocaleData
2173:             1             16  sun.util.resources.LocaleData$LocaleDataResourceBundleControl
Total      36548385<PHONE_NUMBER>
Heap
 PSYoungGen      total 2048K, used 1939K [0x00000000ff580000, 0x00000000ff980000, 0x0000000100000000)
  eden space 1536K, 92% used [0x00000000ff580000,0x00000000ff6e4cd8,0x00000000ff700000)
  from space 512K, 100% used [0x00000000ff700000,0x00000000ff780000,0x00000000ff780000)
  to   space 512K, 0% used [0x00000000ff900000,0x00000000ff900000,0x00000000ff980000)
 ParOldGen       total 5632K, used 260K [0x00000000fe000000, 0x00000000fe580000, 0x00000000ff580000)
  object space 5632K, 4% used [0x00000000fe000000,0x00000000fe041040,0x00000000fe580000)
 Metaspace       used 3464K, capacity 4728K, committed 4864K, reserved 1056768K
  class space    used 380K, capacity 424K, committed 512K, reserved 1048576K

It is these "[B" and "[J" classes that have grown tremendously.
-> OrientDB gurus: can you please comment if this is normal/expected behaviour, which might eventually lead to an OOM situation?

@rdelangh such amount of byte arrays should not be the case.  Could you create heap dump and analyze what is the GC root of all those byte array instances?

hi @laa, heap dump will be massive and will take very long time, but I will try.
Can you please elaborate how I can "analyze what is the GC root" ?

it depends what kind of heap dump analyzer you will have but in general GC root is the object which is on the top of a hierarchy  , one which contains other objects.  This is example from yourkit  https://www.yourkit.com/docs/java/help/gc_roots.jsp

done (taking heap dump took 7 hours...)
Using Exclipse's MAT, pls find in attach the different reports (zip files, containing HTML and img files; so to be viewed with a browser)
heap-20170411170911_Leak_Suspects.zip
heap-20170411170911_System_Overview.zip
heap-20170411170911_Top_Components.zip

@rdelangh Inspected the provided heap information. Seems like something is wrong with Lucene, its internals occupied 2GB+ memory out of total 3.3GB and spawned 300+ threads out of total 425. Are you extensively using Lucene indexes?

@rdelangh please can you share more info about your db structure:

number of classes (AFAIK you are using a class per week of data)
number of record for each class
number of indexes per class, expecially lucene
size of the database
sizes of lucene indexes, they are stored in the database folder under luceneIndexes subfolder
whatever can help to understand your data structure and usage :)

Our lucene indexes are keeped opened, even if no more data are going to be stored inside.
If I'm not wrong, you are using old data mainly in read-only way, isn't it?

@taburet : yes, extensively using Lucene indexes
@robfrank :

number of classes: currently in two databases: "cdrarch" (67 classes) and "mobile" (78 classes); indeed one class per week of data, in each of two databases
number of records: different classes for different purposes, containing a few K records till some 100's M records
number of indexes per class: in "cdrarch" dbase, 8 indexes/class, of which 6 Lucene; in "mobile" dbase, 2 indexes/class of which 1 Lucene
size of dbase: "cdrarch" 1.4 TB, "mobile" 680 GB
sizes of Lucene indexes: 160 GB in "cdrarch" dbase, 36 GB in "mobile" dbase
data structure is very basic: nearly 'flat', schema-full classes; mainly heavily and continuous loading of new records; counting of these records is done 4 times per week and results are stored in (much) smaller classes

AFAIU you create 6 different single field lucene indexes for each class.
So on cdarch you have more than 360 indexes.
My suggestion is to create a single multi-field index per class, this will reduce the number of indexes to  60, and so the number of threads. BUT, lucene eats ram, ad every other.
By default we use   memory mapped MMapDirectory, but you can configure the index to work with  NIOFSDirectory:
http://orientdb.com/docs/last/Full-Text-Index.html#lucene-writer-fine-tuning-expert
I don't know if the latter would increase performance/decrease ram occupation,but for sure a single index per class should be better. The only backdraw is that maybe you need to rewrite some queries, just to qualify the field name(s) if you are using different analyzers.

hi @robfrank , thanks for your valueable feedback!
The issue #7220 is not asserting me that such multi-field Lucene indexes really work. If that issue is resolved, then I will know if and how they can be used, and I will certainly run some tests with them.

Moreover, apart from reducing the memory-usage by using few multi-field Lucene indexes instead of several single-field Lucene indexes, how will this avoid that the Old-heap keeps on growing as we see, until an OOM?

What I noticed in the past days, is the following:

running only data-loading programs which do plain "INSERT", seems to keep the memory usage of ODB stable; these programs load approx 20M records per day in one database "cdrarch", plus some 22M records per day in the other database "mobile"; the pace of insertion for such records is then approx 1-3msec per record
when I activate another program that does "INSERT" of records, plus a lookup/INSERT/UPDATE of another record, all this in a so-called script, into the dbase "mobile" at a pace of some 100M records per day, it becomes way too much for ODB to be able to absorb; the pace of absorbing records becomes 8-15msec per record
at any time, ODB seems not to be able to make use of multiple CPU-cores in the machine: only a single core gets used, of course at approx 100% of capacity for that core leading me to assume that currently the bottleneck is CPU bound

I could try to raise the memory limits of the ODB server even a little higher from currently max 24GB to -say- 30GB, but maybe this will cause very long garbage collections of a few/many seconds
On top of all this, it is still not possible to spread the load over more than a single hardware server. See issue #6666 (open for months!)

I have again this situation where the server is freezing, altough:
as I wrote already above, this ODB server is dealing with two big databases, one named "cdrarch" and another named "mobile". The access to "cdrarch" is still fine, programs can stuff their records to this database at a normal pace.
A data-loading program to dbase "mobile" was also busy, altough much slower, until I launched a query to that dbase in parallel. That query was clearly slowing down very much the data-loading program to "mobile", to the point where I interrupted the query. This did not allow the data-loading program to pick up its pace, even worse: it was granted to a halt altogether ! So I stopped that program as well.
Meanwhile the data-loading to dbase "cdrarch" was still running fine.
I have no clue what the ODB server engine is doing now, regarding this dbase "mobile":

the machine is not out of RAM (still 20GB of the installed 128GB is unused)
I can capture "jstack" info, which looks like it busy very normally:

 S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     GCT
393216.0 393216.0  0.0   57083.8 2359296.0 754055.5 11537408.0 4442292.2  38400.0 37226.9 4096.0 3742.7  40217 1727.860   6     14.630 1742.490
393216.0 393216.0 35303.4  0.0   2359296.0 1424549.3 11537408.0 4448764.3  38400.0 37226.9 4096.0 3742.7  40218 1727.903   6     14.630 1742.533
...

I can start "console.sh", and connect to dbase "mobile", and run a small query
still, none of the data loading programs to "mobile", can send any single record to the ODB server; they can connect, send a batch of INSERT statements, but they hang waiting for any reply from ODB

-> what else can I do to find out why the ODB server is stuck for inserting records to one database ("mobile"), while it happily allows inserting records to another dbase?

And after some 20 minutes of total freeze of data-loading program into dbase "mobile", it finally started with its first records. Its speed is abnormally slow, however.
On the ODB server, system resources are fine (plenty of unused RAM, CPU fairly idle, disks not over-used)

@rdelangh so right now the speed of freeze is very slow right at this moment?

Looking at your last jstat -gc report you have had 6 fullgc with a  FGCT of 14.630 secs.  Has this number increased, and is time up to 34 secs or more (maybe you have had more fullgc in the meantime).  It would be better to confirm with a detailed gc log.  Can use to enable without restart:  jinfo -flag +PrintGCDateStamps;  jinfo -flag +PrintGCTimeStamps  jinfo -flag -XX:+PrintGCID  jinfo -flag -XX:+PrintGCDetails   And to disable use jinfo -flag -XX:-PrintGCDetails and so on

Meanwhile I did not see any reasonably fast solution apart from migrating this database "mobile" onto another, separate hardware server named "orient1".
At moments, the data-loading is still freezing, while another program runs a query on the same data.
The "jstat" output at this moment:
$ /usr/lib/jvm/jdk1.8.0_92/bin/jstat -gc -h10 1057 10s
 S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     GCT
393216.0 393216.0 143922.5  0.0   2359296.0 45404.0  11538432.0 6056022.2  37632.0 36230.9 4096.0 3653.8<PHONE_NUMBER>.991  13     52.214 7606.205
393216.0 393216.0 143922.5  0.0   2359296.0 1318014.1 11538432.0 6056022.2  37632.0 36230.9 4096.0 3653.8<PHONE_NUMBER>.991  13     52.214 7606.205
393216.0 393216.0 143922.5  0.0   2359296.0 1772914.5 11538432.0 6056022.2  37632.0 36230.9 4096.0 3653.8<PHONE_NUMBER>.991  13     52.214 7606.205
...

The commands you gave can only partially run successful:
$ /usr/lib/jvm/jdk1.8.0_92/bin/jinfo -flag +PrintGCDateStamps 1057
$ /usr/lib/jvm/jdk1.8.0_92/bin/jinfo -flag +PrintGCTimeStamps 1057
$ /usr/lib/jvm/jdk1.8.0_92/bin/jinfo -flag -XX:+PrintGCID 1057
Exception in thread "main" com.sun.tools.attach.AttachOperationFailedException: flag 'XX:+PrintGCID' cannot be changed

        at sun.tools.attach.LinuxVirtualMachine.execute(LinuxVirtualMachine.java:229)
        at sun.tools.attach.HotSpotVirtualMachine.executeCommand(HotSpotVirtualMachine.java:261)
        at sun.tools.attach.HotSpotVirtualMachine.setFlag(HotSpotVirtualMachine.java:234)
        at sun.tools.jinfo.JInfo.flag(JInfo.java:144)
        at sun.tools.jinfo.JInfo.main(JInfo.java:81)
orientdb@orient1:~$ /usr/lib/jvm/jdk1.8.0_92/bin/jinfo -flag -XX:+PrintGCDetails 1057
Exception in thread "main" com.sun.tools.attach.AttachOperationFailedException: flag 'XX:+PrintGCDetails' cannot be changed

        at sun.tools.attach.LinuxVirtualMachine.execute(LinuxVirtualMachine.java:229)
        at sun.tools.attach.HotSpotVirtualMachine.executeCommand(HotSpotVirtualMachine.java:261)
        at sun.tools.attach.HotSpotVirtualMachine.setFlag(HotSpotVirtualMachine.java:234)
        at sun.tools.jinfo.JInfo.flag(JInfo.java:144)
        at sun.tools.jinfo.JInfo.main(JInfo.java:81)

Where would I get this additional GC logging details? The output of "jstat" is still showing the same output columns:
$ /usr/lib/jvm/jdk1.8.0_92/bin/jstat -gc -h10 1057 10s
 S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     GCT
393216.0 393216.0  0.0   393193.3 2359296.0 1785915.9 11538432.0 6881507.2  37632.0 36231.5 4096.0 3653.8<PHONE_NUMBER>.596  13     52.214 7610.810
393216.0 393216.0 332867.2  0.0   2359296.0 1069934.1 11538432.0 7173228.7  37632.0 36231.5 4096.0 3653.8<PHONE_NUMBER>.410  13     52.214 7611.623
393216.0 393216.0 163408.9  0.0   2359296.0 1004431.1 11538432.0 7360261.5  37632.0 36231.5 4096.0 3653.8<PHONE_NUMBER>.936  13     52.214 7612.149
393216.0 393216.0 63640.0  0.0   2359296.0 1011200.5 11538432.0 7478940.9  37632.0 36231.5 4096.0 3653.8<PHONE_NUMBER>.389  13     52.214 7612.602
...

Soory, my mistake, due to cut&paste of complete options, recommended before. Syntax is without -XX, when used with jinfo. Only the name of the option, should be used, if manageable option. The jstat is a tool which provide only statistics from the start, alway with the same format, as requested with option (-gc). The detailed gclog is going to stdout, if not redirected with  -Xloggc:$ORIENTDBHOME/log/gc%p_%t.log , at startup.    Anyway, I see you have had FGC=13   FGCT=52.214

There is also the opposite option: -XX:+PrintGCApplicationConcurrentTime
See https://blogs.oracle.com/jonthecollector/the-unspoken-application-times

hi @cmassi , thx a lot for your time ! Highly appreciated because we are really struggling with ODB under high load.
Both of my attempts to set this extra logging option are failing:
$ /usr/lib/jvm/jdk1.8.0_92/bin/jinfo -flag -XX:+PrintGCApplicationStoppedTime 1057
Exception in thread "main" com.sun.tools.attach.AttachOperationFailedException: flag 'XX:+PrintGCApplicationStoppedTime' cannot be changed

        at sun.tools.attach.LinuxVirtualMachine.execute(LinuxVirtualMachine.java:229)
        at sun.tools.attach.HotSpotVirtualMachine.executeCommand(HotSpotVirtualMachine.java:261)
        at sun.tools.attach.HotSpotVirtualMachine.setFlag(HotSpotVirtualMachine.java:234)
        at sun.tools.jinfo.JInfo.flag(JInfo.java:144)
        at sun.tools.jinfo.JInfo.main(JInfo.java:81)

$ /usr/lib/jvm/jdk1.8.0_92/bin/jinfo -flag +PrintGCApplicationStoppedTime 1057
Exception in thread "main" com.sun.tools.attach.AttachOperationFailedException: flag 'PrintGCApplicationStoppedTime' cannot be changed

        at sun.tools.attach.LinuxVirtualMachine.execute(LinuxVirtualMachine.java:229)
        at sun.tools.attach.HotSpotVirtualMachine.executeCommand(HotSpotVirtualMachine.java:261)
        at sun.tools.attach.HotSpotVirtualMachine.setFlag(HotSpotVirtualMachine.java:234)
        at sun.tools.jinfo.JInfo.flag(JInfo.java:140)
        at sun.tools.jinfo.JInfo.main(JInfo.java:81)

It seems that these options can only be set in the command-line when starting up the ODB server.
-> please confirm that this is very valuable information, so that I recycle my server process.

I've added to the above list also the flag to activate from jinfo (which is not needed at startup): jinfo -flag +PrintGC pid_jvm
For the stopped time, it is not a manageable option.
See how to ask jvm which options : java -XX:+PrintFlagsFinal -version | grep manageable

I have not done any restart of the server process yet (need to wait a timeslot later today), but I get meanwhile the following GC log messages in the output, every 5 secs or so:
...
 [Times: user=0.91 sys=0.00, real=0.04 secs]
 [Times: user=1.39 sys=0.01, real=0.07 secs]
 [Times: user=0.80 sys=0.00, real=0.03 secs]
 [Times: user=0.89 sys=0.01, real=0.04 secs]
...

Please remember to add: jinfo -flag +PrintGC pid_jvm

done the "jinfo"
I see now in the output of the server process, almost every second a new line like these:

[GC (Allocation Failure)  6240876K->3881448K(14286848K), 0.0616845 secs]
[GC (Allocation Failure)  6240744K->3887007K(14286848K), 0.0433432 secs]
[GC (Allocation Failure)  6246303K->3877727K(14286848K), 0.0496305 secs]
[GC (Allocation Failure)  6236444K->3882282K(14286848K), 0.0267033 secs]
[GC (Allocation Failure)  6241578K->3887636K(14286848K), 0.0271547 secs]
...

The numbers are total_allocation_before->total_allocation_after (total capacity)
With the help of jstat -gc for current capacity of each generation (in your configuration capacity of generations can change dynamically) you have every second to clean the new area (eden+survivor) and promote something to old, so throughput is more or less 2.4g/s but all these objects are cleaned in up to 0.0.2 secs , which is a good job. Promoted objects are cleaned only with a fullgc in old, when nearly full, and to avoid as much as possible promotion in old, you should study it with PrintTenuringDistribution, and set size of survivor ratio.  With the setup of detailed gclog you are interested in time spent in each fullgc, to verify they are not too slow, and rule out gc activity and config sizes from the responsible activities of your hang issue (20 minutes)

May I close this issue? Does the "laziness" of lucene indexes improved the situation?

hello Frank,
absolutely positive impact of the "laziness" of Lucene indexes, many thanks for that!
We still encounter hanging server processes when -for example- some queries have been launched which are trying to access too much records. I guess that causes an OOM situation, but the server logfiles do not mention anything about the fact that they are stuck : no more client processes (console, or REST-api) can connect, the clean "shutdown" fails to get a connection, a gentle "kill" signal is not trapped...


[C++] Kernel matches read requests to pending commitments

Blocked by #906

PR #907 provides matching (without hints) in the ordering circuit and even makes the chopping for every read (which is clearly wrong as we should do that only when corresponding nullifiers are present.)

Done in #907


confict with metamask

Describe the bug
scam by metamask, bridge.ton.space
Expected behavior
scam by metamask, bridge.ton.space
Current behavior
scam by metamask, bridge.ton.space
Steps to Reproduce
scam by metamask, bridge.ton.space
Environment
scam by metamask, bridge.ton.space
Additional context

?

?


adding headers to request headers

Bug Description
I am trying to put DOWNSTREAM_PEER_SUBJECT value into new header and forward it to next hop but its not working since DOWNSTREAM_PEER_SUBJECT is not part of request headers. I can see DOWNSTREAM_PEER_SUBJECT in access logs via other EnvoyFilter. Any suggestions how to extract DOWNSTREAM_PEER_SUBJECT in envoyfilter? This is how the filter looks
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
annotations:
name: lua-filter
namespace: istio-system
resourceVersion: "24022983"
uid: f86043e6-c17a-4988-9f56-b621c1234c7e
spec:
configPatches:

applyTo: HTTP_FILTER
match:
context: GATEWAY
listener:
filterChain:
filter:
name: envoy.filters.network.http_connection_manager
subFilter:
name: envoy.filters.http.router
patch:
operation: INSERT_BEFORE
value:
name: envoy.lua
typed_config:
'@type': type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
inlineCode: |
function envoy_on_request(request_handle)
request_handle:headers():add("customer-id", "alice")
headers = request_handle:headers()
request_handle:headers():add("authority", request_handle:headers():get(":authority"))
headervalue = request_handle:headers():get("x-forwarded-client-cert")
request_handle:headers():add("roku_xfcc", headervalue)
headervalue = request_handle:headers():get("downstream_peer_subject")      ------------> getting null here
request_handle:headers():add("downstream_peer_subject", headervalue)
request_handle:logCritical("test1: "..request_handle:headers():get("x-forwarded-client-cert"))
request_handle:logCritical("test2: "..request_handle:headers():get("downstream_peer_subject"))
request_handle:logCritical("test3: "..request_handle:headers():get(":authority"))
end
workloadSelector:
labels:
istio: ingressgateway

try this code:
request_handle:headers():replace(":authority", new_autho)

thanks i updated the issue with more details. seems like for xfcc u need not to put : and that did the trick. Now trying to see how to extract downstream_peer_subject

found the way request_handle:streamInfo():downstreamSslConnection():subjectPeerCertificate()


SA1519 not detecting double using statement without braces

The following code does not produce SA1503 SA1519 for the first using where it probably should:
namespace Test
{
  using System.IO;

  public class TestClass
  {
    public void TestMethod()
    {
      using (var s = File.OpenRead("Test.data"))
      using (var b = new BinaryReader(s))
      {
        // ...
      }
    }
  }
}

:memo: It looks like it's coded to give SA1519 in this scenario.

Update: SA1519 does not currently consider using statements, hence no warning.

FWIW, StyleCop classic doesn't mind double using statements. I reckon this is probably intentional. I for one would have preferred to keep it that way.

@oatkins Sounds like a reasonable thing to make an option. Can you open a new issue proposing that?


[Testing] migrate from golang/mock to uber-go/mock

Objective
[ What? Describe the issue in 1-2 sentences]
Origin Document
[ Why? Justify the issue in 1-2 sentences. Consider adding a link or a screenshot.]
Goals

...

Deliverables

...

Non-goals / Non-deliverables

...

General deliverables

[ ] Comments: Add/update TODOs and comments alongside the source code so it is easier to follow.
[ ] Testing: Add new tests (unit and/or E2E) to the test suite.
[ ] Makefile: Add new targets to the Makefile to make the new functionality easier to use.
[ ] Documentation: Update architectural or development READMEs; use mermaid diagrams where appropriate.

Creator: @bryanchriswhite
Co-Owners: @Olshansk

@bryanchriswhite Because you ask me to remind you - friendly reminder that you can just insert screenshots as origin documents.

Closing because it's unlikely this will be in scope anytime soon. Can always reopen if need be.


Cards that are created from the find dialog don't show up in Recent

If you create stub cards while working on a card, and then go to recent, the stub cards don't show up, presumably because recent is about substantive edits, and the creation of a card isn't considered substantive? (Although it totally should be)

This is also true for concept cards if you're viewing everything/concepts and create another concept card via Find Card to Reference. That implies it's not a substantive edit problem, but a more general one

This is also true for concept cards if you're viewing everything/concepts and create another concept card via Find Card to Reference. That implies it's not a substantive edit problem, but a more general one


[VAX-1069] Output info about pg connection url when running starter scripts

The Postgres connection URL can be configured using an environment variable (DATABASE_URL). When working with multiple shells to interact with Electric, it's easy to overlook setting the variable in some of the shells.
The minimal patch would be to raise that the script is not using the default DATABASE_URL. "Ideally, it should display the URL details while obfuscating the password.
Note: Issue #480 is concerned with renaming DATABASE_URL to ELECTRIC_DATABASE_URL
From SyncLinear.com | VAX-1069

Check this is done in VAX-1087

Closing this as this has been implemented in #505.
yarn db:migrate and yarn db:psql now output the connection url without disclosing the username or password.

Reopened by accident.


Upgrading to Rails 4.2.7 breaks funcitonality.

Using Rails 4.2.7 caused my connection to mashape  that I use in my seed file to fail and generates the  warning:
[warning] The response contained in an RestClient::Exception is now a RestClient::Response instead of a Net::HTTPResponse, please update your code
Downgrading to 4.2.6 fixes issue. Not sure it's an actual Unirest issue or a Rails issue, so posting just FYI.

+1


keybase chat cli/gui timeout over proxy - OSX

I'm on OSX 10.11.6. Proxy settings are set on the command line with launchctl setenv HTTP_PROXY http://proxy:port for both HTTP and HTTPS environment variables, as well as via network settings in system preferences. Chat works perfectly when at home without a proxy, but is unusable when under a proxy at work in spite of me being able to login, view my profile and preferences, etc. (Have not tested KBFS extensively yet). Running keybase chat ls in the command line returns a network timeout, while the chat tab in the GUI is stuck at securing..., and after a few moments, contacts I've chatted with on the sidebar disappear from view (they return after a restart). Restarting the app via keybase ctl restart or unsetting and resetting proxy settings in the system does not solve the problem. Any thoughts? I've referred to issue #4071 when setting up my proxies.
my log id: df5bf19773edce762385431c

This might be a known issue, where chat conversations go blank after ctl restart. We're working on a fix.
If you restart the Keybase.app... pkill Keybase; open /Applications/Keybase.app it should workaround that issue.

I was able to test out this command at work today. Unfortunately, restarting Keybase with that command lets it appear logged in for about 30 seconds, only to say "Keybase is currently unreachable. Trying to reconnect you...", with me now unable to access any part of the app. This persists over several restarts, and is only fixed with keybase ctl restart (which brings me back to the original problem of the chat not loading, of course).
I'd also like to note that restarting my computer with proxy settings already set gives me the same message ("Keybase is currently unreachable..."), in that Keybase only seems to work with my proxy settings after having restarted it via keybase ctl restart. Odd.

@maxtaco Perhaps Gregor connections don't go over the proxy properly?

yeah that could very well be...
cc: @mmaxim  on that too....

oh the reachability check might not use proxy settings either

I reported and issue with the proxy support in #4557. It seems that DNS requests are never made over proxy - some in some cases it fails for corporate proxies. Maybe this is the problem you're having.

This is unfortunately a little difficult to fix, we have an internal ticket to try and address it, but there is no ETA as of yet. Sorry!

@mmaxim I don't have any insight into the code - but from what I've read, all you really need to do is not to try and do a DNS lookup when a proxy is defined. The proxy will resolve the host for the application making a query. Since wget, for example, supports this, I suppose there's some example code to look at there at least.
Of course you might have other things to consider that I'm not aware of - but the process itself seems fairly straight forward.


Chat - Inconsistency in send message button turning green in mweb& Android.

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: 1.4.63 - 0
Reproducible in staging?: Y
Reproducible in production?: Y
If this was caught during regression testing, add the test name, ID and link from TestRail: https://expensify.testrail.io/index.php?/tests/view/4499497
Logs: https://stackoverflow.com/c/expensify/questions/4856
Issue reported by: Applause-Internal team
Action Performed:

Open a report on both mweb & Android
Enter a character
Note send message button
In mweb, enter one more character

Expected Result:
There must not be inconsistency in send message button turning green in mweb& Android.
Actual Result:
After entering a character, in Android send message button turns green but in mweb after entering 2nd character only send message button turns green. Inconsistency in send message button turning green in mweb& Android.
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android: Native
[x] Android: mWeb Chrome
[ ] iOS: Native
[x] iOS: mWeb Safari
[ ] MacOS: Chrome / Safari
[ ] MacOS: Desktop

Screenshots/Videos
https://github.com/Expensify/App/assets/115492554/0d0a2f53-c7de-466a-9d7a-8fc29db3b378

View all open jobs on GitHub

We think this issue might be related to the #vip-vsb.

Huh, trying to test this in browserstack, but I can't seem got get mWeb to move on from the splash page. Checking in bug-zero.

Can't repro on 1.4.63 - 21 with a real device. Browserstack wasn't allowing me to access mweb and when I asked support they said that the App Live version isn't meant to work with mobile web?!

This has been labelled "Needs Reproduction". Follow the steps here: https://stackoverflowteams.com/c/expensify/questions/16989

Issue not reproducible during KI retests. (First week)


fixed access to localized strings in bundle

My hero! Thanks, man. The lack of documentation there had us confused.


Codeunit 370 Bank Acc. Reconciliation Post - CloseBankAccLedgEntry - OnBeforeCloseBankAccLedgEntry

Dear Development Team,
We would like to request a publisher in Codeunit 370 Bank Acc. Reconciliation Post in the CloseBankAccLedgEntry function right before modifying the Bank Account Ledger Entry.
In this Publisher, we would like to have access to both "Bank Account Ledger Entry" and "Bank Account Recon. Line" tables.

Much appreciated.
Cheers,
Marton Nagy

The best way to request a new event is to implement it locally, verify how it works for you and then send to us the code snippets. It does not guarantee we will simply copy&paste but it helps understand what you really need.
I assume you need to pass BankAccLedgEntry by reference and BankAccReconLine by value?

Hi Stan,
OK, will do next time.
Yes, please.
Cheers,
Marton


Separate page or space for archived projects

Currently, dormant and inaccessible projects are features on the Projects page alongside active ones. Now that the page is getting crowded, we need to be able to mark some projects as "archived" to send them to a separate page for projects that aren't currently usable and aren't seeking development help.

I added an "archived: true" flag to the YAML files for some defunct projects, and made them not appear on the projects page. We still need an "archived projects" page where they will appear.

overtaken by events


Creation of Base Model

#1 Working iteration of ETH Price prediction tool built with jupyther notebook and facebook prophet and visualised with pyplot on Datapane.
-Creation of README.md
-Creation of ethPrediction.ipynb
-Creation of prophetplot.svg
-gitignore .env files
-exploratory data analysis of recent data for trends
-prediction of ETH Price for next day
-data visualisation and interactive graphs

Credit to Benedict Neo for his impeccable write-up and Tina Huang for the inspiration to do this project as well as the recent crypto crash in emptying my wallet and giving me the motivation to do something about it with my programming knowledge. Will improve on this model and consider testing and benchmarking with paper trading.
Milestone 2 will focus on integration into a web application of sort / telegram bot. (Depends on my mood) Expect user interfaces (wow!) and other cool buttons and user forms. Might add support to BTC. For now, I will work on this iteratively.
Links to interactive forecasted graph of ETH Price:
https://datapane.com/reports/Y3YpzzA/time-series-plot-of-ethereum-open-price/
https://datapane.com/reports/E7ydbp3/eth-price-forecast/
https://datapane.com/reports/9ArQlBk/eth-price-components/


On every HUD, esc key should close the window.

This key behavior is missing into the following HUDS

[x] Explore

[x] Profile

[x] Settings

[x] Help

[ ] Avatar editor

Weird thing, avatar editor closes with ESC for me. Can you double check it's still happening @RominaMarchetti ?
The rest are indeed not being closed by the key.


Index out of range trying to load nameservers

It seems to have problems to fetch the DNS list
╰─$ nameinator -debug                                                                                [2]
starting NAMEinator - version custom
understood the following configuration: {numberOfDomains:100 debug:true contest:true nameserver:}
-------------
NOTE: as this is an alpha - we rely on feedback - please report bugs and featurerequests to https://github.com/mwiora/NAMEinator/issues and provide this output
OS: linux ARCH: amd64
-------------
DEBUG: nslookup output
Server:         <IP_ADDRESS>
Address:        <IP_ADDRESS>#53

Non-authoritative answer:
*** Can't find .: No answer

DEBUG: dns server
<IP_ADDRESS>
trying to load nameservers from datasrc/nameserver-globals.csv
panic: runtime error: index out of range [1] with length 1

goroutine 1 [running]:
main.readNameserversFromFile(0xc000096bd0, 0x0, 0x0, 0x0, 0x55e1bf1af54a, 0x1e)
        NAMEinator/loader.go:52 +0x45f
main.prepareBenchmarkNameservers(0xc000096bd0, 0x0, 0x0, 0x0)
        NAMEinator/loader.go:17 +0xb9
main.prepareBenchmark(0xc000096bd0, 0x0, 0x0, 0x0, 0xc000096c00, 0x0, 0x0, 0x0)
        NAMEinator/main.go:126 +0x45
main.main()
        NAMEinator/main.go:164 +0xad

Had the same issue. I just compiled it myself & that fixed it :)
NAMEinator.zip

Had the same issue. I just compiled it myself & that fixed it :)
NAMEinator.zip

did you change anything to make it work? Cause your .exe is not very useful on linux ^^

Oh, sorry; I figured you were using Windows. I think it just needs the go project directory in it's original position to find the nameservers list (the compiled binary alone threw a file not found error). I currently have WSL (windows subsystem linux) ubuntu on my computer, so I'll give the same process a try on Linux & update you on how it goes

it also works when you execute it with the "datasrc" folder inside the working dir - but thanks anyway :)

it also works when you execute it with the "datasrc" folder inside the working dir - but thanks anyway :)

I'm glad you figured it out! :)

Same cause as: https://github.com/mrwiora/NAMEinator/issues/56#issuecomment-1312380623
You can follow the steps there if you need a bit more guidance.


problem with ptest.print()

I've been using pyRiemann for several days and it's working very well - very impressive.  I did have an issue with the ptest.print() function - maybe I am not initializing something correctly.   Anyway, here's a demo:

for me, it produces this output:

Thanks Alex,
Jonathan Miller.

Hi Jonahathan,
Thank you for your feedback.
I actually made a mistake in my example code, because i rename the print() method to summary() to mimic the standard API.
so, instead of ptest.print(), use print(ptest.summary())
Again, sorry about that.
I'm closing the issue.

Thanks Alex,
Works fine.
Jonathan.


Some unit tests fail in the build intermittently

I've had a few PR builds fail for unit tests which were unaffected by my changes.  They appear to be intermittent failures.  Creating this issue to track.  I suspect that the individual tests may not be to blame, but rather some environmental issue.  Maybe some other test doesn't clean things up properly or something?  Not sure of the root cause.
Builds that failed

https://github.com/microsoft/MixedRealityToolkit-Unity/pull/5949/checks?check_run_id=230208221

PR: #6074
Microsoft.MixedReality.Toolkit.Tests.BoundingBoxTests.ScaleViaNearInteration
Unhandled log message: '[Assert] Assertion failed'. Use UnityEngine.TestTools.LogAssert.Expect
Microsoft.MixedReality.Toolkit.Tests.<ScaleViaNearInteration>d__5:MoveNext() (at Assets/MixedRealityToolkit.Tests/PlayModeTests/BoundingBoxTests.cs:119)
UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)

Microsoft.MixedReality.Toolkit.Tests.BaseCursorTests.CursorContextMove

PR #6075
A3CursorContextMove1 (0.350s)
---
Expected: MoveCross
  But was:  None
---
at Microsoft.MixedReality.Toolkit.Tests.BaseCursorTests.VerifyCursorContext (Microsoft.MixedReality.Toolkit.Input.IMixedRealityCursor cursor, Microsoft.MixedReality.Toolkit.Input.CursorContextEnum context) [0x0000f] in H:\mrtk.dev\Assets\MixedRealityToolkit.Tests\PlayModeTests\BaseCursorTests.cs:89
at Microsoft.MixedReality.Toolkit.Tests.BaseCursorTests.VerifyCursorContextFromPointers (System.Collections.Generic.IEnumerable`1[T] pointers, Microsoft.MixedReality.Toolkit.Input.CursorContextEnum context) [0x00024] in H:\mrtk.dev\Assets\MixedRealityToolkit.Tests\PlayModeTests\BaseCursorTests.cs:81
at Microsoft.MixedReality.Toolkit.Tests.BaseCursorTests+<CursorContextMove>d__10.MoveNext () [0x00309] in H:\mrtk.dev\Assets\MixedRealityToolkit.Tests\PlayModeTests\BaseCursorTests.cs:246
at UnityEngine.TestTools.EnumerableTestMethodCommand+<ExecuteEnumerableAndRecordExceptions>c__Iterator1.MoveNext () [0x0002f] in C:\buildslave\unity\build\Extensions\TestRunner\UnityEngine.TestRunner\NUnitExtensions\Commands\EnumerableTestMethodCommand.cs:58

https://github.com/microsoft/MixedRealityToolkit-Unity/pull/5949/checks?check_run_id=233372911

PR #6075
Microsoft.MixedReality.Toolkit.Tests.InteractableTests.TestInputActionMenuInput
Unhandled log message: '[Exception] AssertionException:   Transform target object was not translated by action.
  Expected: True
  But was:  False
'. Use UnityEngine.TestTools.LogAssert.Expect

Microsoft.MixedReality.Toolkit.Tests.InteractableTests.TestInputActionSelectInput
Unhandled log message: '[Error] No EventSystem detected. UI events will not be propagated to Unity UI.'. Use UnityEngine.TestTools.LogAssert.Expect

https://github.com/microsoft/MixedRealityToolkit-Unity/pull/6069/checks?check_run_id=236048579

Not failing in latest stabilization. 100/100 tests passed.
Microsoft.MixedReality.Toolkit.Tests.PointerBehaviorTests.TestGrab
Incorrect state for Left Line Pointer.IsInteractionEnabled
  Expected: False
  But was:  True

InteractableTests.TestVoiceInputOnPrefab

PR #6082
TestVoiceInputOnPrefab (0.397s)
---
Transform target object was not translated by action.
  Expected: True
  But was:  False
---
at Microsoft.MixedReality.Toolkit.Tests.InteractableTests+<CheckButtonTranslation>d__21.MoveNext () [0x0008b] in H:\mrtk.dev\Assets\MixedRealityToolkit.Tests\PlayModeTests\InteractableTests.cs:560
at UnityEngine.TestTools.EnumerableTestMethodCommand+<ExecuteEnumerableAndRecordExceptions>c__Iterator1.MoveNext () [0x0002f] in C:\buildslave\unity\build\Extensions\TestRunner\UnityEngine.TestRunner\NUnitExtensions\Commands\EnumerableTestMethodCommand.cs:58
---
No main camera found. The Mixed Reality Toolkit requires at least one camera in the scene. One will be generated now.

ManipulationHandlerOneHandMoveFar

PR #6084
A13ManipulationHandlerOneHandMoveFar0 (9.220s)
---
Unhandled log message: '[Assert] Grab point on object changed during rotation, expected (-0.780, 0.732, 4.830), was (-0.327, 0.751, 5.057)'. Use UnityEngine.TestTools.LogAssert.Expect
---
Microsoft.MixedReality.Toolkit.Tests.TestUtilities:AssertAboutEqual(Vector3, Vector3, String, Single) (at Assets/MixedRealityToolkit.Tests/TestUtilities.cs:195)
Microsoft.MixedReality.Toolkit.Tests.<ManipulationHandlerOneHandMoveFar>d__8:MoveNext() (at Assets/MixedRealityToolkit.Tests/PlayModeTests/ManipulationHandlerTests.cs:477)
UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)
---

Marking these are blocking / urgency soon because they are impacting teams' pull request completion velocity.

I'm seeing this locally too; when I run tests on my local machine on 2018.4.x, I get very sporadic test failures. Locally, the two tests that just now sporadically failed are:

NearInteractionTouchableTests - NearInteractionTouchableSetTouchableCollider
FocusProviderTests - TestGazeProviderTargetUnityUI

Hope this can help diagnose this.