Trial tracemalloc as memory tracker (replacement v2 PR)

WIP DO NOT MERGE THIS, EVER
Some modified code for memory benchmarks, evaluating "tracemalloc" against our existing Linux process-RSS measuring technique.
Additional context : sample output from a desktop run
$ asv run -b Memcheck* --quick -e -E existing:/tmp/persistent/newconda-envs/iris3/bin/python
Couldn't load asv.plugins._mamba_helpers because
No module named 'libmambapy'
· Ignoring ASV setting(s): `python`. Benchmark environment management is delegated to third party script(s).
· Discovering benchmarks
· Running 4 total benchmarks (1 commits * 1 environments * 4 benchmarks)
[ 0.00%] ·· Benchmarking existing-py_tmp_persistent_newconda-envs_iris3_bin_python
[ 0.00%] ··· Importing benchmark suite produced output:
[ 0.00%] ···· /net/home/h05/itpp/git/iris/iris_main/benchmarks/benchmarks/generate_data/__init__.py:52: UserWarning: No BENCHMARK_DATA env var, defaulting to /net/home/h05/itpp/git/iris/iris_main/benchmarks/.data. Note that some benchmark files are GB in size.
              warn(message)

[12.50%] ··· memtrace_evaluation.MemcheckBlocksAndWorkers.time_calc                                                                                ok
[12.50%] ··· ========= ========== ========= =========
             --                   nworkers
             --------- ------------------------------
              nblocks      1          4         9
             ========= ========== ========= =========
                 1      568±0ms    543±0ms   514±0ms
                 4      144±0ms    181±0ms   151±0ms
                 9      55.9±0ms   116±0ms   109±0ms
             ========= ========== ========= =========

[25.00%] ··· memtrace_evaluation.MemcheckBlocksAndWorkers.track_addmem_calc                                                                                ok
[25.00%] ··· ========= ======= ======= =======
             --                nworkers
             --------- -----------------------
              nblocks     1       4       9
             ========= ======= ======= =======
                 1      305.3   305.3   305.3
                 4       19.2    76.4    76.4
                 9       4.0     15.2    33.9
             ========= ======= ======= =======

[37.50%] ··· memtrace_evaluation.MemcheckRunstyles.time_calc                                                                                ok
[37.50%] ··· ============= ================= ================= =================== ===================
             --                                          runtype / ysize
             ------------- ---------------------------------------------------------------------------
                measure     threads / 10000   threads / 40000   processes / 10000   processes / 40000
             ============= ================= ================= =================== ===================
              tracemalloc       91.2±0ms          228±0ms            40.2±0ms            82.4±0ms
                  rss           87.0±0ms          222±0ms            45.5±0ms            81.5±0ms
             ============= ================= ================= =================== ===================

[50.00%] ··· memtrace_evaluation.MemcheckRunstyles.track_addmem_calc                                                                                ok
[50.00%] ··· ============= ================= ================= =================== ===================
             --                                          runtype / ysize
             ------------- ---------------------------------------------------------------------------
                measure     threads / 10000   threads / 40000   processes / 10000   processes / 40000
             ============= ================= ================= =================== ===================
              tracemalloc         17.0              67.9               0.2                 0.2
                  rss             9.1               34.4               0.9                 0.9
             ============= ================= ================= =================== ===================

What I think we can deduce from the benchmark results

neither "rss" or "tracemalloc" tracks memory in subprocesses

compare MemcheckRunstyles.track_addmem_calc , any "threads" with "processes"

"rss" method is not picking up on memory used in multiple threads, but "tracemalloc" is

compare MemcheckRunstyles.track_addmem_calc , "threads" results ..
.. these are ~2 times as big for "tracemalloc" as for "rss" (this for nblocks=6, nworkers=4)

Updated ideas
@trexfeathers pointed out that the above were run on a machine with only 4 CPUs (hyperthread), hence nworkers=4/9 can be expected to be questionable.
Also a bit difficult to interpret -- questioned whether the nworkers/nblocks axes were correct !
Hence changes in https://github.com/SciTools/iris/pull/5946/commits/d039ad03bc2b131a9691f4f6e7e2bdbb3b74f9cb

n_workers defaults  to 3
n_workers = [1, 2, 3, 4] in BlocksAndWorkers testing
MemcheckBlocksAndWorkers_Rss repeats the n/n matrix with the Rss measure

New results
typical run ...
$ asv run -b Memcheck --quick -e -E existing:/tmp/persistent/newconda-envs/iris3/bin/python
---
    Couldn't load asv.plugins._mamba_helpers because
    No module named 'libmambapy'
    · Ignoring ASV setting(s): `python`. Benchmark environment management is delegated to third party script(s).
    · Discovering benchmarks
    · Running 6 total benchmarks (1 commits * 1 environments * 6 benchmarks)
    [ 0.00%] ·· Benchmarking existing-py_tmp_persistent_newconda-envs_iris3_bin_python
    [ 0.00%] ··· Importing benchmark suite produced output:
    [ 0.00%] ···· /net/home/h05/itpp/git/iris/iris_main/benchmarks/benchmarks/generate_data/__init__.py:52: UserWarning: No BENCHMARK_DATA env var, defaulting to /net/home/h05/itpp/git/iris/iris_main/benchmarks/.data. Note that some benchmark files are GB in size.
                  warn(message)

    [ 8.33%] ··· memtrace_evaluation.MemcheckBlocksAndWorkers.time_calc                                                                                ok
    [ 8.33%] ··· ========= ========== ========== ========= =========
                 --                         nworkers
                 --------- -----------------------------------------
                  nblocks      1          2          3         4
                 ========= ========== ========== ========= =========
                     1      546±0ms    508±0ms    518±0ms   517±0ms
                     4      120±0ms    156±0ms    151±0ms   139±0ms
                     9      54.6±0ms   71.2±0ms   141±0ms   122±0ms
                 ========= ========== ========== ========= =========

    [16.67%] ··· memtrace_evaluation.MemcheckBlocksAndWorkers.track_addmem_calc                                                                                ok
    [16.67%] ··· ========= ======= ======= ======= =======
                 --                    nworkers
                 --------- -------------------------------
                  nblocks     1       2       3       4
                 ========= ======= ======= ======= =======
                     1      305.3   305.3   305.3   305.3
                     4       19.2    38.3    57.3    76.4
                     9       4.0     7.7     11.5    15.2
                 ========= ======= ======= ======= =======

    [25.00%] ··· memtrace_evaluation.MemcheckBlocksAndWorkers_Rss.time_calc                                                                                ok
    [25.00%] ··· ========= ========== ========== ========== ==========
                 --                          nworkers
                 --------- -------------------------------------------
                  nblocks      1          2          3          4
                 ========= ========== ========== ========== ==========
                     1      501±0ms    518±0ms    570±0ms    597±0ms
                     4      121±0ms    150±0ms    132±0ms    161±0ms
                     9      52.4±0ms   67.5±0ms   64.3±0ms   64.0±0ms
                 ========= ========== ========== ========== ==========

    [33.33%] ··· memtrace_evaluation.MemcheckBlocksAndWorkers_Rss.track_addmem_calc                                                                                ok
    [33.33%] ··· ========= ======= ======= ======= =======
                 --                    nworkers
                 --------- -------------------------------
                  nblocks     1       2       3       4
                 ========= ======= ======= ======= =======
                     1      305.8   305.8   305.8   305.8
                     4       19.8    38.9    20.0    20.8
                     9       4.6     8.0     11.6    15.3
                 ========= ======= ======= ======= =======

    [41.67%] ··· memtrace_evaluation.MemcheckRunstyles.time_calc                                                                                ok
    [41.67%] ··· ============= ================= ================= =================== ===================
                 --                                          runtype / ysize
                 ------------- ---------------------------------------------------------------------------
                    measure     threads / 10000   threads / 40000   processes / 10000   processes / 40000
                 ============= ================= ================= =================== ===================
                  tracemalloc       97.5±0ms          208±0ms            39.9±0ms            94.1±0ms
                      rss           49.5±0ms          203±0ms            51.5±0ms            91.6±0ms
                 ============= ================= ================= =================== ===================

    [50.00%] ··· memtrace_evaluation.MemcheckRunstyles.track_addmem_calc                                                                                ok
    [50.00%] ··· ============= ================= ================= =================== ===================
                 --                                          runtype / ysize
                 ------------- ---------------------------------------------------------------------------
                    measure     threads / 10000   threads / 40000   processes / 10000   processes / 40000
                 ============= ================= ================= =================== ===================
                  tracemalloc         12.8              51.0               0.2                 0.2
                      rss             13.2              51.1               0.8                 0.8
                 ============= ================= ================= =================== ===================

Some interpretation
RunStyle checks are now more understandable.
MemcheckRunstyles.time_calc (run **WITHOUT** --quick)
      --                                          runtype / ysize
      ------------- ---------------------------------------------------------------------------
         measure     threads / 10000   threads / 40000   processes / 10000   processes / 40000
      ============= ================= ================= =================== ===================
       tracemalloc       58.2±8ms          179±5ms            44.3±1ms            86.4±3ms
           rss           71.1±6ms          171±8ms            43.7±3ms            91.2±3ms
      ============= ================= ================= =================== ===================

BUT N.B. results are still not very stable from time to time
      ============= ================= ================= =================== ===================
       tracemalloc       62.9±5ms          172±10ms           41.8±4ms            89.1±6ms
           rss           62.3±7ms          173±9ms            49.1±4ms           91.3±0.9ms
      ============= ================= ================= =================== ===================
       tracemalloc       58.2±3ms          152±20ms           44.9±5ms            91.8±5ms
           rss          58.7±10ms          170±7ms            46.1±2ms            91.6±7ms
      ============= ================= ================= =================== ===================

((but generally, ARE about the same for both methods, as we would expect))
conclude :

~same for methods
processes slightly quicker than threads generally (??odd?)

MemcheckBlocksAndWorkers
General picture
memtrace_evaluation.MemcheckBlocksAndWorkers.time_calc
     --                         nworkers
     --------- -----------------------------------------
      nblocks      1          2          3         4
     ========= ========== ========== ========= =========
         1      546±0ms    508±0ms    518±0ms   517±0ms
         4      120±0ms    156±0ms    151±0ms   139±0ms
         9      54.6±0ms   71.2±0ms   141±0ms   122±0ms
     ========= ========== ========== ========= =========

memtrace_evaluation.MemcheckBlocksAndWorkers.track_addmem_calc
     --                    nworkers
     --------- -------------------------------
      nblocks     1       2       3       4
     ========= ======= ======= ======= =======
         1      305.3   305.3   305.3   305.3
         4       19.2    38.3    57.3    76.4
         9       4.0     7.7     11.5    15.2
     ========= ======= ======= ======= =======

now makes more sense : can see memory reducing with nblocks, and increasing with nworkers
compare methods
memtrace_evaluation.MemcheckBlocksAndWorkers_Rss.track_addmem_calc
     --                    nworkers
     --------- -------------------------------
      nblocks     1       2       3       4
     ========= ======= ======= ======= =======
         1      305.8   305.8   305.8   305.8
         4       19.8    38.9    20.0    20.8
         9       4.6     8.0     11.6    15.3
     ========= ======= ======= ======= =======

conclude

results are mostly similar,
... except RSS tends to under-read ..
.. and EXCEPT for nblocks=4
* TYPICAL (see e.g. below)

/nworkers
1
2
3
4

tracemalloc
19.2
38.3
57.3
76.4

rss
19.8
38.9
20.0
20.8

POSSIBLY

RSS shows re-use of blocks within the operation ?
.. or RSS "misses" some pages ?

generally, it is NOT like RSS is only tracking 1 worker

since, with many blocks, mem-usage generally grows with nworkers, just as for tracemalloc

Outcome

methods are ~equivalent in scope,

both can measure across threads
neither can measure across processes

but tracemalloc is probably more reliable

rss results are les repeatable
rss seems to under-read in some cases

Conclusions:
Since we fixed the task-size scaling (and added a process-based BlocksAndWorkers), the results are now roughly :

Runstyle checks show that old + new memory schemes delivery roughly the same numbers

both methods can track memory in thread workers, but not subprocesses
(we find that tracemalloc delivers more stable answers -- but this was not re-retested)

threaded workers : space required reduces with nblocks + increases with nworkers, as you would expect
threaded workers : time taken is ~constant :  this seems to indicate that due to Python GIL requirements, they don't actually run in parallel
parallel workers : time taken scales as expected (they do run in parallel, and task division gives speedups)

Note: the results reported in the whole-benchmark runs behave a little differently.

the parameter tables are laid out differently in some cases, which is odd but maybe due to some additional config in the GHA action ?
the memory scaling for threaded workers is different -- I suspect the number of available CPUs is different

Added
Investigate repeated ops with simulated memory 'leak'.
Just for reference.
A sample output ...
$ python bm_runner.py custom run -- -b "MemcheckTaskRepeats" -e --quick -E existing:/tmp/persistent/newconda-envs/iris3/bin/python
 . . .
    BM_RUNNER DEBUG: asv run -b MemcheckTaskRepeats -e --quick -E existing:/tmp/persistent/newconda-envs/iris3/bin/python
 . . .
· Ignoring ASV setting(s): `python`. Benchmark environment management is delegated to third party script(s).
· Discovering benchmarks
· Running 1 total benchmarks (1 commits * 1 environments * 1 benchmarks)
[ 0.00%] ·· Benchmarking existing-py_tmp_persistent_newconda-envs_iris3_bin_python
[ 0.00%] ··· Importing benchmark suite produced output:
[ 0.00%] ···· /net/home/h05/itpp/git/iris/iris_main/benchmarks/benchmarks/generate_data/__init__.py:52: UserWarning: No BENCHMARK_DATA env var, defaulting to /net/home/h05/itpp/git/iris/iris_main/benchmarks/.data. Note that some benchmark files are GB in size.
              warn(message)

[100.00%] ··· memtrace_evaluation.MemcheckTaskRepeats.track_mem                                                                                ok
[100.00%] ··· ======= =======
               nreps
              ------- -------
                 1      76.8
                 2      77.1
                 3     102.9
                 4     103.3
              ======= =======

Some combined results
                   nreps
                  ------- -------
                     1      76.8,  76.8,  76.9,  76.8,  76.8,  76.8,
                     2      77.1,  77.0,  77.1,  77.0,  77.0,  77.1,
                     3     102.8, 103.0, 103.0, 103.0, 103.0, 102.9,
                     4     103.3, 103.2, 103.5, 103.4, 103.2, 103.3,
                  ======= =======
(( incremental differences, from above numbers ))
                     1-2     0.3   0.2   0.2   0.2   0.3
                     2-3    25.7  26.0  25.9  26.0  25.8
                     3-4     0.5   0.2   0.5   0.4   0.4
                  ======= =======

Conclusions
The allocation pattern is a trifle hard to explain.
Sizes don't relate very directly to the actual allocations
This does bring into question how stable results are likely to be.


Datastore: Allow empty lists in protobuf array value

Fixes #3152.
@lukesneeringer This was your fix, so I believe you should be able to get me an LGTM soon right?  :)
@dhermes I would like to ask that you are ok with it since I mostly referred to your test_system code.
@tseaver Do you mind looking it over just so I didn't miss anything?
I consolidated all of our responses and comments and made this pull request.
I backed out the code change and the errors happen.
Separately, I have also investigated that
array_values{
  values{
  }
}

would raise ValueError when put into an entity.  It might be desired behavior, but we might need to deal with it at some point.  The issue is that len(array_value.values) is 1 and the result for this was not [] as I had expected in the following code:
        result = [_get_value_from_value_pb(value)
                  for value in value_pb.array_value.values]

Anyway, shall we get this merged soon please?

I'll try again later.

Superseded by #4915.


ethofs.calculateCost(readableStreamForFile, options)

The cost calculation should not require the upload of a file, but only make a simple calculation and return the cost based on a number of bytes.

Integrated on https://github.com/Ether1Project/ethofs-sdk/commit/32e9cdf3134d474288bb18a72b99d6a41253092d

Implemented


Actually log out user from AWS Cognito upon /logout action

Currently, only the browser session seems to be invalidated with the OAuth token still remaining active. Hence, after having logged out by just clicking "Sign in" one is immediately logged in again without AWS Cognito asking for username and password once more.

Not an issue but working as intended because Cognito works as a a SSO endpoint here and the user might want to be logged into other apps at the same time and not want to log out from those as well.


[BLOG] Building your first ML Model with ML Commons

Describe the blog post your would like to write
There should a blog that describes end to end how to create, use, and update a ML Commons model in OpenSearch. This blog should show the surface level work that is required as well as describe what is happening behind the scenes.
What is the title of the blog post?
Building your first ML Model with ML Commons
Who are the authors?
@ylwu-amzn and @dtaivpp
What is the proposed posting date?
7/15/2022

any progress on this @dtaivpp?


registry: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers), retrying

Description
The following error is appearing repeatedly in the registry logs since our yesterday upgrade to 2.4:
registry_1    | 2019-03-14T09:58:01.231650321Z time="2019-03-14T09:58:01Z" level=error msg="retryingsink: error writing events: httpSink{https://registry.redacted.com/v2/webhooks/events}: error posting: Post https://registry.redacted.com/v2/webhooks/events: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers), retrying"
Steps to reproduce

Stop all containers with docker-compose down -v to remove volumes as well
Update docker-compose.yml to use 2.4 images
Start all containers with docker-compose up

Expected behavior: no errors in logs
Actual behavior: The error shown above keeps appearing every ~3 seconds

I started to look at the logs because user reported that a deleted image still appeared in the web (which seems logical if the webhook requests are failing I suppose).
Deployment information
** deployed using docker-compose like in the examples folder
Configuration:

certificates provided by LetsEncrypt
entering the container and resolving the hostname from it works OK, no apparent connectivity issues in this regard between registry and portus
no configuration changes with respect to the 2.3 deployment

Portus version:  2.4.2@f4d6914850a43f26049501a49c4dde2af4e8b415 (from docker image 2.4)

I'm deploying Portus internally and this error started to appear as soon as I had finished to push an image to test if push and pull was working (which are working).
Portus version: 2.4.3@5a616c0ef860567df5700708256f42505cdb9952 (Docker image 2.4.3)

@marc0s @L30Bola could you paste logs? The logs from the registry is stating that it cannot access Portus (maybe it's down?)

Hey @mssola, I missed the notification. Will try to get those logs and report them back to you. Thanks!

I restarted all the containers to make sure all of them are up and running and the very same error log appears.
Can't the registry log which event is it trying to post to try to know which action triggers the event?
Another error that repeats itself is
registry_1    | 2019-04-11T22:32:12.942885504Z time="2019-04-11T22:32:12Z" level=warning msg="error authorizing context: insufficient scope" go.version=go1.7.6 http.request.host=registry.redacted.com http.request.id=e3072236-d360-4ba8-9981-0fdd94eeec51 http.request.method=GET http.request.remoteaddr=<IP_ADDRESS> http.request.uri="/v2/namespace/imagetags/list?n=100" http.request.useragent=Ruby instance.id=e7cbd865-9c61-4f26-a3dd-42986c7c6876 vars.name="namespace/image" version=v2.6.2-14-ga66a4c3

I don't know if it might be related.
Thanks and let me know if I can run something else to debug it further.


Change amount planning type details in budget page

Add tooltip like screenshot above

Design Link: https://www.figma.com/file/mKfQAsd8vVPi0GwmNKG9BY/Cost-Explorer?node-id=25636%3A438084


[SourceGen] Add MSBuild prop to disable source-generated function metadata

Issue describing the changes in this PR
resolves #1155
Pull request checklist

[x] My changes do not require documentation changes

[ ] Otherwise: Documentation issue linked to PR

[ ] My changes should not be added to the release notes for the next release

[ ] Otherwise: I've added my notes to release_notes.md

[x] My changes do not need to be backported to a previous version

[ ] Otherwise: Backport tracked by issue/PR #issue_or_pr

[x] I have added all required tests (Unit tests, E2E tests)

Additional information
Using guidance from this section of the generators cookbook.
For manual testing, I recommend creating a local nuget package of the SDK and referencing that in a simple app (like http trigger template app).

bump to avoid closure.

/check-enforcer evaluate


Error importing

Just following up the example in the README:
Python 3.12.7 (main, Oct  1 2024, 02:05:46) [Clang 16.0.0 (clang-16<IP_ADDRESS>)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from simplepycons import all_icons, PythonIcon
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/xxx/venv/lib/python3.12/site-packages/simplepycons/__init__.py", line 15, in <module>
    from .all import *  # noqa: F403
    ^^^^^^^^^^^^^^^^^^
  File "/xxx/venv/lib/python3.12/site-packages/simplepycons/all.py", line 19, in <module>
    from ._rubysinatra import RubySinatraIcon
  File "/xxx/venv/lib/python3.12/site-packages/simplepycons/_rubysinatra.py", line 18, in <module>
    from .base_icon import Icon
  File "/xxx/venv/lib/python3.12/site-packages/simplepycons/base_icon.py", line 26, in <module>
    class Icon(ABC):
  File "/xxx/venv/lib/python3.12/site-packages/simplepycons/base_icon.py", line 28, in Icon
    @abstractmethod
     ^^^^^^^^^^^^^^
  File "<frozen abc>", line 24, in abstractmethod
AttributeError: attribute '__isabstractmethod__' of 'property' objects is not writable

The fix is now available on main and will be delivered with next simple-icons version coming tomorrow or on monday.

No release :/ ?

Sorry, I had a medicinical emergency in the family. Last week was a huge mess.

Sorry to know :(
Best wishes for the family!


Collision Avoidance Feature (aka Basic obstacle avoidance)

In this pull request, we have implemented a basic collision avoidance feature.
This is referred to as "Basic obstacle avoidance" in the relevant assignment.

unrelated things in the same pull request but instead

This is such a big pull request for such a small feature (at least according to the title).
It's not a good practice to do so. :)
You should avoid merging unrelated things in the same pull request but instead commit and merge often. Additionally, you should point out where the assignment's task is implemented.

@platisd I appreciate your comment.
The reason for the pull request being overly large is that we had to build the foundation of the code that will run on the car, and, in effect, the entire car folder is our response to the assignment. That said however, I will re-submit the assignment, with a new pull request, that contains only one sketch, which meets the requirements of the task, but does not contain additional code.

Splitting it up in smaller pull requests would be the right thing to do. :)

Splitting it up in smaller pull requests would be the right thing to do. :)

Thank you

@istanbuly You realize by pushing ece2653 you just put IDE-internal files and build binaries right?


DC on hlx5

Description
Upgrading to aem.live from hlx.live as per the product requirements and docs.
This is not meant to be a complete PR but a spike to ensure you have everything you need and should be carefully QA'd
Todo's

This will need proper QA
This will need to get merged
This will need an origin swap on the akamai level to consume aem.page and aem.live
Once we switched the origin, we can delete references to hlx.live or hlx.page

Test URLs
All 4 variations should work: hlx.page aem.page hlx.live aem.live
https://hlx-5--dc--adobecom.aem.live/uk/acrobat/business/industries/government
https://stage--dc--adobecom.aem.live/uk/acrobat/business/industries/government
https://stage--dc--adobecom.hlx.live/uk/acrobat/business/industries/government

Codecov Report
Attention: Patch coverage is 91.54930% with 6 lines in your changes missing coverage. Please review.

Project coverage is 81.17%. Comparing base (6b2c83e) to head (7e157f2).
Report is 1 commits behind head on stage.

Files with missing lines
Patch %
Lines

acrobat/blocks/unity/unity.js
0.00%
4 Missing :warning:

...-converter-widget-ootb/dc-converter-widget-ootb.js
80.00%
1 Missing :warning:

acrobat/blocks/verb-widget/pdfAssetManager.js
0.00%
1 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##            stage     #917      +/-   ##
==========================================
- Coverage   81.31%   81.17%   -0.15%
==========================================
  Files          49       49
  Lines        6429     6501      +72
==========================================
+ Hits         5228     5277      +49
- Misses       1201     1224      +23

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Improve false positive UX

[ ] Set a vulnerability as false positive from the vscode plugin

Pull Request - https://github.com/ZupIT/horusec-extensions/pull/25

Pull Request - https://github.com/ZupIT/horusec-extensions/pull/25


A mutex type that doesn't disable interrupts while locked

I would like a way to use mutable globals without unsafe at usage (i.e. contrary to using static mut). I'm in a single core environment and I know that I don't have re-entrancy (i.e. if I have a mutable reference to the global, I will only access it through that reference). I currently have 2 options.
Option 1: Implement a Mutex type myself
use portable_atomic::{AtomicBool, Ordering::SeqCst};
static LOCKED: AtomicBool = AtomicBool::new(false);
static mut DATA: FooBar = FooBar::new();
fn lock<R>(operation: impl FnOnce(&mut FooBar) -> R) -> R {
    assert!(!LOCKED.swap(true, SeqCst));
    let result = operation(unsafe { &mut DATA });
    assert!(LOCKED.swap(false, SeqCst));
    result
}

This is nice because if interrupts are disabled it's only for a very short amount of time when swapping the atomic bool. But this is not nice because I have to use unsafe and implement the whole thing myself. And I'm asking myself: is this even sound?
Option 2: Use critical_section::Mutex
This is nice because I can reuse something existing without unsafe. This is not nice because interrupts are disabled for the whole critical operation. If I know there are no re-entrancy issues (and I'm fine panicking if that's the case because it's a bug), I would prefer interrupts to only be disabled during lock and unlock operations, not during the whole critical section.
Questions

Does this assessment make sense? (I'm not even sure the single core requirement is of any use, probably just the re-entrancy is enough, or the guarantee that I won't lock the mutex before next unlock.)
Did I miss an existing crate that would solve my problem?
Would critical-section be a good candidate to provide such feature? (e.g. providing a PanicMutex or SyncRefCell) Should I ask portable-atomic instead? (but the name does not match) Or should it be a different crate altogether?

Thanks a lot!

To be more precise, I'm thinking something like this (I would love to know if it's sound, besides knowing if it's useful to others):
pub struct Mutex<T> {
    lock: AtomicBool,
    data: UnsafeCell<T>,
}
pub struct MutexGuard<'a, T>(&'a Mutex<T>);

unsafe impl<T: Send> Sync for Mutex<T> {}

impl<T> Mutex<T> {
    pub const fn new(data: T) -> Self {
        Mutex { lock: AtomicBool::new(false), data: UnsafeCell::new(data) }
    }

    pub fn try_lock(&self) -> Option<MutexGuard<'_, T>> {
        match self.lock.swap(true, SeqCst) {
            false => Some(MutexGuard(self)),
            true => None,
        }
    }

    pub fn lock(&self) -> MutexGuard<'_, T> {
        self.try_lock().unwrap()
    }
}

impl<'a, T> Deref for MutexGuard<'a, T> {
    type Target = T;

    fn deref(&self) -> &Self::Target {
        unsafe { &*self.0.data.get() }
    }
}

impl<'a, T> DerefMut for MutexGuard<'a, T> {
    fn deref_mut(&mut self) -> &mut Self::Target {
        unsafe { &mut *self.0.data.get() }
    }
}

impl<'a, T> Drop for MutexGuard<'a, T> {
    fn drop(&mut self) {
        assert!(self.0.lock.swap(false, SeqCst));
    }
}

IIUC it seems you need a Mutex that panics instead of spinning or blocking when Mutex is locked from others. I think you could use spin crate's Mutex::try_lock (with the portable_atomic feature if needed) + unwrap. (You might want to write a wrapper so that you don't accidentally call a method that does the spinning.)

Thanks! That's exactly it. And it doesn't change the binary size negatively (actually slightly positively) compared to my implementation provided above.


Are API keys required?

Hello. It is unclear what the API's are being used for... so, if it is not critical to the functionality, can they be made optional?

The keys are used to pull metadata for your media collection. The application cannot function without first gathering information using the APIs. Otherwise, the app will look blank


bug list index out of range

Maybe for i in range(len(l) - 2)？

good catch. welcome to make a PR to fix it. thanks Asunatan


Method getTypeDefByName not working when server is configured with no local repository

In EnterpriseOMRSMetadataCollection.class method getTypeDefByName is using activeTypes(that are the types in the local repo) instead of allTypes loaded by enterprise Connector querying all members.
512 -> TypeDef  result = super.filterTypeDefsByName(repositoryHelper.getActiveTypeDefs(), name);
Thus when server is configured with no local repo, the types are not recognized.

Picking this up as no-one working on it and causing problems

Fix is in following commit
https://github.com/mandy-chessell/egeria/commit/370f6aab793aecf4eb990ce0894b248fbcb7617a

I checked the change you did Mandy and is still not getting the types when I am trying to retrieve them.  It gets them when I put allTypes.getTypeDefs() instead of repositoryHelper.getKnownAttributeTypeDefs().

Do you mean getKnownTypeDefs() rather than getKnownAttributeTypeDefs()?

I think this is fixed?  Please reopen if there is still a problem


MySQL enum type should use arrow dictionary type

Close by: https://github.com/datafusion-contrib/datafusion-table-providers/pull/119


Thrift Client Configuration Modification

This P/R includes Stratos thrift-client-config.xml file change to read DAS thrift client configuration from the same file and  thrift config parser and publisher changes for the thrift-client-config.xml modification.. DAS configuration read from this file will be used in CC and AS to publish thrift events.

Closing this P/R to send it to tenant-isolation branch.


Update to 1.21.1

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. This mod is not updated to 1.21.1.
Describe the solution you'd like
Update this mod to 1.21.1.
Describe alternatives you've considered
Additional context

Now when Better Combat has been updated to 1.21.1, it's weird to see that its required dependency isn't available on this version.

1.21 mods work on 1.21.1

@mpustovoi 1.21 mods work on 1.21.1

Then mark it as compatible?

@KosmX yeah, it should be marked as compatible.


Run performance benchmarks in CI

Following up on informalsystems/apalache#152 and informalsystems/apalache-tests#1, we want to run the benchmarks and long running integration test in the CI here.
At minimum, we'd like to run the tests weekly, to ensure we catch regressions early, and, ideally, they should also run before the release process.
I'll post a followup here scoping out what needs doing in greater detail, but I think the following issues are prerequisites for beginning this (or, at least, must be accomplished in the process):
informalsystems/apalache-tests#1
informalsystems/apalache-tests#2

Oop, I think this transfer was actually ill advised. We'll still want to incorporate this into the CI pipeline in the main repo. Sorry for the noise.


Automatic PR from RestAPI-PR1724

Generated from RestAPI PR: https://github.com/Azure/azure-rest-api-specs/pull/1724

Codecov Report

Merging #1491 into master will increase coverage by <.01%.
The diff coverage is 100%.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3c2d296...87990e6. Read the comment docs.


Use args instead of variables in Dockerfile

This PR fixes #321.

@dealancer Note that VERSION is later used in SetServerInfo():
https://github.com/jwilder/docker-gen/blob/4edc190faa34342313589a80e3a736cafb45919b/context.go#L37
You might want to set the environment variable based on the build argument:
ARG VERSION=0.7.4
ENV VERSION ${VERSION}


Fix non own-users to edit delete

What does this pull request aim to achieve? 💯
Added own-user check to prevent non eligible users to edit/delete
Please be concise and link any related issue(s):
fixes #1188

[ ] I have checked that no one else is working on similar changes.
[ ] I have read the latest README and understand the development workflow.
[ ] The name of this branch is: feature/<branch_name> (need to have cloned the repo not forked).
[ ] This branch is rebased/merged with the latest master.
[ ] There are no merge conflicts.
[ ] There are no console warnings and errors.
[ ] On the right hand side of this PR, I have linked the appropriate issue(s) under "Issues" and added a project progress under "Projects".

@idjevm let's do this 🔥


Explanation of App pages

In the prototype there is a walkthrough for using the app. The user can skip the walkthrough with a "skip" button
acceptance criteria:

maybe just for first use (could be annoying in the long run)
short text with explanation
Next Button
Skip Button

Moved to Sprint 2 due to more important issues being worked on before this one.


Unable to update or download

I have been getting these messages today. Any ideas why?
C:\Users\John>"C:\Program Files\Serviio\lib\youtube-dl.exe" https://www.youtube.
com/watch?v=e-hggs0GDjc
[youtube] e-hggs0GDjc: Downloading webpage
ERROR: Unable to download webpage: <urlopen error [Errno 10038] An operation was
attempted on something that is not a socket> (caused by URLError(error(10038, '
An operation was attempted on something that is not a socket'),))
C:\Users\John>"C:\Program Files\Serviio\lib\youtube-dl.exe" -U
ERROR: can't find the current version. Please try again later.

Most likely a connectivity problem. Do you experience it now?

HMM. Restoring my system fixed it. Strange. Thanks.


Python 3/2 backwards compatibility

Description
Given the fact that Python 2 will be EOL as of 2020, it is a good idea to move development of new libraries to Python 3. Ensuring backwards compatibility with Python 2 is essential however since there are a number of developers who are in a situation where they do not have control over the environment and also to ensure that the Python 3 port won't break existing installs. This PR migrates the library to a state where it can be run 100% on Python 3 & 2.
Motivation and Context
See above.
How Has This Been Tested?
By running unit testing in both environments:
Python 2
[~/d/nexpose-client-python]─[⎇ 4-py3-backwards-compat]─> python2 --version
Python 2.7.13
[~/d/nexpose-client-python]─[⎇ 4-py3-backwards-compat]─> virtualenv --version
15.1.0
[~/d/nexpose-client-python]─[⎇ 4-py3-backwards-compat]─> virtualenv -p python2 ../virtualenv/nx-2
Running virtualenv with interpreter /usr/bin/python2
New python executable in /home/dhaynes/development/virtualenv/nx-2/bin/python2
Also creating executable in /home/dhaynes/development/virtualenv/nx-2/bin/python
Installing setuptools, pip, wheel...done.
[~/d/nexpose-client-python]─[⎇ 4-py3-backwards-compat]─> source ../virtualenv/nx-2/bin/activate.fish
(nx-2) [~/d/nexpose-client-python]─[⎇ 4-py3-backwards-compat]─> pip install -r requirements.txt
Collecting lxml==3.6.0 (from -r requirements.txt (line 1))
Collecting future==0.16.0 (from -r requirements.txt (line 2))
Installing collected packages: lxml, future
Successfully installed future-0.16.0 lxml-3.6.0
(nx-2) [~/d/nexpose-client-python]─[⎇ 4-py3-backwards-compat]─> pip install -r requirements_test.txt
Requirement already satisfied: lxml==3.6.0 in /home/dhaynes/development/virtualenv/nx-2/lib/python2.7/site-packages (from -r requirements_test.txt (line 1))
Collecting py==1.4.31 (from -r requirements_test.txt (line 2))
  Using cached py-1.4.31-py2.py3-none-any.whl
Collecting pytest==2.9.2 (from -r requirements_test.txt (line 3))
  Using cached pytest-2.9.2-py2.py3-none-any.whl
Requirement already satisfied: future==0.16.0 in /home/dhaynes/development/virtualenv/nx-2/lib/python2.7/site-packages (from -r requirements_test.txt (line 4))
Installing collected packages: py, pytest
Successfully installed py-1.4.31 pytest-2.9.2
(nx-2) [~/d/nexpose-client-python]─[⎇ 4-py3-backwards-compat]─> py.test -v tests
============================================================================================================= test session starts =============================================================================================================
platform linux2 -- Python 2.7.13, pytest-2.9.2, py-1.4.31, pluggy-0.3.1 -- /home/dhaynes/development/virtualenv/nx-2/bin/python2
cachedir: .cache
rootdir: /home/dhaynes/development/nexpose-client-python, inifile:
collected 27 items

tests/test_LoadFixture.py::LoadFixtureTestCase::testThatLoadingInvalidFixtureTypeResultsInAnException PASSED
tests/test_LoadFixture.py::LoadFixtureTestCase::testThatLoadingNonExistingFixtureResultsInAnException PASSED
tests/test_LoadFixture.py::LoadFixtureTestCase::testThatOurFixturesWillLoadCorrectly PASSED
tests/test_NexposeNode.py::NexposeNodeTestCase::testCreateFromJSON PASSED
tests/test_NexposeReportConfigurationSummary.py::NexposeReportConfigurationSummaryTestCase::testCreateFromXML PASSED
tests/test_NexposeReportConfigurationSummary.py::NexposeReportSummaryTestCase::testCreateFromXML PASSED
tests/test_NexposeSession.py::NexposeSessionTestCase::testConstructionOfLoginRequest PASSED
tests/test_NexposeSession.py::NexposeSessionTestCase::testConstructionOfURI_APIv1d1 PASSED
tests/test_NexposeSession.py::NexposeSessionTestCase::testCorrectLogin PASSED
tests/test_NexposeSession.py::NexposeSessionTestCase::testDefaultConstructionOfURI_APIv1d1 PASSED
tests/test_NexposeSession.py::NexposeSessionTestCase::testIncorrectLogin PASSED
tests/test_NexposeSession.py::NexposeSessionTestCase::testLoginWithInvalidHtmlReply PASSED
tests/test_NexposeSession.py::NexposeSessionTestCase::testRequestSiteDeviceListing PASSED
tests/test_NexposeSession.py::NexposeSessionTestCase::testRequestSiteListing PASSED
tests/test_NexposeSession.py::NexposeSessionTestCase::testRequestSiteScanHistory PASSED
tests/test_NexposeSession.py::NexposeSessionTestCase::testRequestSystemInformation PASSED
tests/test_NexposeSession.py::NexposeSessionTestCase::testShouldNotLoginIfSessionIsOpen PASSED
tests/test_NexposeTag.py::NexposeTagAttributeTestCase::testCreate PASSED
tests/test_NexposeTag.py::NexposeTagTestCase::testCreate PASSED
tests/test_NexposeTag.py::NexposeTagTestCase::testCreateCustom PASSED
tests/test_NexposeTag.py::NexposeTagTestCase::testCreateFromJSON PASSED
tests/test_NexposeTag.py::NexposeTagTestCase::testCustomTagWithAttributes PASSED
tests/test_NexposeUserAuthenticatorSummary.py::NexposeUserAuthenticatorSummaryTestCase::testCreateFromXML PASSED
tests/test_NexposeVulnerability.py::NexposeVulnerabilitySummaryTestCase::testCreatingFromXML PASSED
tests/test_NexposeVulnerability.py::NexposeVulnerabilitySummaryTestCase::testIfAnEmptyXmlElementResultsInDefaultValues PASSED
tests/test_NexposeVulnerability.py::NexposeVulnerabilityDetailTestCase::testCreatingFromXML PASSED
tests/test_NexposeVulnerability.py::NexposeVulnerabilityDetailTestCase::testIfAnEmptyXmlElementResultsInDefaultValues PASSED

========================================================================================================== 27 passed in 0.17 seconds ==========================================================================================================
(nx-2) [~/d/nexpose-client-python]─[⎇ 4-py3-backwards-compat]─> deactivate

Python 3
[~/d/nexpose-client-python]─[⎇ 4-py3-backwards-compat]─(1)-> python3 --version
Python 3.6.2
[~/d/nexpose-client-python]─[⎇ 4-py3-backwards-compat]─> virtualenv -p python3 ../virtualenv/nx-3
Running virtualenv with interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /home/dhaynes/development/virtualenv/nx-3/bin/python3
Not overwriting existing python script /home/dhaynes/development/virtualenv/nx-3/bin/python (you must use /home/dhaynes/development/virtualenv/nx-3/bin/python3)
Installing setuptools, pip, wheel...done.
[~/d/nexpose-client-python]─[⎇ 4-py3-backwards-compat]─> source ../virtualenv/nx-3/bin/activate.fish
(nx-3) [~/d/nexpose-client-python]─[⎇ 4-py3-backwards-compat]─> pip install -r requirements.txt
Collecting lxml==3.6.0 (from -r requirements.txt (line 1))
Collecting future==0.16.0 (from -r requirements.txt (line 2))
Installing collected packages: lxml, future
Successfully installed future-0.16.0 lxml-3.6.0
(nx-3) [~/d/nexpose-client-python]─[⎇ 4-py3-backwards-compat]─> pip install -r requirements_test.txt
Requirement already satisfied: lxml==3.6.0 in /home/dhaynes/development/virtualenv/nx-3/lib/python3.6/site-packages (from -r requirements_test.txt (line 1))
Collecting py==1.4.31 (from -r requirements_test.txt (line 2))
  Using cached py-1.4.31-py2.py3-none-any.whl
Collecting pytest==2.9.2 (from -r requirements_test.txt (line 3))
  Using cached pytest-2.9.2-py2.py3-none-any.whl
Requirement already satisfied: future==0.16.0 in /home/dhaynes/development/virtualenv/nx-3/lib/python3.6/site-packages (from -r requirements_test.txt (line 4))
Installing collected packages: py, pytest
Successfully installed py-1.4.31 pytest-2.9.2
(nx-3) [~/d/nexpose-client-python]─[⎇ 4-py3-backwards-compat]─> py.test -v tests
============================================================================================================= test session starts =============================================================================================================
platform linux -- Python 3.6.2, pytest-2.9.2, py-1.4.31, pluggy-0.3.1 -- /home/dhaynes/development/virtualenv/nx-3/bin/python3
cachedir: .cache
rootdir: /home/dhaynes/development/nexpose-client-python, inifile:
collected 27 items

tests/test_LoadFixture.py::LoadFixtureTestCase::testThatLoadingInvalidFixtureTypeResultsInAnException PASSED
tests/test_LoadFixture.py::LoadFixtureTestCase::testThatLoadingNonExistingFixtureResultsInAnException PASSED
tests/test_LoadFixture.py::LoadFixtureTestCase::testThatOurFixturesWillLoadCorrectly PASSED
tests/test_NexposeNode.py::NexposeNodeTestCase::testCreateFromJSON PASSED
tests/test_NexposeReportConfigurationSummary.py::NexposeReportConfigurationSummaryTestCase::testCreateFromXML PASSED
tests/test_NexposeReportConfigurationSummary.py::NexposeReportSummaryTestCase::testCreateFromXML PASSED
tests/test_NexposeSession.py::NexposeSessionTestCase::testConstructionOfLoginRequest PASSED
tests/test_NexposeSession.py::NexposeSessionTestCase::testConstructionOfURI_APIv1d1 PASSED
tests/test_NexposeSession.py::NexposeSessionTestCase::testCorrectLogin PASSED
tests/test_NexposeSession.py::NexposeSessionTestCase::testDefaultConstructionOfURI_APIv1d1 PASSED
tests/test_NexposeSession.py::NexposeSessionTestCase::testIncorrectLogin PASSED
tests/test_NexposeSession.py::NexposeSessionTestCase::testLoginWithInvalidHtmlReply PASSED
tests/test_NexposeSession.py::NexposeSessionTestCase::testRequestSiteDeviceListing PASSED
tests/test_NexposeSession.py::NexposeSessionTestCase::testRequestSiteListing PASSED
tests/test_NexposeSession.py::NexposeSessionTestCase::testRequestSiteScanHistory PASSED
tests/test_NexposeSession.py::NexposeSessionTestCase::testRequestSystemInformation PASSED
tests/test_NexposeSession.py::NexposeSessionTestCase::testShouldNotLoginIfSessionIsOpen PASSED
tests/test_NexposeTag.py::NexposeTagAttributeTestCase::testCreate PASSED
tests/test_NexposeTag.py::NexposeTagTestCase::testCreate PASSED
tests/test_NexposeTag.py::NexposeTagTestCase::testCreateCustom PASSED
tests/test_NexposeTag.py::NexposeTagTestCase::testCreateFromJSON PASSED
tests/test_NexposeTag.py::NexposeTagTestCase::testCustomTagWithAttributes PASSED
tests/test_NexposeUserAuthenticatorSummary.py::NexposeUserAuthenticatorSummaryTestCase::testCreateFromXML PASSED
tests/test_NexposeVulnerability.py::NexposeVulnerabilitySummaryTestCase::testCreatingFromXML PASSED
tests/test_NexposeVulnerability.py::NexposeVulnerabilitySummaryTestCase::testIfAnEmptyXmlElementResultsInDefaultValues PASSED
tests/test_NexposeVulnerability.py::NexposeVulnerabilityDetailTestCase::testCreatingFromXML PASSED
tests/test_NexposeVulnerability.py::NexposeVulnerabilityDetailTestCase::testIfAnEmptyXmlElementResultsInDefaultValues PASSED

========================================================================================================== 27 passed in 0.13 seconds ==========================================================================================================
(nx-3) [~/d/nexpose-client-python]─[⎇ 4-py3-backwards-compat]─> deactivate

Screenshots (if appropriate):
See above.
Types of changes

New feature (non-breaking change which adds functionality): py3 support

Checklist:

[x] I have updated the documentation accordingly (if changes are required).
[] I have added tests to cover my changes.
^ not necessary as testing py3 involves just running tests in a python 3 enviornment
[x] All new and existing tests passed.

Closes #4

I want to set aside some time to do a live usage test before merging, but this looks good to me otherwise. Thank you for doing this!
Did you find any notable differences or "gotchas" for environment setup in py3 compared to py2? In other words would a new user on py3 need additional/different steps than a py2 user when creating a virtual environment, etc?

There shouldn't be anything additional that someone would need to do depending on what version of Python they use.
A situation that comes to mind is if in the future someone decides to contribute code that utilizes a Python 3 specific feature such as ipaddress or asyncio . In that case, in order to ensure Python 2 users aren't affected, they would need to include a back-ported package as a requirement. Luckily the repo in it's current state uses almost exclusively the Python standard library which is fairly interoperable on py2/3.

I would suggest adding the travis-ci bot to PRs to do this checking as well as noting in documentation that contributions must be backwards compatible.

I've made some changes to get things working on travis-ci, please rebase (or re-merge upstream/master) when you get a chance. Thanks!

Nevermind, I was able to resolve the conflicts through github.

Huh, a pass in 3.6 but fail in 3.5. I'll look a it.

You can replicate the issue with:
[~/d/nexpose-client-python]─[⎇ 4-py3-backwards-compat]─> python
Python 2.7.13 (default, Jun 26 2017, 10:20:05)
[GCC 7.1.1 20170622 (Red Hat 7.1.1-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> d = {'x': 1, 'y': 2}
>>> d.items()
[('y', 2), ('x', 1)]

The test assertion may be too strict as the Nexpose API doesn't actually care about the order of the attributes, but that is interesting to see it behave differently on 3.5 specifically!
I think I'm okay with ignoring that failure for now and dealing with it another time.
As for testing, today I started using pyenv and pyenv-virtualenv to install and manage multiple python versions. Give it a try if you don't like waiting on travis-ci runs.

Oh cool I wan't aware of those tools, I'll have to check them out.
If it's no big deal then I won't worry about it, I was just trying to be thorough and not merge in any huge bugs.

Did some local testing; Even with OrderedDict it's doing odd stuff and only on 3.5 (although I didn't check 3.3 or 3.4). It doesn't happen on every run either.
I think this can be "fixed" from a testing standpoint by setting the PYTHONHASHSEED environment variable to a static value in travis-ci. Locally it seems to work out that way at least.
You can probably just remove the OrderedDict change and then change .travis.yml like so:
script:
  - if [[ $TRAVIS_PYTHON_VERSION == 3.5 ]]; then export PYTHONHASHSEED=3140657756; fi
  - py.test tests

Re-ran it a few times on travis and it passed on the last try 🙄

passed on the last try

Oh wow that's interesting..
I think the ideal solution should be to:

Revert the OrderedDict commit
Refactor that test to check for either:

<LoginRequest password="nxpassword" user-id="nxadmin"/>
<LoginRequest user-id="nxadmin" password="nxpassword"/>

Given that the Nexpose API does not care about ordering, just that the attributes are present.
Worst case then fall back to:
script:
  - if [[ $TRAVIS_PYTHON_VERSION == 3.5 ]]; then export PYTHONHASHSEED=3140657756; fi
  - py.test tests

Should be good now.
pyenv was a lifesaver there thanks for the share.


Berserker character animation

Task description
A new creature type is to be added to the game for an upcoming update, the berserker.
It will be a variant of the basic human warrior, but with the armour restyled to a more red colour, and with larger horns on the helmet than the warrior, and the armour covers more of the body.
Recommended to look at the agonite armour task first: https://github.com/Arcanorum/dungeonz/issues/66
Berserkers are more aggressive than the warrior, and attack more often as their health decreases.
Their armour reskin is to showcase the new metal item category, Agonite, that will also come with the same update.
References/notes
Warrior sprites: https://github.com/Arcanorum/dungeonz/tree/master/client/src/assets/images/entities/characters/warrior
A basic concept I drew

Similar concept art
https://i.pinimg.com/originals/04/1b/33/041b335c498b98e0da20012f3a3b65a7.jpg
Uses a tower style shield, but with pointed top and bottom edges
https://upload.wikimedia.org/wikipedia/commons/6/64/Ag-obj-5959-001-pub-large.jpg
https://www.stainedglassexpress.com/var/images/product/300.300/84023.jpg
The code for this creature is already in place using the warrior sprites. Just needs new new assets to use.
Acceptance criteria
Visual

A warrior variant with reddish armour, with more exaggerated features.
3 animation frames for each direction (up, down, left, right (left but flipped)), to be played in the basic humanoid animation pattern used by other entities: 1 (standing), 2 (right step), 1 (standing), 3 (left step), repeating.
Uses the DawnBringer 32 colour palette used by the rest of the sprites.
Expected files:
berserker-up-1.png
berserker-up-2.png
berserker-up-3.png
berserker-down-1.png
berserker-down-2.png
berserker-down-3.png
berserker-left-1.png
berserker-left-2.png
berserker-left-3.png
berserker-right-1.png
berserker-right-2.png
berserker-right-3.png

Code

Update the berserker entity client file to use the new animation set.
https://github.com/Arcanorum/dungeonz/blob/master/client/src/game/entities/characters/Berserker.js#L11

Done. Added to update 10.


shared-mime-info: 2.2 -> 2.3

Description of changes
https://gitlab.freedesktop.org/xdg/shared-mime-info/-/releases/2.3
Also adds a patch for clang_16 compat.
Supersedes #252576.
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[x] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Related to #234710.

The clang16 patches have been merged upstream and there is still an open MR to update the changelog and the version number: https://gitlab.freedesktop.org/xdg/shared-mime-info/-/merge_requests/250/diffs?commit_id=e24fe6127370880596b9f06a4dd4a671ad530f85
I'll set this to draft for now in case the tag will be rewritten.


Unexpected mis-matched flows in eLCI.csv

In the latest eLCI model inventories (e.g., solar, solarthermal, wind), emissions are already mapped to the FEDEFL, which is causing our flow mapping to fail. For example, '1,3-Butadiene' does not match any source flows in eLCI, but matches a target flow.
https://github.com/USEPA/fedelemflowlist/blob/7f2550a383a026ade3d23c8b0e3f6d1ba411bdd3/fedelemflowlist/flowmapping/eLCI.csv#L1206
The result is that mapped flows are being dropped from our inventories.
It would be nice if all the mapped FEDEFL flows were also listed in the sources (i.e., no change; map to itself). If there is an alternative recommendation to updating the CSV, I'm happy to take suggestions.

Can you remind me where this is done in the code? I think it could make more sense to pull out the flows in the dataframe which are already in the FEDEFL or have UUID's and only map the remaining? rather than change the mapping file to try to map flows to itself

L232 in the current development branch of combinator.py's concat_map_upstream_databases method:
https://github.com/KeyLogicLCA/ElectricityLCI/blob/c2c7194ee1d3adf79ab0705d4149e7b6f57f1521/electricitylci/combinator.py#L154
This basically throws away all non-mapped flows, including those already mapped because they're not in the target list.
Your solution seems practical.

Another option would be to use esupy's apply_flow_mapping() which was intended to help streamline mapping operations across packages. I think in this case you could use keep_unmapped_rows=True, but that would only work if you want to keep all unmapped flows, not just those flows that are already FEDEFL.
Though seems reasonable to add to that function another parameter to ignore existing FEDEFL before mapping, much like I suggest above.


Cannot add TreeItem from signal item_selected()

Godot version
3.3.2 official
System information
Windows 10
Issue description
I have created a Tree node and added one TreeItem.  By pressing a button I can add a child TreeItem to the first TreeItem.  However, if I want to add a TreeItem to any existing TreeItem by selecting it, Godot crashes.
I checked to see if a valid TreeItem is loaded when I use the item_selected() signal and call .get_selected(). This checks out fine.  However, for some reason, from the item_selected() function, I can't add a child TreeItem to the one selected.
Steps to reproduce
First:

run provided Test project.
press button and see a TreeItem created under the top TreeItem.
all is well

close and run again:

This time click on the TreeItem.
Godot crashes with the error that the created TreeItem is NULL

Minimal reproduction project
Test.zip

It's not C#/Mono specific issue, it would behave the same in GDScript.
The cause is that Tree::create_item fails when Tree is internally in a blocked state and in such a case it returns nullptr instead of newly created TreeItem:
https://github.com/godotengine/godot/blob/14e7e7ad719b4041b684907cf412a1925af2d384/scene/gui/tree.cpp#L2938-L2939
And Tree is "blocked" when e.g. mouse item selection is being handled:
https://github.com/godotengine/godot/blob/14e7e7ad719b4041b684907cf412a1925af2d384/scene/gui/tree.cpp#L2568-L2570
Documenting that Tree.create_item can fail and return nothing is a must with the current implementation. Users shouldn't assume that it will always successfully create a TreeItem. Not idea if that blocking is neccessary though and if it could be somehow changed.

Tree::clear, Tree:set_columns, Tree::create_item will fail if the Tree is blocked.
Also it is not only the case for the item_selected signal, but others like item_edited.
I would like to document it for now, but should a note added to each method, which can fail or just a general note in the description? And how precise should the condition to be discribed, just while the Tree is processing specific mouse input or under specific conditions.


is this trying to grab my ip

Ok so when i look at the code for the tokenstealer it shows that i need something called public-ip to get it to work
i'm pretty new to this stuff, so i'm not too sure what it's trying to do when i run it and theres some ip stuff in there

Hello @FireKoopaX,
The reason It has a library for grabbing an IP is when someone open the compiled version It will also send that someone's IP address, computer host name, username & etc in the Discord webhook that is set. Also I'm not sure If It will still work because Discord added some extra protection.

oh alright, thanks for clearing that up


Override datatype mapping

Fixes #69

Thanks Tom, I agree with you.  I didn't like it either :-).  I think it may be worth making this small adjustment to the current db-common as this portion won't be changed in 10.


any difference between lucidrains's implemantation?

Hi, may I ask any difference between lucidrains's implementation and this repo?
https://github.com/lucidrains/uniformer-pytorch
Thanks a lot!

Thanks for your question! I have not seriously checked the code.
Since the code for video may not be easy separated, if you want to use code for video, maybe you can find it here.
I also give a demo in app.py.

As there is no more activity, I am closing the issue, don't hesitate to reopen it if necessary.


FFMPEG stderr

Hi,
is it possible to see somewhere stderr from ffmpeg or at least the final ffmpeg command  used to capture the camera?
I'm getting an error " - Camera: Domek was terminated, respawning after 10 seconds..." over and over but I'm able to copy stream from camera manually by ffmpeg (e.g.: ffmpeg -i "rtsp://<IP_ADDRESS>:5544/live0.264" -c:v copy /mnt/md0/test.mp4)
So there should be something ffmpeg is complaining about and I need to know what.
Thanks

Hi @Rumczeis
try removing stimeout: '30000000' from the config, I cant quite re-call, but depending on the ffmpeg version, this command arg, is not implemented / or has been removed (its removed from v3)

Thanks @marcus-j-davies , removing the stimeout: '30000000'  and also adding config param: an: '', to  inputConfig section helped too as the camera was sending incompatible audio codec for mp4 container.


How to reformat the layout when keyboard pops up.

I have a login screen with 2 text fields and 2 buttons. Whenever I click on the text field key board pops up and  the screen is pushed up and a red box appears that hides my button and goes away when the key board is closed. How do I avoid the red box?Do I need to realign the layout whenever the screen is pushed up? If so how do I do it whenever the screen is pushed up?

Hi, thanks for the question! Do you happen to have a small sample that illustrates this issue? A small reproducible case?

This is the normal window.

This is what what happens when the keyboard pops up

the red box in the second pic is what i m talking about

What's happening is that your Column contains more stuff than you have room to display, so it's overflowing (and clipping). You could swap it out for a ListView or some such.

You are saying instead of a column I should use a ListView?

Depends what you want to have happen. What would you want to have your UI look like if the screen wasn't tall enough for everything in your dialog?

(e.g. turn the device on it's side then bring up the keyboard, what do you want to see?)

I just want that user should be able to see whatever he/she is typing. The way the layout is right now works for me. But the only problem is that red box that appears. That should go away.

If the screen was ~100 pixels less tall than your screenshot, when the keyboard comes up after the user taps the password field, the password field would no longer be visible. The red line is warning you of that.
Hence my question, what would you want it to look like?
If you want it to scroll, then yes I would recommend using a ListView instead of a Column.
By the way the red only appears in debug builds, when you release your app it won't be visible.

When the user taps on username field I want the user to see the username field and when the user types in password field then he/she should be able to see that.

Yeah, you definitely want it to be in a ListView. The actual scrolling to show the right thing in view won't work until we fix the last few parts of https://github.com/flutter/flutter/issues/10826#issuecomment-311220245.

Ok. let me try that.

No this doesn't work.I can't set the background nor is the screen pushed up.The parent element in this screen is stack.

Yeah, the screen won't push up properly until https://github.com/flutter/flutter/issues/10826#issuecomment-311220245 is fixed.
Not sure what you mean about the background though.

I meant that the faded background isn't visible

I'm trying this too.

As far as I can tell this is two issues, one is #7032, and the other is #10826.
Closing as a duplicate of those issues.


optional: add deconstruct method

A method to deconstruct a Value as a separate value and bool will enable interoperability with any other "optional" types that might be out there. For example:
func (val Value[T]) Get() (T, bool) {
    return val.v, val.ok
}

var other otheropt.Of(val.Get())

Alternative names:

Unpack
Unzip

Decided on Unpack.

Reopening to add Result.Unpack.


Feature/add debug query method

resolves #2751
Description
In the base debug task, there was a hard-coded test query that wouldn't work in some databases (e.g. Oracle). This new debug_query method on the base adapter allows plugin authors to specify their own debug query.
Checklist

[x] I have signed the CLA
[ ] I have run this code in development and it appears to resolve the stated issue
[x] This PR includes tests, or tests are not required/relevant for this PR
[ ] I have updated the CHANGELOG.md and added information about my change to the "dbt next" section.

First time committer to any open source project here! I believe my change to debug.py needs a parameter in the new method, but I'm unsure what it should be. If any of you had a suggestion as to what it needs to be I'll revise the PR and open it again. Thanks!

@zmac12 I think you're on the right track! debug_query should go in core/dbt/adapters/base/impl.py rather than connections.py, and in the general case I it can be as simple as:
def debug_query(self):
    self.execute('select 1 as id')

Then plugin authors can override it by reimplementing the adapter method for themselves, e.g. in core/dbt/adapters/oracle/impl.py:
def debug_query(self):
    self.execute('select 1 from dual')


Custom string with the api

Our use case is we have a plugin that acts as a bridge for communication between our mobile app and the camera. For images, we call the plugin's api which has a parameter that accepts a string, the plugin calls the camera's api to take picture and once done, update the resulting image's exif data with the string received with the api. We tried the same with the video, but after the camera's stop recording api is called, it exits, we do not receive any response from the camera. The recording stops, but we do not get to update the video's exif data with the string received in the plugin's api. We also tried adding a 15-20 seconds wait, but even after waiting it does nothing, the code exits, no breakpoints are hit.
Can you please help us with understanding what we are doing wrong?
or any other way to pass a custom string with the start or stop recording api calls?

From reading the forum, it looks like you're using the WebAPI from inside the plugin.
This is what is in the forum

So the app calls the plugin's api which is a custom api,
<IP_ADDRESS>:8888/video_start
This triggers the plugin to call the camera api to start recording. The api called by the plugin is,

<IP_ADDRESS>/osc/commands/execute

"name", "camera.startCapture"

@iFieldSmart-Tech @codetricity
I am sorry for our late checking for your issue.
I read the post too, and we need to say that Theta X plugin does not support WebAPI to start/stop video recording. I am sorry that this spec difference between Theta Z1 and X is came from the difference of plugin architecture.
If you want to do video recording in plugin, using the Camera API is strongly recommended. This repository is showing how to use Camera API to execute take picture and start/stop video recording.
Best regards,
Daisuke Hohjoh

@DaisukeHohjoh oh, this is interesting information.  I tried for many days to use Flutter with the webAPI to start/stop the video inside the plugin.  I had assumed it was a limitation of the Flutter architecture.
Do you know of other restrictions with the webAPI used in the THETA X plugin?
https://github.com/ricohapi/theta-api-specs/blob/main/ricoh-theta-plugin/web-api.md
Is there an example of how @iFieldSmart-Tech can get the file location of the video file using the camera api and then feed that into another method to process the file (such as append metadata to the notes field).

@codetricity

Do you know of other restrictions with the webAPI used in the THETA X plugin?

Mainly video recording feature / video related Options are unavailable for Theta X. As you said, RIcoh need to add description about this limitation to https://github.com/ricohapi/theta-api-specs/blob/main/ricoh-theta-plugin/web-api.md .
If plugin use Camera API / MediaRecorder API to do video recording, you need to specify the file name yourself.
https://github.com/ricohapi/theta-plugin-camera-api-sample-x/blob/5115c7ff53449a716d8d0c300911c27472e7b51d/app/src/main/java/com/theta360/sample/camera/MainActivity.kt#L469
I am sorry for NOT my sharing this information in public.
Best regards,
Daisuke Hohjoh

So using camera api is suggested, but the main reason for using the web api is to send a string with the plugin's webapi which needs to be inserted in the output video's exif data. We cannot do it with the camera api, right? In order to send a string along with the api, we have no choice but to use a web api that triggers the camera api, right? Or is there any other way to achieve our use case? The end goal is just to pass a string along with the video recording start api. This api will be called from our mobile app and the plugin will in turn start the recording and update the video exif data once the file is ready.

@iFieldSmart-Tech when you setOutputFile(mFilepath), you know where the video file is going to be.
If you search on google, there seems to be many solutions to read/write metadata of the mp4 container.  i have not tried any.  However, this one looks like it can do it.  https://github.com/sannies/mp4parser

@DaisukeHohjoh If i try another way; is it possible to fetch the video metadata without having to use a plugin in between? Is it possible using only the camera and its api?
Can i get the exif data from a video directly from the camera using the camera api and without having to download the video on the phone first?
If that is possible that provides a workaround for my use case

@DaisukeHohjoh
So our first goal was to insert exif data into the image/video while the file is still on the camera, no need to download it to the phone to insert the data. We noticed an exif parameter "ImageDescription" which is always empty so decided to insert a random string in it so later when the file is uploaded to our portal, we can read that parameter and determine which metadata that file belongs to. We were able to do it successfully with images, but were struggling with videos. The ImageDescription parameter is in the videos as well and just like the images, it is always empty and can be used for our custom purpose.
As we were failing at inserting the data for video files we thought as a workaround maybe we try reading all the exif data and save the whole data in our db which can later be used for mapping. But regardless, yesterday we were able to get it to work without the plugin. We can get list of files directly using the camera api, so we decided to ditch the plugin for videos. We are now using the camera api to start and stop recording and then get the filename once the recording stops and the file is created. We will rely on the filename for the mapping.

@iFieldSmart-Tech
Thanks for sharing it with us.
So.. does following my understanding match your direction ?

You will NOT develop your own Theta plugin.

This means also, you decided NOT to insert some strings to exif ImageDescription tag of photo and video.

Your mobile phone app send WebAPI to execute command camera.takePicture (photo) and camera.startCapture (video).
Your mobile phone app get metadata information of photo and video, by WebAPI camera.getMetadata and camera.listFiles.

Your mobile phone app will use these information for some kind of mapping between filename and its contents.

The ImageDescription parameter is in the videos as well and just like the images, it is always empty and can be used for our custom purpose.

This is just for your information, if you develop plugin to capture photo and/or video with using Camera API, you can insert ImageDescription tag by following Camera.Parameter setting.
        camera?.apply {
            val p = parameters
            p.set(
                Camera.Parameters.PLUGIN_INFO,
                "You can put any String"
            )
            parameters = p
}

With this setting, photo and/or video will have ImageDescription as your setting String value.
Best regards,
Daisuke Hohjoh

Hi @DaisukeHohjoh
Thank you for the detailed explanation. However, we went ahead and did it without the plugin and using only the camera api. We use to camera api to start and stop capture, then stop capture returns the filename, we save that filename and the timestamp to our database. When user uploads the file later, we use the filename and the exif data timestamp to map the two. This still leaves an issue where in case if the user changed the filename before uploading the video, but for now, we need to live with that issue and request the user to not change the file names.

@DaisukeHohjoh I needed help with something else too. The ricoh desktop app has the option to perform stitching as well as top bottom correction on videos. We figured out the stitching part but not sure how to do the topbottom correction at our end. Is there any api that can do that? It does not have to be done on the camera, we can do it on our server after the user uploads the video. But is there any way to programmatically figure out which video needs correction and how to do it?

@iFieldSmart-Tech

we went ahead and did it without the plugin and using only the camera api.

I understood your direction. Thank you.

how to do the topbottom correction at our end. Is there any api that can do that? ....

Theta X
Theta X can do topbottom correction on the camera. Default value is ON. So I expect that you need not consider about it for Theta X.

UI

WebAPI
https://github.com/ricohapi/theta-api-specs/blob/main/theta-web-api-v2.1/options/_top_bottom_correction.md

Theta Z1
On the other hand, Theta Z1 cannot topbottom correction on the camera for video recording mode. If you want to do it at your end (such as on your server), without Ricoh app, followings are option.

Calculate camera atitude with using IMU sensor data which is stored in MP4 file, and do post-processing at your end. Please refer to here for meta data information.

is there any way to programmatically figure out which video needs correction and how to do it?

Yes MP4 file has such Ricoh specified metadata but it is not disclosed now. If you need such information, I think Ricoh can discuss about it.
Best regards,
Daisuke Hohjoh


Randomly getting error

Hi, I'm getting this error randomly when trying to change the state of my ACs, and temperatures are not properly updating I assume due to this error, anyone else experience this?
2022-03-13 16:29:56 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [139695595371632] Invalid response, status code: 500 - Data: {"message":"PCPF Communication error","code":5003}
Traceback (most recent call last):
  File "/config/custom_components/panasonic_cc/panasonic.py", line 21, in wrapper_call
    func(*args, **kwargs)
  File "/config/custom_components/panasonic_cc/panasonic.py", line 346, in set_device
    self._api.set_device(
  File "/usr/local/lib/python3.9/site-packages/pcomfortcloud/session.py", line 389, in set_device
    raise ResponseError(response.status_code, response.text)
pcomfortcloud.session.ResponseError: Invalid response, status code: 500 - Data: {"message":"PCPF Communication error","code":5003}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 192, in handle_call_service
    await hass.services.async_call(
  File "/usr/src/homeassistant/homeassistant/core.py", line 1636, in async_call
    task.result()
  File "/usr/src/homeassistant/homeassistant/core.py", line 1673, in _execute_service
    await cast(Callable[[ServiceCall], Awaitable[None]], handler.job.target)(
  File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 204, in handle_service
    await self.hass.helpers.service.entity_service_call(
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 668, in entity_service_call
    future.result()  # pop exception if have
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 931, in async_request_call
    await coro
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 705, in _handle_entity_call
    await result
  File "/config/custom_components/panasonic_cc/climate.py", line 126, in async_set_hvac_mode
    await self._api.turn_off()
  File "/config/custom_components/panasonic_cc/panasonic.py", line 227, in turn_off
    await self.hass.async_add_executor_job(
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/config/custom_components/panasonic_cc/panasonic.py", line 24, in wrapper_call
    func(*args, **kwargs)
  File "/config/custom_components/panasonic_cc/panasonic.py", line 346, in set_device
    self._api.set_device(
  File "/usr/local/lib/python3.9/site-packages/pcomfortcloud/session.py", line 389, in set_device
    raise ResponseError(response.status_code, response.text)
pcomfortcloud.session.ResponseError: Invalid response, status code: 500 - Data: {"message":"PCPF Communication error","code":5003}

This issue has been inactive for a while, so we’re closing it for now. If you believe this issue is still relevant, please feel free to reopen it or create a new one.


fix: TreeWidgets issues

Description
This Fixes issues with Both Single Tree Select and Multi Tree Select.

Clearing Values
SelectedOptionLabels for both widgets

Fixes #16367
Fixes #16365
Fixes #15466
Type of change

Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Unit Test

Checklist:

[ ] My code follows the style guidelines of this project
[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes

/ok-to-test sha=cbee2c3

All the reported issues pertaining to TreeSelect & MultiTreeSelect widgets are fixed and working fine.
https://www.loom.com/share/72b543e08d984647a758e168da972c0e


Mongodb constantly at 100%+ of CPU

I've got a 2-core system running on Digital Ocean, and after ~3 days of tracking data, everything totally freezes up with Mongodb stuck at 100% on both cores.  I'm using redis-oplog for oplog tailing, but that doesn't seem to help :(
Anyone else experiencing similar?

  Strangely both of them have query: {} so it's hard to say what they're running.
{
        "inprog" : [
                {
                        "desc" : "rsSync",
                        "threadId" : "140578758936320",
                        "active" : true,
                        "opid" : 5,
                        "secs_running" : 468,
                        "microsecs_running" : NumberLong(468308691),
                        "op" : "none",
                        "ns" : "local.oplog.rs",
                        "query" : {

                        },
                        "numYields" : 0,
                        "locks" : {

                        },
                        "waitingForLock" : false,
                        "lockStats" : {
                                "Global" : {
                                        "acquireCount" : {
                                                "r" : NumberLong(19),
                                                "w" : NumberLong(3),
                                                "R" : NumberLong(1),
                                                "W" : NumberLong(1)
                                        }
                                },
                                "Database" : {
                                        "acquireCount" : {
                                                "r" : NumberLong(7),
                                                "w" : NumberLong(2),
                                                "W" : NumberLong(1)
                                        }
                                },
                                "Collection" : {
                                        "acquireCount" : {
                                                "r" : NumberLong(6)
                                        }
                                },
                                "Metadata" : {
                                        "acquireCount" : {
                                                "w" : NumberLong(1)
                                        }
                                },
                                "oplog" : {
                                        "acquireCount" : {
                                                "r" : NumberLong(1),
                                                "w" : NumberLong(2)
                                        }
                                }
                        }
                },
                {
                        "desc" : "ReplBatcher",
                        "threadId" : "140578607867648",
                        "active" : true,
                        "opid" : 9,
                        "secs_running" : 468,
                        "microsecs_running" : NumberLong(468305992),
                        "op" : "none",
                        "ns" : "local.oplog.rs",
                        "query" : {

                        },
                        "numYields" : 0,
                        "locks" : {

                        },
                        "waitingForLock" : false,
                        "lockStats" : {
                                "Global" : {
                                        "acquireCount" : {
                                                "r" : NumberLong(2)
                                        }
                                },
                                "Database" : {
                                        "acquireCount" : {
                                                "r" : NumberLong(1)
                                        }
                                },
                                "oplog" : {
                                        "acquireCount" : {
                                                "r" : NumberLong(1)
                                        }
                                }
                        }
                }
        ],
        "ok" : 1
}

Are you using the modifications included with PR #16 & #17? I saw the same insane mongo container CPU usage, I didn't quite fix it but reduced it to manageable levels by reducing processing intervals.
Before those changes it would fail due to running out of CPU resources after a couple days. Now I can keep 7 days worth of data on a t2.micro unlimited container, and CPU is usage sits around ~90%, so my web UI stays responsive enough for me. This is still kind of expensive ($13/month for excess CPU credits in addition to $8/month for the instance) but works well enough for now.

I have high CPU usage too but the UI response fine, so I didn't spend to much time on investigation.
Maybe some mongodb indexes are missing and the queries are taking to long?

Hi @jehartzog and @lmachens thanks for getting back.  After reporting this I did a reboot, and for the last 24 hours it's been hanging out around 100% (of a possible 200%), so things are running fairly smoothly.  Perhaps some indexes just needed some time to work out; or, similarly, redis-oplog had to kick into gear.  The good news is that it seems to be working fairly stable on a $15/month Digital Ocean droplet (2gb ram and 2 CPU).  Will close this issue for now, and re-open if it crashes again.

Okay so about 5 hours ago it looks like mongodb might have crashed because of excessive CPU usage? Not sure... this is all I see in the logs:
(node:17383) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): MongoError: connection 0 to <IP_ADDRESS>:3001 timed out
Grr.

Double check that you are using code from the two PR's I added above, they helped manage (but not completely fix) this exact issue for me.

@jehartzog Yep, I most certainly am running those -- I only set this server up last week.  Here's what I tried:

A mup deploy, which we've discussed in another thread. This worked for a bit, but then stopped
So, I decided to set up an Ubuntu server and run Meteor on there directly rather than via mup. This worked swell for ~3 days, but stopped because the mongodb instance was using 100% CPU.
I upgraded the server to have 2 cores and made sure I had oplog working well. This seemed to peg mongodb at 100%, and again it worked for a few days.
Then yesterday it stopped, and the logs showed that MongoError above. So, I rebooted everything 13 hours ago when I made that post, and it seems to have worked since then

Stumped for now.

You can adjust metricsLifetime. If you limit it to like 3 days, the cpu usage should be better.

My settings.json already has it set to 3 days:
{
        "metricsLifetime": 259200000
}

My Digital Ocean droplet has 60gb of space -- of which it says 6gb has been used so far.  The Meteor node process is at ~1% CPU and the mongodb one consistently at 100%.

My CPU usage is always at 100% too, but it is fine for me because of 2 cores. No issues in the UI. The aggregations are the reason for the high CPU.
Maybe you can try out what happens when you deactivate/comment out some aggregations in rma server?

I went ahead and changed the first contsant from 60 * 1000 to 5 * 60 * 1000 and when my server restarted mongodb went from 100% to 200% (so using max of both cores).  I'm guessing that's because of this:
Meteor.startup(() => {
  runShort();
  runMedium();
  runLong();
});

Which gets it to run when Meteor first reboots.  So I gave it a few minutes to settle down, but... 30 minutes later and it hasn't died down yet.

Looks like my mongodb is now at 200% again -- both processors completely hogged up and the UI pretty much unresponsive.  I'm going to expand to 3 CPUs and see what happens.

Been running it with 4 cores and 4gb of ram and she's finally happy. But I'm not super happy about the $40/month that's setting me back! Going to shop around for another host at some point.

Unless you're experiencing some massively different customer usage than I'm used to (~200 peak active simultaneous users), you may want to dig more into fixing the issue causing the mongo container CPU usage. I don't think you'll be able to find a significantly cheaper hosting service that can do the CPU processing that is currently required.

Okay so here's an observation -- in the server/rma/server.js file, each of runShort, runMedium and runLong get fired at the same time when Meteor starts. I put a little timer on each.  At startup, this is how long each one took to run:

runMedium = 196 seconds
runShort = 216 seconds
runLong = 250 seconds

Once they were done, runShort was able to run on its own again and took a total of 348 seconds!  As soon as it was done, it started instantly again -- this time taking 480 seconds.  Sure enough, the instant it was done another runShort fired, this time taking 579 seconds. So... getting longer each time, heading in the direction of 10+ minutes per runShort().  Not so short, hey!
Each one of them takes up at least 100% of a CPU -- and in my case, with 4 cores, the CPU was at 400%.  It also looks like whenever runMedium fires (30 minutes), runShort will also fire.  And whenever runLong fires (3 hours), both runShort and runMedium will fire.  So I've constantly got my CPU maxed out at 100% with runShort(), then every 30 minutes it bumps up to 200%+, and then every 3 hours it gets hammered at a full 400%.
Some steps to take to figure this out and mitigate the issue:

Make sure none of them run at exactly the same time
Figure out why the runShort(), which is set to run every 60 seconds, takes more like 600 seconds

I'll report back if I make any progress -- this is my first time digging into what actually happens with each aggregation, so if you've got some thoughts in the meanwhile that'd be great :)

Even with 4 cores, my mongodb keeps crashing and the whole site becomes unusable :(
This could be because I've got so many methods / publications?  For example, there'll often be 25k to 35k methods per hour that it has to aggregate?

WOW. Okay so I added some indexes and it's like night and day.  Instead of runShort() taking 2+ minutes, it now takes ~500 milliseconds.  Meanwhile my mongodb CPU is at almost 0%. I'll put in a pull request for the changes.

@jasongrishkoff This looks great, thanks for keeping up the hunt to figure this out!
Are you 100% sure those indexes are the only things you changed to fix the CPU usage? For one thing, number of those indexes already exist. I also added the indexes you recommend and I'm not seeing a change in CPU usage.

@jehartzog you added the indexes in this pull request, right? https://github.com/lmachens/meteor-apm-server/pull/23
I'm 100% positive that was the only change, and it was like night and day.
When you fire db.methodsTraces.getIndexes() do you get one that looks like this?
        {
                "v" : 1,
                "key" : {
                        "id" : 1
                },
                "name" : "id_1",
                "ns" : "meteor.methodsTraces"
        }

I did use the indexes you added in #23, the only thing I did different was add them in the background.
These are the only 4 collections that you edited in the PR, but methodsTraces is not one of those changed. I'm not sure why I would check indexes on that collection to ensure I applied the PR correctly :D.
  ErrorMetrics.rawCollection().createIndex({id:1});
  SystemMetrics.rawCollection().createIndex({id:1});
  MethodsMetrics.rawCollection().createIndex({id:1});
  PubMetrics.rawCollection().createIndex({id:1});

@jehartzog my apologies, I definitely should have said db.methodsMetrics.getIndexes(), which should give you a response as such:
        {
                "v" : 1,
                "key" : {
                        "id" : 1
                },
                "name" : "id_1",
                "ns" : "meteor.methodsMetrics"
        }

I tested it by doing db.methodsMetrics.dropIndex('id_1') and the negative impact was instant:
I20180226-06:49:30.941(0)? finished runShort 444ms
I20180226-06:52:21.866(0)? finished runShort 111369ms

I went back and reverified everything, don't know what I was smoking before but your solution is 100% effective. I had somehow missed the adding the index for pubMetrics, even though I had a history of the correct mongo command and had verified it. I probably had switched db at some point and not realized it. Thanks for working with me on it!

@jehartzog Nice!  I created a new version with your changes. https://github.com/lmachens/meteor-apm-server/blob/master/CHANGELOG.md#108---2018-02-27


runtime: non-cooperative goroutine preemption does not work when built using c-archive or c-shared

Non-cooperative goroutine preemption does not work when built using c-archive or c-shared, because of this PR not install signal go handler and cause signal preemption disabled.

What version of Go are you using (go version)?

$ 1.14

Does this issue reproduce with the latest release?
yeah.
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/fakang.wfk/.cache/go-build"
GOENV="/home/fakang.wfk/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"

What did you do?

test.c
#include <stdio.h>
#include <unistd.h>

#include"cgo.h"

int main(){
    sleep(1);
    printf("start cgo\n");
    test();
    printf("exit cgo\n");
    sleep(5);
    return 0;
}

cgo.go
package main

import (
        "runtime"
)

import "C"

func init() {
        runtime.GOMAXPROCS(1)
        go func() {
                println("init ok")
                for {
                }
        }()
}

//export test
func test() {
        println("I got scheduled!")
}

func main() {}

run:
go build -o cgo.so -buildmode=c-shared cgo.go
gcc -o main main.c  cgo.so
What did you expect to see?
./main
init ok
start cgo
I got scheduled!
exit cgo

What did you see instead?
./main
init ok
start cgo

@ianlancetaylor The signal _SIGPROF may also need to be added because CPUProfile depends on it.

Please always write plain text as plain text, never as an image.  Plain text is much easier to read and it is possible to cut and paste it.  Thanks.
If Go profiling is requested a signal handler will be installed for SIGPROF if necessary.  We definitely don't want c-archive or c-shared Go code to override the SIGPROF handling of the main program; we only want to use the Go handling if the Go program specifically requests it.

Please always write plain text as plain text, never as an image. Plain text is much easier to read and it is possible to cut and paste it. Thanks.
If Go profiling is requested a signal handler will be installed for SIGPROF if necessary. We definitely don't want c-archive or c-shared Go code to override the SIGPROF handling of the main program; we only want to use the Go handling if the Go program specifically requests it.

Um, sorry. Because I didn't have permission to comment directly on the go-review.googlesource.com platform, so do it.
I have a question that if we don't want c-archive or c-shared Go code to override the SIGPROF handling of the main program, but why is it necessary to check  SIGPROF in runtime·cgoSigtramp?
https://github.com/golang/go/blob/6b3f4d388fe5602172f45361f438edf54699b953/src/runtime/sys_linux_amd64.s#L436-L440

The cgoSigtramp function is used in all build modes, not just c-archive and c-shared.  Also, to be clear, I am saying that for c-archive and c-shared we must not override the handling of SIGPROF by default.  If the program itself calls Go functions like runtime/pprof.StartCPUProfile, then the Go standard library will go ahead and override the handling of SIGPROF.  And in that case of course it's important that cgoSigtramp handle SIGPROF correctly.
Further discussion of these points should take place in a forum, not here on the issue tracker.  See https://golang.org/wiki/Questions.  Thanks.


Undo/redo in a loop causes exponential blowup of history

Describe the bug
Repeatedly doing undo/redo of the same update causes slowdown and room disconnect.
To Reproduce
Steps to reproduce the behavior:

Open whiteboard demo. https://liveblocks.io/examples/collaborative-whiteboard/nextjs
Create rectangle
Move rectangle
Press undo followed by redo multiple times. (5 times seems enough)
One instance of whiteboard reconnects in a loop.

Expected behavior
Rectangle moves between two positions.
Environment (please complete the following information):

<EMAIL_ADDRESS><EMAIL_ADDRESS>
Additional context
It seems that history entries are duplicated during undo/redo. I can see in network debugger that websocket messages are growing exponentially in size and contain duplicate entries.

Upon brief inspection code in _applyUpdate seems suspicious. reverseUpdate is first added to reverse array on line 378 and later added to the same array on line 426

@slon Your intuition about the reverse was correct. We've fixed this in #902, added a test,  and published v1.0.11 to resolve this issue. Thank you for the report!