Object reference not set to an instance of an object

when I try to publish my world it happened like this every time. I also update Udonsharp and SDK to the lastest

If you can post your world ID here so I can verify the issue and or message me on discord @ Superbstingray#6437 so I can look into this further I would appricate it!

I add you on Discord and and my name is Jerker


Remove rendering settings from profile

The ForceFullRepaintRendering and SoftwareRendering are global only and for some reason were in profile. This commit removes them.
Reference: https://github.com/microsoft/terminal/pull/11416#discussion_r742030103

Build log confirms -- Terminal.Settings.Model.dll still builds. I'm admin-merging this to avoid another signoff cycle./


storage/eos: set --comment when running eos commands

We can add --comment flag to the eos command to include the request id that will be logged on EOS for debugging purposes.

When executing commands involving the eos binary:
cmd := exec.CommandContext(ctx, "/usr/bin/eos", "-r", unixUser.Uid, unixUser.Gid, "rm", "-r", path)

We need to add a comment option
reqid := getReqID()
cmd := exec.CommandContext(ctx, "/usr/bin/eos", "-r", unixUser.Uid, unixUser.Gid, "rm", "-r", path, "--comment", reqid)

To not repeat the code, you can create a new execute function like executeEOS add adds this option.
The current execute call is also used for the xrdcopy call and this binary does not support the --comment flag.
To obtain the reqid you need to import the appctx package.


 For e.g. in human eval there are 164 examples which can take 30-60 minutes to generate depending on hardware/model, sometimes its helpful to evaluate your model on just 10-20 examples as a sanity check.
Currently this is not possible due to the assert statements here
https://github.com/evalplus/evalplus/blob/b7f2961f1223a1c488977ac24db5c162401ead08/evalplus/evaluate.py#L174-L175
But can be easily achieved by just taking the subset of examples out of the full dataset here
https://github.com/evalplus/evalplus/blob/b7f2961f1223a1c488977ac24db5c162401ead08/evalplus/evaluate.py#L132
I was wondering if this is a feature that can be useful, if yes maybe this can be added as a flag. What do you think? I can contribute if needed

@RahulSChand Thanks for the suggestion, this sounds like a great feature to have. Feel free to open a PR with your implemention and we can take a look.

There is a bit of complication here since the key to each task (i.e., task_id) can be strings such as HumanEval/k which makes it inconvenient to type as CLI args and existing ground-truth uses a cache mechanism based on the MD5 fingerprint of the raw dataset file. Meanwhile, issues can be foreseen in the future when evalplus integrates other datasets using a different task_id mechanism. Therefore, IMO the best workaround is to simply truncate and override the dataset.
Consequently, I made d4ff6c4e4faa2ffc632e4cce6b0331dbd896a9a2 a compatible way to add this feature:

Create a sub-dataset file at your own flavor (by cutting the lines of tasks you are not interested in)
Use the env variable HUMANEVAL_OVERRIDE_PATH to point to that path

More concretely:
# Get top-10 tasks to quick-start.jsonl
cat ~/.cache/evalplus/HumanEvalPlus-v0.1.6.jsonl | head -n 10 > quick-start.jsonl
# Run with envar set
HUMANEVAL_OVERRIDE_PATH=$(realpath quick-start.jsonl) $YOUR_EVALPLUS_COMMAND # evalplus.evaluate...

Let me know your thoughts on this workaround. Thanks.

@ganler Thanks for this. I think this workaround is enough. No need of changes in the code logic & won't be affected if the keys of the dataset change.


Close #40

cg_iRIC_Read_Grid2d_Str_Size etc. returns invalid data when 3d grid exists.

Hi Scott,
This is the pull request to fix the issue that you've reported with FaSTMECH source code.
Can you make sure that after merging the pull request, cg_iRIC_Read_Grid2d_Str_Size etc. returns valid data in case 3D grid exists?
Thank you,
Keisuke Inoue

Hi Keisuke,
It fixes the unit test (unittests_cgnsfile):
case_GridRead_FaSTMECH_3d_results
but now it fails the unit tests (unittests_cgnsfile):
case_SolWriteStd_hdf5
case_SolWriteDivide_hdf5
Also, the fortran unit test (test1) is now failing in the develop_v4 branch.
It looks like it was broken by commit 257d28d.  It causes an hdf5 error
when calling cg_iRIC_Write_ErrorCode.  It works in commit 07bd0f8.
Thanks,
Scott

Hi Scott,
I've pushed a commit so that the following issue is fixed:

but now it fails the unit tests (unittests_cgnsfile):
case_SolWriteStd_hdf5
case_SolWriteDivide_hdf5

Also, the fortran unit test (test1) is now failing in the develop_v4 branch.
It looks like it was broken by commit https://github.com/i-RIC/iriclib_v4/commit/257d28d754a129a8bd8768592fc6ed79e80f9911. It causes an hdf5 error
when calling cg_iRIC_Write_ErrorCode. It works in commit https://github.com/i-RIC/iriclib_v4/commit/07bd0f8e5ee2a1682d9d54b842a71445fc3eb23a.

I tried to reproduce the issue with this branch.
I compiled iric.f90 to generate iric.mod, and then compiled test1.f90.
It outputs the following error log:
test1.f90(204): warning #5117: Bad # preprocessor line
#if 0
-^
test1.f90(266): warning #5117: Bad # preprocessor line
#endif
-^
test1.f90(208): error #6634: The shape matching rules of actual arguments and dummy arguments have been violated.   [CONDINDICES]
                        condindices(condid:condid, 1:2, 1:condindexlen(condid)), ier)
------------------------^
test1.f90(219): error #6634: The shape matching rules of actual arguments and dummy arguments have been violated.   [FUNC_PARAM]
                        func_param(condid:condid, 1:func_size(condid)), &
------------------------^
test1.f90(220): error #6634: The shape matching rules of actual arguments and dummy arguments have been violated.   [FUNC_VALUE]
                        func_value(condid:condid, 1:func_size(condid)), ier)
------------------------^
test1.f90(223): error #6634: The shape matching rules of actual arguments and dummy arguments have been violated.   [FUNC_REAL_PARAM]
                        func_real_param(condid:condid, 1:func_size(condid)),  &
------------------------^
test1.f90(224): error #6634: The shape matching rules of actual arguments and dummy arguments have been violated.   [FUNC_REAL_VALUE]
                        func_real_value(condid:condid, 1:func_size(condid)), ier)
------------------------^
test1.f90(227): error #6634: The shape matching rules of actual arguments and dummy arguments have been violated.   [FUNC_WITHNAME_PARAM]
                        func_withname_param(condid:condid, 1:func_size(condid)), ier)
------------------------^
test1.f90(230): error #6634: The shape matching rules of actual arguments and dummy arguments have been violated.   [FUNC_WITHNAME_VALUE]
                        func_withname_value(condid:condid, 1:func_size(condid)), ier)
------------------------^
compilation aborted for test1.f90 (code 1)

It looks that the result is different. Can you make sure you can build and link test1.F90 with the newest iric.f90?
Thank you,
Keisuke

Hi Keisuke,
If you use cmake to build and test iriclib it automatically turns on the fortran preprocessor.  You can turn it on manually using the following:

I checked your latest issue-40 branch and it's still throwing an hdf5 error here:

HDF5-DIAG: Error detected in HDF5 (1.8.21) thread 0:
  #000: D:\a\iricdev-2019\iricdev-2019\lib\src\CMake-hdf5-1.8.21\hdf5-1.8.21\src\H5G.c line 312 in H5Gcreate2(): unable to create group
    major: Symbol table
    minor: Unable to initialize object
  #001: D:\a\iricdev-2019\iricdev-2019\lib\src\CMake-hdf5-1.8.21\hdf5-1.8.21\src\H5Gint.c line 192 in H5G__create_named(): unable to create and link to group
    major: Symbol table
    minor: Unable to initialize object
  #002: D:\a\iricdev-2019\iricdev-2019\lib\src\CMake-hdf5-1.8.21\hdf5-1.8.21\src\H5L.c line 1636 in H5L_link_object(): unable to create new link to object
    major: Links
    minor: Unable to initialize object
  #003: D:\a\iricdev-2019\iricdev-2019\lib\src\CMake-hdf5-1.8.21\hdf5-1.8.21\src\H5L.c line 1880 in H5L_create_real(): can't insert link
    major: Symbol table
    minor: Unable to insert object
  #004: D:\a\iricdev-2019\iricdev-2019\lib\src\CMake-hdf5-1.8.21\hdf5-1.8.21\src\H5Gtraverse.c line 859 in H5G_traverse(): internal path traversal failed
    major: Symbol table
    minor: Object not found
  #005: D:\a\iricdev-2019\iricdev-2019\lib\src\CMake-hdf5-1.8.21\hdf5-1.8.21\src\H5Gtraverse.c line 639 in H5G_traverse_real(): traversal operator failed
    major: Symbol table
    minor: Callback failed
  #006: D:\a\iricdev-2019\iricdev-2019\lib\src\CMake-hdf5-1.8.21\hdf5-1.8.21\src\H5L.c line 1672 in H5L_link_cb(): name already exists
    major: Symbol table
    minor: Object already exists

Thanks,
Scott

Hi Scott,
thank you for your information to reproduce the issue. I’ll check and fix it.
I have a task to finish this week, so I think I’ll work on it next week. In my understanding, the issue does not affect your work to migrate FaSTMECH to v4. Am I right?
Thank you,
Keisuke

Hi Keisuke,
I don't think it affects Fastmech v4 migration.
Thanks,
Scott

Hi Scott,
I've pushed another commit so that building test1.f90 succeeds.
Honestly speaking, I did not succeed to reproduce the situation that you've reported. I think I'm not familiar enough to CMake.
I've built test1.f90 manually using Intel OneAPI fortran compiler. With older iric.f90, compiling test1.f90 failed, and with new iric.f90, it succeeds. I've also made sure that test1.exe runs successfully.
Can you make sure if test1.f90 can be built successfully, and run without errors, please?
If not, maybe we can investigate what's happening in the next web meeting, by sharing our screens.
Thank you,
Keisuke

Hi Keisuke,
Today is a holiday in the US.  I will try to test it tomorrow.
Thanks,
Scott

Hi Scott,
Thank you for your response. No problem. Thank you for reviewing.
Thanks,
Keisuke
2022年6月21日(火) 8:21 Charlton, Scott R. @.***>:

Hi Keisuke,
Today is a holiday in the US. I will try to test it tomorrow.
Thanks,
Scott
—
Reply to this email directly, view it on GitHub
https://github.com/i-RIC/iriclib_v4/pull/41#issuecomment-1160944705, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AA6YIUKY7SO4ZSGJYJIFHULVQD4G5ANCNFSM5YNKTIFA
.
You are receiving this because you authored the thread.Message ID:
@.***>

Hi Scott,
Thank you, I’ve got it!
Thanks,
Keisuke
2022年6月22日(水) 4:38 Charlton, Scott R. @.***>:

Merged #41 https://github.com/i-RIC/iriclib_v4/pull/41 into develop_v4.
—
Reply to this email directly, view it on GitHub
https://github.com/i-RIC/iriclib_v4/pull/41#event-6850463439, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AA6YIUIPSKQW7V3QJXCOBW3VQIK47ANCNFSM5YNKTIFA
.
You are receiving this because you authored the thread.Message ID:
@.***>


Only the first cell is visible with certain files

I just found this program, and am having difficulty with certain notebooks. The ones I have saved myself work fine, but those produced by the Julia package Literate.jl don't seem to work.
This package turns files like example.jl into notebooks like example.ipynb. I attach an example of what I see. Only the first cell is shown by nbviewer-app, while nteract (the window behind) shows more.
I don't know which end has the problem here -- it only happens for me with this viewer and such files together. Files created directly in Jupyter work fine. And Jupyter itself works fine with these files.

Interesting... If you open the same file in Jupyter and then save it, it will correctly show also in nbviewer-app. This is actually a problem with the JSON produced from Literate.jl.. I could trace the problem to the first output cell, where nbviewer.js expects an array, but only a string literal is found (here, for example).
Should be an easy fix. I'll look into it.

That I did not think to try, thanks for taking a look.
Is this something Literate.jl ought to change, to conform to standard, or more of a grey area?

Looking through the notebook file format docs, the behavior of Literate.jl is totally fine:

If you intend to work with notebook files directly, you must allow multi-line string fields to be either a string or list of strings.

So this should be fixed in nbviewer.js, which is used by nbviewer-app to render the notebooks.

Thanks for reporting this, I think I've fixed it (and one other related thing), see #28.

Thanks for fixing this, @kokes. I'll merge an updated version of nbviewer.js when #5 is addressed. I'll file a PR if you don't beat me to it 😉

Fixed in 43929a71e7951f51176507570105d972adc125cc


bug fix thermal fwhm equation

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 100.00%. Comparing base (59a2dfa) to head (546b82d).
Report is 13 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff            @@
##              main       #14   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files            5         5
  Lines          161       212   +51
=========================================
+ Hits           161       212   +51

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Skip 64-bit json test on browser platforms

Fixes https://github.com/dart-lang/protobuf/issues/111

:shipit:


Recommend vim-fetch for Neovim

I spent some time trying to get file-line to work with this plugin on Neovim but couldn't. However I found another plugin that does basically the same thing and that worked out of the box on my setup: https://github.com/wsdjeg/vim-fetch
Great plugin btw

Thank you
I'll try it when I have time and add it to the documentation.

After some consideration, we decided to reduce our reliance on plugins.
Neither file-line plugin nor vim-fetch plugin is required for operation


x/tools/gopls: strange error in type checking

This was reported in https://github.com/golang/go/issues/60551#issuecomment-1588750612, but I think is a separate bug (in the ordinary type checking, not analysis component:
2023/06/13 10:49:34 computing the shared import graph: import failed for "vendor/golang.org/x/crypto/chacha20poly1305": internal error while importing "vendor/golang.org/x/crypto/chacha20poly1305" (conflicting names chacha20poly1305 and server for package "vendor/golang.org/x/crypto/chacha20poly1305"); please report an issue
I wonder whether we're making spurious hits in an ID or PackagePath-keyed import map because the current package is called "command-line-arguments"?

We believe this was fixed by https://go.dev/cl/505219.


Functions that accept an EntityClient as an argument should also accept a cache

I want functions that accept an EntityClient as an argument to also accept a cache so that I can use them in EntityApi.mutate invocations.

This issue is obsolete because we have decided not to expose the EntityClient to applications


Multi-project-runner fails with one project

Do you want to request a feature or report a bug?
Bug.
What is the current behavior?
If I remove one of two projects in package.json, test runner gets super confused. Symptoms are similar to https://github.com/facebook/jest/issues/4037. It's basically as if it runs without configuration at all.
What is the expected behavior?
Removing one project from projects doesn't cause the other one to fail.
Please provide your exact Jest configuration and mention your Jest, node, yarn/npm version and operating system.
You can reproduce with this React commit: https://github.com/facebook/react/tree/171149a4b02a399a986c411b264515de45e70521.
Remove one of projects in package.json and run npm test.

i have a fix for it but it needs some more work to do https://github.com/facebook/jest/pull/4083

Fixed in #5176


[docs] updated list of external repos in README

Refer to #1074.
Closed #985.

@chivee All CIs are broken now 😢
#1849 and #1855 should be merged first to repair CIs.


PR: Fix concurrency for installers-conda workflow

Recent concurrency specifications in Spyder GitHub workflows (spyder-ide/spyder#20254) do not prevent build-installers workflows in installers-conda.yml from running under a certain circumstance. Example PR: Cancel runs in progress when pushing new commits (CI) #250
Concurrence cancels workflows that are pending or currently running. However, build-installers workflows are neither pending nor running until after build-matrix and build-noarch-conda-pkgs workflows are all complete. In the circumstance that one or more of these workflows is not complete, they will be cancelled but build-installers workflows will then commence.
This PR should remedy this by replacing the conditional for build-installers workflows with explicit dependence on the results of the needed workflows. ! failure() returns true for "success", "skipped", and "cancelled", and therefore did not achieve the desired effect.
Note that if build-installers workflows are already pending/in process they will not be cancelled immediately. This is because the new build-installers workflows will not be queued until completion of the build-matrix and build-noarch-pkgs workflows. Upon queueing the new build-installers workflows, the old ones will be cancelled.

The appropriate effect was confirmed by the following workflows

PR: Fix concurrency for installers-conda workflow #259, while build-installers was in progress, was cancelled by...
PR: Fix concurrency for installers-conda workflow #260. While build-noarch-pkgs was in progress, was cancelled by...
PR: Fix concurrency for installers-conda workflow #261. While build-noarch-pkgs was in progress, was cancelled by...
PR: Fix concurrency for installers-conda workflow #262

However, I do not understand why an entire workflow is not cancelled, rather than cancelling individual jobs...
It seems that the language used in the documentation is a bit mis-leading.


Consider internationalization

I would be useful if the strings in English that are displayed to users were extracted to allow internationalization. Currently, these strings are scattered throughout the code, such as in the autocomplete or in the interface.

Seems like a reasonable thing to do. The thing that's holding me back is the integration of 3rd party UI libs (e.g. datatables, Google chart, the pivot table plugin,and perhaps other in the future). These may or may not support internationalization..

These are YASR plugins, right? In YASQE, internationalization would be related mostly to itself, so it may be simpler to achieve.
YASR already nicely exposes ways to localize plugins, such as YASR.plugins.table.defaults.datatable.language for DataTables.


Tidying

I noticed a few files in pacehrh/tests that shouldn't have been committed. Now deleted.
Also deleted some commented-out code that's no longer needed.

Added mods to .gitignore so it's more selective.


Zeroed values every 15min

Hoymiles MQTT version: 0.4.0
Python version: 3.10
Operating System: Ubuntu 22.10 Linux 5.19.0-31-generic
DTU-PRO-S V00.02.15 Hardware Ver.H09.06.01
Hoymiles HMS-2000-4T

Description
Thank you for excellent piece of software!
Code works well but described elsewhere limitations in stability of Hoymiles DTU-PRO-S are significant. I observe every 4-5 days periodic drop-downs in readings during the day.
I have attached the image of diagram presenting the issue.
My startup parameters are:
python -m hoymiles_mqtt --microinverter-type HM --mqtt-broker <IP_ADDRESS> --mqtt-user xxx --mqtt-password xxx --dtu-host <IP_ADDRESS> --comm-close-comm-on-error True --query-period 100
To avoid total hang-ups I reset DTU every night at 2:13am. That solution resulted with stable work of DTU except the issue I describe here.
Observations:

Drop-downs happens every 15min (obviously connected with 15min period of sending data to Hoymiles cloud)
Single drop-down can usually influence 1-4 of 5 inverters I have, not affected inverters have all values good
Single drop-down zeroes all values for affected inverter (power, voltage, all ports,...)
the issue is related to interval between queries set. Having default value 60s caused issue every day. 100s period resulted with issue every 4-5 days.
no entries in log appear when values are zeroed.
(module installed as systemd service, log check with: sudo journalctl -u hoymiles_mqtt@homeassistant)

Knowing above I believe that it is possible to modify code of module in a way that faulty readings will be skipped - not send to mqqt. I strongly prefer to have no values at all than having them wrongly zeroed.
I'm ready to modify code myself in quite primitive way checking the value of grid frequency against zero. Hopefully more smart solution exists!

Hi, have you tried --expire-after option? Set the value to more than double the query period (for example 220 in your case)

Hello.
I did not. I believe that default is 'expire-after=infinite'. This is how I understand docs on this page: https://www.home-assistant.io/integrations/sensor.mqtt/#expire_after
Anyway I will try. I have just set this param. I will wait for failure and let you know.
Still the issue is not about missing data but wrong data received via modbus. At least once a week there is a day when almost every 15min zeroed values arrive from mqtt. This is single wrong sample every 15min. Next query results proper values. See attached data below.
I'm aware that that it will be difficult to differentiate in code logic between DTU failure and some regular situations.

Please try the option anyway. What is the link status when it happens?

Yes, I have configured 'expired' option yesterday evening.
Link status is still 1.
But Operating status is 0. So it goes down from 3 to 0 when failure happens.

Hi again.
I set "--expire-after" option to 10 minutes. But it did not help itself. Zeroed values still appeared in states and on graphs.
Still you inspired me. I modified code (ha.py) in a way to not send selected sensitive states to mqtt when operating status of inverter is other than 3. On time of failure data are not send to mqtt. For 10minutes last values are used by HA. This is more than enough to cover the DTU failure.
Issue seems solved for me. :)
My DTU seems overloaded. It serves 5 inverters - 4 panels each, power meter via wire and modbus inquries via tcp. Unfortunately my DTU is 1Ghz version and I can't replace it with openDTU.

I'm closing the issue.

@jtkacz sorry for not responding for a long time. I can add an option to ignore values when the operating status differs from 3.

Thank you for your attention and preparing special experimental version :)
But this is more complicated. Applying your patch results with numerous errors in log of HA when new code starts to ignore selected values due to operating_status==3:

Template variable warning: 'dict object' has no attribute 'pv_voltage' when rendering '{{ value_json.pv_voltage }}'
Template variable warning: 'dict object' has no attribute 'pv_current' when rendering '{{ value_json.pv_current }}'
Template variable warning: 'dict object' has no attribute 'pv_power' when rendering '{{ value_json.pv_power }}'

That's because code uses common state_topic and common json payload for all values for each device_serial/port combination.  HA can't find ignored values and rises errors.
I have corrected that issue with change to sensor's config definition in ha.py: https://github.com/jtkacz/hoymiles-mqtt/commit/125d79fbd8726a0d63b8d21f5e9cd143399b883a

There are some smaller issues still. Give me some time for testing. I will come back in a few days.

Can I make a suggestion? Could you please at the end of the function "get_state" in ha.py check the value returned from the DTU for 'pv_power' and see whether it is the same as the sum of the 'pv_power' values from all of the microinverters?
In my experience the DTU continues collecting correct values for each microinverter, but "relatively often" (maybe once every 15 minutes as reported here) the value returned by the DTU for 'pv_power' is zero (or simply not consistent with what the microinverters report).
I know, ha.py is just trying to report the raw values so I could "fix" this in my mqtt application (in my case home assistant) - but it seems "cleaner" to do something in the code which tries to get these values from modbus rather than exposing to home assistant.

@w-a-p  According to DTU modbus specification the only option to gather power value for whole plant is to read each microinverter and its port one by one, then sum everything to get total power. And that is what hoymiles_modbus\client.py  does to calculate PlantData.pv_power.
I fully share you opinion to correct DTU issue on lower level. Thus currently I fine-tune the code of hoymiles_mqtt package to hide zeroed values resulting from DTU under-performance. The main idea is that it is better to not send value at all than send wrongly zeroed value.
I make experiments on fork https://github.com/jtkacz/hoymiles-mqtt/tree/ignore_values

I haven't made any more updates to the code. Summarizing changes:

Skip sending ALL inverter port's measures to mqtt when operating_status <> 3. For temporary DTU issues it is better solution than setting states to unavailable, especially with parameter: --expire-after 600
Benefit: avoid wrong data in database and ugly graphs with zeroed values;)
Temperature, grid voltage and frequency set to unavailable when operating_status <> 3
Benefit: see p1
Send total values of energy production (from internal cache) to mqtt only if ALL inverters report operating_status == 3
Benefit: avoid issues in reporting of energy production by HA. State class set to total increasing hates incidental drop-downs.
Update module's internal energy cache when new value is larger than value currently stored in cache
Benefit: see p3
Do not send to mqtt energy data on level of microinverter's ports (i.e. sensor.port_1_116481317589_today_production) when  energy is zero or operating_status == 3
Benefit: see p3

For me setup is working quite well now.
@wasilukm Thank you again for this code. It is clean and well structured. It was pleasure to dig into it and update to my needs. If you find anything valuable in my changes please feel free to use ideas or code snippets.

@jtkacz Thank you so much for your feedback, I will review your propositions and apply where feasible.

@jtkacz why is important for you to prevent zero values, is operating_status<>3 not enough for today_production and total_production?
I also have doubts regarding p3, it would mean if one inverter is broken (which happened to me already) no data will be available for the whole installation. I think making sure that new data must be greater than cached is sufficient.

why is important for you to prevent zero values, is operating_status<>3 not enough for today_production and total_production?

I'm geek on diagrams ;) Look on diagram I attached on begin of this thread (power per panel).

I also have doubts regarding p3, it would mean if one inverter is broken (which happened to me already) no data will be available for the whole installation. I think making sure that new data must be greater than cached is sufficient.

I believe cache is not sufficient. Cache has limited persistence, restart of process clears it. I have 5 inverters, they wake up one after another  in the morning. If cache has been restarted at night then reported energy production increases in steps. Total increasing sensors interprets this steps wrongly as energy production.
But I admit there is no one good way to solve this issues.

Hi. I must admit that I lost patience to Hoymiles DTU after two devices were broken without any discoverable reason. And I have confirmed over other forums that this issue is quite frequent.
Since June I use OpenDTU. It works flawlessly for me after proper configuration on HA - no drops to zero, no hangs, no tricks.
I propose to close this issue.

Hi, I understand your frustration. Unfortunately, OpenDTU doesn't support my inverters, so I will have to stick to this solution,
Thank, you for all the feedback, recently I had no time to work on this project but I will keep this issue open at least for some time as I want to make some improvements based on this.


Feature Request: Add the ability to render arrow scroll buttons on scrollbar.

Currently, there is no option to render arrow scroll buttons.  It would be nice (and relatively straightforward, looking at the code) to add an option to render these buttons.
I have an implementation locally, and I'm wondering if you (@malte-wessel) are accepting PRs for this feature.  I'd like to run the rough implementation by you before finalizing things.
Zack

yes sure, feel free to create a PR :)


Code Refactor: Auth GitHub Once

Improve the codebase (DRY) by only authing the user to GitHub once to retrieve their GitHub Personal Access Token for the DB and then pass the values to other .js files

Closed as of #70.


Comparison of approaches to tackle index memory usage

Describe the issue
As stated in #1988 the memory consumption of the index is one of the biggest issues with memory consumption of restic. The main point is the actual implementation of in-memory index storage in internal/repository/index.go
I would therefore like to start a new issue to discuss strategies that could remidy the high memory consumption.
As a starting point I opened a branch index-alternatives which implements the following strategies:

Standard: index as actually used in restic
Reload: Don't save anything. For any index operation reload the file and use the actual implementation for the temporily created index.
Low-memory index: Store only full index data for tree blobs. For data blobs only use a IDSet to save which data blobs are present. This allows most index operations. For the missing ones do as in Reload (at least for data blobs)
Bolt: Use a DB on disc via bbolt

Index strategy can be chosen by
restic -i <default|reload|low-mem|bolt>
The branch is considered WIP. I added a general Interface FileIndex which should be fulfilled by new index strategies. Also the actual implementation just loads index files (file-per-file) into the standard Index struct and then reloads them into a new data structure allowing the GC to clean up. Newly created index files are not affected, so there should be only an effect for repositories with data in it.
Please note that most implementations are quick-and-dirty and miss serious error handling, clean-up etc.
More index strategies can be easily added by adding a new implementation for FileIndex and adding them in repository.go (where loading takes place) and cmd/restic/global.go (where the flag -i is interpreted)
I appreciate to get feedback on the proposed index strategies.
In this task I would also like to collect good test settings where different index strategies can be compared to each other.

I like to start with the setup I used to test during the implementation:
I created test data with the following script
#!/bin/sh
mkdir data
for i in `seq 1 1000`; do
        echo $i
        mkdir data/$i
        for j in `seq 1 100`; do
                echo $i$j > data/$i/$j
        done
done

This creates 100.000 small different files in 1000 directories. Therefore this should give a lot of data blobs and still quite some tree blobs. Can be changed to simulate even more files..
I ran the inital backup
restic -r /path/to/repo init
restic -r /path/to/repo backup /path/to/data

and did a repeated backup run with unchanged data for all index strategies:
/usr/bin/time restic -r /path/to/repo -i <strategy> backup /path/to/data

besides measuring with time I also added memory profiling.
Here are the results:
Standard index

Files:           0 new,     0 changed, 100000 unmodified
Dirs:            0 new,     0 changed,     2 unmodified
Added to the repo: 0 B

processed 100000 files, 567.676 KiB in 0:08
snapshot d7a3f88b saved
13.14user 1.71system 0:11.16elapsed 133%CPU (0avgtext+0avgdata 90564maxresident)k
0inputs+64outputs (0major+1756minor)pagefaults 0swaps

(pprof) top
Showing nodes accounting for 15.21MB, 97.56% of 15.59MB total
Dropped 84 nodes (cum <= 0.08MB)
Showing top 10 nodes out of 59
      flat  flat%   sum%        cum   cum%
   12.52MB 80.26% 80.26%    12.52MB 80.26%  github.com/restic/restic/internal/repository.(*Index).store
    1.01MB  6.46% 86.72%     1.01MB  6.50%  github.com/restic/chunker.NewWithBoundaries
    0.39MB  2.52% 89.23%     0.39MB  2.52%  reflect.New
    0.36MB  2.29% 91.53%     0.37MB  2.34%  github.com/restic/restic/internal/restic.NodeFromFileInfo
    0.29MB  1.89% 93.41%     1.10MB  7.05%  github.com/restic/restic/internal/archiver.(*Archiver).SaveDir
    0.28MB  1.80% 95.22%     0.28MB  1.80%  bytes.makeSlice
    0.15MB  0.94% 96.15%     0.15MB  0.94%  github.com/restic/restic/internal/archiver.(*TreeSaver).Save
    0.09MB   0.6% 96.76%     0.09MB   0.6%  strings.(*Builder).grow
    0.08MB   0.5% 97.26%     0.08MB   0.5%  github.com/restic/restic/internal/restic.BlobSet.Insert
    0.05MB   0.3% 97.56%     0.38MB  2.46%  encoding/json.Marshal

Here, the index already is the huges part of memory. You can also see that time reports 90MB used. This is IMO the garbage collector setting that has been talked about already.
Reload index
This seemes to work. However it displayed a ETA of about 20 minutes while constantly using 100% CPU... This can be easily explained as it constantly rereads (and decrypts!) the index files..
However, I consider this index strategy is not worth a deeper look....
Low-Memory index
<snip: same output as above>
processed 100000 files, 567.676 KiB in 0:08
snapshot f8822cee saved
13.47user 2.03system 0:11.90elapsed 130%CPU (0avgtext+0avgdata 109584maxresident)k
0inputs+72outputs (0major+1945minor)pagefaults 0swaps

(pprof) top
Showing nodes accounting for 7.68MB, 94.71% of 8.11MB total
Dropped 88 nodes (cum <= 0.04MB)
Showing top 10 nodes out of 63
      flat  flat%   sum%        cum   cum%
    4.84MB 59.68% 59.68%     4.84MB 59.68%  github.com/restic/restic/internal/restic.IDSet.Insert
    1.01MB 12.50% 72.18%     1.02MB 12.58%  github.com/restic/chunker.NewWithBoundaries
    0.41MB  5.05% 77.23%     0.41MB  5.05%  reflect.New
    0.35MB  4.26% 81.48%     0.35MB  4.26%  github.com/restic/restic/internal/restic.NodeFromFileInfo
    0.28MB  3.47% 84.95%     0.28MB  3.47%  bytes.makeSlice
    0.27MB  3.31% 88.26%     1.01MB 12.46%  github.com/restic/restic/internal/archiver.(*Archiver).SaveDir
    0.19MB  2.33% 90.60%     0.19MB  2.33%  github.com/restic/restic/internal/archiver.(*TreeSaver).Save
    0.18MB  2.22% 92.81%     4.94MB 60.93%  github.com/restic/restic/internal/repository.(*IndexLowMem).store
    0.08MB  0.96% 93.78%     0.08MB  0.96%  github.com/restic/restic/internal/restic.BlobSet.Insert
    0.08MB  0.93% 94.71%     0.40MB  4.92%  encoding/json.Marshal

Total memory reported by the profiler is halved! Morover, the 12,5 MB Index data set is replaced by a less than 5 MB IDSet. I would estimate that with this approach the index memory requirement can be reduced to about 40%!
Trade-Off is a slightly higher total backup time (13,5s as compared to 13,1s with standard index).
I'm really interested how this index strategy performs in other test settings!
Bolt index - first run
<snip: same output as above>
processed 100000 files, 567.676 KiB in 0:30
snapshot 9ff3d71f saved
37.01user 3.25system 0:34.54elapsed 116%CPU (0avgtext+0avgdata 282816maxresident)k
0inputs+114096outputs (0major+5436minor)pagefaults 0swaps

(pprof) top
Showing nodes accounting for 2592.31kB, 90.31% of 2870.50kB total
Dropped 79 nodes (cum <= 14.35kB)
Showing top 10 nodes out of 81
      flat  flat%   sum%        cum   cum%
 1030.83kB 35.91% 35.91%  1037.16kB 36.13%  github.com/restic/chunker.NewWithBoundaries
  390.02kB 13.59% 49.50%   390.02kB 13.59%  reflect.New
  278.60kB  9.71% 59.20%   278.60kB  9.71%  github.com/restic/restic/internal/restic.NodeFromFileInfo
  274.66kB  9.57% 68.77%   933.59kB 32.52%  github.com/restic/restic/internal/archiver.(*Archiver).SaveDir
  244.05kB  8.50% 77.27%   244.05kB  8.50%  bytes.makeSlice
  169.81kB  5.92% 83.19%   169.81kB  5.92%  github.com/restic/restic/internal/archiver.(*TreeSaver).Save
      80kB  2.79% 85.98%       80kB  2.79%  github.com/restic/restic/internal/restic.BlobSet.Insert
   44.17kB  1.54% 87.52%   292.20kB 10.18%  github.com/restic/restic/internal/archiver.(*TreeSaver).save
   40.16kB  1.40% 88.91%    40.16kB  1.40%  strings.(*Builder).grow
      40kB  1.39% 90.31%       40kB  1.39%  github.com/restic/restic/internal/restic.NewBlobBuffer

It also creates a 72 MB my.db file where the index is saved.
As expected, the memory consumption is basically gone!
This however comes with a big tradeoff in backup time (37s as compared to 13.1s with standard index).
I think it would be interesting to see how this performs with bigger data sets. No clue if this performance issue just adds a factor of about 3 or is even growing more than linear.
Moreover, it seems to me that much more testing and fine-tuning should be considered with storage-based index strategies...
Bolt index - second run
As in the bolt setting the db is not deleted after the run, I reran with an already present db:
processed 100000 files, 567.676 KiB in 0:08
snapshot b364fba4 saved
14.26user 1.94system 0:12.07elapsed 134%CPU (0avgtext+0avgdata 133464maxresident)k
0inputs+88outputs (0major+2601minor)pagefaults 0swaps

Memory profile is almost identical to the first run which is as expected.
I really wonder why this second run is so much faster than the first. Is it the fs cache that is working now? Or does boltdb somehow uses the info of the already stored index to speed things up?
Generally this may indicate that with storage-based index we might have to think about a permanent stored index, because it is able to boost performance quite a lot!

Maybe worth trying an LSM-based Key/Value store too. LSMs have quite different performance and disk-footprint tradeoffs compared to BTrees, and may work better for the persistent index usecase. I'd personally use RocksDB but there are probably good-enough Go-native stores too.

Thank you for prototyping this! Looks very encouraging.
The index files are currently encrypted, so whatever disk store we use should be encrypted too. I looked at https://github.com/dgraph-io/badger - it's Go native, LSM-based and supports encryption.
Low memory index looks good for backups. I'm not sure if it will scale for restores from repos with huge number of blobs though.

Thanks for the discussion so far.
I have to admit that I didn't spend much time in trying other storage-based index options. (badger is still on my to-do-list)
The reason is the following: I see that there will be quite some issues to arise when seriously trying to bring this in production-grade state. I see issues about encryption, how to clean up files ans how to deal with aborted operations and so on.
This is why I started thinking about another possibility for optimizing in-memory index. The result is already in my branch index-alternative and named low-mem2. The idea is to store all index data in sorted tables (saving the overhead of the map) an try to not save anything duplicated. The pack ID for instance is saved in a separate table and only referenced in the main table.
I did not run much tests so far but it seems that performance is just a litte worse than the standard index. On the other side memory consumption is comparable to low-mem, that is saving around 60% compared to standard index in my test setting, see above.
Considering that for most operations (e.g. backup) only specific information of the index is needed, this can be further improved.
So I would suggest the following: I will try to make a PR to replace (or optionally replace) the standard index by the memory-optimized one (low-mem2). This will hopefully give a fast improvement about the memory issue.
The possibilities to use a disc-based index is then open for the future.
What is your opinion about it? Maybe some of the core maintainers can give me a hint about the right direction to work on. Thank you!

Cutting the memory usage in half, sound like a reasonable short-term solution to me. I'm currently a bit worried about a possible performance hit by keeping the data in a sorted array.  The current implementation of the MasterIndex simply iterates over a list containing the indexes. For larger repositories, with a few thousand index files, restic spends a large fraction of the time for backups / checks with iterating over the different indexes, see #2284.
Regarding your low-mem2 branch: I'm a bit confused by the SortByBlob and SortByPack functions. Shouldn't it be possible to just sort by blobID and then just iterate over the whole index for lookups based on the packID (as is done by the current implementation)? (My preference would be to get completely rid of the locking once an index has been finished)
The code currently is just able to convert an existing Index to the new format. It would be rather nice if it were able to convert a finalized Index to the more efficient format, to get the improved memory consumption also during the initial backup.

@MichaelEischer: Thanks for pointing out this issue in masterindex. As far as I can see, there are three possible ways the index is used:

read from index files and then read-only
created during the run an then read-only
Operations to modfy the index (this is only purging/recovering)

As 1. is the primary use case and 2. is quite similar I agree that we should optimize for this. I would propose to also change the way master index is working and allow an index structure to be filled by the index data of many index files that have been read (and also for the index files created and then only used read-only).
About low-mem2:

I added SortByBlob and SortByPack as there is a use-case for sorted packs. But I also agree we should try to have unblocked parellel operations on the index. As far as I can see, the choice whether to search by blob or search by pack is determined by the main command. Hence even if we do block for resorting, it would be efficiently non-blocking for the main operation. And, of course, we should use RWMutex.
Maybe search for packs in a sorted-by-blob list is also ok, just needs some testing..
About the performance considerations. The fact that restic at the moment just reads linearly through all index files makes me quite confident that one sorted list of blob of all index files and bisection search can even outperform the current implementation for large indices, i.e. for many index files ;-)

So how do we continue? As already mentioned I can prepare a PR to exchange the current index/masterindex implementation by an optimized one.
@MichaelEischer I would appreciate to get early feedback and improvement suggestions - may I give you some early parts of the work for review? Maybe we'll find also some people testing the changes with respect to performance and memory usage for real-life scenarios...

I did some test runs with the check command and something strange happens. Using the restic master the command completes in 6 minutes (327.25 real 613.08 user 208.48 sys), with the index changes it takes more than an hour (4516.77 real 896.17 user 531.71 sys). The time is spent somewhere after printing "check snapshots, trees and blobs", but didn't have time to take a closer look yet.

This should be fixed now.

I'll give it a try in the next days.

Race condition in index_new/FindBlob: Index might be resorted between the call to SortByBlob and reacquiring the read lock. Currently I don't see a nice way how this could be fixed when using a RWMutex. The same problem also exists in FindBlobByPack.

I hope I found a nice way to handle this. I basically test sortedByBlob twice if it is set to false. This issue hence should be fixed now ..

This still leaves the possibility of a collision between SortByBlob and AddJSONIndex. The check that the index is sorted properly and the actual index search must happen in the same critical section. Unluckily the RWMutex does not support an atomic downgrade from a write to a read lock, which would be the easiest way to keep the lock after sorting. I currently see two possibilities:

Take the readlock, check that the index is sorted and run the search. If the index is not sorted, release the readlock and get the writelock, sort the index and complete just that search while holding the writelock.
Take the readlock, check that the index is sorted and run the search. If the index is not sorted, release the readlock and get the writelock, sort the index and retry from the start.

MasterIndex.Lookup/Has/Count/... : After the lookup in mi.mainIndex, mi.idx might be saved and added to the mainIndex before the idxMutex is acquired.

It seems like you've missed MasterIndex.Lookup.

MasterIndex.Save: This seems to block the masterIndex while the index upload is pending. The old implementation in Repository.SaveIndex seems to work without blocking.

This issue is still open. I did not find a nice way to handle it with just one unfinished index.
I also do not know the impact of this with respect to performance. Should only affect backup. I hope to see some tests if this is relevant..

Hmm, the main blocker here is that you can't get the index pack id before the upload call completed. It would be rather easy to solve by allowing multiple indexes to be unfinished while only the last one may be written to and the predecessors are currently uploading.

This still leaves the possibility of a collision between SortByBlob and AddJSONIndex. The check that the index is sorted properly and the actual index search must happen in the same critical section. Unluckily the RWMutex does not support an atomic downgrade from a write to a read lock, which would be the easiest way to keep the lock after sorting. I currently see two possibilities:
* Take the readlock, check that the index is sorted and run the search. If the index is not sorted, release the readlock and get the writelock, sort the index and complete just that search while holding the writelock.

* Take the readlock, check that the index is sorted and run the search. If the index is not sorted, release the readlock and get the writelock, sort the index and retry from the start.

Oops - I was in fact so focused on SortByBlobs that I didn't consider the methods it is called :-(
Thanks for your suggestions! I used the second option now.

It seems like you've missed MasterIndex.Lookup.

Is fixed now.

Hmm, the main blocker here is that you can't get the index pack id before the upload call completed. It would be rather easy to solve by allowing multiple indexes to be unfinished while only the last one may be written to and the predecessors are currently uploading.

I also fixed this. I now encode/decrypt the index and write it to the main index while blocked but write the file to the backend after releasing the block.
Moreover I found another optimization that reduces the memory consumption quite a lot (and also fixes the append-problem): I exchanged the large array by a structure that implements a "paged" list of blobs. Here a list of "pages" is maintained and if it needs to be enlarged, just a new page is allocated. This allows to append without needing to recopy all old elements. Also the overhead is fixed and maximum the page size whereas with append you get quite a large overhead (the array gets an increasing "capacity reserve" to ensure it doesn't need to recopy too often). This optimiziation seems to save another ~15-20% of memory.
I'm happy to get feedback and even more like to see performance tests with large repositories,,

Here are the results of my first comparison.
I used a data dir with 550.000 files (results in 550.000 data blobs) using the script
#!/bin/sh
mkdir data
for i in `seq 1 550`; do
        echo $i
        mkdir data/$i
        for j in `seq 1 1000`; do
                echo $i$j > data/$i/$j
        done
done

Then I backuped this data dir three times and run check after:
/usr/bin/time/restic -r /path/to/repo --mem-profile . backup /path/to/data
/usr/bin/time/restic -r /path/to/repo --mem-profile . backup /path/to/data
/usr/bin/time/restic -r /path/to/repo --mem-profile . backup /path/to/data -f
/usr/bin/time/restic -r /path/to/repo --mem-profile . check

These steps I run once with the master branch (compiled to restic.old) and my branch (compiled to restic.new).
I just used my old laptop with local SSD.
Here are the results:
time_backup1_old.txt
mem_backup1_old.pdf
time_backup1_new.txt
mem_backup1_new.pdf
time_backup2_old.txt
mem_backup2_old.pdf
time_backup2_new.txt
mem_backup2_new.pdf
time_backup3_old.txt
mem_backup2_old.pdf
time_backup3_new.txt
mem_backup3_new.pdf
time_check_old.txt
mem_check_old.pdf
time_check_new.txt
mem_check_new.pdf
To summarize:

The new index implementation uses only 22% of the memory used by the index implementation im master (!!!)
whith check index related memory usage is even decreased more (the IDSet was removed)
With the new implementation the index memory usage is no longer the one outstanding memory consumer making optimizations of other parts worthwhile.
Speed and CPU usage was comparable with the new implementation often a bit faster. The only case where the new implementation was a bit slower was the third backup run which highly uses index lookups. However, we are talking about a newly generated repository with only few index files. Hence the main drawback of the master index implemetation (looping over all index files) didn't play a big role here. I believe that things completely change with many index files and that the new implementation then will outperform the current one clearly.

I'm finally done with my tests. It took longer than estimated because I had to re-run my tests because of some mysterious results. I wanted to compare results between 1 and multiple cores. Interestingly the VM with only 1 vCPU was a lot faster with almost all operations. Maybe @fd0 knows what's going on.
results for VM1
1 vCPU
2GB RAM
Note that I've made 1 CPU and 1 memory profiling run for each restic version. Time shown (in seconds) in this table is the average for both runs.

restic v.0.9.6
restic new-index
difference

init repo
2,24
2,27
1,12%

1st backup
1574,36
1542,80
-2,00%

2nd backup
556,95
541,71
-2,74%

3rd backup (--force)
1192,90
1195,53
0,22%

forget
0,51
0,52
2,97%

prune
43,87
44,02
0,34%

check
30,77
31,59
2,68%

list blobs
2,92
3,36
14,90%

cat blob
2,58
2,60
0,97%

find blob
22,86
21,17
-7,39%

restore
895,01
883,57
-1,28%

results for VM8
8 vCPU
32GB RAM
Note that I've made 1 CPU and 1 memory profiling run for each restic version. Time shown (in seconds) in this table is the average for both runs.

restic v.0.9.6
restic new-index
difference

init repo
2,11
2,09
-0,95%

1st backup
1894,47
1832,65
-3,26%

2nd backup
827,95
776,38
-6,23%

3rd backup (--force)
1414,60
1411,98
-0,19%

forget
0,60
0,56
-6,72%

prune
93,10
89,84
-3,50%

check
70,22
68,44
-2,53%

list blobs
3,86
7,24
87,68%

cat blob
3,88
3,69
-4,90%

find blob
30,95
29,11
-5,95%

restore
1150,49
1089,66
-5,29%

Additional information
$ uname -r
5.4.15-200.fc31.x86_64

$restic-orig version
debug enabled
restic 0.9.6 (v0.9.6-40-gd70a4a93) compiled with go1.13.6 on linux/amd64

$ restic-newindex version
debug enabled
restic 0.9.6 (v0.9.6-43-g5db7c80f) compiled with go1.13.6 on linux/amd64

files changed added for backup 1:
Files:       487211 new,     0 changed,     0 unmodified
Dirs:            2 new,     0 changed,     0 unmodified
Added to the repo: 62.069 GiB

files changed added for backup 2:
Files:       166940 new,     0 changed, 487211 unmodified
Dirs:            0 new,     2 changed,     0 unmodified
Added to the repo: 6.805 GiB

files changed added for backup 3:
Files:       654215 new,     0 changed,     0 unmodified
Dirs:            2 new,     0 changed,     0 unmodified
Added to the repo: 4.029 MiB

Logs
logs_vm1_cpu.zip
logs_vm1_mem.zip
logs_vm8_cpu.zip
logs_vm8_mem.zip

@dimejo Thank you very much for your tests!
First regarding your question about increased time for the 8 CPU setting: You are comparing the "user time" that is the total CPU seconds used. With parallel processing this is (and should be higher) than using a single CPU as there is always some overhead due to parallelization. In fact your 8 CPU runs were much faster, see the elapsed time.
You used the commit 5db7c80f for your newindex tests. Is it possible that you retest with the latest commit 26ec33dd? There should be another decrease in memory usage and (if I did everything right) maybe even better performance.
In general I'm quite satisfied with these results! Seem my implementation is using much less memory while still having similar performance than the original index implementation. (BTW: I'll fix the obviously open issue with list blobs)

First regarding your question about increased time for the 8 CPU setting: You are comparing the "user time" that is the total CPU seconds used. With parallel processing this is (and should be higher) than using a single CPU as there is always some overhead due to parallelization. In fact your 8 CPU runs were much faster, see the elapsed time.

Thanks for the explanation.

You used the commit 5db7c80f for your newindex tests. Is it possible that you retest with the latest commit 26ec33dd? There should be another decrease in memory usage and (if I did everything right) maybe even better performance.

Sure.
Results
elapsed time
Note that I've listed elapsed time (in mm:ss) now. Again showing average time for both runs.

v0.9.6
newindex
newindex2
difference (v0.9.6 and newindex2)

init repo
00:42,4
01:11,6
00:21,5
-49,27%

1st backup
26:42,7
26:58,2
31:00,4
16,08%

2nd backup
10:18,9
09:54,1
09:03,7
-12,14%

3rd backup (--force)
17:32,6
17:05,8
20:37,1
17,52%

forget
00:00,8
00:00,9
00:00,8
0,61%

prune
01:02,5
00:56,5
00:51,2
-18,16%

check
00:31,6
00:30,8
00:31,0
-1,74%

list blobs
00:05,1
00:06,2
00:05,6
9,44%

cat blob
00:02,2
00:02,3
00:02,1
-3,39%

find blob
00:28,5
00:28,2
00:23,0
-19,28%

restore
13:02,0
12:23,3
14:09,1
8,58%

memory usage
This table shows total memory usage (in MB) from the pprof files. Hope that's the correct one.

v0.9.6
newindex
newindex2
difference (v0.9.6 and newindex2)

init repo
32,01
32,01
32,00
-0,03

1st backup
227,50
178,48
174,77
-23,18

2nd backup
227,89
157,92
149,52
-34,39

3rd backup (--force)
103,99
147,15
136,06
30,83

forget
32,25
32,09
32,26
0,03

prune
167,55
108,96
130,60
-22,05

check
163,29
75,06
70,34
-56,92

list blobs
33,22
30,21
25,47
-23,33

cat blob
113,79
48,86
32,23
-71,68

find blob
79,48
30,40
25,64
-67,74

restore
226,08
176,54
198,19
-12,33

Additional information
VM8
8 vCPU
32GB RAM

$ restic-newindex2 version
debug enabled
restic 0.9.6 (v0.9.6-44-g26ec33dd) compiled with go1.13.6 on linux/amd64

Logs
logs_test3.zip

@aawsome : I still have problems with the check command not caching tree blobs. The old code called Repository.SetIndex which also triggered the calls to PrepareCache. Now the call to SetIndex is removed, however, I couldn't find a replacement for the call to PrepareCache.

@dimejo Thanks you very much for the tests! This gives very good information and the results regarding memory consumption look already really good!
I tried to understand the memory consumption results for v0.9.6, 3rd backup, but I was not able to reproduce your results. May it be that you used the memory consumption of the index (102 MB) instead of the total memory (204 MB)?
I also do not understand why newindex2 shows higher memory consuption than newindex for prune and restore. In the memory profile you can see that the memory used by the index is lower and the higher memory consumption is due to encoding/json.Marshal and restic.NewBlobBuffer in pruneand  restorer.(*packCache).get in restore. Also the memory consumption showed by time is lower for newindex2 than it is for newindex.
So I believe that newindex2 also has lower memory consumption for these two commands than newindex, but I don't understand why the profiling shows these extra (non-index related) parts consuming memory..
About the total time: I'm a bit irritated that newindex and newindex2 differ so much and that the differences are sometimes positive and sometimes negative. I tried to look at the cpu profiles of some runs and did not see any index related cpu usage. For instance with the three backup runs the cpu consuming parts in all (v0.9.6 and newindex2) runs are totally dominated by sha256, chunker, syscall and runtime and I do not see why there should be differences due to a different index implementation.
Can it be that the machine doesn't have constant cpu power and the differences are not caused by different implementations but by different machine circumstances? When you compared v0.9.6 and newindex, the results did not vary much and as far as it looks these two runs were done at approximately the same time.
With other words: is it possible that you rerun the cpu measurement again for v0.9.6 and newindex2 at the same time? Then we could check if variation in cpu timings is common and maybe get better insights about performance changes due to the changed implementation.

@MichaelEischer Thank you for reporting this still unfixed issue and sorry that it doesn't work so far.
The call to PrepareCache is in internal/index/masterindex.go:Load which is called by internal/repository/repository.go:LoadIndex.
I will have to debug that part of the code.

Just leaving another ad-hoc results of the branch with a big-ish repo (~500GB/1500+snapshots) + small-ish backup (1.3GB). Very promising :+1:
default-restic.txt
patched.txt

@MichaelEischer I found the problem with not using the cache with the check command: As check rebuilds the functionality of masterindex.Load() the PrepareCache has to be called explicitly (was in repository.UseIndex()) This is fixed now.
I also fixed the performance issue with list blobs

@seqizz Thanks for testing! I'm quite satisfied with the memory consumption :smile:
Do you have an explanation why the elapsed time in the patched run is much higher than in the default-restic run? I can't explain this behavior by the index changes. Did you see reduced parallelization due to the different implementation or is this an effect that can be explained from within the system you used?
BTW: total CPU times seem to be comparable between the two runs. This is exactly what I was expecting.

@aawsome I'd say this was the side effect of the improper testing by me. Even though I forget the created snapshots between runs, seems like it was slow because I ran the patched version first. Maybe OS caching on our storage-side (Minio) affected the reply time.
Now as a control run, I've built the upstream restic (instead of previous release version, since I've build your branch over upstream) and ran this upstream version first. It shows you can disregard the elapsed times, since it depends on the order..
restic-upstream.txt
restic-patched.txt

I tried to understand the memory consumption results for v0.9.6, 3rd backup, but I was not able to reproduce your results. May it be that you used the memory consumption of the index (102 MB) instead of the total memory (204 MB)?

Good catch - probably a copy & past error. It's now corrected in my post above.

I also do not understand why newindex2 shows higher memory consuption than newindex for prune and restore. In the memory profile you can see that the memory used by the index is lower and the higher memory consumption is due to encoding/json.Marshal and restic.NewBlobBuffer in pruneand restorer.(*packCache).get in restore (which are basically not present in the profile for v0.9.6).
So I believe that newindex2 also has lower memory consumption for these two commands than newindex, but I don't understand why the profiling shows these extra (non-index related) parts consuming memory..

TBH, I have no idea what happened there. I tried to reproduce the results on the very same VM but was unable to.

Also the memory consumption showed by time is lower for newindex2 than it is for newindex.

I've compared the results shown by time from all runs from my 4. test (see below), and they don't seem to be too reliable. For some tests the memory consumption between cpu and mem profiling run varies by 15-20%.
restic-orig_test4_cpu_findblob: 253,12MB
restic-orig_test4_mem_findblob: 307,44MB
restic-newindex2_test4_cpu_catblob: 184,12MB
restic-newindex2_test4_mem_catblob: 216,56MB
restic-newindex2_test4_cpu_findblob: 174,81MB
restic-newindex2_test4_mem_findblob: 202,38MB

Can it be that the machine doesn't have constant cpu power and the differences are not caused by different implementations but by different machine circumstances? When you compared v0.9.6 and newindex, the results did not vary much and as far as it looks these two runs were done at approximately the same time.
With other words: is it possible that you rerun the cpu measurement again for v0.9.6 and newindex2 at the same time? Then we could check if variation in cpu timings is common and maybe get better insights about performance changes due to the changed implementation.

The VM I used doesn't have dedicated cores. Especially with 8 cores restic isn't even close to using all available CPU power. That's why I thought it wouldn't matter much for my tests. I've now repeated the test with dedicated CPUs and the results are much more even. Looks like that's what caused the discrepancy.
Results
elapsed time (in mm:ss)

v0.9.6
newindex
newindex2
difference (v0.9.6 and newindex2)

init repo
00:26,2
00:48,7
00:58,4
+122,79%

1st backup
32:51,8
32:12,1
32:49,0
-0,14%

2nd backup
08:40,2
08:31,2
08:30,3
-1,90%

3rd backup (--force)
21:40,5
21:43,6
21:41,0
+0,04%

forget
00:00,8
00:00,7
00:00,8
-7,98%

prune
00:43,7
00:43,8
00:42,2
-3,56%

check
00:26,3
00:26,9
00:26,4
+0,30%

list blobs
00:03,5
00:04,4
00:04,6
+29,46%

cat blob
00:01,7
00:01,8
00:01,8
+9,04%

find blob
00:17,7
00:17,3
00:16,7
-5,95%

restore
12:40,2
13:20,3
12:24,1
-2,12%

memory usage (in MB)

v0.9.6
newindex
newindex2
difference (v0.9.6 and newindex2)

init repo
32,05
32,02
32,02
-0,09%

1st backup
242,38
170,70
161,80
-33,25%

2nd backup
224,63
154,78
146,69
-34,70%

3rd backup (--force)
218,38
159,48
150,07
-31,28%

forget
32,01
32,24
32,05
+0,12%

prune
183,05
110,84
113,29
-38,11%

check
195,05
74,28
70,33
-63,94%

list blobs
33,32
29,45
25,45
-23,62%

cat blob
92,94
49,02
33,13
-64,35%

find blob
111,51
29,62
25,41
-77,21%

restore
292,94
217,41
206,77
-29,42%

Additional information
VM used:
8 vCPU (dedicated CPUs)
32GB RAM

$ restic-orig version
debug enabled
restic 0.9.6 (v0.9.6-40-gd70a4a93) compiled with go1.13.6 on linux/amd64

$ restic-newindex version
debug enabled
restic 0.9.6 (v0.9.6-43-g5db7c80f) compiled with go1.13.6 on linux/amd64

$ restic-newindex2 version
debug enabled
restic 0.9.6 (v0.9.6-44-g26ec33dd) compiled with go1.13.6 on linux/amd64

Logs
logs_test4.zip

@dimejo Good work - thank you very much!
The list blob performance (and memory usage) is improved a lot after commit 908c8977df05ea88dcad86c37f39d201468446ad.
The other results look already pretty good. I'm currently looking if the index can save some memory for backup and restore by using small refactoring of these commands (as already done with check).
Then I'll work on the tests and make a PR.
@rawtaz My question about which part of this work should be separated to make a PR acceptable is still open. I'll need some directions here if the changes are too large for maintainers to handle..
I'mm still convinced that the complete reimplementation of the index here is key to these good results in memory consumption reduction.

@aawsome Thanks for fixing the check command. I'll give it a try.
Regarding your optimizations for the check command, you might want to take a look at my pull request #2328 which manages to reduce the memory usage even a bit further and is much faster for repositories containing more than a few snapshots.
After looking at the diff between master and refactor-index I get the impression that nearly half of the changes are caused by moving code between repository and index. However, this is not clear when looking at the commits. Is this move actually necessary?
I'm also not really convinced to have a reference to the repository stored in the MasterIndex. The previous way of letting the Repository control the MasterIndex seems to be better separated in myview. Although, it's probably a good idea to move the fallback from the current to the old index format into the index implementation.
I agree that it is probably not possible to reduce the total amount of changes very much without loosing functionality, but it would probably help a lot to just have to look at a set of commits with less than a few hundred lines each.

@aawsome There's still a bug with caching treePacks. The line https://github.com/aawsome/restic/blob/908c8977df05ea88dcad86c37f39d201468446ad/internal/repository/repository.go#L97 should read treePacks.Insert(pb.PackID). This should also reduce the memory usage a bit further.
The current implementation in restic just caches packs that contain only trees. This is equivalent to the new implementation for packs created with recent versions of restic. However, in old restic versions (I think < 0.8) data and tree blobs are mixed together. These data blobs should not be cached. Therefore, I would suggest to add a second iteration over all blobs and remove packs that contains data blobs again.

@MichaelEischer
Thanks for finding another caching bug - I hope it now really is the last one. And sorry for me not able to find it; I so far used only local backend and there is no standard test which checks if caching is set up correctly... Maybe such a test should be added...
This (and the old repo format issue) is fixed in the last commit.
Also thank you very much for your other comment!
About check I agree that there is even more memory optimization potential (and of course speed). I was already thinking about making the index able to save different user-defined flags for blobs. That way you can save huge IDSets or user-defined maps as you save the blob ID only once. This is what I was thinking about when optimizing prune in a second step.
Your PR #2328 can then easily adapted to use index flags instead of the user-defined map and we get a additional memory reduction.
I agree about your design issue and removed repo from MasterIndex. Hence also Load and Save commands went back to repository.go making the diif to master a bit smaller.
However, I do not think that the main goal should be to have a as small as possible merge. I would like to implement the best design. About where to locate the index implementation we already have two places in master: internal/index and internal/repository. As index related stuff is quite some code, I thought it should be separated from repository and so I chose internal/index to be the location to place all about index (and also define a new interface in internal/restic/index.go). All code in index_new.go, index_old_format.go and master_index.go is written from scratch and index.go is heavily adapted while still having the original functionality used in rebuild-index and prune.
On the other side all index related stuff (except loading/saving) is removed from internal/repository.
About optimizing backup and restore:

With backup I added the possibility to add "known" blobs to the index. As these are removed once the "known" blob is added to the index, this should save a lot of memory used in backup. I expect that the main memory usage should now be to the space reserved for the parallel chunker, if you do not have a really HUGE repository.
For restore I did not find any index-related optimization. It seem that all memory usage is due to the way restore works internally. I do have the feeling that there are optimization possibilities, but this should be done in a separated work.

So for me, only working on the test implementation is open (if there are no other bugs that get reported). Then I will make a PR.

@aawsome Hi, sorry for taking long to reply. I feel I'm unable to give you relevant detailed directions, but what I was thinking of was to split things like "refactoring code in preparation for upcoming changes" (such as changing the index handling) and "refactoring code to optimize or add features" (such as the actual optimizations and stuff you're aiming for here) into separate commits or even PRs (depends on the code).
I'm trying to get you some eyes on this that have better things to say than I do, because I do think that your work here is looking interesting!
Also, I appreciate that you have started this contribution by opening an issue (instead of just opening a PR), thank you for that! Whenever possible it's good if we can have a discussion about suggestions before the actual implementation is done. By doing so we can find a common ground and settle on a direction that has everyone on board (and oftentimes unexpected issues or perspectives that change how the implementation would be done arise, which makes for less wasted work in the end).

@rawtaz Thank you for your reply!
You are right that during development I tried to move fast and be able to try out what things do work and what doesn't work. For now, it seems that the code change is mature enough to discuss how it can be best integrated into master.
I think it helps for the discussion to give an overview of the changes I've made and the reasons why I did each change. The implementation covers the following changes:
Major changes:

the structure of the codebase, namely which functionality resides in internal/repository and internal/index
structure of interface definitions in internal/restic
internal structure of MasterIndex including the handling of "full" index files
the index-file related data structure
support for old format index files (index_old.go)
the new in-memory index data structure (index_new.go)
change of index access where needed

Minor additional changes:

changed Lookup  and added LookupAll
replaced 'Store' by StorePack
add CheckSetKnown to handle known blobs during backup
remove the blob IDSet from checker.go and replace by index functionality
use new index functionality in 'list blobs'

The major changes are IMO the key to achieve low memory consumption, good performance and a clean(er) code design (compared to the actual implementation). The minor changes are quick wins that could also be independent PRs. However if they are assumed to be good enhancements I would prefer keeping them in the main PR, else they probably need to be implemented either twice or need to be postponed.
Structure of code base
I moved most index related functionality from internal/repository to repository/index. IMO this makes the code base structure clearer and also clearly separates what code is in which directory (as opposed to the actual implementation.
Question to maintainers: Do you acknowledge this principal change?
Structure of interface
I also separated interface definitions in internal/restic so that it is clear what's related to Index  and what to Repository`. This also allows clearer changes of the interfaces in future.
Question to maintainers: Do you acknowledge this principal change?
Structure of MasterIndex
Currently, MasterIndex combines index-file related data structures and usually iterates over all these sub-structures for index operations. Each of the index-file related data structures can be full (meaning full enough for saving as a full index file to the repo) or finished (meaning being already present as index file).
I completely changed this behavior. Now MasterIndex uses a main in-memory index where the majority of the index entries should be present - especially all those which are also present as index files in the repo - and a second "unfinished" data structure which is only used to save new index entries and is inserted into the main in-memory index once the contents are written as index file to the repo.
Question to maintainers: Do you acknowledge this principal change?
Index-file related data structure
Currently, there are three index-file related data structure definitions: one JSON-defined in internal/index, one JSON-defined in internal/repository and a lookup-optimized one in internal/repository.
I decided to use the JSON data structure from internal/index and added the needed methods to use it in MasterIndex (and completely removed the implementations from internal/repository). This completely removes converting internal data structures for saving index files during backup. On the other side, the lookup-operations need to loop over all index entries. Hence, it is cruical that this data structure does not contain lots of entries during lookup, i.e. it is inserted regularly to the main index.
Question to maintainers: Do you acknowledge this principal change?
Support for old index file format
The support for new index file formats is now outsourced to index_old_format.go.
Question to maintainers: Do you acknowledge this principal change?
New in-memory index data structure (index_new.go)
This obviously is the biggest change. In principal one big blob list is maintained per blob type. The list is sorted by blob ID such that lookup can be done using binary search. As optimization aims for many entries in this list, the size of each entry is minimized as much as possible. This means that the pack ID is stored in a separate table and only a reference is saved and offset and length are uint32 (effectively limiting maximum pack size to 4TB).
A simple sorted array already saves the memory overhead which is within a map but still generates some memory and performance overhead within append: An extra "buffer" is reserved for future appends and data needs to be copied. Hence I implemented a "paged" array with a small and limited memory overhead and no need to copy for append.
The tests that have been done indicate that this index is a bit faster than the standard MasterIndex implementation while saving more than 75% of the memory.
Question to maintainers: Does this index implementation fulfill your expectations?
Changed Lookup  and added LookupAll
Lookupactually returns more than one results (but not if they are spread over many index files, BTW). In many cases, only the first result is used. Hence I changed Lookup to return only one result and added LookupAll to be used when more than one result is used. Makes code a bit clearer and also saves some unneeded cpu cycles.
Question to maintainers: Is it ok to keep this change or would you prefer to remove it from the index PR?
Replaced 'Store' by StorePack
Store stores only one blob into the index but is only called for a whole pack content. I therefore changed it to a StorePack functionality where the whole pack content gets saved. This makes code more clear, storing to the JSON format much easier and also more performant.
Question to maintainers: Is it ok to keep this change or would you prefer to remove it from the index PR?
add CheckSetKnown to handle known blobs during backup
The tests showed that for many new files backup uses quite some memory to store which blobs have been added during this backup run. This is to not save blobs twice by parallel savers. I moved this functionality to the index as this "known blob" information can be removed once the corresponding entry is added to the index. With this change only very few blobs are "known" but not (yet) in the index.
Question to maintainers: Is it ok to keep this change or would you prefer to remove it from the index PR?
Remove the blob IDSet from checker.go and replace by index functionality
Checker builds its own in-memory index and additionally keeps all blob IDs in an additional IDSet. As all blobs are already known to the index, this can be replaced by the Each method of the index.
This saves additional memory within check.
Question to maintainers: Is it ok to keep this change or would you prefer to remove it from the index PR?
Use new index functionality in 'list blobs'
Instead of reading the whole index into memory and then looping over all entries to print the blob list, each index file can be loaded separately into memory and loop over these contents. Makes list blobs much less memory-hungry.
Question to maintainers: Is it ok to keep this change or would you prefer to remove it from the index PR?

A small test result @aawsome , I am not able to backup into an existing repository after latest commit.
Values from time:

---
"Normal" backup
fix masterindex.Load() commit
last commit

Memory usage
~650Mb
~320Mb
~55Mb

Backup time
~12Sec
~12Sec
(gave up after 5mins)

Storage reports it stops activity after querying indexes. (Sorry if that was intended due to format change)

@seqizz Thanks for sending the bug report. I have to admit that I only tested the last commit with go run build.go -T which didn't show any issue... I fixed it in the last commit 5b4009fcd275794e3ce05304fce255d5fa3e1864

@aawsome Thanks for writing that summary comment. One of the things I as a non-core developer (I'm just a maintainer or simple things) experience is that during the time that core developer time is spent on other parts of the project, issues like this one just keeps growing. Then whenever a core developer looks at it, it's a lot or in some cases too much to dig into at the time. The less there is to read for them, the higher are the chances that we can make good use of their time. I took the liberty of editing the initial post in this issue to add a reference to your summary comment.
Regarding your questions, I personally cannot answer them. I will have to defer them a while, and I kindly ask for your patience. Our main focus right now with the available time we have is some other stuff, but I can tell you that this issue and your PR is on my radar and I am trying to get eyes on it eventually.
I think that with your summary comment, besides fixing bugs and whatever you can find that doesn't deviate from what you wrote in your comment, the most productive thing is to let this rest a bit. The next step would be to get core developer eyes on it, and until that happens it's probably more counter-productive to grow the discussion more. Let me know if my rationale with this doesn't seem to make sense, so I can clarify what I mean.

I finally found time to test the performance of the backup command on a 8TB, 40M files repository. To keep the overall runtime low I've just backed up a 6GB directory. CPU usage of restic was limited using GOMAXPROCS=1.
current master: cpu-idx-master.pdf
this pr: cpu-idx-table.pdf
With this PR restic uses around 6.5GB memory instead of 18GB for the current master, which is a gigantic improvement. The index takes around 9 minutes to load in both cases (the host does not support AES-NI so decryption is a bit slow). The backup takes 34mins for this PR and 18mins for the current master. Taking a look at the cpu profiling graphs shows the difference: The indexTable spends 16 minutes just for sorting the in-memory index. The index is large enough that a single sort call for the in-memory index takes about 75 seconds (!) to complete. During this time the whole backup processing is blocked as the index is locked exclusively.
The in-memory index probably needs a way to merge a new index without resorting everything, maybe even implemented incrementally. This should be possible by sorting the new index beforehand and then merge it into the MasterIndex with a single scan over the indexTable.

@MichaelEischer Thank you very much for doing your test! In fact, the current implementation needs a resort of the index table each time an index file is written. So if I calculated correctly, 13 new index files have been created, right?
The good message is that the Lookups in IndexJSON (which are unsorted and thus require linear search) do not show up at all, so this seems to be a good way to handle "unfinished" index files
Actually my judgement is that resorting an almost sorted table with sort.Sort should be almost optimal. (However, trying out sort.Stable or a pure mergesort should still be worth a try...) I do think that the main issue is: If we want to have one sorted index array and insert things into it we need to move ~6.5GB of data in the memory around.
So I propose to separate the index entries read from index files from those newly generated (and already saved). This makes three data structures in the MasterIndex implementation. I already changed this in my refactor-index branch. Would you mind re-testing your setup?

When I zoom into the profile data IndexJSON takes 1.22 seconds in total. So this seems fine for now. The worst case would be a lot of small files in case a map might be faster.
The backup run actually created 16 new index files. The 75 seconds were a guess from a check run which only triggers re-sorting once. Both quicksort (used by sort.Sort) and mergesort have a best-case time complexity of O(n log n), thus presorted data does not improve performance. As a side-note the in-memory index requires less space than the on-disk representation ^^ .
I've rerun the test with the updated code and the performance is a lot better: cpu-idx-table-split.pdf
However, using a separate index for new index entries merely hides but does not solve the underlying problem: Just imagine a backup run that adds a few million new files or a large initial backup. The current implementation has O(n * n*log n) complexity. The first n relates to the number of appends to the index which scales somewhat linear with the amount of new backup data. And the latter part of n*log n is the already mentioned complexity for sorting the index. By replacing quicksort with a linear merge this is reduced to O(n * n) which could be good enough for index sizes below several dozen gigabytes. Beyond that it won't scale, but I guess Petabyte repositories are a problem of their own. The linear merge could even be implemented to work incrementally. I have an idea how to reduce the complexity down to O(n * log n) but that would increase the lookup costs to O(log^2 n) and require some extensive code changes.
I wonder whether there is a data structure that would perfectly fit our needs:

it should be an 'implicit data structure' or 'succinct data structure', that is have a low space overhead
amortized sub-linear query speed, preferably log n or less
amortized sub-linear insertion costs,  preferably log n or less. With n insertions this would then lead to costs of O(n * log n) for a whole backup run

When dropping the requirement for a really compact in-memory representation, hashmaps are a perfect fit, as their queries and updates happen inO(1). However, they require a lot more memory: My current prototype uses about 70% more memory than this PR.
That said it is probably best to wait for feedback from @fd0 before making any extensive changes. (Sorry for growing the discussing again.)

I wonder whether there is a data structure that would perfectly fit our needs:

I'd say there are "many" more or less satisfying these needs, but don't know how much effort it would be to switch to them in restic. Basically any B-tree or generally tree structures should satisfy these needs (hash maps are really inefficient for such scenarios as you also found out yourself in your measurements).

If I'm not mistaken a B-Tree only guarantees that it's nodes are at least half filled. And it would require pointers which would more or less add a 8 byte pointer for every 48bytes of data. If a B-Tree guarantees 70% space usage on average then it would end up with a similar memory overhead as my hashmap based implementation. There might be some variants like B*-Trees which are more efficient though.

@michaeldorner yeah, that's the "old traditional B-tree" - as you wrote, there are many (I've seen at least ten quite different ones so far) B-tree like structures each having different properties. Other well known tree-like structures include e.g. red-black trees (popular for maps which must guarantee consistent insertion/deletion time & space; this is in contrast with hash-based maps which might suffer from "jumps" or other excessive peaks both in time & space eventually leading to severe security/performance problems). There are also various "compressed" maps - feel free to search for them. So there is really a number of options.
Though I'm not against using a small temporary file like discussed in this topic and like e.g. sqlite does - btw. sqlite is highly efficient and I've used it as storage just for the purpose of extremely low memory usage while offering quite high throughput despite using hard disks as temporary cache - feel free to try it out, it might surprise you.
Just my 2 cents :wink:.

The sorted, densely packed array of index entries is basically the gold-standard for memory usage. Yes, you could apply some compression, but that would only help with the blob size/offset and pack id idx (approx. 16bytes) the remaining 32bytes are the blob id (sha256 hashes) which I'd assume to be basically incompressible. I'm not sure whether the potential for another 20-30% lower memory usage would justify the even higher complexity.
Trees always require one (and some unused space like with B-Trees) or two pointers (red-black-tree and others) per elements which would already add 8 or 16 bytes overhead to the approx. 48 bytes of the index entry itself. I'm also not sure how well the Go garbage collector would handle millions or even billions of index entries.
To save the honor of hash tables: cuckoo hashing (constant lookup time) together with 4 entries per bucket is said to achieve over 90% space utilization. If you just use ten of these hash tables then the total memory overhead would also stay below 33% even while growing a hashmap.
I think the main question is more whether we want a single data structure (tree/hashmap) with a certain memory overhead. Or whether we want the lower overhead of a sorted packed array, which will need an auxiliary data structure to provide a reasonable insertion speed. If that auxiliary data structure just contains 10-20% of the overall index data, then it won't matter much whether a data structure has 30% or 70% overhead and in that case Go's builtin hashmap would be the simplest solution.
Using an on-disk index has it's own share of problems: If the index is too large to be kept in memory, then the index performance will probably implode on HDDs (SSDs should be much better off) unless the index is able to benefit from locality between blobs for e.g. one folder/file (in a fully sorted list of blob ids, the access pattern would basically be uniformly at random).

@MichaelEischer thanks for the wrap up. I'd still though encourage you to try disk cache (just to get a glimpse of possible performance, not to be incorporated in Restic) using sqlite3 with different setups (e.g. both with and without index, with increased max memory constraints than the very low defaults, using just one table as key-value map-like storage, etc.) while leveraging some sqlite3 pecularities like builtin integer primary key and multithreaded access without explicit locking (sqlite handles it itself) and using one huge transaction for all accesses and not using WAL as it's significantly slower for reads (but slightly faster for writes).
Using disc cache might also quite significantly benefit from stream-oriented processing, but don't know whether it's a good fit for a quick tryout. Just an idea based on my good experience with sqlite3-based key-value storage :wink:.

@aawsome Could you split your "list blobs" optimization into a separate PR?
I don't plan to merge the blob IDSet removal from checker.go as #2328 also provides that optimization but without having to involve the master index.
That leaves LookupAll as the last minor change. That function would be useful when adding support for damaged repositories to prune. However, without some index performance optimizations first, it will probably just reduce the overall performance of restic. So this will have to wait.
Regarding the major changes, these also have to stay on hold for now.
@dumblob The problem with using sqlite or any other generic database for an on-disk index is that without optimizations that make use of locality between blobs in a file (i.e. blobs within a file end up in nearby packs and are usually accessed together), we'll end up with a pseudo-random access to all parts of the index which is larger than the main memory and therefore has to load data from random locations on the disk. That is a generic database implementation has no chance at all to provide a good performance.

@aawsome Could you split your "list blobs" optimization into a separate PR?

Sure. PR will come soon.

That leaves LookupAll as the last minor change. That function would be useful when adding support for damaged repositories to prune. However, without some index performance optimizations first, it will probably just reduce the overall performance of restic. So this will have to wait.

In fact, the actual implementation of Lookup is a (partly) LookupAll in the sense that it returns all results within an index file, but only the results of the first index file that has a match.
As the usual case is that there are no duplicates within one index files, this actually does not give duplicates even in cases where they might be needed.
I will prepare a PR to change Lookup and add LookupAll where it is useful. Then we can discuss if this is helpful or not.


2024-10-10_01-35-07_master-next_python3-botocore

Automatically created.

Successfully created backport PR for kirkstone-next:

#9881

Successfully created backport PR for scarthgap-next:

#9882

Successfully created backport PR for styhead-next:

#9883


I2C: Support I2C1/2/3

The current I2C API only supports I2C0. Extending it to support the other I2C peripherals should be relatively straight-forward, and the USART API shows how this can be done.

Support has been added in #202.


Update sbt to 1.9.4

About this PR
📦 Updates org.scala-sbt:sbt from 1.9.3 to 1.9.4
📜 GitHub Release Notes - Version Diff
Usage
✅ Please merge!
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

⚙ Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.scala-sbt", artifactId = "sbt" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "30 days" },
  dependency = { groupId = "org.scala-sbt", artifactId = "sbt" }
}]

labels: library-update, early-semver-patch, semver-spec-patch, version-scheme:early-semver, commit-count:1

Codecov Report
Patch coverage has no change and project coverage change: -0.06% :warning:

Comparison is base (752e134) 89.58% compared to head (6d4299f) 89.52%.
 Consider uploading reports for the commit d145f36 to get more accurate results

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #1140      +/-   ##
===========================================
- Coverage    89.58%   89.52%   -0.06%
===========================================
  Files          124      124
  Lines        10028    10028
  Branches       782      782
===========================================
- Hits          8984     8978       -6
- Misses        1044     1050       +6

see 12 files with indirect coverage changes

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Add DNS tests

What this PR does
Tests the common DNS records created for WCs
Checklist

[x] Update changelog in CHANGELOG.md.

Trigger e2e tests

/run cluster-test-suites

/run cluster-test-suites

@mnitchev any ideas when you can pick it up again 🙏🏻?

/run cluster-test-suites

/run cluster-test-suites

I've removed the wildcard and ingress domains. These can be checked later with a test app that uses the ingress. Please re-review

/run cluster-test-suites

/run cluster-test-suites

/run cluster-test-suites
Hopefully for the last time

The CAPA dns took nearly 30 minutes before it was available. The CAPZ one was immediately available. 🤔

/run cluster-test-suites

/run cluster-test-suites
The new release of cluster-aws should solve the previous problem on CAPA

/run cluster-test-suites

/run cluster-test-suites

/run cluster-test-suites

/run cluster-test-suites


Added the ability for column filter to be callable

This PR adds the ability make filter callable.
'filter' => function (QueryBuilder $queryBuilder, $field, $search) {
	$users = explode(',', str_replace(' ', '', $search));

    $queryBuilder
    	->andWhere($queryBuilder->expr()->in($field, ':users'))
		->setParameter('users', $users);
}

Thanks, and my apologies for not noticing it before, but I'm closing this PR. I see what you want to achieve but it's not the right way to do so - you should just implement your own filter based on AbstractFilter. As PHP now has anonymous classes this is just as convenient as adding a callable option here which would just complicate our core code.

@curry684 AbstractFilters cant handle IN queries. Do you have any way around this?


Remote attestation failure due to invalid number of Auditor app signatures

I have a new Pixel 6a with GrapheneOS installed yesterday.
I set up remote attestation and it was successful for the first few attestations. It has now stopped functioning with the following error from the Auditor app;
Remote verification failure
Failed to submit remote attestation.
java.security.GeneralSecurityException: Invalid number of Auditor app signatures.

This is a known firmware bug. You need to reboot the device.


⚠ Payment API has degraded performance

In f9fc421, Payment API ($PAYMENT_API_URL) experienced degraded performance:

HTTP code: 504
Response time: 29832 ms

Resolved: Payment API performance has improved in 1eb8475.


Awkward translation of Korean in Disk Resize Doc

Ref Doc : https://learn.microsoft.com/en-us/azure/virtual-machines/windows/expand-os-disk#expand-without-downtime
Hi, I found awkward translation in Korean which can lead misunderstanding of information.

Per this comment, If Disk is beyond 4TiB(>4TiB, Not >=4TiB), then can expand to it without deallocation
The document states that disks exceeding 4TB(>4TiB, Not >=4TiB) can be changed to a size beyond 4TB. In the case, If disk size is exactly 4TB(==4TiB), deallocation is required.
There is no issue on this English doc.
But In Korean Translation Doc, It translated like this
_If a disk is 4 TiB or less, you should deallocate your VM and detach the disk before expanding it beyond 4 TiB. If a disk is already greater than 4 TiB, you can expand it without deallocating the VM and detaching the disk.
→ 디스크가 4TiB 이하인 경우 VM의 할당을 취소하고 디스크를 분리한 후 4TiB 이상으로 확장해야 합니다. 디스크가 이미 4TiB보다 큰 경우에는 VM의 할당을 취소하거나 디스크를 분리하지 않고도 디스크를 확장할 수 있습니다._
The bold text means like this.
이하 : =< 4TiB
이상 : >= 4TiB
If we translate backward from Korean description to English,
Then It said like this.
If a disk is 4TiB or less, you should deallocate your VM and detach the disk before expanding it to 4 TiB or upper size.
If a disk is already greater than 4TiB, you can expand it without deallocating the VM and detaching the disk.
Per this Korean translation, Even disk size is exactly 4TiB(==4TiB), can change disk size to upper size( >4TiB) without deallocation.
So we need to correct this translation which can lead misunderstanding.
So need to change word "이상" (>=4TiB) to "초과" (> 4TiB).
My suggestion about this Korean translation is like this.
→ 디스크가 4TiB 혹은 그 이하의 사이즈인 경우 VM을 할당을 취소하고 디스크를 분리한 후 4TiB 초과하는 사이즈로 확장해야 합니다. 디스크가 이미 4TiB 보다 큰 경우에는 VM 할당을 취소하거나 디스크를 분리하지 않고도 디스크를 확장할 수 있습니다.
If you need any help for this Korean translation, please feel free to ping me.
<EMAIL_ADDRESS>[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: ccc58c74-7913-f1f6-8681-1823bbd8f050
Version Independent ID: c11bab28-a82a-29a1-84bf-fb3dfb9a5cb2
Content: Expand virtual hard disks attached to a Windows VM in an Azure - Azure Virtual Machines
Content Source: articles/virtual-machines/windows/expand-os-disk.md
Service: azure-disk-storage
GitHub Login: @kirpasingh
Microsoft Alias: kirpas

@hynmoo
Thanks for your feedback! We will investigate and update as appropriate.

@hynmoo
Thanks for your feedback! I've assigned this issue to the author who will investigate and update as appropriate.

Thanks for your contribution to our documentation
We are moving to a new user feedback system. As part of that effort, we are creating a content request in our backlog to address this gap and are closing this GitHub issue. The updated content will be published shortly.

Please continue to provide feedback about the documentation. We appreciate your contributions to our community.
#please-close


Use ore dictionary logs for recipes

I hear that dark oak logs and silverwood logs don't work. Need to fix this.

Yeah you are using the log block, and in vanilla there is 2 log blocks (log
and log2). Just register it as an ore dict recipe with logWood.
On Feb 10, 2016 10:00 AM, "zerofall"<EMAIL_ADDRESS>wrote:

I hear that dark oak logs and silverwood logs don't work. Need to fix this.
—
Reply to this email directly or view it on GitHub
https://github.com/zerofall/EZStorage/issues/6.


Add support for private credentials.

Finalzed in https://github.com/Contrast-Security-OSS/sheepdog/pull/3/commits/28bcc760f81cfdcbfa3ca1a7839145f14c5bc415


get_current_user() py2 userid different than google admin directory user id

Environment details

OS: Ubuntu
Python version: 2.7 vs 3.7
pip version:
google-api-python-client version:

Steps to reproduce

Currently migrating a python2.7 app on google app engine to python 3.7
from google.appengine.api.users import get_current_user returns a User object with a property  "user_id": {  "stringValue": "101342104126303287634" }
When I query the same user with the google admin directory api, I receive the response'id': '108828824026963599070'

Why are these id's different if they are the same google user?
Thanks!

Hi @koskorya,
I'm going to close this issue due to inactivity. If you're still having trouble please re-open it with more information.


feat(prom/exp/snmp): add support to pass multiple SNMP config files to prometheus.exporter.snmp

PR Description
Which issue(s) this PR fixes

Closes #916
Notes to the Reviewer
I decided not to add another test case into snmp_test.go as it doesn't test whole component with argument but rather just a unit test for LoadSNMP function. Incoming integration test in #954 will address this.
PR Checklist

[x] CHANGELOG.md updated
[x] Documentation added
[x] Tests updated
[x] Config converters updated

Hi,
does this change support globbing, like the mentioned upstream PR (https://github.com/prometheus/snmp_exporter/pull/970)? This will be needed to make the cloud integration have a better out of the box experience for users running with their own credentials (instead of using "public" community string)

Will this get merged sometime in the future? really lookin forward to use multiple files instead of prejoining the files i generated into one large file, with mixed config&secrets.